2023-09-29T21:39:13.572 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-29T21:39:13.572 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284 --verbose --description rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi} --name yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/orig.config.yaml 2023-09-29T21:39:13.606 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T21:39:13.682 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284 branch: wip-yuri11-testing-2023-09-28-1611-reef description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi} email: yweinste@redhat.com first_in_suite: false job_id: '7406284' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri11-testing-2023-09-28-1611-reef ceph: conf: client: debug rgw: 20 debug rgw sync: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync data inject err probability: 0.1 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph client.0: rgw data notify interval msec: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - enabled_features: - resharding endpoints: - c1.client.0 is_default: true is_master: true name: a zones: - endpoints: - c1.client.0 is_default: true is_master: true name: a1 - endpoints: - c1.client.1 name: a2 - enabled_features: - resharding endpoints: - c2.client.0 is_default: true name: b zones: - endpoints: - c2.client.0 is_default: true is_master: true name: b1 - endpoints: - c2.client.1 name: b2 rgw-multisite-tests: args: - tests.py workunit: branch: wip-yuri11-testing-2023-09-28-1611-reef sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 seed: 4212 sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 sleep_before_teardown: 0 subset: 111/120000 suite: rgw suite_branch: wip-yuri11-testing-2023-09-28-1611-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 targets: smithi112.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_12:36:11 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-29T21:39:13.682 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa; will attempt to use it 2023-09-29T21:39:13.683 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks 2023-09-29T21:39:13.683 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-29T21:39:13.685 INFO:teuthology.task.internal:Checking packages... 2023-09-29T21:39:13.705 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '38c3646d535f6f917a872ef9baef468589ba9f32' 2023-09-29T21:39:13.705 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-29T21:39:13.705 INFO:teuthology.packaging:ref: None 2023-09-29T21:39:13.706 INFO:teuthology.packaging:tag: None 2023-09-29T21:39:13.706 INFO:teuthology.packaging:branch: wip-yuri11-testing-2023-09-28-1611-reef 2023-09-29T21:39:13.706 INFO:teuthology.packaging:sha1: 38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T21:39:13.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri11-testing-2023-09-28-1611-reef 2023-09-29T21:39:13.843 INFO:teuthology.task.internal:Found packages for ceph version 18.2.0-524-g38c3646d-1focal 2023-09-29T21:39:13.843 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-29T21:39:13.849 INFO:teuthology.task.internal:no buildpackages task found 2023-09-29T21:39:13.850 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-29T21:39:13.861 INFO:teuthology.task.internal:Saving configuration 2023-09-29T21:39:13.874 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-29T21:39:13.880 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T21:39:13.903 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:33:23.921292', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-29T21:39:13.925 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:33:23.923073', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-09-29T21:39:13.925 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-29T21:39:13.931 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2023-09-29T21:39:13.931 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2023-09-29T21:39:13.932 INFO:teuthology.run_tasks:Running task console_log... 2023-09-29T21:39:14.001 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2983e14040>, signals=[15]) 2023-09-29T21:39:14.001 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-29T21:39:14.007 INFO:teuthology.task.internal:Opening connections... 2023-09-29T21:39:14.007 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T21:39:14.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:39:14.057 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T21:39:14.058 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:39:14.106 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-29T21:39:14.112 DEBUG:teuthology.orchestra.run.smithi112:> uname -m 2023-09-29T21:39:14.140 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2023-09-29T21:39:14.141 DEBUG:teuthology.orchestra.run.smithi112:> cat /etc/os-release 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Ubuntu" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:ID=ubuntu 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE=debian 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="20.04" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-29T21:39:14.189 INFO:teuthology.orchestra.run.smithi112.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-29T21:39:14.190 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_CODENAME=focal 2023-09-29T21:39:14.190 INFO:teuthology.orchestra.run.smithi112.stdout:UBUNTU_CODENAME=focal 2023-09-29T21:39:14.190 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-09-29T21:39:14.210 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2023-09-29T21:39:14.217 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2023-09-29T21:39:14.217 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2023-09-29T21:39:14.263 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="Ubuntu" 2023-09-29T21:39:14.263 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-09-29T21:39:14.263 INFO:teuthology.orchestra.run.smithi131.stdout:ID=ubuntu 2023-09-29T21:39:14.263 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE=debian 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="20.04" 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://www.ubuntu.com/" 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_CODENAME=focal 2023-09-29T21:39:14.264 INFO:teuthology.orchestra.run.smithi131.stdout:UBUNTU_CODENAME=focal 2023-09-29T21:39:14.264 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2023-09-29T21:39:14.285 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-29T21:39:14.294 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-29T21:39:14.299 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-29T21:39:14.300 DEBUG:teuthology.orchestra.run.smithi112:> test '!' -e /home/ubuntu/cephtest 2023-09-29T21:39:14.302 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2023-09-29T21:39:14.307 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-29T21:39:14.313 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-29T21:39:14.313 DEBUG:teuthology.orchestra.run.smithi112:> test -z $(ls -A /var/lib/ceph) 2023-09-29T21:39:14.316 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2023-09-29T21:39:14.327 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-29T21:39:14.380 INFO:teuthology.run_tasks:Running task kernel... 2023-09-29T21:39:14.395 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-29T21:39:14.395 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-29T21:39:14.395 DEBUG:teuthology.orchestra.run.smithi112:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T21:39:14.396 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-29T21:39:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:39:14.401 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-09-29T21:39:14.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:39:14.402 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2023-09-29T21:39:14.408 INFO:teuthology.orchestra.run.smithi112.stdout:5.4.0-124-generic 2023-09-29T21:39:14.409 INFO:teuthology.task.kernel:Running kernel on smithi112: 5.4.0-124-generic 2023-09-29T21:39:14.409 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2023-09-29T21:39:14.410 INFO:teuthology.orchestra.run.smithi131.stdout:5.4.0-124-generic 2023-09-29T21:39:14.410 INFO:teuthology.task.kernel:Running kernel on smithi131: 5.4.0-124-generic 2023-09-29T21:39:14.410 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get clean 2023-09-29T21:39:14.514 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2023-09-29T21:39:14.518 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get update 2023-09-29T21:39:14.681 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T21:39:14.684 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T21:39:14.684 INFO:teuthology.orchestra.run.smithi131.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-29T21:39:14.684 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-29T21:39:14.702 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-09-29T21:39:14.750 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-29T21:39:14.767 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-09-29T21:39:14.788 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-09-29T21:39:15.074 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-29T21:39:15.079 INFO:teuthology.orchestra.run.smithi131.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-29T21:39:15.157 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-29T21:39:15.196 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-29T21:39:15.199 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-29T21:39:15.199 INFO:teuthology.orchestra.run.smithi131.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-29T21:39:15.200 INFO:teuthology.orchestra.run.smithi131.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-29T21:39:15.213 INFO:teuthology.orchestra.run.smithi131.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,852 kB] 2023-09-29T21:39:15.216 INFO:teuthology.orchestra.run.smithi131.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-29T21:39:15.219 INFO:teuthology.orchestra.run.smithi131.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-29T21:39:15.219 INFO:teuthology.orchestra.run.smithi131.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-29T21:39:15.225 INFO:teuthology.orchestra.run.smithi131.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-29T21:39:15.228 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-29T21:39:15.229 INFO:teuthology.orchestra.run.smithi131.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-29T21:39:15.230 INFO:teuthology.orchestra.run.smithi131.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-29T21:39:15.231 INFO:teuthology.orchestra.run.smithi131.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-29T21:39:15.232 INFO:teuthology.orchestra.run.smithi131.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-29T21:39:15.232 INFO:teuthology.orchestra.run.smithi131.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-29T21:39:15.245 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-29T21:39:15.248 INFO:teuthology.orchestra.run.smithi131.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-29T21:39:15.249 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-29T21:39:15.249 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-29T21:39:15.249 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-29T21:39:15.266 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-29T21:39:15.270 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-29T21:39:15.270 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-29T21:39:15.276 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-29T21:39:15.288 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-29T21:39:15.290 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-29T21:39:15.294 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-29T21:39:15.294 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-29T21:39:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-29T21:39:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-29T21:39:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-29T21:39:15.295 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-29T21:39:15.313 INFO:teuthology.orchestra.run.smithi131.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [885 kB] 2023-09-29T21:39:15.313 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-29T21:39:15.314 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-29T21:39:15.314 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-29T21:39:15.315 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-29T21:39:15.322 INFO:teuthology.orchestra.run.smithi131.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [469 kB] 2023-09-29T21:39:15.327 INFO:teuthology.orchestra.run.smithi131.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.1 kB] 2023-09-29T21:39:15.328 INFO:teuthology.orchestra.run.smithi131.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,330 kB] 2023-09-29T21:39:15.343 INFO:teuthology.orchestra.run.smithi131.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [34.7 kB] 2023-09-29T21:39:15.344 INFO:teuthology.orchestra.run.smithi131.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [325 kB] 2023-09-29T21:39:15.346 INFO:teuthology.orchestra.run.smithi131.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [572 B] 2023-09-29T21:39:15.346 INFO:teuthology.orchestra.run.smithi131.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [748 kB] 2023-09-29T21:39:15.351 INFO:teuthology.orchestra.run.smithi131.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,117 kB] 2023-09-29T21:39:15.358 INFO:teuthology.orchestra.run.smithi131.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [267 kB] 2023-09-29T21:39:15.360 INFO:teuthology.orchestra.run.smithi131.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.6 kB] 2023-09-29T21:39:15.360 INFO:teuthology.orchestra.run.smithi131.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.9 kB] 2023-09-29T21:39:15.361 INFO:teuthology.orchestra.run.smithi131.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,436 B] 2023-09-29T21:39:15.373 INFO:teuthology.orchestra.run.smithi131.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-09-29T21:39:15.373 INFO:teuthology.orchestra.run.smithi131.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-09-29T21:39:15.374 INFO:teuthology.orchestra.run.smithi131.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-09-29T21:39:15.374 INFO:teuthology.orchestra.run.smithi131.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-09-29T21:39:15.374 INFO:teuthology.orchestra.run.smithi131.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-09-29T21:39:15.390 INFO:teuthology.orchestra.run.smithi131.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-09-29T21:39:15.391 INFO:teuthology.orchestra.run.smithi131.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-09-29T21:39:15.391 INFO:teuthology.orchestra.run.smithi131.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-09-29T21:39:15.421 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [652 kB] 2023-09-29T21:39:15.744 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,464 kB] 2023-09-29T21:39:15.932 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [385 kB] 2023-09-29T21:39:15.946 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-09-29T21:39:15.947 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [33.3 kB] 2023-09-29T21:39:15.948 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,218 kB] 2023-09-29T21:39:16.015 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [309 kB] 2023-09-29T21:39:16.021 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [572 B] 2023-09-29T21:39:16.022 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [615 kB] 2023-09-29T21:39:16.032 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [886 kB] 2023-09-29T21:39:16.230 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [185 kB] 2023-09-29T21:39:16.644 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.1 kB] 2023-09-29T21:39:16.650 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-09-29T21:39:16.657 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,196 B] 2023-09-29T21:39:16.660 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-09-29T21:39:16.662 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-09-29T21:39:18.992 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 17.4 MB in 2s (7,057 kB/s) 2023-09-29T21:39:19.184 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 17.4 MB in 3s (6,661 kB/s) 2023-09-29T21:39:20.436 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2023-09-29T21:39:20.463 DEBUG:teuthology.orchestra.run.smithi131:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-29T21:39:20.537 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2023-09-29T21:39:20.541 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2023-09-29T21:39:20.566 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-29T21:39:20.634 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2023-09-29T21:39:20.739 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2023-09-29T21:39:20.739 INFO:teuthology.orchestra.run.smithi131.stdout:Reading state information... 2023-09-29T21:39:20.834 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2023-09-29T21:39:20.835 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2023-09-29T21:39:20.921 INFO:teuthology.orchestra.run.smithi131.stdout:The following additional packages will be installed: 2023-09-29T21:39:20.922 INFO:teuthology.orchestra.run.smithi131.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-29T21:39:20.922 INFO:teuthology.orchestra.run.smithi131.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-29T21:39:20.922 INFO:teuthology.orchestra.run.smithi131.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-29T21:39:20.923 INFO:teuthology.orchestra.run.smithi131.stdout:Suggested packages: 2023-09-29T21:39:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-29T21:39:20.923 INFO:teuthology.orchestra.run.smithi131.stdout:Recommended packages: 2023-09-29T21:39:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: thermald 2023-09-29T21:39:20.987 INFO:teuthology.orchestra.run.smithi131.stdout:The following NEW packages will be installed: 2023-09-29T21:39:20.988 INFO:teuthology.orchestra.run.smithi131.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-29T21:39:20.988 INFO:teuthology.orchestra.run.smithi131.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-29T21:39:20.988 INFO:teuthology.orchestra.run.smithi131.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-29T21:39:20.989 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages will be upgraded: 2023-09-29T21:39:20.989 INFO:teuthology.orchestra.run.smithi131.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-29T21:39:21.022 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2023-09-29T21:39:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-29T21:39:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-generic linux-image-5.4.0-163-generic 2023-09-29T21:39:21.023 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-5.4.0-163-generic linux-modules-extra-5.4.0-163-generic 2023-09-29T21:39:21.024 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2023-09-29T21:39:21.024 INFO:teuthology.orchestra.run.smithi112.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-09-29T21:39:21.024 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2023-09-29T21:39:21.025 INFO:teuthology.orchestra.run.smithi112.stdout: thermald 2023-09-29T21:39:21.047 INFO:teuthology.orchestra.run.smithi131.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-29T21:39:21.047 INFO:teuthology.orchestra.run.smithi131.stdout:Need to get 77.1 MB of archives. 2023-09-29T21:39:21.047 INFO:teuthology.orchestra.run.smithi131.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-29T21:39:21.048 INFO:teuthology.orchestra.run.smithi131.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-29T21:39:21.072 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2023-09-29T21:39:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: linux-headers-5.4.0-163 linux-headers-5.4.0-163-generic 2023-09-29T21:39:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: linux-image-5.4.0-163-generic linux-modules-5.4.0-163-generic 2023-09-29T21:39:21.073 INFO:teuthology.orchestra.run.smithi112.stdout: linux-modules-extra-5.4.0-163-generic 2023-09-29T21:39:21.074 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2023-09-29T21:39:21.074 INFO:teuthology.orchestra.run.smithi112.stdout: linux-generic linux-headers-generic linux-image-generic 2023-09-29T21:39:21.257 INFO:teuthology.orchestra.run.smithi112.stdout:3 upgraded, 5 newly installed, 0 to remove and 263 not upgraded. 2023-09-29T21:39:21.257 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 77.1 MB of archives. 2023-09-29T21:39:21.257 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 380 MB of additional disk space will be used. 2023-09-29T21:39:21.257 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-163-generic amd64 5.4.0-163.180 [15.0 MB] 2023-09-29T21:39:21.401 INFO:teuthology.orchestra.run.smithi131.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-29T21:39:21.495 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-29T21:39:22.066 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-29T21:39:22.066 INFO:teuthology.orchestra.run.smithi131.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-29T21:39:22.066 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-29T21:39:22.308 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-29T21:39:22.328 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-29T21:39:22.377 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-163-generic amd64 5.4.0-163.180 [10.5 MB] 2023-09-29T21:39:22.649 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-163-generic amd64 5.4.0-163.180 [39.2 MB] 2023-09-29T21:39:22.994 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 77.1 MB in 1s (57.8 MB/s) 2023-09-29T21:39:23.182 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-29T21:39:24.000 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.163.160 [1,904 B] 2023-09-29T21:39:24.000 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.163.160 [2,604 B] 2023-09-29T21:39:24.001 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163 all 5.4.0-163.180 [11.0 MB] 2023-09-29T21:39:24.287 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-163-generic amd64 5.4.0-163.180 [1,358 kB] 2023-09-29T21:39:24.316 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.163.160 [2,468 B] 2023-09-29T21:39:24.870 INFO:teuthology.orchestra.run.smithi131.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-29T21:39:24.875 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:24.962 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:24.974 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 77.1 MB in 3s (23.8 MB/s) 2023-09-29T21:39:25.164 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-5.4.0-163-generic. 2023-09-29T21:39:26.867 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-09-29T21:39:26.872 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../0-linux-modules-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:26.879 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-29T21:39:26.892 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:26.961 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:26.992 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:28.073 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-29T21:39:28.086 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:28.148 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:28.962 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-image-5.4.0-163-generic. 2023-09-29T21:39:28.978 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../1-linux-image-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:29.097 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:30.198 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-163-generic. 2023-09-29T21:39:30.212 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:30.256 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:32.872 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T21:39:32.997 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T21:39:33.314 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T21:39:33.459 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T21:39:33.787 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-29T21:39:33.803 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-29T21:39:33.845 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-29T21:39:35.077 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../3-linux-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T21:39:35.212 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T21:39:35.580 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T21:39:35.690 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-image-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T21:39:35.985 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.4.0-163. 2023-09-29T21:39:36.000 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../5-linux-headers-5.4.0-163_5.4.0-163.180_all.deb ... 2023-09-29T21:39:36.060 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-29T21:39:39.209 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-29T21:39:39.232 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:39.276 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:41.566 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package linux-headers-5.4.0-163-generic. 2023-09-29T21:39:41.589 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../6-linux-headers-5.4.0-163-generic_5.4.0-163.180_amd64.deb ... 2023-09-29T21:39:41.632 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:41.934 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T21:39:42.061 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T21:39:42.413 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:43.616 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-29T21:39:43.742 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:44.313 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.163.160_amd64.deb ... 2023-09-29T21:39:44.441 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking linux-headers-generic (5.4.0.163.160) over (5.4.0.124.125) ... 2023-09-29T21:39:44.775 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:44.971 INFO:teuthology.orchestra.run.smithi131.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-29T21:39:44.971 INFO:teuthology.orchestra.run.smithi131.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-29T21:39:45.619 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:45.745 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:45.971 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.4.0-163 (5.4.0-163.180) ... 2023-09-29T21:39:46.097 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:46.938 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-29T21:39:47.064 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-29T21:39:47.190 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-29T21:39:47.242 INFO:teuthology.orchestra.run.smithi112.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-163-generic 2023-09-29T21:39:47.242 INFO:teuthology.orchestra.run.smithi112.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-163-generic 2023-09-29T21:39:47.342 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:47.374 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:47.414 INFO:teuthology.orchestra.run.smithi131.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-29T21:39:47.414 INFO:teuthology.orchestra.run.smithi131.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-29T21:39:47.500 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-modules-extra-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:49.111 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-image-generic (5.4.0.163.160) ... 2023-09-29T21:39:49.237 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-headers-generic (5.4.0.163.160) ... 2023-09-29T21:39:49.363 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up linux-generic (5.4.0.163.160) ... 2023-09-29T21:39:49.506 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for linux-image-5.4.0-163-generic (5.4.0-163.180) ... 2023-09-29T21:39:49.576 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-09-29T21:39:49.576 INFO:teuthology.orchestra.run.smithi112.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-163-generic 2023-09-29T21:40:09.312 INFO:teuthology.orchestra.run.smithi131.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-29T21:40:09.313 INFO:teuthology.orchestra.run.smithi131.stdout:Sourcing file `/etc/default/grub' 2023-09-29T21:40:09.324 INFO:teuthology.orchestra.run.smithi131.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-29T21:40:09.341 INFO:teuthology.orchestra.run.smithi131.stdout:Generating grub configuration file ... 2023-09-29T21:40:09.644 INFO:teuthology.orchestra.run.smithi131.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-29T21:40:09.667 INFO:teuthology.orchestra.run.smithi131.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-29T21:40:10.083 INFO:teuthology.orchestra.run.smithi131.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-29T21:40:10.098 INFO:teuthology.orchestra.run.smithi131.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-29T21:40:11.296 INFO:teuthology.orchestra.run.smithi112.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-09-29T21:40:11.296 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub' 2023-09-29T21:40:11.307 INFO:teuthology.orchestra.run.smithi112.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-29T21:40:11.318 INFO:teuthology.orchestra.run.smithi112.stdout:Generating grub configuration file ... 2023-09-29T21:40:11.635 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-29T21:40:11.658 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-29T21:40:11.930 INFO:teuthology.orchestra.run.smithi131.stdout:done 2023-09-29T21:40:12.069 INFO:teuthology.orchestra.run.smithi112.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-29T21:40:12.081 INFO:teuthology.orchestra.run.smithi112.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-29T21:40:12.326 DEBUG:teuthology.orchestra.run.smithi131:> dpkg -s linux-image-generic 2023-09-29T21:40:12.373 INFO:teuthology.orchestra.run.smithi131.stdout:Package: linux-image-generic 2023-09-29T21:40:12.373 INFO:teuthology.orchestra.run.smithi131.stdout:Status: install ok installed 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Priority: optional 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Section: kernel 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Installed-Size: 20 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Maintainer: Ubuntu Kernel Team 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: amd64 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Source: linux-meta 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Version: 5.4.0.163.160 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-29T21:40:12.374 INFO:teuthology.orchestra.run.smithi131.stdout:Recommends: thermald 2023-09-29T21:40:12.375 INFO:teuthology.orchestra.run.smithi131.stdout:Description: Generic Linux kernel image 2023-09-29T21:40:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: This package will always depend on the latest generic kernel image 2023-09-29T21:40:12.375 INFO:teuthology.orchestra.run.smithi131.stdout: available. 2023-09-29T21:40:12.376 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-29T21:40:12.376 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T21:40:12.376 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T21:40:12.376 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2023-09-29T21:40:12.376 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T21:40:12.376 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get clean 2023-09-29T21:40:12.481 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get update 2023-09-29T21:40:12.609 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T21:40:12.615 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-29T21:40:12.619 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-29T21:40:12.632 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-29T21:40:14.353 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2023-09-29T21:40:14.378 DEBUG:teuthology.orchestra.run.smithi131:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-29T21:40:14.444 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2023-09-29T21:40:14.646 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2023-09-29T21:40:14.647 INFO:teuthology.orchestra.run.smithi131.stdout:Reading state information... 2023-09-29T21:40:14.800 INFO:teuthology.orchestra.run.smithi112.stdout:done 2023-09-29T21:40:14.854 INFO:teuthology.orchestra.run.smithi131.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-29T21:40:14.854 INFO:teuthology.orchestra.run.smithi131.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-29T21:40:14.856 DEBUG:teuthology.orchestra.run.smithi131:> dpkg -s linux-image-generic 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Package: linux-image-generic 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Status: install ok installed 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Priority: optional 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Section: kernel 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Installed-Size: 20 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Maintainer: Ubuntu Kernel Team 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: amd64 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Source: linux-meta 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Version: 5.4.0.163.160 2023-09-29T21:40:14.879 INFO:teuthology.orchestra.run.smithi131.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-29T21:40:14.880 INFO:teuthology.orchestra.run.smithi131.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-29T21:40:14.880 INFO:teuthology.orchestra.run.smithi131.stdout:Recommends: thermald 2023-09-29T21:40:14.880 INFO:teuthology.orchestra.run.smithi131.stdout:Description: Generic Linux kernel image 2023-09-29T21:40:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: This package will always depend on the latest generic kernel image 2023-09-29T21:40:14.880 INFO:teuthology.orchestra.run.smithi131.stdout: available. 2023-09-29T21:40:14.880 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2023-09-29T21:40:14.885 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.C9BE9lwDuw 2023-09-29T21:40:14.885 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub/grub.cfg /tmp/tmp.C9BE9lwDuw 2023-09-29T21:40:14.895 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.C9BE9lwDuw 2023-09-29T21:40:14.973 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.C9BE9lwDuw is 9KB 2023-09-29T21:40:14.988 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.C9BE9lwDuw 2023-09-29T21:40:14.994 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-29T21:40:15.053 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:40:15.053 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-29T21:40:15.053 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-29T21:40:15.100 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-29T21:40:15.101 DEBUG:teuthology.orchestra.run.smithi131:> sudo update-grub 2023-09-29T21:40:15.241 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 20 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2023-09-29T21:40:15.285 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.4.0.163.160 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2023-09-29T21:40:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2023-09-29T21:40:15.287 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-163-generic 2023-09-29T21:40:15.287 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-29T21:40:15.287 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-29T21:40:15.287 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2023-09-29T21:40:15.287 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi112.front.sepia.ceph.com, path=None, version=distro) 2023-09-29T21:40:15.287 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get clean 2023-09-29T21:40:15.381 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2023-09-29T21:40:15.513 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-29T21:40:15.625 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T21:40:15.707 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-29T21:40:15.789 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-29T21:40:16.391 INFO:teuthology.orchestra.run.smithi131.stderr:Sourcing file `/etc/default/grub' 2023-09-29T21:40:16.398 INFO:teuthology.orchestra.run.smithi131.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-29T21:40:16.407 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2023-09-29T21:40:16.634 INFO:teuthology.orchestra.run.smithi131.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-29T21:40:16.646 INFO:teuthology.orchestra.run.smithi131.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-29T21:40:17.039 INFO:teuthology.orchestra.run.smithi131.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-29T21:40:17.050 INFO:teuthology.orchestra.run.smithi131.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-29T21:40:17.250 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2023-09-29T21:40:17.275 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-09-29T21:40:17.342 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2023-09-29T21:40:17.543 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2023-09-29T21:40:17.544 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2023-09-29T21:40:17.756 INFO:teuthology.orchestra.run.smithi112.stdout:linux-image-generic is already the newest version (5.4.0.163.160). 2023-09-29T21:40:17.757 INFO:teuthology.orchestra.run.smithi112.stdout:0 upgraded, 0 newly installed, 0 to remove and 263 not upgraded. 2023-09-29T21:40:17.759 DEBUG:teuthology.orchestra.run.smithi112:> dpkg -s linux-image-generic 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Package: linux-image-generic 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Status: install ok installed 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Priority: optional 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Section: kernel 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Installed-Size: 20 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Maintainer: Ubuntu Kernel Team 2023-09-29T21:40:17.780 INFO:teuthology.orchestra.run.smithi112.stdout:Architecture: amd64 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout:Source: linux-meta 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout:Version: 5.4.0.163.160 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout:Depends: linux-image-5.4.0-163-generic, linux-modules-extra-5.4.0-163-generic, linux-firmware, intel-microcode, amd64-microcode 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout:Recommends: thermald 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout:Description: Generic Linux kernel image 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout: This package will always depend on the latest generic kernel image 2023-09-29T21:40:17.781 INFO:teuthology.orchestra.run.smithi112.stdout: available. 2023-09-29T21:40:17.781 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-09-29T21:40:17.787 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.TGburzyZ6R 2023-09-29T21:40:17.787 DEBUG:teuthology.orchestra.run.smithi112:> sudo cp /boot/grub/grub.cfg /tmp/tmp.TGburzyZ6R 2023-09-29T21:40:17.796 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.TGburzyZ6R 2023-09-29T21:40:17.860 INFO:teuthology.orchestra.run.smithi131.stderr:done 2023-09-29T21:40:17.861 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2023-09-29T21:40:17.873 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.TGburzyZ6R is 9KB 2023-09-29T21:40:17.887 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.TGburzyZ6R 2023-09-29T21:40:17.893 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-09-29T21:40:17.950 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:40:17.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-09-29T21:40:17.950 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-09-29T21:40:17.992 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-163-generic 2023-09-29T21:40:17.992 DEBUG:teuthology.orchestra.run.smithi112:> sudo update-grub 2023-09-29T21:40:19.296 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub' 2023-09-29T21:40:19.301 INFO:teuthology.orchestra.run.smithi112.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-09-29T21:40:19.317 INFO:teuthology.orchestra.run.smithi112.stderr:Generating grub configuration file ... 2023-09-29T21:40:19.556 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-163-generic 2023-09-29T21:40:19.568 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-163-generic 2023-09-29T21:40:19.956 INFO:teuthology.orchestra.run.smithi112.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-09-29T21:40:19.968 INFO:teuthology.orchestra.run.smithi112.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-09-29T21:40:21.623 INFO:teuthology.orchestra.run.smithi112.stderr:done 2023-09-29T21:40:21.625 DEBUG:teuthology.orchestra.run.smithi112:> sudo shutdown -r now 2023-09-29T21:40:47.890 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2023-09-29T21:40:47.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-09-29T21:40:47.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:40:51.632 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2023-09-29T21:40:51.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-09-29T21:40:51.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:41:06.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.131 2023-09-29T21:41:10.138 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-09-29T21:41:15.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-09-29T21:41:15.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:41:18.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.131 2023-09-29T21:41:19.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-09-29T21:41:19.141 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:41:22.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-09-29T21:41:30.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-09-29T21:41:30.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:41:34.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-09-29T21:41:34.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:42:02.050 DEBUG:teuthology.orchestra.run.smithi131:> true 2023-09-29T21:42:02.678 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-09-29T21:42:02.678 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "5.4.0-163-generic"... 2023-09-29T21:42:02.678 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2023-09-29T21:42:02.685 INFO:teuthology.orchestra.run.smithi131.stdout:5.4.0-163-generic 2023-09-29T21:42:02.686 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-29T21:42:02.686 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T21:42:02.686 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-29T21:42:03.687 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2023-09-29T21:42:03.687 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T21:42:03.770 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2023-09-29T21:42:03.798 DEBUG:teuthology.parallel:result is None 2023-09-29T21:42:05.900 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-09-29T21:42:06.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-09-29T21:42:06.502 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "5.4.0-163-generic"... 2023-09-29T21:42:06.502 DEBUG:teuthology.orchestra.run.smithi112:> uname -r 2023-09-29T21:42:06.509 INFO:teuthology.orchestra.run.smithi112.stdout:5.4.0-163-generic 2023-09-29T21:42:06.509 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-163-generic vs 5.4.0-163-generic 2023-09-29T21:42:06.509 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-29T21:42:06.509 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-09-29T21:42:07.510 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2023-09-29T21:42:07.510 DEBUG:teuthology.orchestra.run.smithi112:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-29T21:42:07.592 INFO:teuthology.orchestra.run.smithi112.stdout:ttyS1 2023-09-29T21:42:07.628 DEBUG:teuthology.parallel:result is None 2023-09-29T21:42:07.628 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-29T21:42:07.635 INFO:teuthology.task.internal:Creating test directory... 2023-09-29T21:42:07.635 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T21:42:07.638 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-29T21:42:07.644 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-29T21:42:07.650 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-29T21:42:07.660 INFO:teuthology.task.internal:Creating archive directory... 2023-09-29T21:42:07.660 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T21:42:07.662 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-29T21:42:07.689 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-29T21:42:07.744 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-29T21:42:07.745 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T21:42:07.747 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-29T21:42:07.762 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:42:07.772 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:42:07.780 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:42:07.791 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-29T21:42:07.793 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-29T21:42:07.800 INFO:teuthology.task.internal:Configuring sudo... 2023-09-29T21:42:07.800 DEBUG:teuthology.orchestra.run.smithi112:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T21:42:07.802 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-29T21:42:07.818 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-29T21:42:07.827 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-29T21:42:07.827 DEBUG:teuthology.orchestra.run.smithi112:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T21:42:07.829 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-29T21:42:07.866 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T21:42:07.880 DEBUG:teuthology.orchestra.run.smithi112:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T21:42:07.887 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:42:07.887 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T21:42:07.909 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-29T21:42:07.916 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-29T21:42:07.966 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:42:07.966 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-29T21:42:07.988 DEBUG:teuthology.orchestra.run.smithi112:> sudo service rsyslog restart 2023-09-29T21:42:07.990 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2023-09-29T21:42:08.042 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-29T21:42:08.049 INFO:teuthology.task.internal:Starting timer... 2023-09-29T21:42:08.049 INFO:teuthology.run_tasks:Running task pcp... 2023-09-29T21:42:08.111 INFO:teuthology.run_tasks:Running task selinux... 2023-09-29T21:42:08.119 DEBUG:teuthology.task.selinux:Excluding smithi112: OS 'ubuntu' does not support SELinux 2023-09-29T21:42:08.119 DEBUG:teuthology.task.selinux:Excluding smithi131: OS 'ubuntu' does not support SELinux 2023-09-29T21:42:08.119 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-29T21:42:08.119 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-09-29T21:42:08.119 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-29T21:42:08.120 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-29T21:42:08.128 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-09-29T21:42:08.134 INFO:teuthology.repo_utils:Fetching main from origin 2023-09-29T21:42:08.252 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-29T21:42:08.265 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-29T21:42:08.266 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi112.front.sepia.ceph.com,smithi131.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-29T21:47:48.018 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi112.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com')] 2023-09-29T21:47:48.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-09-29T21:47:48.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi112.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:47:48.068 DEBUG:teuthology.orchestra.run.smithi112:> true 2023-09-29T21:47:48.187 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-09-29T21:47:48.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-09-29T21:47:48.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-29T21:47:48.238 DEBUG:teuthology.orchestra.run.smithi131:> true 2023-09-29T21:47:48.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2023-09-29T21:47:48.360 INFO:teuthology.run_tasks:Running task clock... 2023-09-29T21:47:48.373 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-29T21:47:48.373 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T21:47:48.373 DEBUG:teuthology.orchestra.run.smithi112:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T21:47:48.377 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-29T21:47:48.377 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T21:47:48.407 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-29T21:47:48.407 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Command line: ntpd -gq 2023-09-29T21:47:48.409 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: proto: precision = 0.116 usec (-23) 2023-09-29T21:47:48.409 INFO:teuthology.orchestra.run.smithi112.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-29T21:47:48.410 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-29T21:47:48.410 INFO:teuthology.orchestra.run.smithi112.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: restrict ::: KOD does nothing without LIMITED. 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listen and drop on 0 v6wildcard [::]:123 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listen normally on 2 lo 127.0.0.1:123 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listen normally on 3 enp3s0f1 172.21.15.112:123 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listen normally on 4 lo [::1]:123 2023-09-29T21:47:48.411 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:cb69%5]:123 2023-09-29T21:47:48.413 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:48 ntpd[22683]: Listening on routing socket on fd #22 for interface updates 2023-09-29T21:47:48.440 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-09-29T21:47:48.440 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Command line: ntpd -gq 2023-09-29T21:47:48.442 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: proto: precision = 0.087 usec (-23) 2023-09-29T21:47:48.442 INFO:teuthology.orchestra.run.smithi131.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-29T21:47:48.442 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-09-29T21:47:48.443 INFO:teuthology.orchestra.run.smithi131.stderr:restrict ::: KOD does nothing without LIMITED. 2023-09-29T21:47:48.443 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: restrict ::: KOD does nothing without LIMITED. 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listen and drop on 0 v6wildcard [::]:123 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listen normally on 2 lo 127.0.0.1:123 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listen normally on 3 enp3s0f1 172.21.15.131:123 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listen normally on 4 lo [::1]:123 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:72f9%5]:123 2023-09-29T21:47:48.444 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:48 ntpd[22751]: Listening on routing socket on fd #22 for interface updates 2023-09-29T21:47:49.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:49.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:49 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:49.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:49 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:49.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:49 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:50 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:50.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:50 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:50 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:50.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:50 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:50 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:50.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:50 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:50.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:50.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:50.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:50.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:50.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:50 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:51.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:51 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:51.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:51 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:51.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:51 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:51.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:51 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:51.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:51 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:51.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:51 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:51 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:51.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:51 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:51 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:51 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:51 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:51.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:51 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:52.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:52 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:52.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:52 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:52.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:52 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:52.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:52 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:52.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:52 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:52.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:52 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:52.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:52.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:52.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:52.445 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:52.445 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:52 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:53.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:53.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:53.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:53.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:53.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:53.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:53 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:53.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:53 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:53.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:53 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:53.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:53 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:53.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:53 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:53.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:53 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:53.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:53 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:54.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:54 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:54.408 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:54 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:54.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:54 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:54.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:54 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:54.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:54 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:54.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:54 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:54.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:54.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:54.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:54.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:54.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:54.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:54 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:55.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:55.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:55.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:55.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:55.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:55.409 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stdout:29 Sep 21:47:55 ntpd[22683]: ntpd: time slew +0.000377 s 2023-09-29T21:47:55.410 INFO:teuthology.orchestra.run.smithi112.stdout:ntpd: time slew +0.000377s 2023-09-29T21:47:55.411 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-29T21:47:55.411 INFO:teuthology.orchestra.run.smithi112.stderr:29 Sep 21:47:55 ntpd[22683]: can't open /var/log/ntpstats/loopstats.20230929: Permission denied 2023-09-29T21:47:55.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:55 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:55.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:55 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:55.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:55 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:55.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:55 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:55.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:55 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:55.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:55 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2023-09-29T21:47:55.473 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2023-09-29T21:47:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:55.473 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:56.441 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: can't open /var/log/ntpstats/rawstats.20230929: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-09-29T21:47:56.442 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: can't open /var/log/ntpstats/peerstats.20230929: Permission denied 2023-09-29T21:47:56.443 INFO:teuthology.orchestra.run.smithi131.stdout:29 Sep 21:47:56 ntpd[22751]: ntpd: time slew -0.000532 s 2023-09-29T21:47:56.443 INFO:teuthology.orchestra.run.smithi131.stdout:ntpd: time slew -0.000532s 2023-09-29T21:47:56.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-09-29T21:47:56.443 INFO:teuthology.orchestra.run.smithi131.stderr:29 Sep 21:47:56 ntpd[22751]: can't open /var/log/ntpstats/loopstats.20230929: Permission denied 2023-09-29T21:47:56.503 INFO:teuthology.orchestra.run.smithi131.stdout: remote refid st t when poll reach delay offset jitter 2023-09-29T21:47:56.503 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================== 2023-09-29T21:47:56.503 INFO:teuthology.orchestra.run.smithi131.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:56.503 INFO:teuthology.orchestra.run.smithi131.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:56.503 INFO:teuthology.orchestra.run.smithi131.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:56.503 INFO:teuthology.orchestra.run.smithi131.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-09-29T21:47:56.504 INFO:teuthology.run_tasks:Running task install... 2023-09-29T21:47:56.513 DEBUG:teuthology.task.install:project ceph 2023-09-29T21:47:56.513 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}} 2023-09-29T21:47:56.513 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'} 2023-09-29T21:47:56.513 INFO:teuthology.task.install:Using flavor: default 2023-09-29T21:47:56.519 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-29T21:47:56.519 INFO:teuthology.task.install:extra packages: [] 2023-09-29T21:47:56.519 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-key list | grep Ceph 2023-09-29T21:47:56.520 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-key list | grep Ceph 2023-09-29T21:47:56.614 INFO:teuthology.orchestra.run.smithi131.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-29T21:47:56.623 INFO:teuthology.orchestra.run.smithi112.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-09-29T21:47:56.673 INFO:teuthology.orchestra.run.smithi131.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-29T21:47:56.673 INFO:teuthology.orchestra.run.smithi131.stdout:uid [ unknown] Ceph.com (release key) 2023-09-29T21:47:56.674 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-29T21:47:56.675 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T21:47:56.682 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-09-29T21:47:56.682 INFO:teuthology.orchestra.run.smithi112.stdout:uid [ unknown] Ceph.com (release key) 2023-09-29T21:47:56.683 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-09-29T21:47:56.683 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T21:47:56.829 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default/ 2023-09-29T21:47:56.830 INFO:teuthology.task.install.deb:Package version is 18.2.0-524-g38c3646d-1focal 2023-09-29T21:47:56.904 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default/ 2023-09-29T21:47:56.904 INFO:teuthology.task.install.deb:Package version is 18.2.0-524-g38c3646d-1focal 2023-09-29T21:47:56.968 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:47:56.968 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-29T21:47:56.985 DEBUG:teuthology.orchestra.run.smithi131:> sudo apt-get update 2023-09-29T21:47:57.081 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:47:57.081 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-09-29T21:47:57.098 DEBUG:teuthology.orchestra.run.smithi112:> sudo apt-get update 2023-09-29T21:47:57.191 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-29T21:47:57.223 INFO:teuthology.orchestra.run.smithi131.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal InRelease 2023-09-29T21:47:57.244 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-29T21:47:57.266 INFO:teuthology.orchestra.run.smithi131.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal Release.gpg 2023-09-29T21:47:57.288 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T21:47:57.288 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main i386 Packages [3,408 B] 2023-09-29T21:47:57.291 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-09-29T21:47:57.328 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2023-09-29T21:47:57.369 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-29T21:47:57.400 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal InRelease 2023-09-29T21:47:57.402 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal InRelease 2023-09-29T21:47:57.436 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-09-29T21:47:57.451 INFO:teuthology.orchestra.run.smithi131.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-29T21:47:57.473 INFO:teuthology.orchestra.run.smithi112.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal Release.gpg 2023-09-29T21:47:57.484 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-09-29T21:47:57.509 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main i386 Packages [3,408 B] 2023-09-29T21:47:57.566 INFO:teuthology.orchestra.run.smithi112.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-09-29T21:47:57.574 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 Packages [17.3 kB] 2023-09-29T21:47:57.870 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 28.4 kB in 1s (42.4 kB/s) 2023-09-29T21:47:57.958 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 28.4 kB in 1s (43.8 kB/s) 2023-09-29T21:47:58.910 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2023-09-29T21:47:58.936 DEBUG:teuthology.orchestra.run.smithi131:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.0-524-g38c3646d-1focal cephadm=18.2.0-524-g38c3646d-1focal ceph-mds=18.2.0-524-g38c3646d-1focal ceph-mgr=18.2.0-524-g38c3646d-1focal ceph-common=18.2.0-524-g38c3646d-1focal ceph-fuse=18.2.0-524-g38c3646d-1focal ceph-test=18.2.0-524-g38c3646d-1focal ceph-volume=18.2.0-524-g38c3646d-1focal radosgw=18.2.0-524-g38c3646d-1focal python3-rados=18.2.0-524-g38c3646d-1focal python3-rgw=18.2.0-524-g38c3646d-1focal python3-cephfs=18.2.0-524-g38c3646d-1focal python3-rbd=18.2.0-524-g38c3646d-1focal libcephfs2=18.2.0-524-g38c3646d-1focal libcephfs-dev=18.2.0-524-g38c3646d-1focal librados2=18.2.0-524-g38c3646d-1focal librbd1=18.2.0-524-g38c3646d-1focal rbd-fuse=18.2.0-524-g38c3646d-1focal 2023-09-29T21:47:58.985 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2023-09-29T21:47:59.004 INFO:teuthology.orchestra.run.smithi131.stdout:Reading package lists... 2023-09-29T21:47:59.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.0-524-g38c3646d-1focal cephadm=18.2.0-524-g38c3646d-1focal ceph-mds=18.2.0-524-g38c3646d-1focal ceph-mgr=18.2.0-524-g38c3646d-1focal ceph-common=18.2.0-524-g38c3646d-1focal ceph-fuse=18.2.0-524-g38c3646d-1focal ceph-test=18.2.0-524-g38c3646d-1focal ceph-volume=18.2.0-524-g38c3646d-1focal radosgw=18.2.0-524-g38c3646d-1focal python3-rados=18.2.0-524-g38c3646d-1focal python3-rgw=18.2.0-524-g38c3646d-1focal python3-cephfs=18.2.0-524-g38c3646d-1focal python3-rbd=18.2.0-524-g38c3646d-1focal libcephfs2=18.2.0-524-g38c3646d-1focal libcephfs-dev=18.2.0-524-g38c3646d-1focal librados2=18.2.0-524-g38c3646d-1focal librbd1=18.2.0-524-g38c3646d-1focal rbd-fuse=18.2.0-524-g38c3646d-1focal 2023-09-29T21:47:59.082 INFO:teuthology.orchestra.run.smithi112.stdout:Reading package lists... 2023-09-29T21:47:59.181 INFO:teuthology.orchestra.run.smithi131.stdout:Building dependency tree... 2023-09-29T21:47:59.182 INFO:teuthology.orchestra.run.smithi131.stdout:Reading state information... 2023-09-29T21:47:59.259 INFO:teuthology.orchestra.run.smithi112.stdout:Building dependency tree... 2023-09-29T21:47:59.259 INFO:teuthology.orchestra.run.smithi112.stdout:Reading state information... 2023-09-29T21:47:59.349 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages were automatically installed and are no longer required: 2023-09-29T21:47:59.350 INFO:teuthology.orchestra.run.smithi131.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-29T21:47:59.350 INFO:teuthology.orchestra.run.smithi131.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-29T21:47:59.350 INFO:teuthology.orchestra.run.smithi131.stdout:The following additional packages will be installed: 2023-09-29T21:47:59.351 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-29T21:47:59.351 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-09-29T21:47:59.352 INFO:teuthology.orchestra.run.smithi131.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-29T21:47:59.353 INFO:teuthology.orchestra.run.smithi131.stdout: qttranslations5-l10n socat xmlstarlet 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout:Suggested packages: 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-29T21:47:59.354 INFO:teuthology.orchestra.run.smithi131.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-29T21:47:59.355 INFO:teuthology.orchestra.run.smithi131.stdout:Recommended packages: 2023-09-29T21:47:59.355 INFO:teuthology.orchestra.run.smithi131.stdout: btrfs-tools 2023-09-29T21:47:59.426 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages were automatically installed and are no longer required: 2023-09-29T21:47:59.427 INFO:teuthology.orchestra.run.smithi112.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-09-29T21:47:59.427 INFO:teuthology.orchestra.run.smithi112.stdout:Use 'sudo apt autoremove' to remove them. 2023-09-29T21:47:59.428 INFO:teuthology.orchestra.run.smithi112.stdout:The following additional packages will be installed: 2023-09-29T21:47:59.428 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-09-29T21:47:59.428 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-09-29T21:47:59.428 INFO:teuthology.orchestra.run.smithi112.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-09-29T21:47:59.428 INFO:teuthology.orchestra.run.smithi112.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-09-29T21:47:59.428 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-09-29T21:47:59.429 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-09-29T21:47:59.430 INFO:teuthology.orchestra.run.smithi112.stdout: qttranslations5-l10n socat xmlstarlet 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout:Suggested packages: 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-09-29T21:47:59.431 INFO:teuthology.orchestra.run.smithi112.stdout:Recommended packages: 2023-09-29T21:47:59.432 INFO:teuthology.orchestra.run.smithi112.stdout: btrfs-tools 2023-09-29T21:47:59.467 INFO:teuthology.orchestra.run.smithi131.stdout:The following NEW packages will be installed: 2023-09-29T21:47:59.467 INFO:teuthology.orchestra.run.smithi131.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-29T21:47:59.467 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-29T21:47:59.467 INFO:teuthology.orchestra.run.smithi131.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-09-29T21:47:59.470 INFO:teuthology.orchestra.run.smithi131.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-09-29T21:47:59.471 INFO:teuthology.orchestra.run.smithi131.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-09-29T21:47:59.472 INFO:teuthology.orchestra.run.smithi131.stdout: xmlstarlet 2023-09-29T21:47:59.472 INFO:teuthology.orchestra.run.smithi131.stdout:The following packages will be upgraded: 2023-09-29T21:47:59.472 INFO:teuthology.orchestra.run.smithi131.stdout: librados2 librbd1 2023-09-29T21:47:59.543 INFO:teuthology.orchestra.run.smithi112.stdout:The following NEW packages will be installed: 2023-09-29T21:47:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-09-29T21:47:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-09-29T21:47:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-09-29T21:47:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-09-29T21:47:59.543 INFO:teuthology.orchestra.run.smithi112.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-09-29T21:47:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-09-29T21:47:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-09-29T21:47:59.544 INFO:teuthology.orchestra.run.smithi112.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-09-29T21:47:59.545 INFO:teuthology.orchestra.run.smithi112.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout: xmlstarlet 2023-09-29T21:47:59.546 INFO:teuthology.orchestra.run.smithi112.stdout:The following packages will be upgraded: 2023-09-29T21:47:59.547 INFO:teuthology.orchestra.run.smithi112.stdout: librados2 librbd1 2023-09-29T21:47:59.618 INFO:teuthology.orchestra.run.smithi131.stdout:2 upgraded, 110 newly installed, 0 to remove and 245 not upgraded. 2023-09-29T21:47:59.618 INFO:teuthology.orchestra.run.smithi131.stdout:Need to get 207 MB of archives. 2023-09-29T21:47:59.618 INFO:teuthology.orchestra.run.smithi131.stdout:After this operation, 1,027 MB of additional disk space will be used. 2023-09-29T21:47:59.618 INFO:teuthology.orchestra.run.smithi131.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.0-524-g38c3646d-1focal [3,431 kB] 2023-09-29T21:47:59.662 INFO:teuthology.orchestra.run.smithi131.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-29T21:47:59.690 INFO:teuthology.orchestra.run.smithi112.stdout:2 upgraded, 110 newly installed, 0 to remove and 245 not upgraded. 2023-09-29T21:47:59.690 INFO:teuthology.orchestra.run.smithi112.stdout:Need to get 207 MB of archives. 2023-09-29T21:47:59.690 INFO:teuthology.orchestra.run.smithi112.stdout:After this operation, 1,027 MB of additional disk space will be used. 2023-09-29T21:47:59.690 INFO:teuthology.orchestra.run.smithi112.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.0-524-g38c3646d-1focal [3,431 kB] 2023-09-29T21:47:59.737 INFO:teuthology.orchestra.run.smithi112.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-09-29T21:47:59.850 INFO:teuthology.orchestra.run.smithi131.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.0-524-g38c3646d-1focal [3,988 kB] 2023-09-29T21:47:59.992 INFO:teuthology.orchestra.run.smithi131.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.0-524-g38c3646d-1focal [723 kB] 2023-09-29T21:48:00.024 INFO:teuthology.orchestra.run.smithi131.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.0-524-g38c3646d-1focal [377 kB] 2023-09-29T21:48:00.037 INFO:teuthology.orchestra.run.smithi131.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.0-524-g38c3646d-1focal [22.5 kB] 2023-09-29T21:48:00.037 INFO:teuthology.orchestra.run.smithi131.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.0-524-g38c3646d-1focal [184 kB] 2023-09-29T21:48:00.043 INFO:teuthology.orchestra.run.smithi131.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.0-524-g38c3646d-1focal [50.9 kB] 2023-09-29T21:48:00.046 INFO:teuthology.orchestra.run.smithi131.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.0-524-g38c3646d-1focal [355 kB] 2023-09-29T21:48:00.057 INFO:teuthology.orchestra.run.smithi131.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.0-524-g38c3646d-1focal [6,564 kB] 2023-09-29T21:48:00.058 INFO:teuthology.orchestra.run.smithi131.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-29T21:48:00.072 INFO:teuthology.orchestra.run.smithi131.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-29T21:48:00.132 INFO:teuthology.orchestra.run.smithi112.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-09-29T21:48:00.146 INFO:teuthology.orchestra.run.smithi112.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-09-29T21:48:00.155 INFO:teuthology.orchestra.run.smithi131.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-29T21:48:00.230 INFO:teuthology.orchestra.run.smithi112.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-09-29T21:48:00.317 INFO:teuthology.orchestra.run.smithi131.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.0-524-g38c3646d-1focal [104 kB] 2023-09-29T21:48:00.320 INFO:teuthology.orchestra.run.smithi131.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.0-524-g38c3646d-1focal [466 kB] 2023-09-29T21:48:00.336 INFO:teuthology.orchestra.run.smithi131.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.0-524-g38c3646d-1focal [25.1 MB] 2023-09-29T21:48:00.399 INFO:teuthology.orchestra.run.smithi131.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-29T21:48:00.409 INFO:teuthology.orchestra.run.smithi131.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-29T21:48:00.437 INFO:teuthology.orchestra.run.smithi131.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-29T21:48:00.446 INFO:teuthology.orchestra.run.smithi131.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-29T21:48:00.447 INFO:teuthology.orchestra.run.smithi131.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-29T21:48:00.450 INFO:teuthology.orchestra.run.smithi131.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-29T21:48:00.451 INFO:teuthology.orchestra.run.smithi131.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-29T21:48:00.473 INFO:teuthology.orchestra.run.smithi112.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-09-29T21:48:00.484 INFO:teuthology.orchestra.run.smithi112.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-09-29T21:48:00.512 INFO:teuthology.orchestra.run.smithi112.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-09-29T21:48:00.520 INFO:teuthology.orchestra.run.smithi112.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.0-524-g38c3646d-1focal [3,988 kB] 2023-09-29T21:48:00.521 INFO:teuthology.orchestra.run.smithi112.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-09-29T21:48:00.521 INFO:teuthology.orchestra.run.smithi112.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-09-29T21:48:00.525 INFO:teuthology.orchestra.run.smithi112.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-09-29T21:48:00.526 INFO:teuthology.orchestra.run.smithi112.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-09-29T21:48:00.628 INFO:teuthology.orchestra.run.smithi131.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-29T21:48:00.703 INFO:teuthology.orchestra.run.smithi112.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-09-29T21:48:00.876 INFO:teuthology.orchestra.run.smithi131.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-29T21:48:00.951 INFO:teuthology.orchestra.run.smithi112.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-09-29T21:48:01.047 INFO:teuthology.orchestra.run.smithi131.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-29T21:48:01.060 INFO:teuthology.orchestra.run.smithi131.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-29T21:48:01.067 INFO:teuthology.orchestra.run.smithi131.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-29T21:48:01.074 INFO:teuthology.orchestra.run.smithi131.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-29T21:48:01.102 INFO:teuthology.orchestra.run.smithi131.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-29T21:48:01.106 INFO:teuthology.orchestra.run.smithi131.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-29T21:48:01.110 INFO:teuthology.orchestra.run.smithi131.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-29T21:48:01.122 INFO:teuthology.orchestra.run.smithi112.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-09-29T21:48:01.135 INFO:teuthology.orchestra.run.smithi112.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-09-29T21:48:01.140 INFO:teuthology.orchestra.run.smithi131.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-29T21:48:01.143 INFO:teuthology.orchestra.run.smithi112.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-09-29T21:48:01.148 INFO:teuthology.orchestra.run.smithi131.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-29T21:48:01.150 INFO:teuthology.orchestra.run.smithi112.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-09-29T21:48:01.177 INFO:teuthology.orchestra.run.smithi112.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-09-29T21:48:01.182 INFO:teuthology.orchestra.run.smithi112.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-09-29T21:48:01.185 INFO:teuthology.orchestra.run.smithi112.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-09-29T21:48:01.216 INFO:teuthology.orchestra.run.smithi112.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-09-29T21:48:01.223 INFO:teuthology.orchestra.run.smithi112.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-09-29T21:48:01.331 INFO:teuthology.orchestra.run.smithi131.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-29T21:48:01.388 INFO:teuthology.orchestra.run.smithi112.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.0-524-g38c3646d-1focal [723 kB] 2023-09-29T21:48:01.406 INFO:teuthology.orchestra.run.smithi112.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-09-29T21:48:01.422 INFO:teuthology.orchestra.run.smithi131.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.0-524-g38c3646d-1focal [11.0 MB] 2023-09-29T21:48:01.477 INFO:teuthology.orchestra.run.smithi112.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.0-524-g38c3646d-1focal [377 kB] 2023-09-29T21:48:01.546 INFO:teuthology.orchestra.run.smithi112.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.0-524-g38c3646d-1focal [22.5 kB] 2023-09-29T21:48:01.547 INFO:teuthology.orchestra.run.smithi112.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.0-524-g38c3646d-1focal [184 kB] 2023-09-29T21:48:01.609 INFO:teuthology.orchestra.run.smithi112.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.0-524-g38c3646d-1focal [50.9 kB] 2023-09-29T21:48:01.610 INFO:teuthology.orchestra.run.smithi112.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.0-524-g38c3646d-1focal [355 kB] 2023-09-29T21:48:01.643 INFO:teuthology.orchestra.run.smithi131.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-29T21:48:01.655 INFO:teuthology.orchestra.run.smithi131.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-29T21:48:01.678 INFO:teuthology.orchestra.run.smithi112.stdout:Get:32 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.0-524-g38c3646d-1focal [6,564 kB] 2023-09-29T21:48:01.719 INFO:teuthology.orchestra.run.smithi112.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-09-29T21:48:01.723 INFO:teuthology.orchestra.run.smithi131.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.0-524-g38c3646d-1focal [224 kB] 2023-09-29T21:48:01.729 INFO:teuthology.orchestra.run.smithi131.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.0-524-g38c3646d-1focal [133 kB] 2023-09-29T21:48:01.731 INFO:teuthology.orchestra.run.smithi112.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-09-29T21:48:01.734 INFO:teuthology.orchestra.run.smithi131.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.0-524-g38c3646d-1focal [1,551 kB] 2023-09-29T21:48:01.796 INFO:teuthology.orchestra.run.smithi131.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.0-524-g38c3646d-1focal [9,399 kB] 2023-09-29T21:48:01.891 INFO:teuthology.orchestra.run.smithi131.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-29T21:48:01.894 INFO:teuthology.orchestra.run.smithi131.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-29T21:48:01.901 INFO:teuthology.orchestra.run.smithi131.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-29T21:48:01.907 INFO:teuthology.orchestra.run.smithi131.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-29T21:48:01.923 INFO:teuthology.orchestra.run.smithi131.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-29T21:48:01.948 INFO:teuthology.orchestra.run.smithi131.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-29T21:48:01.966 INFO:teuthology.orchestra.run.smithi112.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-09-29T21:48:01.967 INFO:teuthology.orchestra.run.smithi112.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-09-29T21:48:01.975 INFO:teuthology.orchestra.run.smithi112.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-09-29T21:48:01.977 INFO:teuthology.orchestra.run.smithi131.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-29T21:48:01.981 INFO:teuthology.orchestra.run.smithi112.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-09-29T21:48:01.981 INFO:teuthology.orchestra.run.smithi131.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-09-29T21:48:01.997 INFO:teuthology.orchestra.run.smithi112.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.1 [183 kB] 2023-09-29T21:48:02.024 INFO:teuthology.orchestra.run.smithi112.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-09-29T21:48:02.050 INFO:teuthology.orchestra.run.smithi112.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-09-29T21:48:02.056 INFO:teuthology.orchestra.run.smithi112.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-09-29T21:48:02.162 INFO:teuthology.orchestra.run.smithi131.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-29T21:48:02.168 INFO:teuthology.orchestra.run.smithi131.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.0-524-g38c3646d-1focal [31.3 MB] 2023-09-29T21:48:02.235 INFO:teuthology.orchestra.run.smithi112.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-09-29T21:48:02.498 INFO:teuthology.orchestra.run.smithi131.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-29T21:48:02.521 INFO:teuthology.orchestra.run.smithi131.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-29T21:48:02.534 INFO:teuthology.orchestra.run.smithi131.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-29T21:48:02.561 INFO:teuthology.orchestra.run.smithi131.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-29T21:48:02.569 INFO:teuthology.orchestra.run.smithi112.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-09-29T21:48:02.576 INFO:teuthology.orchestra.run.smithi131.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-29T21:48:02.593 INFO:teuthology.orchestra.run.smithi112.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-09-29T21:48:02.605 INFO:teuthology.orchestra.run.smithi112.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-09-29T21:48:02.626 INFO:teuthology.orchestra.run.smithi131.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-29T21:48:02.630 INFO:teuthology.orchestra.run.smithi131.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-29T21:48:02.632 INFO:teuthology.orchestra.run.smithi112.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-09-29T21:48:02.647 INFO:teuthology.orchestra.run.smithi112.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-09-29T21:48:02.649 INFO:teuthology.orchestra.run.smithi131.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-29T21:48:02.658 INFO:teuthology.orchestra.run.smithi131.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-29T21:48:02.697 INFO:teuthology.orchestra.run.smithi112.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-09-29T21:48:02.701 INFO:teuthology.orchestra.run.smithi112.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-09-29T21:48:02.710 INFO:teuthology.orchestra.run.smithi131.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-29T21:48:02.719 INFO:teuthology.orchestra.run.smithi112.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-09-29T21:48:02.729 INFO:teuthology.orchestra.run.smithi112.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-09-29T21:48:02.780 INFO:teuthology.orchestra.run.smithi112.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-09-29T21:48:02.878 INFO:teuthology.orchestra.run.smithi131.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-29T21:48:02.947 INFO:teuthology.orchestra.run.smithi112.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-09-29T21:48:03.038 INFO:teuthology.orchestra.run.smithi112.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.0-524-g38c3646d-1focal [104 kB] 2023-09-29T21:48:03.103 INFO:teuthology.orchestra.run.smithi112.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.0-524-g38c3646d-1focal [466 kB] 2023-09-29T21:48:03.182 INFO:teuthology.orchestra.run.smithi112.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.0-524-g38c3646d-1focal [25.1 MB] 2023-09-29T21:48:03.217 INFO:teuthology.orchestra.run.smithi131.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-29T21:48:03.274 INFO:teuthology.orchestra.run.smithi131.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-29T21:48:03.286 INFO:teuthology.orchestra.run.smithi112.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-09-29T21:48:03.339 INFO:teuthology.orchestra.run.smithi131.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-29T21:48:03.345 INFO:teuthology.orchestra.run.smithi112.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-09-29T21:48:03.346 INFO:teuthology.orchestra.run.smithi131.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-29T21:48:03.361 INFO:teuthology.orchestra.run.smithi131.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-29T21:48:03.410 INFO:teuthology.orchestra.run.smithi112.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-09-29T21:48:03.416 INFO:teuthology.orchestra.run.smithi112.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-09-29T21:48:03.432 INFO:teuthology.orchestra.run.smithi112.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-09-29T21:48:03.506 INFO:teuthology.orchestra.run.smithi131.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.0-524-g38c3646d-1focal [3,868 B] 2023-09-29T21:48:03.506 INFO:teuthology.orchestra.run.smithi131.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.0-524-g38c3646d-1focal [832 kB] 2023-09-29T21:48:03.519 INFO:teuthology.orchestra.run.smithi131.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.0-524-g38c3646d-1focal [2,290 kB] 2023-09-29T21:48:03.535 INFO:teuthology.orchestra.run.smithi131.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.0-524-g38c3646d-1focal [221 kB] 2023-09-29T21:48:03.537 INFO:teuthology.orchestra.run.smithi131.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.0-524-g38c3646d-1focal [126 kB] 2023-09-29T21:48:03.538 INFO:teuthology.orchestra.run.smithi131.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.0-524-g38c3646d-1focal [1,800 kB] 2023-09-29T21:48:03.597 INFO:teuthology.orchestra.run.smithi131.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.0-524-g38c3646d-1focal [7,876 kB] 2023-09-29T21:48:03.885 INFO:teuthology.orchestra.run.smithi131.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-29T21:48:03.977 INFO:teuthology.orchestra.run.smithi112.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-09-29T21:48:03.986 INFO:teuthology.orchestra.run.smithi131.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.0-524-g38c3646d-1focal [3,992 B] 2023-09-29T21:48:03.986 INFO:teuthology.orchestra.run.smithi131.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-29T21:48:03.987 INFO:teuthology.orchestra.run.smithi131.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.0-524-g38c3646d-1focal [50.0 MB] 2023-09-29T21:48:03.987 INFO:teuthology.orchestra.run.smithi131.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-29T21:48:04.004 INFO:teuthology.orchestra.run.smithi131.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-29T21:48:04.004 INFO:teuthology.orchestra.run.smithi131.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-29T21:48:04.009 INFO:teuthology.orchestra.run.smithi112.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-09-29T21:48:04.011 INFO:teuthology.orchestra.run.smithi112.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-09-29T21:48:04.090 INFO:teuthology.orchestra.run.smithi112.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-09-29T21:48:04.090 INFO:teuthology.orchestra.run.smithi112.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-09-29T21:48:04.170 INFO:teuthology.orchestra.run.smithi131.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-29T21:48:04.255 INFO:teuthology.orchestra.run.smithi112.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-09-29T21:48:04.364 INFO:teuthology.orchestra.run.smithi131.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-29T21:48:04.409 INFO:teuthology.orchestra.run.smithi131.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-29T21:48:04.451 INFO:teuthology.orchestra.run.smithi112.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-09-29T21:48:04.472 INFO:teuthology.orchestra.run.smithi131.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-29T21:48:04.492 INFO:teuthology.orchestra.run.smithi131.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-29T21:48:04.495 INFO:teuthology.orchestra.run.smithi112.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-09-29T21:48:04.558 INFO:teuthology.orchestra.run.smithi112.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-09-29T21:48:04.570 INFO:teuthology.orchestra.run.smithi131.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-29T21:48:04.578 INFO:teuthology.orchestra.run.smithi112.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-09-29T21:48:04.581 INFO:teuthology.orchestra.run.smithi131.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-29T21:48:04.601 INFO:teuthology.orchestra.run.smithi131.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-29T21:48:04.657 INFO:teuthology.orchestra.run.smithi112.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-09-29T21:48:04.667 INFO:teuthology.orchestra.run.smithi112.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-09-29T21:48:04.687 INFO:teuthology.orchestra.run.smithi112.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-09-29T21:48:04.696 INFO:teuthology.orchestra.run.smithi131.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-29T21:48:04.726 INFO:teuthology.orchestra.run.smithi131.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-29T21:48:04.748 INFO:teuthology.orchestra.run.smithi131.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-29T21:48:04.783 INFO:teuthology.orchestra.run.smithi112.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-09-29T21:48:04.814 INFO:teuthology.orchestra.run.smithi112.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-09-29T21:48:04.835 INFO:teuthology.orchestra.run.smithi112.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-09-29T21:48:04.920 INFO:teuthology.orchestra.run.smithi131.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-29T21:48:05.006 INFO:teuthology.orchestra.run.smithi112.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-09-29T21:48:05.404 INFO:teuthology.orchestra.run.smithi131.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-29T21:48:05.455 INFO:teuthology.orchestra.run.smithi131.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-29T21:48:05.490 INFO:teuthology.orchestra.run.smithi112.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-09-29T21:48:05.539 INFO:teuthology.orchestra.run.smithi112.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-09-29T21:48:05.684 INFO:teuthology.orchestra.run.smithi131.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-29T21:48:05.695 INFO:teuthology.orchestra.run.smithi131.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-09-29T21:48:05.704 INFO:teuthology.orchestra.run.smithi131.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-29T21:48:05.769 INFO:teuthology.orchestra.run.smithi112.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-09-29T21:48:05.780 INFO:teuthology.orchestra.run.smithi112.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-09-29T21:48:05.790 INFO:teuthology.orchestra.run.smithi112.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-09-29T21:48:05.858 INFO:teuthology.orchestra.run.smithi131.stdout:Get:100 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-29T21:48:05.859 INFO:teuthology.orchestra.run.smithi131.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-29T21:48:05.859 INFO:teuthology.orchestra.run.smithi131.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-29T21:48:05.861 INFO:teuthology.orchestra.run.smithi131.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-29T21:48:05.902 INFO:teuthology.orchestra.run.smithi131.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-29T21:48:05.944 INFO:teuthology.orchestra.run.smithi112.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-09-29T21:48:05.945 INFO:teuthology.orchestra.run.smithi112.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-09-29T21:48:05.945 INFO:teuthology.orchestra.run.smithi112.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-09-29T21:48:05.946 INFO:teuthology.orchestra.run.smithi112.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-09-29T21:48:05.991 INFO:teuthology.orchestra.run.smithi112.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-09-29T21:48:06.066 INFO:teuthology.orchestra.run.smithi131.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-29T21:48:06.156 INFO:teuthology.orchestra.run.smithi112.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-09-29T21:48:06.172 INFO:teuthology.orchestra.run.smithi131.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.0-524-g38c3646d-1focal [223 kB] 2023-09-29T21:48:06.177 INFO:teuthology.orchestra.run.smithi131.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.0-524-g38c3646d-1focal [27.5 kB] 2023-09-29T21:48:06.178 INFO:teuthology.orchestra.run.smithi131.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.0-524-g38c3646d-1focal [13.4 MB] 2023-09-29T21:48:06.454 INFO:teuthology.orchestra.run.smithi131.stdout:Get:109 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-29T21:48:06.456 INFO:teuthology.orchestra.run.smithi131.stdout:Get:110 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-29T21:48:06.478 INFO:teuthology.orchestra.run.smithi131.stdout:Get:111 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-29T21:48:06.538 INFO:teuthology.orchestra.run.smithi112.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-09-29T21:48:06.548 INFO:teuthology.orchestra.run.smithi112.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-09-29T21:48:06.570 INFO:teuthology.orchestra.run.smithi112.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-09-29T21:48:06.575 INFO:teuthology.orchestra.run.smithi131.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.0-524-g38c3646d-1focal [97.5 kB] 2023-09-29T21:48:06.869 INFO:teuthology.orchestra.run.smithi112.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.0-524-g38c3646d-1focal [11.0 MB] 2023-09-29T21:48:07.384 INFO:teuthology.orchestra.run.smithi131.stdout:Fetched 207 MB in 7s (28.5 MB/s) 2023-09-29T21:48:07.420 INFO:teuthology.orchestra.run.smithi112.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.0-524-g38c3646d-1focal [224 kB] 2023-09-29T21:48:07.434 INFO:teuthology.orchestra.run.smithi112.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.0-524-g38c3646d-1focal [133 kB] 2023-09-29T21:48:07.437 INFO:teuthology.orchestra.run.smithi112.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.0-524-g38c3646d-1focal [1,551 kB] 2023-09-29T21:48:07.519 INFO:teuthology.orchestra.run.smithi112.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.0-524-g38c3646d-1focal [9,399 kB] 2023-09-29T21:48:07.564 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-29T21:48:07.620 INFO:teuthology.orchestra.run.smithi131.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-29T21:48:07.624 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-29T21:48:07.704 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-29T21:48:08.020 INFO:teuthology.orchestra.run.smithi112.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.0-524-g38c3646d-1focal [31.3 MB] 2023-09-29T21:48:08.092 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-29T21:48:08.108 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-29T21:48:08.175 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-29T21:48:08.536 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-29T21:48:08.553 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-29T21:48:08.594 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-29T21:48:09.015 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-29T21:48:09.034 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T21:48:09.074 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:48:09.609 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-29T21:48:09.625 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T21:48:09.667 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:48:09.749 INFO:teuthology.orchestra.run.smithi112.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.0-524-g38c3646d-1focal [3,868 B] 2023-09-29T21:48:09.750 INFO:teuthology.orchestra.run.smithi112.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.0-524-g38c3646d-1focal [832 kB] 2023-09-29T21:48:09.765 INFO:teuthology.orchestra.run.smithi112.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.0-524-g38c3646d-1focal [2,290 kB] 2023-09-29T21:48:09.837 INFO:teuthology.orchestra.run.smithi112.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.0-524-g38c3646d-1focal [221 kB] 2023-09-29T21:48:09.853 INFO:teuthology.orchestra.run.smithi112.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.0-524-g38c3646d-1focal [126 kB] 2023-09-29T21:48:09.854 INFO:teuthology.orchestra.run.smithi112.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.0-524-g38c3646d-1focal [1,800 kB] 2023-09-29T21:48:09.936 INFO:teuthology.orchestra.run.smithi112.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.0-524-g38c3646d-1focal [7,876 kB] 2023-09-29T21:48:10.054 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-29T21:48:10.072 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T21:48:10.112 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:48:10.284 INFO:teuthology.orchestra.run.smithi112.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.0-524-g38c3646d-1focal [3,992 B] 2023-09-29T21:48:10.284 INFO:teuthology.orchestra.run.smithi112.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.0-524-g38c3646d-1focal [50.0 MB] 2023-09-29T21:48:10.534 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-29T21:48:10.549 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-29T21:48:10.592 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-29T21:48:10.986 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../007-librbd1_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:11.112 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librbd1 (18.2.0-524-g38c3646d-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-29T21:48:11.790 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../008-librados2_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:11.925 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librados2 (18.2.0-524-g38c3646d-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-29T21:48:12.589 INFO:teuthology.orchestra.run.smithi112.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.0-524-g38c3646d-1focal [223 kB] 2023-09-29T21:48:12.593 INFO:teuthology.orchestra.run.smithi112.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.0-524-g38c3646d-1focal [27.5 kB] 2023-09-29T21:48:12.594 INFO:teuthology.orchestra.run.smithi112.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.0-524-g38c3646d-1focal [13.4 MB] 2023-09-29T21:48:12.646 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libcephfs2. 2023-09-29T21:48:12.662 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../009-libcephfs2_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:12.703 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libcephfs2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:13.022 INFO:teuthology.orchestra.run.smithi112.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2023-09-28-1611-reef/38c3646d535f6f917a872ef9baef468589ba9f32/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.0-524-g38c3646d-1focal [97.5 kB] 2023-09-29T21:48:13.031 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rados. 2023-09-29T21:48:13.047 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../010-python3-rados_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:13.123 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rados (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:13.452 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-29T21:48:13.471 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:13.509 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-ceph-argparse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:13.658 INFO:teuthology.orchestra.run.smithi112.stdout:Fetched 207 MB in 13s (15.4 MB/s) 2023-09-29T21:48:13.743 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cephfs. 2023-09-29T21:48:13.760 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../012-python3-cephfs_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:13.829 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cephfs (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:13.866 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-09-29T21:48:13.922 INFO:teuthology.orchestra.run.smithi112.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141758 files and directories currently installed.) 2023-09-29T21:48:13.926 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-09-29T21:48:14.008 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-29T21:48:14.159 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-29T21:48:14.179 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:14.217 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:14.354 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-09-29T21:48:14.372 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-09-29T21:48:14.413 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-29T21:48:14.568 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-prettytable. 2023-09-29T21:48:14.584 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-09-29T21:48:14.626 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-29T21:48:14.749 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-09-29T21:48:14.765 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-09-29T21:48:14.808 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-29T21:48:14.887 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rbd. 2023-09-29T21:48:14.904 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../015-python3-rbd_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:14.936 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rbd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:15.303 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-09-29T21:48:15.306 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-29T21:48:15.322 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-29T21:48:15.322 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T21:48:15.364 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-29T21:48:15.371 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:48:15.779 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-29T21:48:15.798 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-29T21:48:15.845 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-29T21:48:15.872 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-09-29T21:48:15.890 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T21:48:15.956 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:48:16.163 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-29T21:48:16.180 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-29T21:48:16.213 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-29T21:48:16.325 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-09-29T21:48:16.341 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-09-29T21:48:16.393 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:48:16.666 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package librgw2. 2023-09-29T21:48:16.686 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../019-librgw2_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:16.856 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking librgw2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:17.079 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-09-29T21:48:17.096 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-09-29T21:48:17.137 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-29T21:48:17.476 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../007-librbd1_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:17.574 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librbd1 (18.2.0-524-g38c3646d-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-29T21:48:17.761 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rgw. 2023-09-29T21:48:17.778 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../020-python3-rgw_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:17.819 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:18.130 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package liboath0. 2023-09-29T21:48:18.146 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-29T21:48:18.213 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-29T21:48:18.312 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../008-librados2_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:18.436 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librados2 (18.2.0-524-g38c3646d-1focal) over (15.2.17-0ubuntu0.20.04.4) ... 2023-09-29T21:48:18.566 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libradosstriper1. 2023-09-29T21:48:18.582 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../022-libradosstriper1_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:18.625 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libradosstriper1 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:19.111 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-common. 2023-09-29T21:48:19.128 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../023-ceph-common_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:19.140 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs2. 2023-09-29T21:48:19.159 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../009-libcephfs2_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:19.194 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:19.207 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:19.585 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rados. 2023-09-29T21:48:19.600 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../010-python3-rados_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:19.643 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rados (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:20.012 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-09-29T21:48:20.028 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:20.070 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-argparse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:20.340 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cephfs. 2023-09-29T21:48:20.356 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../012-python3-cephfs_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:20.389 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cephfs (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:20.709 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-ceph-common. 2023-09-29T21:48:20.728 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../013-python3-ceph-common_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:20.776 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:21.121 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-prettytable. 2023-09-29T21:48:21.142 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-09-29T21:48:21.180 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-09-29T21:48:21.458 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rbd. 2023-09-29T21:48:21.475 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../015-python3-rbd_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:21.516 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rbd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:21.734 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-base. 2023-09-29T21:48:21.752 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../024-ceph-base_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:21.802 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-base (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:21.900 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-09-29T21:48:21.916 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-09-29T21:48:21.958 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-29T21:48:22.295 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-09-29T21:48:22.311 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-09-29T21:48:22.353 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-09-29T21:48:22.715 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-09-29T21:48:22.732 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-09-29T21:48:22.790 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-29T21:48:23.159 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package librgw2. 2023-09-29T21:48:23.178 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../019-librgw2_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:23.189 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-29T21:48:23.207 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-29T21:48:23.217 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking librgw2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:23.247 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-29T21:48:23.618 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-natsort. 2023-09-29T21:48:23.634 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-09-29T21:48:23.701 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-29T21:48:24.046 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-logutils. 2023-09-29T21:48:24.066 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-09-29T21:48:24.095 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rgw. 2023-09-29T21:48:24.104 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-29T21:48:24.113 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../020-python3-rgw_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:24.145 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:24.407 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-29T21:48:24.426 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-29T21:48:24.457 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-29T21:48:24.523 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liboath0. 2023-09-29T21:48:24.539 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-09-29T21:48:24.614 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-09-29T21:48:24.785 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-mako. 2023-09-29T21:48:24.801 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-29T21:48:24.843 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-29T21:48:24.976 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libradosstriper1. 2023-09-29T21:48:24.994 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../022-libradosstriper1_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:25.034 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libradosstriper1 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:25.196 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-29T21:48:25.213 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-29T21:48:25.255 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-29T21:48:25.421 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-common. 2023-09-29T21:48:25.436 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../023-ceph-common_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:25.470 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:25.591 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-29T21:48:25.608 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-29T21:48:25.674 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-29T21:48:25.902 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-webob. 2023-09-29T21:48:25.920 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-29T21:48:25.985 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-29T21:48:26.313 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-29T21:48:26.331 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-29T21:48:26.380 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-29T21:48:26.725 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-bs4. 2023-09-29T21:48:26.743 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-09-29T21:48:26.783 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-29T21:48:27.195 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-waitress. 2023-09-29T21:48:27.212 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-29T21:48:27.256 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-29T21:48:27.620 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-tempita. 2023-09-29T21:48:27.639 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-09-29T21:48:27.681 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-29T21:48:27.918 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-base. 2023-09-29T21:48:27.933 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../024-ceph-base_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:27.979 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-base (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:27.993 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-paste. 2023-09-29T21:48:28.012 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-29T21:48:28.051 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-29T21:48:28.413 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-29T21:48:28.430 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-29T21:48:28.471 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-29T21:48:28.791 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-29T21:48:28.808 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-29T21:48:28.857 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-29T21:48:29.186 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-webtest. 2023-09-29T21:48:29.201 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-09-29T21:48:29.243 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-29T21:48:29.456 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cherrypy3. 2023-09-29T21:48:29.471 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-09-29T21:48:29.514 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-09-29T21:48:29.571 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pecan. 2023-09-29T21:48:29.588 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-29T21:48:29.646 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-29T21:48:29.884 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-natsort. 2023-09-29T21:48:29.899 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-09-29T21:48:29.942 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-09-29T21:48:30.026 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-29T21:48:30.044 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-29T21:48:30.084 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-29T21:48:30.246 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-logutils. 2023-09-29T21:48:30.262 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-09-29T21:48:30.296 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-09-29T21:48:30.404 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-29T21:48:30.421 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:30.462 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-modules-core (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:30.541 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-markupsafe. 2023-09-29T21:48:30.558 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-09-29T21:48:30.599 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-09-29T21:48:30.860 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-29T21:48:30.879 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:30.919 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-mako. 2023-09-29T21:48:30.938 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-09-29T21:48:30.950 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libsqlite3-mod-ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:30.977 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-29T21:48:31.304 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cryptography. 2023-09-29T21:48:31.320 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-29T21:48:31.362 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-29T21:48:31.422 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplegeneric. 2023-09-29T21:48:31.442 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-09-29T21:48:31.472 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-09-29T21:48:31.758 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-openssl. 2023-09-29T21:48:31.773 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-29T21:48:31.791 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-singledispatch. 2023-09-29T21:48:31.810 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-09-29T21:48:31.824 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-29T21:48:31.849 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-09-29T21:48:32.202 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr. 2023-09-29T21:48:32.218 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../047-ceph-mgr_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:32.361 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webob. 2023-09-29T21:48:32.378 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-09-29T21:48:32.794 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:32.861 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-09-29T21:48:33.289 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-soupsieve. 2023-09-29T21:48:33.307 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-09-29T21:48:33.339 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-29T21:48:33.482 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mon. 2023-09-29T21:48:33.499 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../048-ceph-mon_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:33.538 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mon (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:33.658 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bs4. 2023-09-29T21:48:33.674 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-09-29T21:48:33.716 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-09-29T21:48:34.078 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-waitress. 2023-09-29T21:48:34.094 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-09-29T21:48:34.140 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-29T21:48:34.482 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-tempita. 2023-09-29T21:48:34.501 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-09-29T21:48:34.540 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-09-29T21:48:34.859 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-osd. 2023-09-29T21:48:34.878 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../049-ceph-osd_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:34.892 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-paste. 2023-09-29T21:48:34.908 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-09-29T21:48:34.942 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-osd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:34.951 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-09-29T21:48:35.296 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-09-29T21:48:35.313 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-09-29T21:48:35.354 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-09-29T21:48:35.690 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastedeploy. 2023-09-29T21:48:35.707 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-09-29T21:48:35.748 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-09-29T21:48:36.068 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webtest. 2023-09-29T21:48:36.085 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-09-29T21:48:36.128 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-09-29T21:48:36.456 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pecan. 2023-09-29T21:48:36.474 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-09-29T21:48:36.548 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-09-29T21:48:36.919 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-werkzeug. 2023-09-29T21:48:36.935 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.1_all.deb ... 2023-09-29T21:48:36.977 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-29T21:48:37.305 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-09-29T21:48:37.321 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:37.363 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-modules-core (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:37.760 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-09-29T21:48:37.776 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:37.818 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libsqlite3-mod-ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:38.079 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph. 2023-09-29T21:48:38.094 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../050-ceph_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:38.162 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cryptography. 2023-09-29T21:48:38.182 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-09-29T21:48:38.212 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:38.229 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-29T21:48:38.573 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-fuse. 2023-09-29T21:48:38.589 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../051-ceph-fuse_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:38.624 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-openssl. 2023-09-29T21:48:38.631 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:38.640 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-09-29T21:48:38.682 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-09-29T21:48:39.035 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mds. 2023-09-29T21:48:39.035 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr. 2023-09-29T21:48:39.050 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../052-ceph-mds_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:39.054 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../047-ceph-mgr_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:39.084 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mds (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:39.118 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:39.621 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package cephadm. 2023-09-29T21:48:39.637 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../053-cephadm_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:39.638 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mon. 2023-09-29T21:48:39.655 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../048-ceph-mon_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:39.679 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:39.696 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mon (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:40.032 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-asyncssh. 2023-09-29T21:48:40.049 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-09-29T21:48:40.090 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-09-29T21:48:40.485 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jinja2. 2023-09-29T21:48:40.501 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-09-29T21:48:40.543 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-29T21:48:40.871 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-29T21:48:40.888 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:40.938 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:41.067 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-osd. 2023-09-29T21:48:41.083 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../049-ceph-osd_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:41.124 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-osd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:41.300 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-29T21:48:41.316 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-29T21:48:41.358 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-29T21:48:41.686 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-jwt. 2023-09-29T21:48:41.701 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-29T21:48:41.744 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-29T21:48:42.097 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-attr. 2023-09-29T21:48:42.113 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-09-29T21:48:42.155 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-29T21:48:42.516 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pluggy. 2023-09-29T21:48:42.533 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-09-29T21:48:42.600 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-29T21:48:42.944 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-py. 2023-09-29T21:48:42.960 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-29T21:48:43.002 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-29T21:48:43.348 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-29T21:48:43.365 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-29T21:48:43.405 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-29T21:48:43.742 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-packaging. 2023-09-29T21:48:43.760 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-09-29T21:48:43.800 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-29T21:48:44.111 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-29T21:48:44.131 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-29T21:48:44.169 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-29T21:48:44.227 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph. 2023-09-29T21:48:44.246 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../050-ceph_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:44.285 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:44.489 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pytest. 2023-09-29T21:48:44.507 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-09-29T21:48:44.547 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-29T21:48:44.669 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-fuse. 2023-09-29T21:48:44.686 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../051-ceph-fuse_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:44.729 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:44.934 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-29T21:48:44.953 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-09-29T21:48:44.992 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-29T21:48:45.182 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mds. 2023-09-29T21:48:45.199 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../052-ceph-mds_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:45.240 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mds (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:45.270 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libjs-underscore. 2023-09-29T21:48:45.288 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-29T21:48:45.328 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-29T21:48:45.630 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-29T21:48:45.648 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-29T21:48:45.688 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-29T21:48:45.785 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package cephadm. 2023-09-29T21:48:45.801 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../053-cephadm_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:45.843 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:46.033 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-routes. 2023-09-29T21:48:46.049 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-09-29T21:48:46.091 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-29T21:48:46.171 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-asyncssh. 2023-09-29T21:48:46.190 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-09-29T21:48:46.229 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-09-29T21:48:46.444 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-29T21:48:46.461 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:46.502 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-dashboard (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:46.615 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jinja2. 2023-09-29T21:48:46.631 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-09-29T21:48:46.673 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-09-29T21:48:47.023 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-decorator. 2023-09-29T21:48:47.034 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-09-29T21:48:47.042 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-29T21:48:47.051 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:47.081 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-29T21:48:47.092 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:47.409 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-29T21:48:47.425 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-29T21:48:47.467 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-29T21:48:47.496 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-bcrypt. 2023-09-29T21:48:47.514 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-09-29T21:48:47.554 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-29T21:48:47.829 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-scipy. 2023-09-29T21:48:47.845 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-29T21:48:47.882 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-jwt. 2023-09-29T21:48:47.901 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-09-29T21:48:47.911 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-29T21:48:47.931 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-29T21:48:48.193 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-attr. 2023-09-29T21:48:48.213 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-09-29T21:48:48.251 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-09-29T21:48:49.037 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pluggy. 2023-09-29T21:48:49.053 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-09-29T21:48:49.104 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-09-29T21:48:49.449 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-py. 2023-09-29T21:48:49.464 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-09-29T21:48:49.506 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-09-29T21:48:49.604 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-29T21:48:49.623 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-29T21:48:49.662 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-29T21:48:49.860 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-atomicwrites. 2023-09-29T21:48:49.875 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-09-29T21:48:49.918 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-09-29T21:48:50.232 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-joblib. 2023-09-29T21:48:50.251 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-09-29T21:48:50.254 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-packaging. 2023-09-29T21:48:50.271 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-09-29T21:48:50.299 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-29T21:48:50.320 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-packaging (20.3-1) ... 2023-09-29T21:48:50.656 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-wcwidth. 2023-09-29T21:48:50.661 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-sklearn. 2023-09-29T21:48:50.672 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-09-29T21:48:50.677 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-29T21:48:50.714 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-29T21:48:50.718 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-29T21:48:51.042 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pytest. 2023-09-29T21:48:51.058 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-09-29T21:48:51.125 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-09-29T21:48:51.487 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-repoze.lru. 2023-09-29T21:48:51.504 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-09-29T21:48:51.511 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-29T21:48:51.530 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:51.545 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-09-29T21:48:51.560 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:51.831 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjs-underscore. 2023-09-29T21:48:51.847 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-09-29T21:48:51.889 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-29T21:48:52.208 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-09-29T21:48:52.224 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-09-29T21:48:52.266 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-29T21:48:52.627 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-routes. 2023-09-29T21:48:52.643 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-09-29T21:48:52.685 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-09-29T21:48:52.697 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-cachetools. 2023-09-29T21:48:52.713 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-09-29T21:48:52.788 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-29T21:48:53.030 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-09-29T21:48:53.049 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:53.088 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-dashboard (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:53.116 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-29T21:48:53.133 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-29T21:48:53.174 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-29T21:48:53.545 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-29T21:48:53.561 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-29T21:48:53.602 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-29T21:48:53.616 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-decorator. 2023-09-29T21:48:53.632 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-09-29T21:48:53.674 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-09-29T21:48:54.002 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-09-29T21:48:54.016 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-rsa. 2023-09-29T21:48:54.018 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-09-29T21:48:54.033 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-09-29T21:48:54.060 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-29T21:48:54.091 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-29T21:48:54.413 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-scipy. 2023-09-29T21:48:54.420 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-google-auth. 2023-09-29T21:48:54.433 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-09-29T21:48:54.439 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-09-29T21:48:54.471 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-09-29T21:48:54.478 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-29T21:48:54.832 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-blinker. 2023-09-29T21:48:54.849 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-29T21:48:54.892 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-29T21:48:55.229 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-29T21:48:55.245 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-29T21:48:55.287 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-29T21:48:55.608 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-29T21:48:55.627 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-29T21:48:55.665 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-29T21:48:55.813 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-09-29T21:48:55.831 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-09-29T21:48:55.871 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-29T21:48:56.011 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-websocket. 2023-09-29T21:48:56.028 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-29T21:48:56.102 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-29T21:48:56.433 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-joblib. 2023-09-29T21:48:56.447 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-29T21:48:56.450 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-09-29T21:48:56.467 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-29T21:48:56.490 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-09-29T21:48:56.505 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-29T21:48:56.902 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-sklearn. 2023-09-29T21:48:56.922 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-09-29T21:48:56.960 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-29T21:48:57.069 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-29T21:48:57.088 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:57.127 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-mgr-k8sevents (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:57.478 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-29T21:48:57.496 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-09-29T21:48:57.537 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-29T21:48:57.702 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-09-29T21:48:57.719 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:48:57.759 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:57.916 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-29T21:48:57.932 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-29T21:48:57.973 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:48:58.277 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package jq. 2023-09-29T21:48:58.293 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-29T21:48:58.376 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:48:58.621 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package socat. 2023-09-29T21:48:58.637 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-09-29T21:48:58.704 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-29T21:48:58.946 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-cachetools. 2023-09-29T21:48:58.965 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-09-29T21:48:59.004 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-09-29T21:48:59.066 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package xmlstarlet. 2023-09-29T21:48:59.084 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-29T21:48:59.124 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-29T21:48:59.340 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1. 2023-09-29T21:48:59.356 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-09-29T21:48:59.423 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-09-29T21:48:59.503 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-test. 2023-09-29T21:48:59.519 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../094-ceph-test_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:48:59.561 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-test (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:48:59.776 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-09-29T21:48:59.792 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-09-29T21:48:59.834 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-29T21:49:00.256 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-rsa. 2023-09-29T21:49:00.272 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-09-29T21:49:00.314 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-rsa (4.0-3) ... 2023-09-29T21:49:00.651 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-google-auth. 2023-09-29T21:49:00.668 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-09-29T21:49:00.709 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-09-29T21:49:01.055 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-blinker. 2023-09-29T21:49:01.074 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-09-29T21:49:01.113 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-29T21:49:01.449 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-oauthlib. 2023-09-29T21:49:01.467 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-09-29T21:49:01.507 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-29T21:49:01.853 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-09-29T21:49:01.868 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-09-29T21:49:01.944 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-29T21:49:02.230 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-websocket. 2023-09-29T21:49:02.250 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-09-29T21:49:02.313 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-09-29T21:49:02.658 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-kubernetes. 2023-09-29T21:49:02.675 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-09-29T21:49:02.716 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-09-29T21:49:03.271 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-09-29T21:49:03.288 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:49:03.329 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-mgr-k8sevents (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:03.674 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libonig5:amd64. 2023-09-29T21:49:03.691 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-09-29T21:49:03.732 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-09-29T21:49:04.076 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjq1:amd64. 2023-09-29T21:49:04.095 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-29T21:49:04.124 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package ceph-volume. 2023-09-29T21:49:04.141 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../095-ceph-volume_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:49:04.159 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:49:04.182 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking ceph-volume (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:04.477 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package jq. 2023-09-29T21:49:04.495 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-09-29T21:49:04.541 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:49:04.575 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-29T21:49:04.595 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-29T21:49:04.632 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:04.786 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package socat. 2023-09-29T21:49:04.803 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-09-29T21:49:04.877 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking socat (1.7.3.3-2) ... 2023-09-29T21:49:05.187 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-29T21:49:05.203 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:05.222 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package xmlstarlet. 2023-09-29T21:49:05.239 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-09-29T21:49:05.244 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libcephfs-dev (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:05.280 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-09-29T21:49:05.660 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-test. 2023-09-29T21:49:05.676 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../094-ceph-test_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:05.697 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package nvme-cli. 2023-09-29T21:49:05.715 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-29T21:49:05.717 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-test (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:05.773 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-29T21:49:06.102 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-09-29T21:49:06.121 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-09-29T21:49:06.160 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-09-29T21:49:06.564 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-29T21:49:06.581 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-29T21:49:06.622 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:08.346 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-29T21:49:08.364 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-29T21:49:08.429 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-29T21:49:08.790 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-webencodings. 2023-09-29T21:49:08.809 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-29T21:49:08.848 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-29T21:49:09.193 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-html5lib. 2023-09-29T21:49:09.210 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-09-29T21:49:09.251 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-29T21:49:09.621 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-29T21:49:09.638 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-29T21:49:09.707 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-29T21:49:10.080 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package ceph-volume. 2023-09-29T21:49:10.097 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../095-ceph-volume_18.2.0-524-g38c3646d-1focal_all.deb ... 2023-09-29T21:49:10.137 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking ceph-volume (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:10.184 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pastescript. 2023-09-29T21:49:10.204 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-09-29T21:49:10.241 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-29T21:49:10.451 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package fonts-mathjax. 2023-09-29T21:49:10.468 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-29T21:49:10.521 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:10.595 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-psutil. 2023-09-29T21:49:10.613 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-29T21:49:10.653 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-29T21:49:11.048 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-29T21:49:11.063 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libcephfs-dev. 2023-09-29T21:49:11.065 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-29T21:49:11.080 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../097-libcephfs-dev_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:11.106 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-29T21:49:11.121 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libcephfs-dev (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:11.443 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package python3-simplejson. 2023-09-29T21:49:11.461 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-29T21:49:11.501 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-29T21:49:11.536 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package nvme-cli. 2023-09-29T21:49:11.552 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-09-29T21:49:11.637 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-09-29T21:49:11.804 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-29T21:49:11.821 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-29T21:49:11.862 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-29T21:49:11.972 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-09-29T21:49:11.989 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-09-29T21:49:12.031 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-09-29T21:49:12.417 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package libjs-mathjax. 2023-09-29T21:49:12.435 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-09-29T21:49:12.475 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:12.625 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package radosgw. 2023-09-29T21:49:12.644 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../110-radosgw_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:12.682 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking radosgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:14.123 INFO:teuthology.orchestra.run.smithi131.stdout:Selecting previously unselected package rbd-fuse. 2023-09-29T21:49:14.141 INFO:teuthology.orchestra.run.smithi131.stdout:Preparing to unpack .../111-rbd-fuse_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:14.181 INFO:teuthology.orchestra.run.smithi131.stdout:Unpacking rbd-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:14.186 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python-natsort-doc. 2023-09-29T21:49:14.205 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-09-29T21:49:14.277 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-09-29T21:49:14.517 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-29T21:49:14.622 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-webencodings. 2023-09-29T21:49:14.640 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-09-29T21:49:14.680 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-29T21:49:14.819 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-29T21:49:14.945 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-29T21:49:15.008 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-html5lib. 2023-09-29T21:49:15.024 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-09-29T21:49:15.066 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-09-29T21:49:15.105 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-29T21:49:15.306 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:15.436 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-09-29T21:49:15.452 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-09-29T21:49:15.494 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-29T21:49:15.985 INFO:teuthology.orchestra.run.smithi131.stdout:Adding system user cephadm....done 2023-09-29T21:49:15.997 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pastescript. 2023-09-29T21:49:16.018 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-09-29T21:49:16.088 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-09-29T21:49:16.189 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:16.315 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-29T21:49:16.458 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-psutil. 2023-09-29T21:49:16.476 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-09-29T21:49:16.516 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-09-29T21:49:16.725 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-29T21:49:16.895 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-pyinotify. 2023-09-29T21:49:16.912 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-09-29T21:49:16.953 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-29T21:49:17.039 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:17.165 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-29T21:49:17.291 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-29T21:49:17.297 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package python3-simplejson. 2023-09-29T21:49:17.316 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-09-29T21:49:17.355 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-29T21:49:17.665 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-29T21:49:17.666 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-09-29T21:49:17.684 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-09-29T21:49:17.724 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-29T21:49:18.071 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-29T21:49:18.389 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-29T21:49:18.454 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package radosgw. 2023-09-29T21:49:18.472 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../110-radosgw_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:18.512 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking radosgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:18.690 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-29T21:49:18.993 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-29T21:49:19.300 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-ceph-argparse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:20.037 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-29T21:49:20.124 INFO:teuthology.orchestra.run.smithi112.stdout:Selecting previously unselected package rbd-fuse. 2023-09-29T21:49:20.140 INFO:teuthology.orchestra.run.smithi112.stdout:Preparing to unpack .../111-rbd-fuse_18.2.0-524-g38c3646d-1focal_amd64.deb ... 2023-09-29T21:49:20.163 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-29T21:49:20.181 INFO:teuthology.orchestra.run.smithi112.stdout:Unpacking rbd-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:20.448 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-29T21:49:20.519 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-attr (19.3.0-2) ... 2023-09-29T21:49:20.732 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-29T21:49:20.844 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-09-29T21:49:20.858 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-29T21:49:20.974 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-09-29T21:49:21.125 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-09-29T21:49:21.217 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-29T21:49:21.326 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:21.544 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-29T21:49:21.845 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-29T21:49:22.004 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user cephadm....done 2023-09-29T21:49:22.179 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-29T21:49:22.182 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:22.308 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-09-29T21:49:22.466 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-29T21:49:22.651 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-09-29T21:49:22.792 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-29T21:49:22.927 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-09-29T21:49:23.046 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-09-29T21:49:23.068 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-29T21:49:23.172 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-09-29T21:49:23.369 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-29T21:49:23.478 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-29T21:49:23.523 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-09-29T21:49:23.629 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-29T21:49:23.947 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-29T21:49:23.975 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-09-29T21:49:24.305 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-09-29T21:49:24.357 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-29T21:49:24.572 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-09-29T21:49:24.725 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-29T21:49:24.874 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-09-29T21:49:25.085 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-29T21:49:25.158 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-argparse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:25.236 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:25.459 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-09-29T21:49:25.571 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:49:25.585 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-09-29T21:49:25.705 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-29T21:49:25.905 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-09-29T21:49:26.059 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-29T21:49:26.185 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-29T21:49:26.187 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-09-29T21:49:26.306 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-09-29T21:49:26.336 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-29T21:49:26.612 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-29T21:49:26.724 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-09-29T21:49:26.930 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:49:27.042 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-09-29T21:49:27.056 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-29T21:49:27.208 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-29T21:49:27.343 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-09-29T21:49:27.562 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-29T21:49:27.686 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-09-29T21:49:27.846 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-29T21:49:27.979 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-09-29T21:49:28.214 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-29T21:49:28.313 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-09-29T21:49:28.339 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-29T21:49:28.589 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-packaging (20.3-1) ... 2023-09-29T21:49:28.624 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-29T21:49:28.865 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-09-29T21:49:28.917 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-29T21:49:29.016 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-09-29T21:49:29.142 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-09-29T21:49:29.234 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-29T21:49:29.401 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-09-29T21:49:29.676 INFO:teuthology.orchestra.run.smithi131.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-29T21:49:29.778 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.1) ... 2023-09-29T21:49:29.956 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-29T21:49:30.090 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-29T21:49:30.146 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-09-29T21:49:30.433 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-29T21:49:30.514 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up socat (1.7.3.3-2) ... 2023-09-29T21:49:30.665 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:30.817 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-29T21:49:30.968 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-29T21:49:31.017 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:49:31.094 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-29T21:49:31.151 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-09-29T21:49:31.445 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-29T21:49:31.494 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-09-29T21:49:31.620 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-09-29T21:49:31.779 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-09-29T21:49:31.780 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-29T21:49:32.089 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-09-29T21:49:32.415 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:49:32.541 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-09-29T21:49:32.669 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-09-29T21:49:33.005 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-09-29T21:49:33.285 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:49:33.289 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-09-29T21:49:33.411 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-29T21:49:33.573 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-09-29T21:49:33.699 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-09-29T21:49:33.796 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-29T21:49:33.975 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-09-29T21:49:34.145 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-29T21:49:34.268 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-09-29T21:49:34.474 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:49:34.572 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-09-29T21:49:34.616 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-29T21:49:34.959 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-29T21:49:34.976 INFO:teuthology.orchestra.run.smithi112.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-09-29T21:49:35.269 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-09-29T21:49:35.274 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-09-29T21:49:35.401 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-09-29T21:49:35.653 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-29T21:49:35.735 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-09-29T21:49:36.004 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-29T21:49:36.086 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-09-29T21:49:36.212 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-09-29T21:49:36.365 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-09-29T21:49:36.683 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-09-29T21:49:36.959 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-09-29T21:49:37.102 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-29T21:49:37.227 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-29T21:49:37.353 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:49:37.504 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-29T21:49:37.797 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-29T21:49:37.923 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-29T21:49:38.233 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-29T21:49:38.448 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:49:38.517 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-29T21:49:38.574 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-09-29T21:49:38.826 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-29T21:49:38.959 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-09-29T21:49:39.187 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librados2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:39.260 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-09-29T21:49:39.330 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-09-29T21:49:39.464 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librgw2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:39.553 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-09-29T21:49:39.590 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libsqlite3-mod-ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:39.678 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-09-29T21:49:39.818 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libcephfs2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:39.935 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-29T21:49:40.021 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-09-29T21:49:40.253 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libradosstriper1 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:40.348 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-09-29T21:49:40.438 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-29T21:49:40.724 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-09-29T21:49:40.748 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up librbd1 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:40.882 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-modules-core (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:40.983 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:41.050 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-09-29T21:49:41.237 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-29T21:49:41.237 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-29T21:49:41.751 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up libcephfs-dev (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:41.852 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rados (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:41.970 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-29T21:49:42.089 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-09-29T21:49:42.214 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-09-29T21:49:42.366 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-09-29T21:49:42.491 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rsa (4.0-3) ... 2023-09-29T21:49:42.784 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-09-29T21:49:42.910 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-09-29T21:49:42.932 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rbd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:43.083 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up rbd-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:43.184 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-rgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:43.261 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-09-29T21:49:43.284 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up python3-cephfs (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:43.427 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:43.587 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-09-29T21:49:43.897 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-09-29T21:49:44.038 INFO:teuthology.orchestra.run.smithi131.stdout:Adding group ceph....done 2023-09-29T21:49:44.249 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librados2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:44.375 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-09-29T21:49:44.443 INFO:teuthology.orchestra.run.smithi131.stdout:Adding system user ceph....done 2023-09-29T21:49:44.534 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librgw2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:44.619 INFO:teuthology.orchestra.run.smithi131.stdout:Setting system user ceph properties....done 2023-09-29T21:49:44.641 INFO:teuthology.orchestra.run.smithi131.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-29T21:49:44.660 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libsqlite3-mod-ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:44.790 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-29T21:49:44.896 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs2 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:45.022 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-routes (2.4.1-2) ... 2023-09-29T21:49:45.109 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-29T21:49:45.315 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libradosstriper1 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:45.433 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-09-29T21:49:45.573 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-test (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:45.733 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up radosgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:45.760 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up librbd1 (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:45.927 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-modules-core (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:45.979 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-29T21:49:45.979 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-29T21:49:46.062 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:46.285 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-29T21:49:46.285 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-09-29T21:49:46.626 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-base (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up libcephfs-dev (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:46.864 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rados (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-09-29T21:49:47.030 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-29T21:49:47.495 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mds (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:47.721 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-29T21:49:47.722 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-29T21:49:47.935 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rbd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.080 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up rbd-fuse (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.163 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.206 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-rgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.357 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up python3-cephfs (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.431 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-29T21:49:48.431 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-29T21:49:48.483 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-common (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.881 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-osd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:48.993 INFO:teuthology.orchestra.run.smithi112.stdout:Adding group ceph....done 2023-09-29T21:49:49.397 INFO:teuthology.orchestra.run.smithi112.stdout:Adding system user ceph....done 2023-09-29T21:49:49.420 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-29T21:49:49.421 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-29T21:49:49.572 INFO:teuthology.orchestra.run.smithi112.stdout:Setting system user ceph properties....done 2023-09-29T21:49:49.597 INFO:teuthology.orchestra.run.smithi112.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-09-29T21:49:49.742 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-09-29T21:49:49.887 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-k8sevents (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:49.993 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-diskprediction-local (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:50.065 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-09-29T21:49:50.129 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mon (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:50.390 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-29T21:49:50.390 INFO:teuthology.orchestra.run.smithi131.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-29T21:49:50.585 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-test (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:50.728 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up radosgw (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:51.006 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:51.036 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-29T21:49:51.037 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-09-29T21:49:51.149 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:51.296 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-mgr-dashboard (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:51.442 INFO:teuthology.orchestra.run.smithi131.stdout:Setting up ceph-volume (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:51.569 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-29T21:49:51.647 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-base (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:51.745 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-29T21:49:52.029 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-09-29T21:49:52.038 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-29T21:49:52.468 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mds (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:52.742 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-29T21:49:52.742 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-09-29T21:49:53.170 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:53.429 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-29T21:49:53.429 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-09-29T21:49:53.863 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-osd (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:54.408 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-29T21:49:54.408 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-09-29T21:49:54.848 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-k8sevents (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:54.914 INFO:teuthology.orchestra.run.smithi131.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-29T21:49:54.974 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-diskprediction-local (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:55.158 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mon (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:55.389 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-29T21:49:55.389 INFO:teuthology.orchestra.run.smithi112.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-09-29T21:49:55.515 INFO:teuthology.orchestra.run.smithi131.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-29T21:49:55.517 DEBUG:teuthology.parallel:result is None 2023-09-29T21:49:55.852 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-cephadm (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:55.995 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:56.146 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-mgr-dashboard (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:56.281 INFO:teuthology.orchestra.run.smithi112.stdout:Setting up ceph-volume (18.2.0-524-g38c3646d-1focal) ... 2023-09-29T21:49:56.408 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-09-29T21:49:56.595 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-09-29T21:49:56.904 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-09-29T21:49:59.774 INFO:teuthology.orchestra.run.smithi112.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-09-29T21:50:00.351 INFO:teuthology.orchestra.run.smithi112.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-09-29T21:50:00.354 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:00.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T21:50:00.581 DEBUG:teuthology.orchestra.run.smithi112:> dpkg-query -W -f '${Version}' ceph 2023-09-29T21:50:00.606 INFO:teuthology.orchestra.run.smithi112.stdout:18.2.0-524-g38c3646d-1focal 2023-09-29T21:50:00.607 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-524-g38c3646d-1focal 2023-09-29T21:50:00.607 INFO:teuthology.task.install:The correct ceph version 18.2.0-524-g38c3646d-1focal is installed. 2023-09-29T21:50:00.608 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=38c3646d535f6f917a872ef9baef468589ba9f32 2023-09-29T21:50:00.828 DEBUG:teuthology.orchestra.run.smithi131:> dpkg-query -W -f '${Version}' ceph 2023-09-29T21:50:00.851 INFO:teuthology.orchestra.run.smithi131.stdout:18.2.0-524-g38c3646d-1focal 2023-09-29T21:50:00.851 INFO:teuthology.packaging:The installed version of ceph is 18.2.0-524-g38c3646d-1focal 2023-09-29T21:50:00.851 INFO:teuthology.task.install:The correct ceph version 18.2.0-524-g38c3646d-1focal is installed. 2023-09-29T21:50:00.852 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-29T21:50:00.852 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:00.852 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T21:50:00.869 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:00.869 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-29T21:50:00.909 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-29T21:50:00.909 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:00.909 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T21:50:00.924 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T21:50:00.982 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:00.982 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/daemon-helper 2023-09-29T21:50:00.999 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-29T21:50:01.055 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-29T21:50:01.056 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:01.056 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T21:50:01.071 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T21:50:01.081 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:01.081 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-29T21:50:01.096 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-29T21:50:01.107 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-29T21:50:01.107 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:01.108 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T21:50:01.139 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T21:50:01.150 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:01.150 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/usr/bin/stdin-killer 2023-09-29T21:50:01.166 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-29T21:50:01.179 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T21:50:01.242 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-29T21:50:01.243 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 777 /var/log/ceph 2023-09-29T21:50:01.245 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /var/log/ceph 2023-09-29T21:50:01.261 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-29T21:50:01.261 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T21:50:01.301 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-29T21:50:01.320 INFO:tasks.ceph:Creating extra log directories... 2023-09-29T21:50:01.321 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T21:50:01.323 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-29T21:50:01.343 INFO:tasks.ceph:Creating ceph cluster c1... 2023-09-29T21:50:01.343 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T21:50:01.343 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284', 'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406284', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4212, 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_12:36:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T21:50:01.343 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-09-29T21:50:01.345 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2023-09-29T21:50:01.379 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T21:50:01.393 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T21:50:01.409 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:01.409 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T21:50:01.414 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T21:50:01.414 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_1 2023-09-29T21:50:01.462 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 359 Links: 1 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-09-29 21:46:15.147575897 +0000 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-09-29 21:46:14.875582171 +0000 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-09-29 21:46:14.875582171 +0000 2023-09-29T21:50:01.463 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-09-29T21:50:01.463 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T21:50:01.478 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-09-29T21:50:01.478 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-09-29T21:50:01.478 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000442317 s, 1.2 MB/s 2023-09-29T21:50:01.479 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T21:50:01.487 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_2 2023-09-29T21:50:01.534 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-09-29 21:46:15.627564825 +0000 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-09-29 21:46:15.371570730 +0000 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-09-29 21:46:15.371570730 +0000 2023-09-29T21:50:01.535 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-09-29T21:50:01.535 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T21:50:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-09-29T21:50:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-09-29T21:50:01.547 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000385998 s, 1.3 MB/s 2023-09-29T21:50:01.548 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T21:50:01.556 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_3 2023-09-29T21:50:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T21:50:01.562 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:01.563 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-09-29T21:50:01.563 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:01.563 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-09-29 21:46:15.851559657 +0000 2023-09-29T21:50:01.563 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-09-29 21:46:15.847559751 +0000 2023-09-29T21:50:01.563 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-09-29 21:46:15.847559751 +0000 2023-09-29T21:50:01.563 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-09-29T21:50:01.563 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T21:50:01.576 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-09-29T21:50:01.576 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-09-29T21:50:01.576 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000384737 s, 1.3 MB/s 2023-09-29T21:50:01.577 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T21:50:01.628 DEBUG:teuthology.orchestra.run.smithi112:> stat /dev/vg_nvme/lv_4 2023-09-29T21:50:01.635 INFO:teuthology.orchestra.run.smithi112.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T21:50:01.635 INFO:teuthology.orchestra.run.smithi112.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:01.635 INFO:teuthology.orchestra.run.smithi112.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-09-29T21:50:01.635 INFO:teuthology.orchestra.run.smithi112.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:01.635 INFO:teuthology.orchestra.run.smithi112.stdout:Access: 2023-09-29 21:46:16.587542681 +0000 2023-09-29T21:50:01.636 INFO:teuthology.orchestra.run.smithi112.stdout:Modify: 2023-09-29 21:46:16.331548586 +0000 2023-09-29T21:50:01.636 INFO:teuthology.orchestra.run.smithi112.stdout:Change: 2023-09-29 21:46:16.331548586 +0000 2023-09-29T21:50:01.636 INFO:teuthology.orchestra.run.smithi112.stdout: Birth: - 2023-09-29T21:50:01.636 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T21:50:01.650 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records in 2023-09-29T21:50:01.650 INFO:teuthology.orchestra.run.smithi112.stderr:1+0 records out 2023-09-29T21:50:01.650 INFO:teuthology.orchestra.run.smithi112.stderr:512 bytes copied, 0.000420988 s, 1.2 MB/s 2023-09-29T21:50:01.651 DEBUG:teuthology.orchestra.run.smithi112:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T21:50:01.704 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:01.704 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T21:50:01.704 INFO:tasks.ceph:Generating config... 2023-09-29T21:50:01.705 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-09-29T21:50:01.706 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-09-29T21:50:01.707 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-09-29T21:50:01.707 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-29T21:50:01.707 INFO:tasks.ceph:[client] setuser = ceph 2023-09-29T21:50:01.707 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-09-29T21:50:01.707 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T21:50:01.707 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T21:50:01.707 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T21:50:01.707 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T21:50:01.707 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T21:50:01.707 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T21:50:01.707 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T21:50:01.707 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-09-29T21:50:01.708 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-09-29T21:50:01.708 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-09-29T21:50:01.708 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-09-29T21:50:01.708 INFO:tasks.ceph:Setting up c1.mon.a... 2023-09-29T21:50:01.708 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2023-09-29T21:50:01.747 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.keyring 2023-09-29T21:50:01.762 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2023-09-29T21:50:01.803 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-29T21:50:01.859 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.112')] 2023-09-29T21:50:01.859 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.112'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-09-29T21:50:01.860 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:01.860 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T21:50:01.865 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.112 --print /home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:50:01.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: generated fsid 1be42293-dd6a-48c9-b3ea-af1a1b1442b0 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:setting min_mon_release = pacific 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:epoch 0 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:fsid 1be42293-dd6a-48c9-b3ea-af1a1b1442b0 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:last_changed 2023-09-29T21:50:01.906481+0000 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:created 2023-09-29T21:50:01.906481+0000 2023-09-29T21:50:01.905 INFO:teuthology.orchestra.run.smithi112.stdout:min_mon_release 16 (pacific) 2023-09-29T21:50:01.906 INFO:teuthology.orchestra.run.smithi112.stdout:election_strategy: 1 2023-09-29T21:50:01.906 INFO:teuthology.orchestra.run.smithi112.stdout:0: [v2:172.21.15.112:3300/0,v1:172.21.15.112:6789/0] mon.a 2023-09-29T21:50:01.906 INFO:teuthology.orchestra.run.smithi112.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2023-09-29T21:50:01.907 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T21:50:01.954 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 1be42293-dd6a-48c9-b3ea-af1a1b1442b0... 2023-09-29T21:50:01.955 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-09-29T21:50:01.957 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2023-09-29T21:50:01.988 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.989 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.990 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-09-29T21:50:01.991 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.112 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 1be42293-dd6a-48c9-b3ea-af1a1b1442b0 2023-09-29T21:50:01.992 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-09-29T21:50:01.993 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.994 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-09-29T21:50:01.995 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-09-29T21:50:01.996 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T21:50:01.997 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-09-29T21:50:01.998 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw sync = 20 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync data inject err probability = 0.1 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2023-09-29T21:50:01.999 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync obj etag verify = True 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data notify interval msec = 0 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-09-29T21:50:02.000 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout:[global] 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: chdir = "" 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: auth supported = cephx 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: filestore xattr use omap = true 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: mon clock drift allowed = 1.000 2023-09-29T21:50:02.002 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: osd crush chooseleaf type = 0 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: auth debug = true 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on old message = true 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on bug = true 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: mon pg warn max object skew = 0 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T21:50:02.003 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default size = 2 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow primary affinity = true 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow pg remap = true 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on legacy crush tunables = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on crush straw calc version zero = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on no sortbitwise = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on osd down out interval zero = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on too few osds = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T21:50:02.004 INFO:teuthology.orchestra.run.smithi131.stdout: mon_allow_pool_size_one = true 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: osd default data pool replay window = 5 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: mon allow pool delete = true 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: mon cluster log file level = debug 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: debug asserts on shutdown = true 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: mon health detail to clog = false 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: mon host = 172.21.15.112 2023-09-29T21:50:02.005 INFO:teuthology.orchestra.run.smithi131.stdout: fsid = 1be42293-dd6a-48c9-b3ea-af1a1b1442b0 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout:[osd] 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd journal size = 100 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub load threshold = 5.0 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub max interval = 600 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd mclock profile = high_recovery_ops 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd recover clone overlap = true 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd recovery max chunk = 1048576 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.006 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug shutdown = true 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug op order = true 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug verify stray on activate = true 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: osd open classes on start = true 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug pg log writeout = true 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: osd deep scrub update digest min age = 30 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: osd map max advance = 10 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: journal zero on create = true 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.007 INFO:teuthology.orchestra.run.smithi131.stdout: filestore ondisk finisher threads = 3 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: filestore apply finisher threads = 3 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: bdev debug aio = true 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug misdirected ops = true 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: debug osd = 20 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: rocksdb delete range threshold = 0 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr] 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: debug mgr = 20 2023-09-29T21:50:02.008 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: mgr/telemetry/nag = false 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout:[mon] 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: debug paxos = 20 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: mon data avail warn = 5 2023-09-29T21:50:02.009 INFO:teuthology.orchestra.run.smithi131.stdout: mon mgr mkfs grace = 240 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd reporter subtree level = osd 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd prime pg temp = true 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T21:50:02.010 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: # 1m isn't quite enough 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: mon_down_mkfs_grace = 2m 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_filestore_osds = false 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout:[client] 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: rgw cache enabled = true 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable ops log = true 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable usage log = true 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T21:50:02.011 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw = 20 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw sync = 20 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt require ssl = False 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms backend = testing 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw curl low speed time = 300 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data log num shards = 4 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw md log max shards = 4 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync data inject err probability = 0.1 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync log trim interval = 0 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T21:50:02.012 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync obj etag verify = True 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout: setgroup = ceph 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout: setuser = ceph 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout:[mon.a] 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data notify interval msec = 0 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2023-09-29T21:50:02.013 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T21:50:02.013 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2023-09-29T21:50:02.014 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2023-09-29T21:50:02.070 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T21:50:02.071 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:02.071 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2023-09-29T21:50:02.077 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:02.077 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2023-09-29T21:50:02.082 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T21:50:02.082 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:02.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/c1.keyring 2023-09-29T21:50:02.082 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-29T21:50:02.104 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:02.104 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:02.110 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T21:50:02.110 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:02.110 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/c1.keyring 2023-09-29T21:50:02.110 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/c1.keyring 2023-09-29T21:50:02.136 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:02.136 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:02.142 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T21:50:02.142 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T21:50:02.142 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2023-09-29T21:50:02.192 INFO:teuthology.orchestra.run.smithi112.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2023-09-29T21:50:02.195 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T21:50:02.195 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T21:50:02.195 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2023-09-29T21:50:02.237 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.client.0.keyring 2023-09-29T21:50:02.248 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2023-09-29T21:50:02.284 INFO:teuthology.orchestra.run.smithi112.stdout:creating /etc/ceph/c1.client.1.keyring 2023-09-29T21:50:02.296 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T21:50:02.297 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T21:50:02.297 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2023-09-29T21:50:02.308 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:02.308 INFO:tasks.ceph:role: c1.osd.0 2023-09-29T21:50:02.308 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T21:50:02.308 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T21:50:02.834 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T21:50:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T21:50:02.834 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T21:50:02.835 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T21:50:02.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-09-29T21:50:02.836 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2023-09-29T21:50:02.939 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2023-09-29T21:50:02.951 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T21:50:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:50:02.952 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2023-09-29T21:50:02.962 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:02.963 INFO:tasks.ceph:role: c1.osd.1 2023-09-29T21:50:02.963 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T21:50:02.963 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T21:50:03.498 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T21:50:03.498 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T21:50:03.498 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T21:50:03.499 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T21:50:03.500 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-09-29T21:50:03.500 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2023-09-29T21:50:03.536 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2023-09-29T21:50:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:50:03.543 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T21:50:03.543 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2023-09-29T21:50:03.599 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:03.599 INFO:tasks.ceph:role: c1.osd.2 2023-09-29T21:50:03.599 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T21:50:03.599 DEBUG:teuthology.orchestra.run.smithi112:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout: = reflink=1 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout: = sunit=0 swidth=0 blks 2023-09-29T21:50:04.078 INFO:teuthology.orchestra.run.smithi112.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T21:50:04.079 INFO:teuthology.orchestra.run.smithi112.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T21:50:04.079 INFO:teuthology.orchestra.run.smithi112.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T21:50:04.079 INFO:teuthology.orchestra.run.smithi112.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T21:50:04.080 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi112.front.sepia.ceph.com -o noatime 2023-09-29T21:50:04.080 DEBUG:teuthology.orchestra.run.smithi112:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2023-09-29T21:50:04.137 DEBUG:teuthology.orchestra.run.smithi112:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2023-09-29T21:50:04.149 INFO:teuthology.orchestra.run.smithi112.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T21:50:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:50:04.150 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:04.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:04.218+0000 7fcb272d1700 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2023-09-29T21:50:04.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:04.222+0000 7fcb272d1700 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2023-09-29T21:50:04.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:04.222+0000 7fcb272d1700 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-29T21:50:04.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:04.222+0000 7fcb272d1700 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-29T21:50:04.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:04.222+0000 7fcb272d1700 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2023-09-29T21:50:04.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:04.226+0000 7fcb272d1700 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2023-09-29T21:50:06.687 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2023-09-29T21:50:06.701 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:06.742+0000 7fc6eedbe700 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2023-09-29T21:50:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:06.742+0000 7fc6eedbe700 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2023-09-29T21:50:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:06.742+0000 7fc6eedbe700 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-29T21:50:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:06.742+0000 7fc6eedbe700 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-29T21:50:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:06.742+0000 7fc6eedbe700 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2023-09-29T21:50:06.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:06.742+0000 7fc6eedbe700 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2023-09-29T21:50:08.963 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2023-09-29T21:50:08.978 DEBUG:teuthology.orchestra.run.smithi112:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:09.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:09.022+0000 7fdd8a49e700 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2023-09-29T21:50:09.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:09.022+0000 7fdd8a49e700 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2023-09-29T21:50:09.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:09.022+0000 7fdd8a49e700 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-29T21:50:09.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:09.022+0000 7fdd8a49e700 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-29T21:50:09.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:09.022+0000 7fdd8a49e700 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2023-09-29T21:50:09.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:50:09.022+0000 7fdd8a49e700 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2023-09-29T21:50:11.238 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2023-09-29T21:50:11.252 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T21:50:11.253 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:11.253 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2023-09-29T21:50:11.266 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:11.266 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2023-09-29T21:50:11.277 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:11.277 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2023-09-29T21:50:11.292 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:11.292 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2023-09-29T21:50:11.306 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:11.306 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2023-09-29T21:50:11.313 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:11.313 DEBUG:teuthology.orchestra.run.smithi112:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2023-09-29T21:50:11.318 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T21:50:11.318 DEBUG:teuthology.orchestra.run.smithi112:> sudo tee -a /etc/ceph/c1.keyring 2023-09-29T21:50:11.329 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr.x] 2023-09-29T21:50:11.329 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCKRhdly9qMCxAAR39luE+NvvJOtMMc4Qycog== 2023-09-29T21:50:11.329 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-09-29T21:50:11.329 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCMRhdloiFsDRAAina7zYqqxWyZ/IL70fNZ6A== 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCORhdlImqXLBAAepu1cdjUK+EoJDD1la7jXw== 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCRRhdlm6WVARAABO3QMB+ZIIoxlB4pEDfvcw== 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCKRhdlvcs8DhAAH1FBELtzyX5yfRYUKE/UfQ== 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout:[client.1] 2023-09-29T21:50:11.330 INFO:teuthology.orchestra.run.smithi112.stdout: key = AQCKRhdla4EIERAArDyU9W1l5y77ZOiTLF74rw== 2023-09-29T21:50:11.331 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T21:50:11.377 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T21:50:11.420 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T21:50:11.469 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T21:50:11.511 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T21:50:11.559 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T21:50:11.664 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T21:50:11.665 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2023-09-29T21:50:11.681 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2023-09-29T21:50:11.747 DEBUG:teuthology.orchestra.run.smithi112:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2023-09-29T21:50:11.762 DEBUG:teuthology.orchestra.run.smithi112:> rm -- /home/ubuntu/cephtest/c1.monmap 2023-09-29T21:50:11.809 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2023-09-29T21:50:11.810 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2023-09-29T21:50:11.810 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2023-09-29T21:50:11.812 INFO:tasks.ceph.c1.mon.a:Started 2023-09-29T21:50:11.812 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2023-09-29T21:50:11.812 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2023-09-29T21:50:11.813 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2023-09-29T21:50:11.815 INFO:tasks.ceph.c1.mgr.x:Started 2023-09-29T21:50:11.815 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T21:50:11.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd crush tunables default 2023-09-29T21:50:12.069 INFO:teuthology.orchestra.run.smithi112.stderr:adjusted tunables profile to default 2023-09-29T21:50:12.079 INFO:tasks.ceph:check_enable_crimson: False 2023-09-29T21:50:12.079 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2023-09-29T21:50:12.079 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:12.079 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2023-09-29T21:50:12.090 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:12.090 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2023-09-29T21:50:12.144 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:12.144 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2023-09-29T21:50:12.159 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new e70c6625-c0f6-446c-ad31-0ee74f70448c 0 2023-09-29T21:50:12.177 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:12.174+0000 7f2107ba42c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T21:50:12.361 INFO:teuthology.orchestra.run.smithi112.stdout:0 2023-09-29T21:50:12.374 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new fbb92071-bc06-44e9-b1ed-926e31545524 1 2023-09-29T21:50:12.573 INFO:teuthology.orchestra.run.smithi112.stdout:1 2023-09-29T21:50:12.591 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd new d2f737d1-26a9-4e90-b6b5-5992389bdb61 2 2023-09-29T21:50:12.791 INFO:teuthology.orchestra.run.smithi112.stdout:2 2023-09-29T21:50:12.805 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:12.802+0000 7f2107ba42c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T21:50:12.809 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2023-09-29T21:50:12.809 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2023-09-29T21:50:12.812 INFO:tasks.ceph.c1.osd.0:Started 2023-09-29T21:50:12.812 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2023-09-29T21:50:12.812 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2023-09-29T21:50:12.814 INFO:tasks.ceph.c1.osd.1:Started 2023-09-29T21:50:12.814 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2023-09-29T21:50:12.815 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2023-09-29T21:50:12.817 INFO:tasks.ceph.c1.osd.2:Started 2023-09-29T21:50:12.817 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-29T21:50:12.885 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:12.882+0000 7f2107ba42c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T21:50:13.012 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:13.012 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":5,"fsid":"1be42293-dd6a-48c9-b3ea-af1a1b1442b0","created":"2023-09-29T21:50:11.952674+0000","modified":"2023-09-29T21:50:12.789489+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T21:50:12.789489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e70c6625-c0f6-446c-ad31-0ee74f70448c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fbb92071-bc06-44e9-b1ed-926e31545524","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d2f737d1-26a9-4e90-b6b5-5992389bdb61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:13.030 INFO:tasks.ceph.ceph_manager.c1:[] 2023-09-29T21:50:13.031 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T21:50:13.062 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:13.058+0000 7f2107ba42c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T21:50:13.194 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-09-29T21:50:13.190+0000 7f871a3b4700 -1 Falling back to public interface 2023-09-29T21:50:13.274 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:13.270+0000 7f2107ba42c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T21:50:13.328 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:13.326+0000 7f2107ba42c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T21:50:13.333 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-29T21:50:13.458 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-09-29T21:50:13.454+0000 7f728d0a0700 -1 Falling back to public interface 2023-09-29T21:50:13.458 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-09-29T21:50:13.454+0000 7f99ee5eb700 -1 Falling back to public interface 2023-09-29T21:50:13.459 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-09-29T21:50:13.454+0000 7f728c06b700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6816/0: (98) Address already in use 2023-09-29T21:50:13.514 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:13.510+0000 7f2107ba42c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T21:50:13.519 INFO:teuthology.misc.health.smithi112.stdout: 2023-09-29T21:50:13.519 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":5,"fsid":"1be42293-dd6a-48c9-b3ea-af1a1b1442b0","created":"2023-09-29T21:50:11.952674+0000","modified":"2023-09-29T21:50:12.789489+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T21:50:12.789489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e70c6625-c0f6-446c-ad31-0ee74f70448c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fbb92071-bc06-44e9-b1ed-926e31545524","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d2f737d1-26a9-4e90-b6b5-5992389bdb61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:13.530 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-29T21:50:13.582 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:13.578+0000 7f2107ba42c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T21:50:13.651 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:13.646+0000 7f2107ba42c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T21:50:14.084 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:14.078+0000 7f2107ba42c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T21:50:14.211 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:14.206+0000 7f2107ba42c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T21:50:14.571 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:14.566+0000 7f2107ba42c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T21:50:14.740 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:14.734+0000 7f2107ba42c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T21:50:14.807 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:14.802+0000 7f2107ba42c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T21:50:14.861 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:14.858+0000 7f2107ba42c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T21:50:15.796 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:15.794+0000 7f2107ba42c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T21:50:15.867 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:15.862+0000 7f2107ba42c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T21:50:16.037 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:16.034+0000 7f2107ba42c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T21:50:16.038 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-09-29T21:50:16.034+0000 7f871a3b4700 -1 osd.0 0 log_to_monitors true 2023-09-29T21:50:16.104 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:16.102+0000 7f2107ba42c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T21:50:16.178 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:16.174+0000 7f2107ba42c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T21:50:16.261 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-09-29T21:50:16.258+0000 7f99ee5eb700 -1 osd.2 0 log_to_monitors true 2023-09-29T21:50:16.320 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-09-29T21:50:16.318+0000 7f728d0a0700 -1 osd.1 0 log_to_monitors true 2023-09-29T21:50:16.354 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:16.350+0000 7f2107ba42c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T21:50:16.565 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:16.562+0000 7f2107ba42c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T21:50:17.213 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:17.210+0000 7f2107ba42c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T21:50:17.876 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-09-29T21:50:17.874+0000 7f87161f3700 -1 osd.0 0 waiting for initial osdmap 2023-09-29T21:50:17.879 INFO:tasks.ceph.c1.osd.0.smithi112.stderr:2023-09-29T21:50:17.874+0000 7f870dca2700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T21:50:18.298 INFO:tasks.ceph.c1.mgr.x.smithi112.stderr:2023-09-29T21:50:18.294+0000 7f20eec30700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-29T21:50:19.120 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-09-29T21:50:19.118+0000 7f99ea42a700 -1 osd.2 0 waiting for initial osdmap 2023-09-29T21:50:19.125 INFO:tasks.ceph.c1.osd.2.smithi112.stderr:2023-09-29T21:50:19.122+0000 7f99e1ed9700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T21:50:19.178 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-09-29T21:50:19.174+0000 7f7288edf700 -1 osd.1 0 waiting for initial osdmap 2023-09-29T21:50:19.184 INFO:tasks.ceph.c1.osd.1.smithi112.stderr:2023-09-29T21:50:19.182+0000 7f728098e700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T21:50:19.833 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2023-09-29T21:50:20.130 INFO:teuthology.misc.health.smithi112.stdout: 2023-09-29T21:50:20.131 INFO:teuthology.misc.health.smithi112.stdout:{"epoch":9,"fsid":"1be42293-dd6a-48c9-b3ea-af1a1b1442b0","created":"2023-09-29T21:50:11.952674+0000","modified":"2023-09-29T21:50:19.301510+0000","last_up_change":"2023-09-29T21:50:19.301510+0000","last_in_change":"2023-09-29T21:50:12.789489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e70c6625-c0f6-446c-ad31-0ee74f70448c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":28957},{"type":"v1","addr":"172.21.15.112:6801","nonce":28957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":28957},{"type":"v1","addr":"172.21.15.112:6803","nonce":28957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":28957},{"type":"v1","addr":"172.21.15.112:6807","nonce":28957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":28957},{"type":"v1","addr":"172.21.15.112:6805","nonce":28957}]},"public_addr":"172.21.15.112:6801/28957","cluster_addr":"172.21.15.112:6803/28957","heartbeat_back_addr":"172.21.15.112:6807/28957","heartbeat_front_addr":"172.21.15.112:6805/28957","state":["exists","up"]},{"osd":1,"uuid":"fbb92071-bc06-44e9-b1ed-926e31545524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":28956},{"type":"v1","addr":"172.21.15.112:6812","nonce":28956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":28956},{"type":"v1","addr":"172.21.15.112:6817","nonce":28956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":28956},{"type":"v1","addr":"172.21.15.112:6823","nonce":28956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":28956},{"type":"v1","addr":"172.21.15.112:6821","nonce":28956}]},"public_addr":"172.21.15.112:6812/28956","cluster_addr":"172.21.15.112:6817/28956","heartbeat_back_addr":"172.21.15.112:6823/28956","heartbeat_front_addr":"172.21.15.112:6821/28956","state":["exists","up"]},{"osd":2,"uuid":"d2f737d1-26a9-4e90-b6b5-5992389bdb61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":28958},{"type":"v1","addr":"172.21.15.112:6810","nonce":28958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6811","nonce":28958},{"type":"v1","addr":"172.21.15.112:6813","nonce":28958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":28958},{"type":"v1","addr":"172.21.15.112:6820","nonce":28958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":28958},{"type":"v1","addr":"172.21.15.112:6816","nonce":28958}]},"public_addr":"172.21.15.112:6810/28958","cluster_addr":"172.21.15.112:6813/28958","heartbeat_back_addr":"172.21.15.112:6820/28958","heartbeat_front_addr":"172.21.15.112:6816/28958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.047567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:20.145 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-29T21:50:20.145 INFO:tasks.ceph:Creating RBD pool 2023-09-29T21:50:20.145 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd pool create rbd 8 2023-09-29T21:50:21.199 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'rbd' created 2023-09-29T21:50:21.212 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T21:50:22.208 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T21:50:22.221 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2023-09-29T21:50:22.222 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2023-09-29T21:50:22.222 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2023-09-29T21:50:22.222 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2023-09-29T21:50:22.223 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T21:50:22.546 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:22.559 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6824","nonce":28771},{"type":"v1","addr":"172.21.15.112:6825","nonce":28771}]},"active_addr":"172.21.15.112:6825/28771","active_change":"2023-09-29T21:50:17.235566+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2301467953}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":2756568148}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.112:0","nonce":657328779}]}]} 2023-09-29T21:50:22.562 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2023-09-29T21:50:22.562 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2023-09-29T21:50:22.562 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-29T21:50:22.854 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:22.855 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":11,"fsid":"1be42293-dd6a-48c9-b3ea-af1a1b1442b0","created":"2023-09-29T21:50:11.952674+0000","modified":"2023-09-29T21:50:22.190160+0000","last_up_change":"2023-09-29T21:50:19.301510+0000","last_in_change":"2023-09-29T21:50:12.789489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-29T21:50:20.483167+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-29T21:50:21.300904+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e70c6625-c0f6-446c-ad31-0ee74f70448c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":28957},{"type":"v1","addr":"172.21.15.112:6801","nonce":28957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":28957},{"type":"v1","addr":"172.21.15.112:6803","nonce":28957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":28957},{"type":"v1","addr":"172.21.15.112:6807","nonce":28957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":28957},{"type":"v1","addr":"172.21.15.112:6805","nonce":28957}]},"public_addr":"172.21.15.112:6801/28957","cluster_addr":"172.21.15.112:6803/28957","heartbeat_back_addr":"172.21.15.112:6807/28957","heartbeat_front_addr":"172.21.15.112:6805/28957","state":["exists","up"]},{"osd":1,"uuid":"fbb92071-bc06-44e9-b1ed-926e31545524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":28956},{"type":"v1","addr":"172.21.15.112:6812","nonce":28956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":28956},{"type":"v1","addr":"172.21.15.112:6817","nonce":28956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":28956},{"type":"v1","addr":"172.21.15.112:6823","nonce":28956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":28956},{"type":"v1","addr":"172.21.15.112:6821","nonce":28956}]},"public_addr":"172.21.15.112:6812/28956","cluster_addr":"172.21.15.112:6817/28956","heartbeat_back_addr":"172.21.15.112:6823/28956","heartbeat_front_addr":"172.21.15.112:6821/28956","state":["exists","up"]},{"osd":2,"uuid":"d2f737d1-26a9-4e90-b6b5-5992389bdb61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":28958},{"type":"v1","addr":"172.21.15.112:6810","nonce":28958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6811","nonce":28958},{"type":"v1","addr":"172.21.15.112:6813","nonce":28958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":28958},{"type":"v1","addr":"172.21.15.112:6820","nonce":28958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":28958},{"type":"v1","addr":"172.21.15.112:6816","nonce":28958}]},"public_addr":"172.21.15.112:6810/28958","cluster_addr":"172.21.15.112:6813/28958","heartbeat_back_addr":"172.21.15.112:6820/28958","heartbeat_front_addr":"172.21.15.112:6816/28958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.047567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.339440+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.242579+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:22.868 INFO:tasks.ceph.ceph_manager.c1:all up! 2023-09-29T21:50:22.869 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2023-09-29T21:50:23.157 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:23.157 INFO:teuthology.orchestra.run.smithi112.stdout:{"epoch":11,"fsid":"1be42293-dd6a-48c9-b3ea-af1a1b1442b0","created":"2023-09-29T21:50:11.952674+0000","modified":"2023-09-29T21:50:22.190160+0000","last_up_change":"2023-09-29T21:50:19.301510+0000","last_in_change":"2023-09-29T21:50:12.789489+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-29T21:50:20.483167+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-29T21:50:21.300904+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e70c6625-c0f6-446c-ad31-0ee74f70448c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6800","nonce":28957},{"type":"v1","addr":"172.21.15.112:6801","nonce":28957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6802","nonce":28957},{"type":"v1","addr":"172.21.15.112:6803","nonce":28957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6806","nonce":28957},{"type":"v1","addr":"172.21.15.112:6807","nonce":28957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6804","nonce":28957},{"type":"v1","addr":"172.21.15.112:6805","nonce":28957}]},"public_addr":"172.21.15.112:6801/28957","cluster_addr":"172.21.15.112:6803/28957","heartbeat_back_addr":"172.21.15.112:6807/28957","heartbeat_front_addr":"172.21.15.112:6805/28957","state":["exists","up"]},{"osd":1,"uuid":"fbb92071-bc06-44e9-b1ed-926e31545524","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6809","nonce":28956},{"type":"v1","addr":"172.21.15.112:6812","nonce":28956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6814","nonce":28956},{"type":"v1","addr":"172.21.15.112:6817","nonce":28956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6822","nonce":28956},{"type":"v1","addr":"172.21.15.112:6823","nonce":28956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6819","nonce":28956},{"type":"v1","addr":"172.21.15.112:6821","nonce":28956}]},"public_addr":"172.21.15.112:6812/28956","cluster_addr":"172.21.15.112:6817/28956","heartbeat_back_addr":"172.21.15.112:6823/28956","heartbeat_front_addr":"172.21.15.112:6821/28956","state":["exists","up"]},{"osd":2,"uuid":"d2f737d1-26a9-4e90-b6b5-5992389bdb61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6808","nonce":28958},{"type":"v1","addr":"172.21.15.112:6810","nonce":28958}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6811","nonce":28958},{"type":"v1","addr":"172.21.15.112:6813","nonce":28958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6818","nonce":28958},{"type":"v1","addr":"172.21.15.112:6820","nonce":28958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.112:6815","nonce":28958},{"type":"v1","addr":"172.21.15.112:6816","nonce":28958}]},"public_addr":"172.21.15.112:6810/28958","cluster_addr":"172.21.15.112:6813/28958","heartbeat_back_addr":"172.21.15.112:6820/28958","heartbeat_front_addr":"172.21.15.112:6816/28958","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.047567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.339440+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:17.242579+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:23.172 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2023-09-29T21:50:23.172 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2023-09-29T21:50:23.173 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2023-09-29T21:50:23.307 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-09-29T21:50:23.307 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-29T21:50:23.312 INFO:teuthology.orchestra.run.smithi112.stdout:38654705667 2023-09-29T21:50:23.312 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-29T21:50:23.327 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2023-09-29T21:50:23.327 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-29T21:50:23.619 INFO:teuthology.orchestra.run.smithi112.stdout:38654705665 2023-09-29T21:50:23.631 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.2 2023-09-29T21:50:23.643 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-09-29T21:50:23.656 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2023-09-29T21:50:23.671 INFO:teuthology.orchestra.run.smithi112.stdout:34359738369 2023-09-29T21:50:23.685 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738369 for osd.0 2023-09-29T21:50:24.633 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-29T21:50:24.657 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-29T21:50:24.686 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-29T21:50:24.981 INFO:teuthology.orchestra.run.smithi112.stdout:34359738369 2023-09-29T21:50:24.995 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738369 for osd.0 2023-09-29T21:50:24.996 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-09-29T21:50:24.997 INFO:teuthology.orchestra.run.smithi112.stdout:38654705665 2023-09-29T21:50:25.010 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705666 for osd.1 2023-09-29T21:50:25.011 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705665 for osd.2 2023-09-29T21:50:25.996 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2023-09-29T21:50:26.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2023-09-29T21:50:26.011 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2023-09-29T21:50:26.313 INFO:teuthology.orchestra.run.smithi112.stdout:38654705666 2023-09-29T21:50:26.320 INFO:teuthology.orchestra.run.smithi112.stdout:38654705667 2023-09-29T21:50:26.327 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705666 got 38654705666 for osd.2 2023-09-29T21:50:26.327 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:26.332 INFO:teuthology.orchestra.run.smithi112.stdout:34359738371 2023-09-29T21:50:26.333 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2023-09-29T21:50:26.333 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:26.346 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2023-09-29T21:50:26.346 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:26.346 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2023-09-29T21:50:26.346 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T21:50:26.631 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:26.634 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-09-29T21:50:26.644 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T21:50:25.247126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81152,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314491648,"statfs":{"total":322122547200,"available":322039447552,"internally_reserved":0,"allocated":303104,"data_stored":76336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.217383+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:22.217383+0000","last_peered":"2023-09-29T21:50:22.217383+0000","last_clean":"2023-09-29T21:50:22.217383+0000","last_became_active":"2023-09-29T21:50:22.213863+0000","last_became_peered":"2023-09-29T21:50:22.213863+0000","last_unstale":"2023-09-29T21:50:22.217383+0000","last_undegraded":"2023-09-29T21:50:22.217383+0000","last_fullsized":"2023-09-29T21:50:22.217383+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:45:32.741449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216919+0000","last_change":"2023-09-29T21:50:22.213796+0000","last_active":"2023-09-29T21:50:22.216919+0000","last_peered":"2023-09-29T21:50:22.216919+0000","last_clean":"2023-09-29T21:50:22.216919+0000","last_became_active":"2023-09-29T21:50:22.213555+0000","last_became_peered":"2023-09-29T21:50:22.213555+0000","last_unstale":"2023-09-29T21:50:22.216919+0000","last_undegraded":"2023-09-29T21:50:22.216919+0000","last_fullsized":"2023-09-29T21:50:22.216919+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:49:23.583986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203966+0000","last_change":"2023-09-29T21:50:22.217056+0000","last_active":"2023-09-29T21:50:23.203966+0000","last_peered":"2023-09-29T21:50:23.203966+0000","last_clean":"2023-09-29T21:50:23.203966+0000","last_became_active":"2023-09-29T21:50:22.216814+0000","last_became_peered":"2023-09-29T21:50:22.216814+0000","last_unstale":"2023-09-29T21:50:23.203966+0000","last_undegraded":"2023-09-29T21:50:23.203966+0000","last_fullsized":"2023-09-29T21:50:23.203966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:37:11.988482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.222005+0000","last_change":"2023-09-29T21:50:22.216868+0000","last_active":"2023-09-29T21:50:22.222005+0000","last_peered":"2023-09-29T21:50:22.222005+0000","last_clean":"2023-09-29T21:50:22.222005+0000","last_became_active":"2023-09-29T21:50:22.216575+0000","last_became_peered":"2023-09-29T21:50:22.216575+0000","last_unstale":"2023-09-29T21:50:22.222005+0000","last_undegraded":"2023-09-29T21:50:22.222005+0000","last_fullsized":"2023-09-29T21:50:22.222005+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:00.192258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3,"reported_epoch":12,"state":"creating+activating","last_fresh":"2023-09-29T21:50:23.204164+0000","last_change":"2023-09-29T21:50:23.204164+0000","last_active":"2023-09-29T21:50:22.190160+0000","last_peered":"2023-09-29T21:50:22.190160+0000","last_clean":"2023-09-29T21:50:22.190160+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T21:50:23.204164+0000","last_undegraded":"2023-09-29T21:50:23.204164+0000","last_fullsized":"2023-09-29T21:50:23.204164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_clean_scrub_stamp":"2023-09-29T21:50:22.190160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:30.258112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216988+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:22.216988+0000","last_peered":"2023-09-29T21:50:22.216988+0000","last_clean":"2023-09-29T21:50:22.216988+0000","last_became_active":"2023-09-29T21:50:22.213922+0000","last_became_peered":"2023-09-29T21:50:22.213922+0000","last_unstale":"2023-09-29T21:50:22.216988+0000","last_undegraded":"2023-09-29T21:50:22.216988+0000","last_fullsized":"2023-09-29T21:50:22.216988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:59.692405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216036+0000","last_change":"2023-09-29T21:50:22.211029+0000","last_active":"2023-09-29T21:50:22.216036+0000","last_peered":"2023-09-29T21:50:22.216036+0000","last_clean":"2023-09-29T21:50:22.216036+0000","last_became_active":"2023-09-29T21:50:22.210748+0000","last_became_peered":"2023-09-29T21:50:22.210748+0000","last_unstale":"2023-09-29T21:50:22.216036+0000","last_undegraded":"2023-09-29T21:50:22.216036+0000","last_fullsized":"2023-09-29T21:50:22.216036+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:33:28.229495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203946+0000","last_change":"2023-09-29T21:50:22.216651+0000","last_active":"2023-09-29T21:50:23.203946+0000","last_peered":"2023-09-29T21:50:23.203946+0000","last_clean":"2023-09-29T21:50:23.203946+0000","last_became_active":"2023-09-29T21:50:22.216391+0000","last_became_peered":"2023-09-29T21:50:22.216391+0000","last_unstale":"2023-09-29T21:50:23.203946+0000","last_undegraded":"2023-09-29T21:50:23.203946+0000","last_fullsized":"2023-09-29T21:50:23.203946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:57:32.843104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216969+0000","last_change":"2023-09-29T21:50:22.213848+0000","last_active":"2023-09-29T21:50:22.216969+0000","last_peered":"2023-09-29T21:50:22.216969+0000","last_clean":"2023-09-29T21:50:22.216969+0000","last_became_active":"2023-09-29T21:50:22.213540+0000","last_became_peered":"2023-09-29T21:50:22.213540+0000","last_unstale":"2023-09-29T21:50:22.216969+0000","last_undegraded":"2023-09-29T21:50:22.216969+0000","last_fullsized":"2023-09-29T21:50:22.216969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:29.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:26.646 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T21:50:26.932 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:26.935 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-09-29T21:50:26.946 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T21:50:25.247126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81152,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314491648,"statfs":{"total":322122547200,"available":322039447552,"internally_reserved":0,"allocated":303104,"data_stored":76336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.217383+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:22.217383+0000","last_peered":"2023-09-29T21:50:22.217383+0000","last_clean":"2023-09-29T21:50:22.217383+0000","last_became_active":"2023-09-29T21:50:22.213863+0000","last_became_peered":"2023-09-29T21:50:22.213863+0000","last_unstale":"2023-09-29T21:50:22.217383+0000","last_undegraded":"2023-09-29T21:50:22.217383+0000","last_fullsized":"2023-09-29T21:50:22.217383+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:45:32.741449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216919+0000","last_change":"2023-09-29T21:50:22.213796+0000","last_active":"2023-09-29T21:50:22.216919+0000","last_peered":"2023-09-29T21:50:22.216919+0000","last_clean":"2023-09-29T21:50:22.216919+0000","last_became_active":"2023-09-29T21:50:22.213555+0000","last_became_peered":"2023-09-29T21:50:22.213555+0000","last_unstale":"2023-09-29T21:50:22.216919+0000","last_undegraded":"2023-09-29T21:50:22.216919+0000","last_fullsized":"2023-09-29T21:50:22.216919+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:49:23.583986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203966+0000","last_change":"2023-09-29T21:50:22.217056+0000","last_active":"2023-09-29T21:50:23.203966+0000","last_peered":"2023-09-29T21:50:23.203966+0000","last_clean":"2023-09-29T21:50:23.203966+0000","last_became_active":"2023-09-29T21:50:22.216814+0000","last_became_peered":"2023-09-29T21:50:22.216814+0000","last_unstale":"2023-09-29T21:50:23.203966+0000","last_undegraded":"2023-09-29T21:50:23.203966+0000","last_fullsized":"2023-09-29T21:50:23.203966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:37:11.988482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.222005+0000","last_change":"2023-09-29T21:50:22.216868+0000","last_active":"2023-09-29T21:50:22.222005+0000","last_peered":"2023-09-29T21:50:22.222005+0000","last_clean":"2023-09-29T21:50:22.222005+0000","last_became_active":"2023-09-29T21:50:22.216575+0000","last_became_peered":"2023-09-29T21:50:22.216575+0000","last_unstale":"2023-09-29T21:50:22.222005+0000","last_undegraded":"2023-09-29T21:50:22.222005+0000","last_fullsized":"2023-09-29T21:50:22.222005+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:00.192258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3,"reported_epoch":12,"state":"creating+activating","last_fresh":"2023-09-29T21:50:23.204164+0000","last_change":"2023-09-29T21:50:23.204164+0000","last_active":"2023-09-29T21:50:22.190160+0000","last_peered":"2023-09-29T21:50:22.190160+0000","last_clean":"2023-09-29T21:50:22.190160+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T21:50:23.204164+0000","last_undegraded":"2023-09-29T21:50:23.204164+0000","last_fullsized":"2023-09-29T21:50:23.204164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_clean_scrub_stamp":"2023-09-29T21:50:22.190160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:30.258112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216988+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:22.216988+0000","last_peered":"2023-09-29T21:50:22.216988+0000","last_clean":"2023-09-29T21:50:22.216988+0000","last_became_active":"2023-09-29T21:50:22.213922+0000","last_became_peered":"2023-09-29T21:50:22.213922+0000","last_unstale":"2023-09-29T21:50:22.216988+0000","last_undegraded":"2023-09-29T21:50:22.216988+0000","last_fullsized":"2023-09-29T21:50:22.216988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:59.692405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216036+0000","last_change":"2023-09-29T21:50:22.211029+0000","last_active":"2023-09-29T21:50:22.216036+0000","last_peered":"2023-09-29T21:50:22.216036+0000","last_clean":"2023-09-29T21:50:22.216036+0000","last_became_active":"2023-09-29T21:50:22.210748+0000","last_became_peered":"2023-09-29T21:50:22.210748+0000","last_unstale":"2023-09-29T21:50:22.216036+0000","last_undegraded":"2023-09-29T21:50:22.216036+0000","last_fullsized":"2023-09-29T21:50:22.216036+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:33:28.229495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203946+0000","last_change":"2023-09-29T21:50:22.216651+0000","last_active":"2023-09-29T21:50:23.203946+0000","last_peered":"2023-09-29T21:50:23.203946+0000","last_clean":"2023-09-29T21:50:23.203946+0000","last_became_active":"2023-09-29T21:50:22.216391+0000","last_became_peered":"2023-09-29T21:50:22.216391+0000","last_unstale":"2023-09-29T21:50:23.203946+0000","last_undegraded":"2023-09-29T21:50:23.203946+0000","last_fullsized":"2023-09-29T21:50:23.203946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:57:32.843104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216969+0000","last_change":"2023-09-29T21:50:22.213848+0000","last_active":"2023-09-29T21:50:22.216969+0000","last_peered":"2023-09-29T21:50:22.216969+0000","last_clean":"2023-09-29T21:50:22.216969+0000","last_became_active":"2023-09-29T21:50:22.213540+0000","last_became_peered":"2023-09-29T21:50:22.213540+0000","last_unstale":"2023-09-29T21:50:22.216969+0000","last_undegraded":"2023-09-29T21:50:22.216969+0000","last_fullsized":"2023-09-29T21:50:22.216969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:29.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:26.947 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T21:50:27.240 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:27.243 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-09-29T21:50:27.254 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T21:50:25.247126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":81152,"kb_used_data":296,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314491648,"statfs":{"total":322122547200,"available":322039447552,"internally_reserved":0,"allocated":303104,"data_stored":76336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.217383+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:22.217383+0000","last_peered":"2023-09-29T21:50:22.217383+0000","last_clean":"2023-09-29T21:50:22.217383+0000","last_became_active":"2023-09-29T21:50:22.213863+0000","last_became_peered":"2023-09-29T21:50:22.213863+0000","last_unstale":"2023-09-29T21:50:22.217383+0000","last_undegraded":"2023-09-29T21:50:22.217383+0000","last_fullsized":"2023-09-29T21:50:22.217383+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:45:32.741449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216919+0000","last_change":"2023-09-29T21:50:22.213796+0000","last_active":"2023-09-29T21:50:22.216919+0000","last_peered":"2023-09-29T21:50:22.216919+0000","last_clean":"2023-09-29T21:50:22.216919+0000","last_became_active":"2023-09-29T21:50:22.213555+0000","last_became_peered":"2023-09-29T21:50:22.213555+0000","last_unstale":"2023-09-29T21:50:22.216919+0000","last_undegraded":"2023-09-29T21:50:22.216919+0000","last_fullsized":"2023-09-29T21:50:22.216919+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:49:23.583986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203966+0000","last_change":"2023-09-29T21:50:22.217056+0000","last_active":"2023-09-29T21:50:23.203966+0000","last_peered":"2023-09-29T21:50:23.203966+0000","last_clean":"2023-09-29T21:50:23.203966+0000","last_became_active":"2023-09-29T21:50:22.216814+0000","last_became_peered":"2023-09-29T21:50:22.216814+0000","last_unstale":"2023-09-29T21:50:23.203966+0000","last_undegraded":"2023-09-29T21:50:23.203966+0000","last_fullsized":"2023-09-29T21:50:23.203966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:37:11.988482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.222005+0000","last_change":"2023-09-29T21:50:22.216868+0000","last_active":"2023-09-29T21:50:22.222005+0000","last_peered":"2023-09-29T21:50:22.222005+0000","last_clean":"2023-09-29T21:50:22.222005+0000","last_became_active":"2023-09-29T21:50:22.216575+0000","last_became_peered":"2023-09-29T21:50:22.216575+0000","last_unstale":"2023-09-29T21:50:22.222005+0000","last_undegraded":"2023-09-29T21:50:22.222005+0000","last_fullsized":"2023-09-29T21:50:22.222005+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:00.192258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3,"reported_epoch":12,"state":"creating+activating","last_fresh":"2023-09-29T21:50:23.204164+0000","last_change":"2023-09-29T21:50:23.204164+0000","last_active":"2023-09-29T21:50:22.190160+0000","last_peered":"2023-09-29T21:50:22.190160+0000","last_clean":"2023-09-29T21:50:22.190160+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-09-29T21:50:23.204164+0000","last_undegraded":"2023-09-29T21:50:23.204164+0000","last_fullsized":"2023-09-29T21:50:23.204164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_clean_scrub_stamp":"2023-09-29T21:50:22.190160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:30.258112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216988+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:22.216988+0000","last_peered":"2023-09-29T21:50:22.216988+0000","last_clean":"2023-09-29T21:50:22.216988+0000","last_became_active":"2023-09-29T21:50:22.213922+0000","last_became_peered":"2023-09-29T21:50:22.213922+0000","last_unstale":"2023-09-29T21:50:22.216988+0000","last_undegraded":"2023-09-29T21:50:22.216988+0000","last_fullsized":"2023-09-29T21:50:22.216988+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:59.692405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216036+0000","last_change":"2023-09-29T21:50:22.211029+0000","last_active":"2023-09-29T21:50:22.216036+0000","last_peered":"2023-09-29T21:50:22.216036+0000","last_clean":"2023-09-29T21:50:22.216036+0000","last_became_active":"2023-09-29T21:50:22.210748+0000","last_became_peered":"2023-09-29T21:50:22.210748+0000","last_unstale":"2023-09-29T21:50:22.216036+0000","last_undegraded":"2023-09-29T21:50:22.216036+0000","last_fullsized":"2023-09-29T21:50:22.216036+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:33:28.229495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203946+0000","last_change":"2023-09-29T21:50:22.216651+0000","last_active":"2023-09-29T21:50:23.203946+0000","last_peered":"2023-09-29T21:50:23.203946+0000","last_clean":"2023-09-29T21:50:23.203946+0000","last_became_active":"2023-09-29T21:50:22.216391+0000","last_became_peered":"2023-09-29T21:50:22.216391+0000","last_unstale":"2023-09-29T21:50:23.203946+0000","last_undegraded":"2023-09-29T21:50:23.203946+0000","last_fullsized":"2023-09-29T21:50:23.203946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:57:32.843104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216969+0000","last_change":"2023-09-29T21:50:22.213848+0000","last_active":"2023-09-29T21:50:22.216969+0000","last_peered":"2023-09-29T21:50:22.216969+0000","last_clean":"2023-09-29T21:50:22.216969+0000","last_became_active":"2023-09-29T21:50:22.213540+0000","last_became_peered":"2023-09-29T21:50:22.213540+0000","last_unstale":"2023-09-29T21:50:22.216969+0000","last_undegraded":"2023-09-29T21:50:22.216969+0000","last_fullsized":"2023-09-29T21:50:22.216969+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:29.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":28016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]}]},{"osd":1,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:27.255 INFO:tasks.ceph.ceph_manager.c1:PG 2.0 is not active+clean 2023-09-29T21:50:27.255 INFO:tasks.ceph.ceph_manager.c1:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 12, 'state': 'creating+activating', 'last_fresh': '2023-09-29T21:50:23.204164+0000', 'last_change': '2023-09-29T21:50:23.204164+0000', 'last_active': '2023-09-29T21:50:22.190160+0000', 'last_peered': '2023-09-29T21:50:22.190160+0000', 'last_clean': '2023-09-29T21:50:22.190160+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-09-29T21:50:23.204164+0000', 'last_undegraded': '2023-09-29T21:50:23.204164+0000', 'last_fullsized': '2023-09-29T21:50:23.204164+0000', 'mapping_epoch': 11, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-09-29T21:50:22.190160+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-09-29T21:50:22.190160+0000', 'last_clean_scrub_stamp': '2023-09-29T21:50:22.190160+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-10-01T07:11:30.258112+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-09-29T21:50:27.255 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 status --format=json 2023-09-29T21:50:27.606 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:27.606 INFO:teuthology.orchestra.run.smithi112.stdout:{"fsid":"1be42293-dd6a-48c9-b3ea-af1a1b1442b0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":15,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":13,"num_osds":3,"num_up_osds":3,"osd_up_since":1696024219,"num_in_osds":3,"osd_in_since":1696024212,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"creating+activating","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":83099648,"bytes_avail":322039447552,"bytes_total":322122547200,"inactive_pgs_ratio":0.1111111119389534},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-09-29T21:50:11.954572+0000","services":{}},"progress_events":{}} 2023-09-29T21:50:27.620 INFO:tasks.ceph.ceph_manager.c1:no progress seen, keeping timeout for now 2023-09-29T21:50:27.620 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T21:50:27.907 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-09-29T21:50:27.921 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2023-09-29T21:50:27.247637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82328,"kb_used_data":1472,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490472,"statfs":{"total":322122547200,"available":322038243328,"internally_reserved":0,"allocated":1507328,"data_stored":1264438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322314+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:23.322314+0000","last_peered":"2023-09-29T21:50:23.322314+0000","last_clean":"2023-09-29T21:50:23.322314+0000","last_became_active":"2023-09-29T21:50:22.213863+0000","last_became_peered":"2023-09-29T21:50:22.213863+0000","last_unstale":"2023-09-29T21:50:23.322314+0000","last_undegraded":"2023-09-29T21:50:23.322314+0000","last_fullsized":"2023-09-29T21:50:23.322314+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:45:32.741449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322383+0000","last_change":"2023-09-29T21:50:22.213796+0000","last_active":"2023-09-29T21:50:23.322383+0000","last_peered":"2023-09-29T21:50:23.322383+0000","last_clean":"2023-09-29T21:50:23.322383+0000","last_became_active":"2023-09-29T21:50:22.213555+0000","last_became_peered":"2023-09-29T21:50:22.213555+0000","last_unstale":"2023-09-29T21:50:23.322383+0000","last_undegraded":"2023-09-29T21:50:23.322383+0000","last_fullsized":"2023-09-29T21:50:23.322383+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:49:23.583986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203966+0000","last_change":"2023-09-29T21:50:22.217056+0000","last_active":"2023-09-29T21:50:23.203966+0000","last_peered":"2023-09-29T21:50:23.203966+0000","last_clean":"2023-09-29T21:50:23.203966+0000","last_became_active":"2023-09-29T21:50:22.216814+0000","last_became_peered":"2023-09-29T21:50:22.216814+0000","last_unstale":"2023-09-29T21:50:23.203966+0000","last_undegraded":"2023-09-29T21:50:23.203966+0000","last_fullsized":"2023-09-29T21:50:23.203966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:37:11.988482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322422+0000","last_change":"2023-09-29T21:50:22.216868+0000","last_active":"2023-09-29T21:50:23.322422+0000","last_peered":"2023-09-29T21:50:23.322422+0000","last_clean":"2023-09-29T21:50:23.322422+0000","last_became_active":"2023-09-29T21:50:22.216575+0000","last_became_peered":"2023-09-29T21:50:22.216575+0000","last_unstale":"2023-09-29T21:50:23.322422+0000","last_undegraded":"2023-09-29T21:50:23.322422+0000","last_fullsized":"2023-09-29T21:50:23.322422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:00.192258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.975248+0000","last_change":"2023-09-29T21:50:23.323780+0000","last_active":"2023-09-29T21:50:23.975248+0000","last_peered":"2023-09-29T21:50:23.975248+0000","last_clean":"2023-09-29T21:50:23.975248+0000","last_became_active":"2023-09-29T21:50:23.323527+0000","last_became_peered":"2023-09-29T21:50:23.323527+0000","last_unstale":"2023-09-29T21:50:23.975248+0000","last_undegraded":"2023-09-29T21:50:23.975248+0000","last_fullsized":"2023-09-29T21:50:23.975248+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_clean_scrub_stamp":"2023-09-29T21:50:22.190160+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:30.258112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322285+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:23.322285+0000","last_peered":"2023-09-29T21:50:23.322285+0000","last_clean":"2023-09-29T21:50:23.322285+0000","last_became_active":"2023-09-29T21:50:22.213922+0000","last_became_peered":"2023-09-29T21:50:22.213922+0000","last_unstale":"2023-09-29T21:50:23.322285+0000","last_undegraded":"2023-09-29T21:50:23.322285+0000","last_fullsized":"2023-09-29T21:50:23.322285+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:59.692405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-29T21:50:22.216036+0000","last_change":"2023-09-29T21:50:22.211029+0000","last_active":"2023-09-29T21:50:22.216036+0000","last_peered":"2023-09-29T21:50:22.216036+0000","last_clean":"2023-09-29T21:50:22.216036+0000","last_became_active":"2023-09-29T21:50:22.210748+0000","last_became_peered":"2023-09-29T21:50:22.210748+0000","last_unstale":"2023-09-29T21:50:22.216036+0000","last_undegraded":"2023-09-29T21:50:22.216036+0000","last_fullsized":"2023-09-29T21:50:22.216036+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:33:28.229495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203946+0000","last_change":"2023-09-29T21:50:22.216651+0000","last_active":"2023-09-29T21:50:23.203946+0000","last_peered":"2023-09-29T21:50:23.203946+0000","last_clean":"2023-09-29T21:50:23.203946+0000","last_became_active":"2023-09-29T21:50:22.216391+0000","last_became_peered":"2023-09-29T21:50:22.216391+0000","last_unstale":"2023-09-29T21:50:23.203946+0000","last_undegraded":"2023-09-29T21:50:23.203946+0000","last_fullsized":"2023-09-29T21:50:23.203946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:57:32.843104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322143+0000","last_change":"2023-09-29T21:50:22.213848+0000","last_active":"2023-09-29T21:50:23.322143+0000","last_peered":"2023-09-29T21:50:23.322143+0000","last_clean":"2023-09-29T21:50:23.322143+0000","last_became_active":"2023-09-29T21:50:22.213540+0000","last_became_peered":"2023-09-29T21:50:22.213540+0000","last_unstale":"2023-09-29T21:50:23.322143+0000","last_undegraded":"2023-09-29T21:50:23.322143+0000","last_fullsized":"2023-09-29T21:50:23.322143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:29.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27636,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829964,"statfs":{"total":107374182400,"available":107345883136,"internally_reserved":0,"allocated":700416,"data_stored":618384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27048,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830552,"statfs":{"total":107374182400,"available":107346485248,"internally_reserved":0,"allocated":98304,"data_stored":24160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:30.922 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2023-09-29T21:50:31.226 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:31.229 INFO:teuthology.orchestra.run.smithi112.stderr:dumped all 2023-09-29T21:50:31.240 INFO:teuthology.orchestra.run.smithi112.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2023-09-29T21:50:29.248179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":82344,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490456,"statfs":{"total":322122547200,"available":322038226944,"internally_reserved":0,"allocated":1523712,"data_stored":1271804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000542"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322314+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:23.322314+0000","last_peered":"2023-09-29T21:50:23.322314+0000","last_clean":"2023-09-29T21:50:23.322314+0000","last_became_active":"2023-09-29T21:50:22.213863+0000","last_became_peered":"2023-09-29T21:50:22.213863+0000","last_unstale":"2023-09-29T21:50:23.322314+0000","last_undegraded":"2023-09-29T21:50:23.322314+0000","last_fullsized":"2023-09-29T21:50:23.322314+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:45:32.741449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322383+0000","last_change":"2023-09-29T21:50:22.213796+0000","last_active":"2023-09-29T21:50:23.322383+0000","last_peered":"2023-09-29T21:50:23.322383+0000","last_clean":"2023-09-29T21:50:23.322383+0000","last_became_active":"2023-09-29T21:50:22.213555+0000","last_became_peered":"2023-09-29T21:50:22.213555+0000","last_unstale":"2023-09-29T21:50:23.322383+0000","last_undegraded":"2023-09-29T21:50:23.322383+0000","last_fullsized":"2023-09-29T21:50:23.322383+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:49:23.583986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203966+0000","last_change":"2023-09-29T21:50:22.217056+0000","last_active":"2023-09-29T21:50:23.203966+0000","last_peered":"2023-09-29T21:50:23.203966+0000","last_clean":"2023-09-29T21:50:23.203966+0000","last_became_active":"2023-09-29T21:50:22.216814+0000","last_became_peered":"2023-09-29T21:50:22.216814+0000","last_unstale":"2023-09-29T21:50:23.203966+0000","last_undegraded":"2023-09-29T21:50:23.203966+0000","last_fullsized":"2023-09-29T21:50:23.203966+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T22:37:11.988482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322422+0000","last_change":"2023-09-29T21:50:22.216868+0000","last_active":"2023-09-29T21:50:23.322422+0000","last_peered":"2023-09-29T21:50:23.322422+0000","last_clean":"2023-09-29T21:50:23.322422+0000","last_became_active":"2023-09-29T21:50:22.216575+0000","last_became_peered":"2023-09-29T21:50:22.216575+0000","last_unstale":"2023-09-29T21:50:23.322422+0000","last_undegraded":"2023-09-29T21:50:23.322422+0000","last_fullsized":"2023-09-29T21:50:23.322422+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:03:00.192258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.975248+0000","last_change":"2023-09-29T21:50:23.323780+0000","last_active":"2023-09-29T21:50:23.975248+0000","last_peered":"2023-09-29T21:50:23.975248+0000","last_clean":"2023-09-29T21:50:23.975248+0000","last_became_active":"2023-09-29T21:50:23.323527+0000","last_became_peered":"2023-09-29T21:50:23.323527+0000","last_unstale":"2023-09-29T21:50:23.975248+0000","last_undegraded":"2023-09-29T21:50:23.975248+0000","last_fullsized":"2023-09-29T21:50:23.975248+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:22.190160+0000","last_clean_scrub_stamp":"2023-09-29T21:50:22.190160+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:30.258112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322285+0000","last_change":"2023-09-29T21:50:22.214163+0000","last_active":"2023-09-29T21:50:23.322285+0000","last_peered":"2023-09-29T21:50:23.322285+0000","last_clean":"2023-09-29T21:50:23.322285+0000","last_became_active":"2023-09-29T21:50:22.213922+0000","last_became_peered":"2023-09-29T21:50:22.213922+0000","last_unstale":"2023-09-29T21:50:23.322285+0000","last_undegraded":"2023-09-29T21:50:23.322285+0000","last_fullsized":"2023-09-29T21:50:23.322285+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:50:59.692405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.327803+0000","last_change":"2023-09-29T21:50:22.211029+0000","last_active":"2023-09-29T21:50:23.327803+0000","last_peered":"2023-09-29T21:50:23.327803+0000","last_clean":"2023-09-29T21:50:23.327803+0000","last_became_active":"2023-09-29T21:50:22.210748+0000","last_became_peered":"2023-09-29T21:50:22.210748+0000","last_unstale":"2023-09-29T21:50:23.327803+0000","last_undegraded":"2023-09-29T21:50:23.327803+0000","last_fullsized":"2023-09-29T21:50:23.327803+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:33:28.229495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.203946+0000","last_change":"2023-09-29T21:50:22.216651+0000","last_active":"2023-09-29T21:50:23.203946+0000","last_peered":"2023-09-29T21:50:23.203946+0000","last_clean":"2023-09-29T21:50:23.203946+0000","last_became_active":"2023-09-29T21:50:22.216391+0000","last_became_peered":"2023-09-29T21:50:22.216391+0000","last_unstale":"2023-09-29T21:50:23.203946+0000","last_undegraded":"2023-09-29T21:50:23.203946+0000","last_fullsized":"2023-09-29T21:50:23.203946+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-09-30T23:57:32.843104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:23.322143+0000","last_change":"2023-09-29T21:50:22.213848+0000","last_active":"2023-09-29T21:50:23.322143+0000","last_peered":"2023-09-29T21:50:23.322143+0000","last_clean":"2023-09-29T21:50:23.322143+0000","last_became_active":"2023-09-29T21:50:22.213540+0000","last_became_peered":"2023-09-29T21:50:22.213540+0000","last_unstale":"2023-09-29T21:50:23.322143+0000","last_undegraded":"2023-09-29T21:50:23.322143+0000","last_fullsized":"2023-09-29T21:50:23.322143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:21.187493+0000","last_clean_scrub_stamp":"2023-09-29T21:50:21.187493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:11:29.255444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27636,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829964,"statfs":{"total":107374182400,"available":107345883136,"internally_reserved":0,"allocated":700416,"data_stored":618384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27644,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829956,"statfs":{"total":107374182400,"available":107345874944,"internally_reserved":0,"allocated":708608,"data_stored":621894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27064,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830536,"statfs":{"total":107374182400,"available":107346468864,"internally_reserved":0,"allocated":114688,"data_stored":31526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:31.241 INFO:tasks.ceph.ceph_manager.c1:clean! 2023-09-29T21:50:31.241 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2023-09-29T21:50:31.241 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2023-09-29T21:50:31.241 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2023-09-29T21:50:31.566 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:31.566 INFO:teuthology.orchestra.run.smithi112.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-29T21:50:31.580 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2023-09-29T21:50:31.581 INFO:teuthology.run_tasks:Running task ceph... 2023-09-29T21:50:31.589 INFO:tasks.ceph:Creating ceph cluster c2... 2023-09-29T21:50:31.589 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-29T21:50:31.589 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284', 'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406284', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'enabled_features': ['resharding'], 'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'a1'}, {'endpoints': ['c1.client.1'], 'name': 'a2'}]}, {'enabled_features': ['resharding'], 'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'b', 'zones': [{'endpoints': ['c2.client.0'], 'is_default': True, 'is_master': True, 'name': 'b1'}, {'endpoints': ['c2.client.1'], 'name': 'b2'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py']}, 'workunit': {'branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'seed': 4212, 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rgw', 'suite_branch': 'wip-yuri11-testing-2023-09-28-1611-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'targets': {'smithi112.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi131.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'client.0': {'rgw data notify interval msec': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '38c3646d535f6f917a872ef9baef468589ba9f32', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_12:36:11', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-29T21:50:31.590 DEBUG:teuthology.orchestra.run.smithi112:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-09-29T21:50:31.592 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2023-09-29T21:50:31.600 DEBUG:teuthology.orchestra.run.smithi112:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T21:50:31.601 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m0777 -- /var/run/ceph 2023-09-29T21:50:31.657 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:31.657 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2023-09-29T21:50:31.663 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-29T21:50:31.663 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2023-09-29T21:50:31.670 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-29T21:50:31.670 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:31.671 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 361 Links: 1 2023-09-29T21:50:31.671 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:31.671 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-09-29 21:46:16.693637338 +0000 2023-09-29T21:50:31.671 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-09-29 21:46:16.413643794 +0000 2023-09-29T21:50:31.671 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-09-29 21:46:16.413643794 +0000 2023-09-29T21:50:31.671 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-09-29T21:50:31.671 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-29T21:50:31.729 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-09-29T21:50:31.730 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-09-29T21:50:31.730 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000431355 s, 1.2 MB/s 2023-09-29T21:50:31.731 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-29T21:50:31.739 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2023-09-29T21:50:31.745 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-29T21:50:31.745 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:31.745 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-09-29T21:50:31.745 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:31.746 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-09-29 21:46:16.933631802 +0000 2023-09-29T21:50:31.746 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-09-29 21:46:16.929631895 +0000 2023-09-29T21:50:31.746 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-09-29 21:46:16.929631895 +0000 2023-09-29T21:50:31.746 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-09-29T21:50:31.746 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-29T21:50:31.759 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-09-29T21:50:31.759 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-09-29T21:50:31.759 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000405888 s, 1.3 MB/s 2023-09-29T21:50:31.760 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-29T21:50:31.814 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2023-09-29T21:50:31.821 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-29T21:50:31.821 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:31.821 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 370 Links: 1 2023-09-29T21:50:31.822 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:31.822 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-09-29 21:46:17.729613445 +0000 2023-09-29T21:50:31.822 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-09-29 21:46:17.441620088 +0000 2023-09-29T21:50:31.822 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-09-29 21:46:17.441620088 +0000 2023-09-29T21:50:31.822 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-09-29T21:50:31.822 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-29T21:50:31.834 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-09-29T21:50:31.834 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-09-29T21:50:31.834 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000397211 s, 1.3 MB/s 2023-09-29T21:50:31.835 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-29T21:50:31.843 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2023-09-29T21:50:31.896 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-29T21:50:31.896 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-29T21:50:31.896 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 6h/6d Inode: 379 Links: 1 2023-09-29T21:50:31.896 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-29T21:50:31.896 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2023-09-29 21:46:18.269600992 +0000 2023-09-29T21:50:31.897 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2023-09-29 21:46:17.993607358 +0000 2023-09-29T21:50:31.897 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2023-09-29 21:46:17.993607358 +0000 2023-09-29T21:50:31.897 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: - 2023-09-29T21:50:31.897 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-29T21:50:31.910 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2023-09-29T21:50:31.910 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2023-09-29T21:50:31.910 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000414611 s, 1.2 MB/s 2023-09-29T21:50:31.911 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-29T21:50:31.919 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:31.919 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi131.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T21:50:31.920 INFO:tasks.ceph:Generating config... 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] debug rgw = 20 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] debug rgw sync = 20 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw data log num shards = 4 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw md log max shards = 4 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw sync data inject err probability = 0.1 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2023-09-29T21:50:31.921 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2023-09-29T21:50:31.922 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2023-09-29T21:50:31.922 INFO:tasks.ceph:[client] setgroup = ceph 2023-09-29T21:50:31.922 INFO:tasks.ceph:[client] setuser = ceph 2023-09-29T21:50:31.922 INFO:tasks.ceph:[client.0] rgw data notify interval msec = 0 2023-09-29T21:50:31.922 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-29T21:50:31.922 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-29T21:50:31.922 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-29T21:50:31.922 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-29T21:50:31.922 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-29T21:50:31.922 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-29T21:50:31.922 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-29T21:50:31.923 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2023-09-29T21:50:31.923 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2023-09-29T21:50:31.923 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2023-09-29T21:50:31.923 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2023-09-29T21:50:31.923 INFO:tasks.ceph:Setting up c2.mon.a... 2023-09-29T21:50:31.923 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2023-09-29T21:50:31.961 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/c2.keyring 2023-09-29T21:50:31.977 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2023-09-29T21:50:32.063 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-29T21:50:32.077 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.131')] 2023-09-29T21:50:32.077 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.131'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'debug rgw sync': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync data inject err probability': 0.1, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'client.0': {'rgw data notify interval msec': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2023-09-29T21:50:32.078 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:32.078 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T21:50:32.083 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.131 --print /home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:32.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:50:32.163 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:50:32.163 INFO:teuthology.orchestra.run.smithi131.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:monmaptool: generated fsid 68f7a9eb-320e-4d10-bd94-d64f23fd55a8 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:setting min_mon_release = pacific 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:epoch 0 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:fsid 68f7a9eb-320e-4d10-bd94-d64f23fd55a8 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:last_changed 2023-09-29T21:50:32.163347+0000 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:created 2023-09-29T21:50:32.163347+0000 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:min_mon_release 16 (pacific) 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:election_strategy: 1 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:0: [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon.a 2023-09-29T21:50:32.164 INFO:teuthology.orchestra.run.smithi131.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2023-09-29T21:50:32.165 DEBUG:teuthology.orchestra.run.smithi131:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-29T21:50:32.170 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 68f7a9eb-320e-4d10-bd94-d64f23fd55a8... 2023-09-29T21:50:32.170 DEBUG:teuthology.orchestra.run.smithi112:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-09-29T21:50:32.173 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2023-09-29T21:50:32.203 INFO:teuthology.orchestra.run.smithi131.stdout:[global] 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: chdir = "" 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: auth supported = cephx 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: filestore xattr use omap = true 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: mon clock drift allowed = 1.000 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: osd crush chooseleaf type = 0 2023-09-29T21:50:32.204 INFO:teuthology.orchestra.run.smithi131.stdout: auth debug = true 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on old message = true 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: ms die on bug = true 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: mon pg warn max object skew = 0 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.205 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default size = 2 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow primary affinity = true 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd allow pg remap = true 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on legacy crush tunables = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on crush straw calc version zero = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on no sortbitwise = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on osd down out interval zero = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon warn on too few osds = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: mon_allow_pool_size_one = true 2023-09-29T21:50:32.206 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: osd default data pool replay window = 5 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: mon allow pool delete = true 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: mon cluster log file level = debug 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: debug asserts on shutdown = true 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: mon health detail to clog = false 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: mon host = 172.21.15.131 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: fsid = 68f7a9eb-320e-4d10-bd94-d64f23fd55a8 2023-09-29T21:50:32.207 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout:[osd] 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd journal size = 100 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub load threshold = 5.0 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd scrub max interval = 600 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd mclock profile = high_recovery_ops 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd recover clone overlap = true 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd recovery max chunk = 1048576 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug shutdown = true 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug op order = true 2023-09-29T21:50:32.208 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug verify stray on activate = true 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: osd open classes on start = true 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug pg log writeout = true 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: osd deep scrub update digest min age = 30 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: osd map max advance = 10 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: journal zero on create = true 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: filestore ondisk finisher threads = 3 2023-09-29T21:50:32.209 INFO:teuthology.orchestra.run.smithi131.stdout: filestore apply finisher threads = 3 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: bdev debug aio = true 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: osd debug misdirected ops = true 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: debug osd = 20 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: rocksdb delete range threshold = 0 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr] 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: debug mgr = 20 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2023-09-29T21:50:32.210 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mgr/telemetry/nag = false 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout:[mon] 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: debug ms = 1 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: debug mon = 20 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: debug paxos = 20 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: debug auth = 20 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mon data avail warn = 5 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mon mgr mkfs grace = 240 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:32.211 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd reporter subtree level = osd 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: mon osd prime pg temp = true 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: # 1m isn't quite enough 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: mon_down_mkfs_grace = 2m 2023-09-29T21:50:32.212 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: mon_warn_on_filestore_osds = false 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout:[client] 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: rgw cache enabled = true 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable ops log = true 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: rgw enable usage log = true 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw = 20 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: debug rgw sync = 20 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt require ssl = False 2023-09-29T21:50:32.213 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms backend = testing 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw curl low speed time = 300 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data log num shards = 4 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw md log max shards = 4 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync data inject err probability = 0.1 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync log trim interval = 0 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: rgw sync obj etag verify = True 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: setgroup = ceph 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout: setuser = ceph 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout:[mon.a] 2023-09-29T21:50:32.214 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout: rgw data notify interval msec = 0 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2023-09-29T21:50:32.215 INFO:teuthology.orchestra.run.smithi131.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout:[global] 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: chdir = "" 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: auth supported = cephx 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: filestore xattr use omap = true 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: mon clock drift allowed = 1.000 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: osd crush chooseleaf type = 0 2023-09-29T21:50:32.216 INFO:teuthology.orchestra.run.smithi112.stdout: auth debug = true 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on old message = true 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: ms die on bug = true 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: mon pg warn max object skew = 0 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: # disable pg_autoscaler by default for new pools 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.217 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default size = 2 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow primary affinity = true 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd allow pg remap = true 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on legacy crush tunables = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on crush straw calc version zero = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on no sortbitwise = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on osd down out interval zero = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon warn on too few osds = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: mon_allow_pool_size_one = true 2023-09-29T21:50:32.218 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: osd default data pool replay window = 5 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: mon allow pool delete = true 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: mon cluster log file level = debug 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: debug asserts on shutdown = true 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: mon health detail to clog = false 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: mon host = 172.21.15.131 2023-09-29T21:50:32.219 INFO:teuthology.orchestra.run.smithi112.stdout: fsid = 68f7a9eb-320e-4d10-bd94-d64f23fd55a8 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout:[osd] 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd journal size = 100 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub load threshold = 5.0 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd scrub max interval = 600 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd mclock profile = high_recovery_ops 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd recover clone overlap = true 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: osd recovery max chunk = 1048576 2023-09-29T21:50:32.220 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug shutdown = true 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug op order = true 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug verify stray on activate = true 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd open classes on start = true 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug pg log writeout = true 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd deep scrub update digest min age = 30 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: osd map max advance = 10 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.221 INFO:teuthology.orchestra.run.smithi112.stdout: journal zero on create = true 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: filestore ondisk finisher threads = 3 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: filestore apply finisher threads = 3 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: bdev debug aio = true 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: osd debug misdirected ops = true 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: debug osd = 20 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: rocksdb delete range threshold = 0 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.222 INFO:teuthology.orchestra.run.smithi112.stdout:[mgr] 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug mgr = 20 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: mgr/telemetry/nag = false 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout:[mon] 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug ms = 1 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug mon = 20 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug paxos = 20 2023-09-29T21:50:32.223 INFO:teuthology.orchestra.run.smithi112.stdout: debug auth = 20 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon data avail warn = 5 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon mgr mkfs grace = 240 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min pgs per osd = 4 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd reporter subtree level = osd 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon osd prime pg temp = true 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon reweight min bytes per osd = 10 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: auth mon ticket ttl = 660 # 11m 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: auth service ticket ttl = 240 # 4m 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: # don't complain about insecure global_id in the test suite 2023-09-29T21:50:32.224 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: # 1m isn't quite enough 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: mon_down_mkfs_grace = 2m 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: mon_warn_on_filestore_osds = false 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout:[client] 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: rgw cache enabled = true 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable ops log = true 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: rgw enable usage log = true 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-29T21:50:32.225 INFO:teuthology.orchestra.run.smithi112.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw = 20 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: debug rgw sync = 20 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt require ssl = False 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms backend = testing 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw curl low speed time = 300 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data log num shards = 4 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw md log max shards = 4 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync data inject err probability = 0.1 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync log trim interval = 0 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync meta inject err probability = 0.1 2023-09-29T21:50:32.226 INFO:teuthology.orchestra.run.smithi112.stdout: rgw sync obj etag verify = True 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: setgroup = ceph 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: setuser = ceph 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout:[mon.a] 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout:[client.0] 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: rgw data notify interval msec = 0 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.0] 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.1] 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 1000 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout:[osd.2] 2023-09-29T21:50:32.227 INFO:teuthology.orchestra.run.smithi112.stdout: osd_max_omap_entries_per_request = 10000 2023-09-29T21:50:32.228 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2023-09-29T21:50:32.229 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2023-09-29T21:50:32.323 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-29T21:50:32.323 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:32.323 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2023-09-29T21:50:32.329 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:32.330 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2023-09-29T21:50:32.334 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T21:50:32.335 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:32.335 DEBUG:teuthology.orchestra.run.smithi112:> sudo dd of=/etc/ceph/c2.keyring 2023-09-29T21:50:32.335 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-29T21:50:32.362 DEBUG:teuthology.orchestra.run.smithi112:> set -ex 2023-09-29T21:50:32.362 DEBUG:teuthology.orchestra.run.smithi112:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:32.366 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T21:50:32.366 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:32.367 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/c2.keyring 2023-09-29T21:50:32.367 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/c2.keyring 2023-09-29T21:50:32.392 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:32.393 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:32.440 INFO:tasks.ceph:Setting up mon nodes... 2023-09-29T21:50:32.440 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-29T21:50:32.440 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2023-09-29T21:50:32.486 INFO:teuthology.orchestra.run.smithi131.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2023-09-29T21:50:32.489 INFO:tasks.ceph:Setting up mds nodes... 2023-09-29T21:50:32.489 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-29T21:50:32.489 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2023-09-29T21:50:32.527 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/c2.client.0.keyring 2023-09-29T21:50:32.540 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2023-09-29T21:50:32.623 INFO:teuthology.orchestra.run.smithi131.stdout:creating /etc/ceph/c2.client.1.keyring 2023-09-29T21:50:32.637 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-29T21:50:32.638 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi112.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi131.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-29T21:50:32.638 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2023-09-29T21:50:32.649 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:32.649 INFO:tasks.ceph:role: c2.osd.0 2023-09-29T21:50:32.649 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T21:50:32.649 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-29T21:50:33.182 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T21:50:33.182 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T21:50:33.183 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T21:50:33.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2023-09-29T21:50:33.185 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2023-09-29T21:50:33.268 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2023-09-29T21:50:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:50:33.280 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T21:50:33.280 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2023-09-29T21:50:33.290 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:33.290 INFO:tasks.ceph:role: c2.osd.1 2023-09-29T21:50:33.290 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T21:50:33.290 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2023-09-29T21:50:33.843 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T21:50:33.844 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T21:50:33.844 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T21:50:33.844 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T21:50:33.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2023-09-29T21:50:33.845 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2023-09-29T21:50:33.877 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2023-09-29T21:50:33.890 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T21:50:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:50:33.891 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2023-09-29T21:50:33.947 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2023-09-29T21:50:33.947 INFO:tasks.ceph:role: c2.osd.2 2023-09-29T21:50:33.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T21:50:33.947 DEBUG:teuthology.orchestra.run.smithi131:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-29T21:50:34.408 INFO:teuthology.orchestra.run.smithi131.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-29T21:50:34.408 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-29T21:50:34.408 INFO:teuthology.orchestra.run.smithi131.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-29T21:50:34.408 INFO:teuthology.orchestra.run.smithi131.stdout: = reflink=1 2023-09-29T21:50:34.408 INFO:teuthology.orchestra.run.smithi131.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-29T21:50:34.408 INFO:teuthology.orchestra.run.smithi131.stdout: = sunit=0 swidth=0 blks 2023-09-29T21:50:34.409 INFO:teuthology.orchestra.run.smithi131.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-29T21:50:34.409 INFO:teuthology.orchestra.run.smithi131.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-29T21:50:34.409 INFO:teuthology.orchestra.run.smithi131.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-29T21:50:34.409 INFO:teuthology.orchestra.run.smithi131.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-29T21:50:34.410 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi131.front.sepia.ceph.com -o noatime 2023-09-29T21:50:34.410 DEBUG:teuthology.orchestra.run.smithi131:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2023-09-29T21:50:34.468 DEBUG:teuthology.orchestra.run.smithi131:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2023-09-29T21:50:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T21:50:34.480 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: /sbin/restorecon: command not found 2023-09-29T21:50:34.480 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:34.559 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:34.555+0000 7f6bf5524700 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2023-09-29T21:50:34.560 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:34.555+0000 7f6bf5524700 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2023-09-29T21:50:34.561 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:34.555+0000 7f6bf5524700 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-29T21:50:34.561 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:34.555+0000 7f6bf5524700 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-29T21:50:34.561 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:34.555+0000 7f6bf5524700 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2023-09-29T21:50:34.564 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:34.559+0000 7f6bf5524700 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2023-09-29T21:50:37.009 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2023-09-29T21:50:37.025 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:37.071 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:37.067+0000 7ffac4689700 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2023-09-29T21:50:37.071 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:37.067+0000 7ffac4689700 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2023-09-29T21:50:37.071 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:37.067+0000 7ffac4689700 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-29T21:50:37.071 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:37.067+0000 7ffac4689700 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-29T21:50:37.071 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:37.067+0000 7ffac4689700 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2023-09-29T21:50:37.072 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:37.067+0000 7ffac4689700 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2023-09-29T21:50:39.280 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2023-09-29T21:50:39.296 DEBUG:teuthology.orchestra.run.smithi131:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:39.383 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:39.379+0000 7f26ae25c700 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2023-09-29T21:50:39.383 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:39.379+0000 7f26ae25c700 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2023-09-29T21:50:39.383 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:39.379+0000 7f26ae25c700 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-29T21:50:39.384 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:39.379+0000 7f26ae25c700 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-29T21:50:39.384 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:39.379+0000 7f26ae25c700 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2023-09-29T21:50:39.384 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:50:39.379+0000 7f26ae25c700 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2023-09-29T21:50:41.592 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2023-09-29T21:50:41.606 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-29T21:50:41.606 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:41.607 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2023-09-29T21:50:41.621 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:41.621 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2023-09-29T21:50:41.681 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:41.681 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2023-09-29T21:50:41.691 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:41.691 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2023-09-29T21:50:41.706 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:41.706 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2023-09-29T21:50:41.755 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:41.755 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2023-09-29T21:50:41.762 INFO:tasks.ceph:Adding keys to all mons... 2023-09-29T21:50:41.762 DEBUG:teuthology.orchestra.run.smithi131:> sudo tee -a /etc/ceph/c2.keyring 2023-09-29T21:50:41.776 INFO:teuthology.orchestra.run.smithi131.stdout:[mgr.x] 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQCoRhdlYpr5HBAABfekltYgejOWagFkt/Kwig== 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.0] 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQCqRhdldmFMIRAAmtwSYrZX7i2RPgbT7UeDKg== 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.1] 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQCtRhdl5444BBAAhf0vnC9roiRJk+nKCXotdQ== 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout:[osd.2] 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQCvRhdlNDTUFhAAgUO+FOC+JykA+/Hp+vpKLw== 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout:[client.0] 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQCoRhdlw05nHxAAV8vRqsHaZ4bToL+g0WRJ/w== 2023-09-29T21:50:41.777 INFO:teuthology.orchestra.run.smithi131.stdout:[client.1] 2023-09-29T21:50:41.778 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQCoRhdlouclJRAACTz7IzYqLin8J5Cas2L2QA== 2023-09-29T21:50:41.778 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-29T21:50:41.826 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T21:50:41.877 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T21:50:41.922 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-29T21:50:41.968 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T21:50:42.024 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-29T21:50:42.074 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-29T21:50:42.074 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2023-09-29T21:50:42.089 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2023-09-29T21:50:42.154 DEBUG:teuthology.orchestra.run.smithi131:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2023-09-29T21:50:42.168 DEBUG:teuthology.orchestra.run.smithi131:> rm -- /home/ubuntu/cephtest/c2.monmap 2023-09-29T21:50:42.174 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2023-09-29T21:50:42.175 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2023-09-29T21:50:42.175 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2023-09-29T21:50:42.177 INFO:tasks.ceph.c2.mon.a:Started 2023-09-29T21:50:42.177 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2023-09-29T21:50:42.177 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2023-09-29T21:50:42.177 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2023-09-29T21:50:42.179 INFO:tasks.ceph.c2.mgr.x:Started 2023-09-29T21:50:42.179 INFO:tasks.ceph:Setting crush tunables to default 2023-09-29T21:50:42.180 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd crush tunables default 2023-09-29T21:50:42.431 INFO:teuthology.orchestra.run.smithi131.stderr:adjusted tunables profile to default 2023-09-29T21:50:42.446 INFO:tasks.ceph:check_enable_crimson: False 2023-09-29T21:50:42.446 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2023-09-29T21:50:42.446 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:42.446 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2023-09-29T21:50:42.458 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:42.458 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2023-09-29T21:50:42.469 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2023-09-29T21:50:42.469 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2023-09-29T21:50:42.478 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd new 2209cedc-2379-478a-9bcc-a46436d716f5 0 2023-09-29T21:50:42.545 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:42.538+0000 7f5da8b9e2c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-29T21:50:42.695 INFO:teuthology.orchestra.run.smithi131.stdout:0 2023-09-29T21:50:42.709 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd new d03e2605-eccc-4c9c-8b6e-e10738fdc772 1 2023-09-29T21:50:42.907 INFO:teuthology.orchestra.run.smithi131.stdout:1 2023-09-29T21:50:42.925 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd new 6c160d35-54f0-4f1a-ac04-b047f4b17d99 2 2023-09-29T21:50:43.127 INFO:teuthology.orchestra.run.smithi131.stdout:2 2023-09-29T21:50:43.140 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2023-09-29T21:50:43.140 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2023-09-29T21:50:43.142 INFO:tasks.ceph.c2.osd.0:Started 2023-09-29T21:50:43.142 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2023-09-29T21:50:43.142 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2023-09-29T21:50:43.144 INFO:tasks.ceph.c2.osd.1:Started 2023-09-29T21:50:43.144 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2023-09-29T21:50:43.145 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2023-09-29T21:50:43.147 INFO:tasks.ceph.c2.osd.2:Started 2023-09-29T21:50:43.147 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-29T21:50:43.208 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:43.202+0000 7f5da8b9e2c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-29T21:50:43.294 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:43.290+0000 7f5da8b9e2c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-29T21:50:43.376 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:43.377 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":5,"fsid":"68f7a9eb-320e-4d10-bd94-d64f23fd55a8","created":"2023-09-29T21:50:42.311295+0000","modified":"2023-09-29T21:50:43.124218+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T21:50:43.124218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2209cedc-2379-478a-9bcc-a46436d716f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d03e2605-eccc-4c9c-8b6e-e10738fdc772","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c160d35-54f0-4f1a-ac04-b047f4b17d99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:43.388 INFO:tasks.ceph.ceph_manager.c2:[] 2023-09-29T21:50:43.388 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-29T21:50:43.463 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:43.458+0000 7f5da8b9e2c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-29T21:50:43.523 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2023-09-29T21:50:43.518+0000 7f4e7ff10700 -1 Falling back to public interface 2023-09-29T21:50:43.672 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:43.666+0000 7f5da8b9e2c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-29T21:50:43.689 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-09-29T21:50:43.726 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:43.722+0000 7f5da8b9e2c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-29T21:50:43.787 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2023-09-29T21:50:43.782+0000 7fe2e8a48700 -1 Falling back to public interface 2023-09-29T21:50:43.810 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2023-09-29T21:50:43.806+0000 7f1a2222a700 -1 Falling back to public interface 2023-09-29T21:50:43.878 INFO:teuthology.misc.health.smithi131.stdout: 2023-09-29T21:50:43.878 INFO:teuthology.misc.health.smithi131.stdout:{"epoch":5,"fsid":"68f7a9eb-320e-4d10-bd94-d64f23fd55a8","created":"2023-09-29T21:50:42.311295+0000","modified":"2023-09-29T21:50:43.124218+0000","last_up_change":"0.000000","last_in_change":"2023-09-29T21:50:43.124218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2209cedc-2379-478a-9bcc-a46436d716f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d03e2605-eccc-4c9c-8b6e-e10738fdc772","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6c160d35-54f0-4f1a-ac04-b047f4b17d99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:43.889 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-29T21:50:43.958 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:43.954+0000 7f5da8b9e2c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-29T21:50:44.025 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:44.018+0000 7f5da8b9e2c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-29T21:50:44.088 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:44.082+0000 7f5da8b9e2c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-29T21:50:44.501 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:44.494+0000 7f5da8b9e2c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-29T21:50:44.618 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:44.614+0000 7f5da8b9e2c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-29T21:50:44.946 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:44.942+0000 7f5da8b9e2c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-29T21:50:45.115 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:45.110+0000 7f5da8b9e2c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-29T21:50:45.180 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:45.174+0000 7f5da8b9e2c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-29T21:50:45.235 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:45.230+0000 7f5da8b9e2c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-29T21:50:46.054 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.050+0000 7f5da8b9e2c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-29T21:50:46.111 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2023-09-29T21:50:46.106+0000 7f1a2222a700 -1 osd.1 0 log_to_monitors true 2023-09-29T21:50:46.121 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.118+0000 7f5da8b9e2c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-29T21:50:46.292 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.286+0000 7f5da8b9e2c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-29T21:50:46.356 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.350+0000 7f5da8b9e2c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-29T21:50:46.430 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.426+0000 7f5da8b9e2c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-29T21:50:46.592 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2023-09-29T21:50:46.586+0000 7fe2e8a48700 -1 osd.2 0 log_to_monitors true 2023-09-29T21:50:46.612 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.606+0000 7f5da8b9e2c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-29T21:50:46.613 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2023-09-29T21:50:46.606+0000 7f4e7ff10700 -1 osd.0 0 log_to_monitors true 2023-09-29T21:50:46.839 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:46.834+0000 7f5da8b9e2c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-29T21:50:47.550 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:47.546+0000 7f5da8b9e2c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-29T21:50:47.958 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2023-09-29T21:50:47.954+0000 7f1a1e069700 -1 osd.1 0 waiting for initial osdmap 2023-09-29T21:50:47.963 INFO:tasks.ceph.c2.osd.1.smithi131.stderr:2023-09-29T21:50:47.958+0000 7f1a15b18700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T21:50:48.605 INFO:tasks.ceph.c2.mgr.x.smithi131.stderr:2023-09-29T21:50:48.598+0000 7f5d8fc2a700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-29T21:50:49.248 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2023-09-29T21:50:49.242+0000 7fe2e4887700 -1 osd.2 0 waiting for initial osdmap 2023-09-29T21:50:49.251 INFO:tasks.ceph.c2.osd.2.smithi131.stderr:2023-09-29T21:50:49.246+0000 7fe2dc336700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T21:50:49.290 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2023-09-29T21:50:49.286+0000 7f4e7bd4f700 -1 osd.0 0 waiting for initial osdmap 2023-09-29T21:50:49.294 INFO:tasks.ceph.c2.osd.0.smithi131.stderr:2023-09-29T21:50:49.290+0000 7f4e737fe700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-29T21:50:50.193 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2023-09-29T21:50:50.476 INFO:teuthology.misc.health.smithi131.stdout: 2023-09-29T21:50:50.476 INFO:teuthology.misc.health.smithi131.stdout:{"epoch":9,"fsid":"68f7a9eb-320e-4d10-bd94-d64f23fd55a8","created":"2023-09-29T21:50:42.311295+0000","modified":"2023-09-29T21:50:49.621223+0000","last_up_change":"2023-09-29T21:50:49.621223+0000","last_in_change":"2023-09-29T21:50:43.124218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2209cedc-2379-478a-9bcc-a46436d716f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":28996},{"type":"v1","addr":"172.21.15.131:6801","nonce":28996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":28996},{"type":"v1","addr":"172.21.15.131:6803","nonce":28996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":28996},{"type":"v1","addr":"172.21.15.131:6807","nonce":28996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":28996},{"type":"v1","addr":"172.21.15.131:6805","nonce":28996}]},"public_addr":"172.21.15.131:6801/28996","cluster_addr":"172.21.15.131:6803/28996","heartbeat_back_addr":"172.21.15.131:6807/28996","heartbeat_front_addr":"172.21.15.131:6805/28996","state":["exists","up"]},{"osd":1,"uuid":"d03e2605-eccc-4c9c-8b6e-e10738fdc772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":28997},{"type":"v1","addr":"172.21.15.131:6817","nonce":28997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":28997},{"type":"v1","addr":"172.21.15.131:6819","nonce":28997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":28997},{"type":"v1","addr":"172.21.15.131:6823","nonce":28997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":28997},{"type":"v1","addr":"172.21.15.131:6821","nonce":28997}]},"public_addr":"172.21.15.131:6817/28997","cluster_addr":"172.21.15.131:6819/28997","heartbeat_back_addr":"172.21.15.131:6823/28997","heartbeat_front_addr":"172.21.15.131:6821/28997","state":["exists","up"]},{"osd":2,"uuid":"6c160d35-54f0-4f1a-ac04-b047f4b17d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":28995},{"type":"v1","addr":"172.21.15.131:6809","nonce":28995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":28995},{"type":"v1","addr":"172.21.15.131:6811","nonce":28995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":28995},{"type":"v1","addr":"172.21.15.131:6815","nonce":28995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":28995},{"type":"v1","addr":"172.21.15.131:6813","nonce":28995}]},"public_addr":"172.21.15.131:6809/28995","cluster_addr":"172.21.15.131:6811/28995","heartbeat_back_addr":"172.21.15.131:6815/28995","heartbeat_front_addr":"172.21.15.131:6813/28995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.078967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:50.489 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-29T21:50:50.489 INFO:tasks.ceph:Creating RBD pool 2023-09-29T21:50:50.489 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd pool create rbd 8 2023-09-29T21:50:51.664 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'rbd' created 2023-09-29T21:50:51.677 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-29T21:50:52.668 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-29T21:50:52.682 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2023-09-29T21:50:52.683 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2023-09-29T21:50:52.683 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2023-09-29T21:50:52.683 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2023-09-29T21:50:52.683 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-29T21:50:53.024 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:53.038 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6824","nonce":28812},{"type":"v1","addr":"172.21.15.131:6825","nonce":28812}]},"active_addr":"172.21.15.131:6825/28812","active_change":"2023-09-29T21:50:47.571207+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph:v18","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":3953288845}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":2618944219}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.131:0","nonce":3548002477}]}]} 2023-09-29T21:50:53.045 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2023-09-29T21:50:53.045 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2023-09-29T21:50:53.045 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-29T21:50:53.357 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:53.357 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":12,"fsid":"68f7a9eb-320e-4d10-bd94-d64f23fd55a8","created":"2023-09-29T21:50:42.311295+0000","modified":"2023-09-29T21:50:52.652631+0000","last_up_change":"2023-09-29T21:50:49.621223+0000","last_in_change":"2023-09-29T21:50:43.124218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-29T21:50:50.818221+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-29T21:50:51.648370+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2209cedc-2379-478a-9bcc-a46436d716f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":28996},{"type":"v1","addr":"172.21.15.131:6801","nonce":28996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":28996},{"type":"v1","addr":"172.21.15.131:6803","nonce":28996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":28996},{"type":"v1","addr":"172.21.15.131:6807","nonce":28996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":28996},{"type":"v1","addr":"172.21.15.131:6805","nonce":28996}]},"public_addr":"172.21.15.131:6801/28996","cluster_addr":"172.21.15.131:6803/28996","heartbeat_back_addr":"172.21.15.131:6807/28996","heartbeat_front_addr":"172.21.15.131:6805/28996","state":["exists","up"]},{"osd":1,"uuid":"d03e2605-eccc-4c9c-8b6e-e10738fdc772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":28997},{"type":"v1","addr":"172.21.15.131:6817","nonce":28997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":28997},{"type":"v1","addr":"172.21.15.131:6819","nonce":28997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":28997},{"type":"v1","addr":"172.21.15.131:6823","nonce":28997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":28997},{"type":"v1","addr":"172.21.15.131:6821","nonce":28997}]},"public_addr":"172.21.15.131:6817/28997","cluster_addr":"172.21.15.131:6819/28997","heartbeat_back_addr":"172.21.15.131:6823/28997","heartbeat_front_addr":"172.21.15.131:6821/28997","state":["exists","up"]},{"osd":2,"uuid":"6c160d35-54f0-4f1a-ac04-b047f4b17d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":28995},{"type":"v1","addr":"172.21.15.131:6809","nonce":28995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":28995},{"type":"v1","addr":"172.21.15.131:6811","nonce":28995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":28995},{"type":"v1","addr":"172.21.15.131:6815","nonce":28995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":28995},{"type":"v1","addr":"172.21.15.131:6813","nonce":28995}]},"public_addr":"172.21.15.131:6809/28995","cluster_addr":"172.21.15.131:6811/28995","heartbeat_back_addr":"172.21.15.131:6815/28995","heartbeat_front_addr":"172.21.15.131:6813/28995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.625017+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.078967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.593405+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:53.370 INFO:tasks.ceph.ceph_manager.c2:all up! 2023-09-29T21:50:53.370 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2023-09-29T21:50:53.663 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:53.663 INFO:teuthology.orchestra.run.smithi131.stdout:{"epoch":13,"fsid":"68f7a9eb-320e-4d10-bd94-d64f23fd55a8","created":"2023-09-29T21:50:42.311295+0000","modified":"2023-09-29T21:50:53.654328+0000","last_up_change":"2023-09-29T21:50:49.621223+0000","last_in_change":"2023-09-29T21:50:43.124218+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-09-29T21:50:50.818221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-09-29T21:50:51.648370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2209cedc-2379-478a-9bcc-a46436d716f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":28996},{"type":"v1","addr":"172.21.15.131:6801","nonce":28996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":28996},{"type":"v1","addr":"172.21.15.131:6803","nonce":28996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":28996},{"type":"v1","addr":"172.21.15.131:6807","nonce":28996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":28996},{"type":"v1","addr":"172.21.15.131:6805","nonce":28996}]},"public_addr":"172.21.15.131:6801/28996","cluster_addr":"172.21.15.131:6803/28996","heartbeat_back_addr":"172.21.15.131:6807/28996","heartbeat_front_addr":"172.21.15.131:6805/28996","state":["exists","up"]},{"osd":1,"uuid":"d03e2605-eccc-4c9c-8b6e-e10738fdc772","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6816","nonce":28997},{"type":"v1","addr":"172.21.15.131:6817","nonce":28997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6818","nonce":28997},{"type":"v1","addr":"172.21.15.131:6819","nonce":28997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6822","nonce":28997},{"type":"v1","addr":"172.21.15.131:6823","nonce":28997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6820","nonce":28997},{"type":"v1","addr":"172.21.15.131:6821","nonce":28997}]},"public_addr":"172.21.15.131:6817/28997","cluster_addr":"172.21.15.131:6819/28997","heartbeat_back_addr":"172.21.15.131:6823/28997","heartbeat_front_addr":"172.21.15.131:6821/28997","state":["exists","up"]},{"osd":2,"uuid":"6c160d35-54f0-4f1a-ac04-b047f4b17d99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6808","nonce":28995},{"type":"v1","addr":"172.21.15.131:6809","nonce":28995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6810","nonce":28995},{"type":"v1","addr":"172.21.15.131:6811","nonce":28995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6814","nonce":28995},{"type":"v1","addr":"172.21.15.131:6815","nonce":28995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6812","nonce":28995},{"type":"v1","addr":"172.21.15.131:6813","nonce":28995}]},"public_addr":"172.21.15.131:6809/28995","cluster_addr":"172.21.15.131:6811/28995","heartbeat_back_addr":"172.21.15.131:6815/28995","heartbeat_front_addr":"172.21.15.131:6813/28995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.625017+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.078967+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-09-29T21:50:47.593405+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-29T21:50:53.677 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2023-09-29T21:50:53.678 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2023-09-29T21:50:53.678 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2023-09-29T21:50:53.819 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2023-09-29T21:50:53.819 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-29T21:50:53.830 INFO:teuthology.orchestra.run.smithi131.stdout:34359738371 2023-09-29T21:50:53.830 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-29T21:50:53.836 INFO:teuthology.orchestra.run.smithi131.stdout:38654705668 2023-09-29T21:50:53.836 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-29T21:50:54.132 INFO:teuthology.orchestra.run.smithi131.stdout:38654705666 2023-09-29T21:50:54.146 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.2 2023-09-29T21:50:54.163 INFO:teuthology.orchestra.run.smithi131.stdout:34359738369 2023-09-29T21:50:54.172 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2023-09-29T21:50:54.176 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.1 2023-09-29T21:50:54.186 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2023-09-29T21:50:55.146 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-29T21:50:55.177 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-29T21:50:55.187 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-29T21:50:55.475 INFO:teuthology.orchestra.run.smithi131.stdout:38654705666 2023-09-29T21:50:55.488 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705666 for osd.2 2023-09-29T21:50:55.502 INFO:teuthology.orchestra.run.smithi131.stdout:34359738369 2023-09-29T21:50:55.502 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2023-09-29T21:50:55.517 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2023-09-29T21:50:55.517 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738369 for osd.1 2023-09-29T21:50:56.490 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2023-09-29T21:50:56.517 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2023-09-29T21:50:56.518 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2023-09-29T21:50:56.798 INFO:teuthology.orchestra.run.smithi131.stdout:38654705667 2023-09-29T21:50:56.811 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705667 got 38654705667 for osd.2 2023-09-29T21:50:56.812 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:56.834 INFO:teuthology.orchestra.run.smithi131.stdout:34359738371 2023-09-29T21:50:56.848 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2023-09-29T21:50:56.848 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:56.852 INFO:teuthology.orchestra.run.smithi131.stdout:38654705668 2023-09-29T21:50:56.866 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.0 2023-09-29T21:50:56.867 DEBUG:teuthology.parallel:result is None 2023-09-29T21:50:56.867 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2023-09-29T21:50:56.867 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-29T21:50:57.156 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:57.159 INFO:teuthology.orchestra.run.smithi131.stderr:dumped all 2023-09-29T21:50:57.170 INFO:teuthology.orchestra.run.smithi131.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T21:50:55.587692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83088,"kb_used_data":1472,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314489712,"statfs":{"total":322122547200,"available":322037465088,"internally_reserved":0,"allocated":1507328,"data_stored":1262687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604988+0000","last_change":"2023-09-29T21:50:52.671614+0000","last_active":"2023-09-29T21:50:53.604988+0000","last_peered":"2023-09-29T21:50:53.604988+0000","last_clean":"2023-09-29T21:50:53.604988+0000","last_became_active":"2023-09-29T21:50:52.670960+0000","last_became_peered":"2023-09-29T21:50:52.670960+0000","last_unstale":"2023-09-29T21:50:53.604988+0000","last_undegraded":"2023-09-29T21:50:53.604988+0000","last_fullsized":"2023-09-29T21:50:53.604988+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:31:42.382615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604936+0000","last_change":"2023-09-29T21:50:52.671730+0000","last_active":"2023-09-29T21:50:53.604936+0000","last_peered":"2023-09-29T21:50:53.604936+0000","last_clean":"2023-09-29T21:50:53.604936+0000","last_became_active":"2023-09-29T21:50:52.671412+0000","last_became_peered":"2023-09-29T21:50:52.671412+0000","last_unstale":"2023-09-29T21:50:53.604936+0000","last_undegraded":"2023-09-29T21:50:53.604936+0000","last_fullsized":"2023-09-29T21:50:53.604936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:37:43.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:52.673789+0000","last_change":"2023-09-29T21:50:52.671733+0000","last_active":"2023-09-29T21:50:52.673789+0000","last_peered":"2023-09-29T21:50:52.673789+0000","last_clean":"2023-09-29T21:50:52.673789+0000","last_became_active":"2023-09-29T21:50:52.671403+0000","last_became_peered":"2023-09-29T21:50:52.671403+0000","last_unstale":"2023-09-29T21:50:52.673789+0000","last_undegraded":"2023-09-29T21:50:52.673789+0000","last_fullsized":"2023-09-29T21:50:52.673789+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:28.539956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604889+0000","last_change":"2023-09-29T21:50:52.671258+0000","last_active":"2023-09-29T21:50:53.604889+0000","last_peered":"2023-09-29T21:50:53.604889+0000","last_clean":"2023-09-29T21:50:53.604889+0000","last_became_active":"2023-09-29T21:50:52.670943+0000","last_became_peered":"2023-09-29T21:50:52.670943+0000","last_unstale":"2023-09-29T21:50:53.604889+0000","last_undegraded":"2023-09-29T21:50:53.604889+0000","last_fullsized":"2023-09-29T21:50:53.604889+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:20.031085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.346979+0000","last_change":"2023-09-29T21:50:52.669561+0000","last_active":"2023-09-29T21:50:53.346979+0000","last_peered":"2023-09-29T21:50:53.346979+0000","last_clean":"2023-09-29T21:50:53.346979+0000","last_became_active":"2023-09-29T21:50:52.668620+0000","last_became_peered":"2023-09-29T21:50:52.668620+0000","last_unstale":"2023-09-29T21:50:53.346979+0000","last_undegraded":"2023-09-29T21:50:53.346979+0000","last_fullsized":"2023-09-29T21:50:53.346979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:45.566671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604835+0000","last_change":"2023-09-29T21:50:52.668470+0000","last_active":"2023-09-29T21:50:53.604835+0000","last_peered":"2023-09-29T21:50:53.604835+0000","last_clean":"2023-09-29T21:50:53.604835+0000","last_became_active":"2023-09-29T21:50:52.668158+0000","last_became_peered":"2023-09-29T21:50:52.668158+0000","last_unstale":"2023-09-29T21:50:53.604835+0000","last_undegraded":"2023-09-29T21:50:53.604835+0000","last_fullsized":"2023-09-29T21:50:53.604835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:23:48.040610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:52.672868+0000","last_change":"2023-09-29T21:50:52.670999+0000","last_active":"2023-09-29T21:50:52.672868+0000","last_peered":"2023-09-29T21:50:52.672868+0000","last_clean":"2023-09-29T21:50:52.672868+0000","last_became_active":"2023-09-29T21:50:52.670725+0000","last_became_peered":"2023-09-29T21:50:52.670725+0000","last_unstale":"2023-09-29T21:50:52.672868+0000","last_undegraded":"2023-09-29T21:50:52.672868+0000","last_fullsized":"2023-09-29T21:50:52.672868+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:42:26.277257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:52.673770+0000","last_change":"2023-09-29T21:50:52.671219+0000","last_active":"2023-09-29T21:50:52.673770+0000","last_peered":"2023-09-29T21:50:52.673770+0000","last_clean":"2023-09-29T21:50:52.673770+0000","last_became_active":"2023-09-29T21:50:52.670960+0000","last_became_peered":"2023-09-29T21:50:52.670960+0000","last_unstale":"2023-09-29T21:50:52.673770+0000","last_undegraded":"2023-09-29T21:50:52.673770+0000","last_fullsized":"2023-09-29T21:50:52.673770+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:51:49.312244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604765+0000","last_change":"2023-09-29T21:50:52.671137+0000","last_active":"2023-09-29T21:50:53.604765+0000","last_peered":"2023-09-29T21:50:53.604765+0000","last_clean":"2023-09-29T21:50:53.604765+0000","last_became_active":"2023-09-29T21:50:52.670839+0000","last_became_peered":"2023-09-29T21:50:52.670839+0000","last_unstale":"2023-09-29T21:50:53.604765+0000","last_undegraded":"2023-09-29T21:50:53.604765+0000","last_fullsized":"2023-09-29T21:50:53.604765+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:20.314402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28140,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829460,"statfs":{"total":107374182400,"available":107345367040,"internally_reserved":0,"allocated":700416,"data_stored":617685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":27317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27892,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829708,"statfs":{"total":107374182400,"available":107345620992,"internally_reserved":0,"allocated":700416,"data_stored":617685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:57.171 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2023-09-29T21:50:57.459 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:57.462 INFO:teuthology.orchestra.run.smithi131.stderr:dumped all 2023-09-29T21:50:57.473 INFO:teuthology.orchestra.run.smithi131.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-09-29T21:50:55.587692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":83088,"kb_used_data":1472,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314489712,"statfs":{"total":322122547200,"available":322037465088,"internally_reserved":0,"allocated":1507328,"data_stored":1262687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604988+0000","last_change":"2023-09-29T21:50:52.671614+0000","last_active":"2023-09-29T21:50:53.604988+0000","last_peered":"2023-09-29T21:50:53.604988+0000","last_clean":"2023-09-29T21:50:53.604988+0000","last_became_active":"2023-09-29T21:50:52.670960+0000","last_became_peered":"2023-09-29T21:50:52.670960+0000","last_unstale":"2023-09-29T21:50:53.604988+0000","last_undegraded":"2023-09-29T21:50:53.604988+0000","last_fullsized":"2023-09-29T21:50:53.604988+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:31:42.382615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604936+0000","last_change":"2023-09-29T21:50:52.671730+0000","last_active":"2023-09-29T21:50:53.604936+0000","last_peered":"2023-09-29T21:50:53.604936+0000","last_clean":"2023-09-29T21:50:53.604936+0000","last_became_active":"2023-09-29T21:50:52.671412+0000","last_became_peered":"2023-09-29T21:50:52.671412+0000","last_unstale":"2023-09-29T21:50:53.604936+0000","last_undegraded":"2023-09-29T21:50:53.604936+0000","last_fullsized":"2023-09-29T21:50:53.604936+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:37:43.546100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:52.673789+0000","last_change":"2023-09-29T21:50:52.671733+0000","last_active":"2023-09-29T21:50:52.673789+0000","last_peered":"2023-09-29T21:50:52.673789+0000","last_clean":"2023-09-29T21:50:52.673789+0000","last_became_active":"2023-09-29T21:50:52.671403+0000","last_became_peered":"2023-09-29T21:50:52.671403+0000","last_unstale":"2023-09-29T21:50:52.673789+0000","last_undegraded":"2023-09-29T21:50:52.673789+0000","last_fullsized":"2023-09-29T21:50:52.673789+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:17:28.539956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604889+0000","last_change":"2023-09-29T21:50:52.671258+0000","last_active":"2023-09-29T21:50:53.604889+0000","last_peered":"2023-09-29T21:50:53.604889+0000","last_clean":"2023-09-29T21:50:53.604889+0000","last_became_active":"2023-09-29T21:50:52.670943+0000","last_became_peered":"2023-09-29T21:50:52.670943+0000","last_unstale":"2023-09-29T21:50:53.604889+0000","last_undegraded":"2023-09-29T21:50:53.604889+0000","last_fullsized":"2023-09-29T21:50:53.604889+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:38:20.031085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":136,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.346979+0000","last_change":"2023-09-29T21:50:52.669561+0000","last_active":"2023-09-29T21:50:53.346979+0000","last_peered":"2023-09-29T21:50:53.346979+0000","last_clean":"2023-09-29T21:50:53.346979+0000","last_became_active":"2023-09-29T21:50:52.668620+0000","last_became_peered":"2023-09-29T21:50:52.668620+0000","last_unstale":"2023-09-29T21:50:53.346979+0000","last_undegraded":"2023-09-29T21:50:53.346979+0000","last_fullsized":"2023-09-29T21:50:53.346979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T00:18:45.566671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604835+0000","last_change":"2023-09-29T21:50:52.668470+0000","last_active":"2023-09-29T21:50:53.604835+0000","last_peered":"2023-09-29T21:50:53.604835+0000","last_clean":"2023-09-29T21:50:53.604835+0000","last_became_active":"2023-09-29T21:50:52.668158+0000","last_became_peered":"2023-09-29T21:50:52.668158+0000","last_unstale":"2023-09-29T21:50:53.604835+0000","last_undegraded":"2023-09-29T21:50:53.604835+0000","last_fullsized":"2023-09-29T21:50:53.604835+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:23:48.040610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:52.672868+0000","last_change":"2023-09-29T21:50:52.670999+0000","last_active":"2023-09-29T21:50:52.672868+0000","last_peered":"2023-09-29T21:50:52.672868+0000","last_clean":"2023-09-29T21:50:52.672868+0000","last_became_active":"2023-09-29T21:50:52.670725+0000","last_became_peered":"2023-09-29T21:50:52.670725+0000","last_unstale":"2023-09-29T21:50:52.672868+0000","last_undegraded":"2023-09-29T21:50:52.672868+0000","last_fullsized":"2023-09-29T21:50:52.672868+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:42:26.277257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:52.673770+0000","last_change":"2023-09-29T21:50:52.671219+0000","last_active":"2023-09-29T21:50:52.673770+0000","last_peered":"2023-09-29T21:50:52.673770+0000","last_clean":"2023-09-29T21:50:52.673770+0000","last_became_active":"2023-09-29T21:50:52.670960+0000","last_became_peered":"2023-09-29T21:50:52.670960+0000","last_unstale":"2023-09-29T21:50:52.673770+0000","last_undegraded":"2023-09-29T21:50:52.673770+0000","last_fullsized":"2023-09-29T21:50:52.673770+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:51:49.312244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-29T21:50:53.604765+0000","last_change":"2023-09-29T21:50:52.671137+0000","last_active":"2023-09-29T21:50:53.604765+0000","last_peered":"2023-09-29T21:50:53.604765+0000","last_clean":"2023-09-29T21:50:53.604765+0000","last_became_active":"2023-09-29T21:50:52.670839+0000","last_became_peered":"2023-09-29T21:50:52.670839+0000","last_unstale":"2023-09-29T21:50:53.604765+0000","last_undegraded":"2023-09-29T21:50:53.604765+0000","last_fullsized":"2023-09-29T21:50:53.604765+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-29T21:50:51.648555+0000","last_clean_scrub_stamp":"2023-09-29T21:50:51.648555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:04:20.314402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":28140,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829460,"statfs":{"total":107374182400,"available":107345367040,"internally_reserved":0,"allocated":700416,"data_stored":617685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27056,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830544,"statfs":{"total":107374182400,"available":107346477056,"internally_reserved":0,"allocated":106496,"data_stored":27317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":27892,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829708,"statfs":{"total":107374182400,"available":107345620992,"internally_reserved":0,"allocated":700416,"data_stored":617685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-29T21:50:57.474 INFO:tasks.ceph.ceph_manager.c2:clean! 2023-09-29T21:50:57.474 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2023-09-29T21:50:57.475 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2023-09-29T21:50:57.475 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2023-09-29T21:50:57.796 INFO:teuthology.orchestra.run.smithi131.stdout: 2023-09-29T21:50:57.796 INFO:teuthology.orchestra.run.smithi131.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-29T21:50:57.810 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2023-09-29T21:50:57.810 INFO:teuthology.run_tasks:Running task rgw... 2023-09-29T21:50:57.880 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2023-09-29T21:50:57.880 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2023-09-29T21:50:57.881 INFO:tasks.rgw:Creating data pools 2023-09-29T21:50:57.881 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2023-09-29T21:50:57.881 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-09-29T21:50:58.981 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' created 2023-09-29T21:50:58.991 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-09-29T21:50:59.999 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T21:51:00.016 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-09-29T21:51:01.006 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' created 2023-09-29T21:51:01.019 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-09-29T21:51:02.036 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T21:51:02.051 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2023-09-29T21:51:02.051 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2023-09-29T21:51:02.344 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.data' already exists 2023-09-29T21:51:02.355 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2023-09-29T21:51:03.013 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T21:51:03.025 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2023-09-29T21:51:03.322 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'default.rgw.buckets.index' already exists 2023-09-29T21:51:03.332 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2023-09-29T21:51:04.035 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T21:51:04.048 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2023-09-29T21:51:04.048 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-09-29T21:51:05.330 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.data' created 2023-09-29T21:51:05.343 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-09-29T21:51:06.358 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T21:51:06.369 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-09-29T21:51:07.341 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.index' created 2023-09-29T21:51:07.351 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-09-29T21:51:08.367 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T21:51:08.383 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2023-09-29T21:51:08.383 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2023-09-29T21:51:08.665 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.data' already exists 2023-09-29T21:51:08.676 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2023-09-29T21:51:09.352 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-09-29T21:51:09.365 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2023-09-29T21:51:09.680 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'default.rgw.buckets.index' already exists 2023-09-29T21:51:09.690 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2023-09-29T21:51:10.365 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-09-29T21:51:10.378 DEBUG:tasks.rgw:Pools created 2023-09-29T21:51:10.378 INFO:tasks.rgw:Configuring compression type = random 2023-09-29T21:51:10.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2023-09-29T21:51:10.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2023-09-29T21:51:10.379 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2023-09-29T21:51:10.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:10.420 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:10.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:10.437+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 realm 2023-09-29T21:51:11.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T21:51:11.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.961+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 10 Creating default zonegroup 2023-09-29T21:51:11.975 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.973+0000 7f78e4dc9f40 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-29T21:51:11.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.993+0000 7f78e4dc9f40 10 WARNING: driver->list_zones() returned r=-2 2023-09-29T21:51:11.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.993+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.993+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.997+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.997+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:11.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.997+0000 7f78e4dc9f40 10 could not read realm id: (2) No such file or directory 2023-09-29T21:51:11.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.997+0000 7f78e4dc9f40 10 WARNING: failed to set zone as default, r=-22 2023-09-29T21:51:11.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.997+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:11.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:11.997+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:12.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:12.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T21:51:12.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:12.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T21:51:12.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 10 Using default name default 2023-09-29T21:51:12.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:12.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T21:51:12.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:12.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T21:51:12.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 zone default found 2023-09-29T21:51:12.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 4 Realm: () 2023-09-29T21:51:12.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 4 ZoneGroup: default (4b91bdce-571f-44c8-abe6-d169b13ae71e) 2023-09-29T21:51:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 4 Zone: default (af27eef6-40e2-4f0f-9ac6-e20c2b3336c5) 2023-09-29T21:51:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T21:51:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 zonegroup default 2023-09-29T21:51:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:12.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:12.001+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:13.992+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:13.992+0000 7f78e4dc9f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:13.992+0000 7f78e4dc9f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:13.992+0000 7f78e4dc9f40 20 started sync module instance, tier type = 2023-09-29T21:51:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:13.992+0000 7f78e4dc9f40 20 started zone id=af27eef6-40e2-4f0f-9ac6-e20c2b3336c5 (name=default) with tier type = 2023-09-29T21:51:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=0 2023-09-29T21:51:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=1 2023-09-29T21:51:16.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=2 2023-09-29T21:51:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=3 2023-09-29T21:51:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=4 2023-09-29T21:51:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=5 2023-09-29T21:51:16.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=6 2023-09-29T21:51:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 add_watcher() i=7 2023-09-29T21:51:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 2 all 8 watchers are set, enabling cache 2023-09-29T21:51:16.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-09-29T21:51:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-09-29T21:51:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.048+0000 7f78e4dc9f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-09-29T21:51:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-09-29T21:51:16.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-09-29T21:51:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-09-29T21:51:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-09-29T21:51:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-09-29T21:51:16.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-09-29T21:51:16.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-09-29T21:51:16.055 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.052+0000 7f78e4dc9f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.060+0000 7f78ca550700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.060+0000 7f78e4dc9f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:16.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:16.060+0000 7f78e4dc9f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.008+0000 7f78e4dc9f40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-29T21:51:19.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.008+0000 7f78e4dc9f40 5 note: GC not initialized 2023-09-29T21:51:19.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.024+0000 7f78c5d47700 20 reqs_thread_entry: start 2023-09-29T21:51:19.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.024+0000 7f78e4dc9f40 20 init_complete bucket index max shards: 11 2023-09-29T21:51:19.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.024+0000 7f78e4dc9f40 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T21:51:19.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.024+0000 7f78c3d43700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 21:51:49 2023 2023-09-29T21:51:19.025 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-09-29T21:51:19.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.024+0000 7f78c3521700 20 reqs_thread_entry: start 2023-09-29T21:51:19.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.056+0000 7f78ca550700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:19.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.060+0000 7f78e4dc9f40 20 remove_watcher() i=0 2023-09-29T21:51:19.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.060+0000 7f78e4dc9f40 2 removed watcher, disabling cache 2023-09-29T21:51:19.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.060+0000 7f78e4dc9f40 20 remove_watcher() i=1 2023-09-29T21:51:19.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.064+0000 7f78e4dc9f40 20 remove_watcher() i=2 2023-09-29T21:51:19.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.064+0000 7f78e4dc9f40 20 remove_watcher() i=3 2023-09-29T21:51:19.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.068+0000 7f78e4dc9f40 20 remove_watcher() i=4 2023-09-29T21:51:19.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.072+0000 7f78e4dc9f40 20 remove_watcher() i=5 2023-09-29T21:51:19.075 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.072+0000 7f78e4dc9f40 20 remove_watcher() i=6 2023-09-29T21:51:19.077 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.076+0000 7f78e4dc9f40 20 remove_watcher() i=7 2023-09-29T21:51:19.082 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T21:51:19.085 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T21:51:19.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:19.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:19.086 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T21:51:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:19.176 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.192+0000 7fe63d698f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:19.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.192+0000 7fe63d698f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:19.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.196+0000 7fe61f618700 20 reqs_thread_entry: start 2023-09-29T21:51:19.211 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:19.211 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "af27eef6-40e2-4f0f-9ac6-e20c2b3336c5", 2023-09-29T21:51:19.212 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default", 2023-09-29T21:51:19.212 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T21:51:19.212 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "default.rgw.control", 2023-09-29T21:51:19.213 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T21:51:19.213 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T21:51:19.213 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "default.rgw.log", 2023-09-29T21:51:19.213 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T21:51:19.214 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T21:51:19.214 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T21:51:19.214 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T21:51:19.215 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T21:51:19.215 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T21:51:19.215 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T21:51:19.216 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T21:51:19.216 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T21:51:19.216 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T21:51:19.216 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-09-29T21:51:19.217 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-09-29T21:51:19.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:19.217 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T21:51:19.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:19.218 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T21:51:19.218 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:51:19.219 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T21:51:19.219 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T21:51:19.219 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T21:51:19.219 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T21:51:19.220 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T21:51:19.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:19.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:19.221 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T21:51:19.221 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T21:51:19.221 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T21:51:19.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:19.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:19.222 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:19.222 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "", 2023-09-29T21:51:19.223 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T21:51:19.223 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:19.225 DEBUG:tasks.util.rgw: json result: {'id': 'af27eef6-40e2-4f0f-9ac6-e20c2b3336c5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T21:51:19.225 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2023-09-29T21:51:19.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2023-09-29T21:51:19.226 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2023-09-29T21:51:19.269 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:19.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:19.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.284+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.284+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.284+0000 7fcec0df1f40 20 realm 2023-09-29T21:51:19.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.284+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T21:51:19.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T21:51:19.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 rados_obj.operate() r=0 bl.length=890 2023-09-29T21:51:19.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.288+0000 7fcec0df1f40 20 searching for the correct realm 2023-09-29T21:51:19.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 RGWRados::pool_iterate: got zonegroup_info.4b91bdce-571f-44c8-abe6-d169b13ae71e 2023-09-29T21:51:19.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 RGWRados::pool_iterate: got zone_info.af27eef6-40e2-4f0f-9ac6-e20c2b3336c5 2023-09-29T21:51:19.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 RGWRados::pool_iterate: got zone_names.default 2023-09-29T21:51:19.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-29T21:51:19.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T21:51:19.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T21:51:19.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 zone default found 2023-09-29T21:51:19.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 4 Realm: () 2023-09-29T21:51:19.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 4 ZoneGroup: default (4b91bdce-571f-44c8-abe6-d169b13ae71e) 2023-09-29T21:51:19.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 4 Zone: default (af27eef6-40e2-4f0f-9ac6-e20c2b3336c5) 2023-09-29T21:51:19.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T21:51:19.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 zonegroup default 2023-09-29T21:51:19.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:19.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:19.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 started sync module instance, tier type = 2023-09-29T21:51:19.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.312+0000 7fcec0df1f40 20 started zone id=af27eef6-40e2-4f0f-9ac6-e20c2b3336c5 (name=default) with tier type = 2023-09-29T21:51:19.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=0 2023-09-29T21:51:19.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=1 2023-09-29T21:51:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=2 2023-09-29T21:51:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=3 2023-09-29T21:51:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=4 2023-09-29T21:51:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=5 2023-09-29T21:51:19.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.336+0000 7fcec0df1f40 20 add_watcher() i=6 2023-09-29T21:51:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.340+0000 7fcec0df1f40 20 add_watcher() i=7 2023-09-29T21:51:19.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.340+0000 7fcec0df1f40 2 all 8 watchers are set, enabling cache 2023-09-29T21:51:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.340+0000 7fcea6578700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.340+0000 7fcec0df1f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:19.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.340+0000 7fcec0df1f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:19.349 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.348+0000 7fcec0df1f40 5 note: GC not initialized 2023-09-29T21:51:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.348+0000 7fcea1d6f700 20 reqs_thread_entry: start 2023-09-29T21:51:19.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.348+0000 7fcec0df1f40 20 init_complete bucket index max shards: 11 2023-09-29T21:51:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.348+0000 7fcec0df1f40 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T21:51:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.348+0000 7fce9fd6b700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 21:51:49 2023 2023-09-29T21:51:19.352 INFO:teuthology.orchestra.run.smithi112.stderr: 2023-09-29T21:51:19.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.352+0000 7fce9f549700 20 reqs_thread_entry: start 2023-09-29T21:51:19.375 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.372+0000 7fcea6578700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.376+0000 7fcec0df1f40 20 remove_watcher() i=0 2023-09-29T21:51:19.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.376+0000 7fcec0df1f40 2 removed watcher, disabling cache 2023-09-29T21:51:19.382 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.380+0000 7fcec0df1f40 20 remove_watcher() i=1 2023-09-29T21:51:19.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.380+0000 7fcec0df1f40 20 remove_watcher() i=2 2023-09-29T21:51:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.384+0000 7fcec0df1f40 20 remove_watcher() i=3 2023-09-29T21:51:19.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.388+0000 7fcec0df1f40 20 remove_watcher() i=4 2023-09-29T21:51:19.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.388+0000 7fcec0df1f40 20 remove_watcher() i=5 2023-09-29T21:51:19.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.392+0000 7fcec0df1f40 20 remove_watcher() i=6 2023-09-29T21:51:19.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.392+0000 7fcec0df1f40 20 remove_watcher() i=7 2023-09-29T21:51:19.402 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T21:51:19.403 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T21:51:19.403 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:19.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:19.404 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T21:51:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:19.497 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.512+0000 7f6348627f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:19.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.512+0000 7f6348627f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:19.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:19.520+0000 7f632a5a7700 20 reqs_thread_entry: start 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "af27eef6-40e2-4f0f-9ac6-e20c2b3336c5", 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default", 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "default.rgw.control", 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T21:51:19.533 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "default.rgw.log", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T21:51:19.534 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T21:51:19.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "", 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T21:51:19.536 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:19.537 DEBUG:tasks.util.rgw: json result: {'id': 'af27eef6-40e2-4f0f-9ac6-e20c2b3336c5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T21:51:19.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2023-09-29T21:51:19.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2023-09-29T21:51:19.537 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2023-09-29T21:51:19.579 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:19.579 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:19.597 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:19.590+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.345 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.338+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.345 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.338+0000 7f9c13ce4f40 20 realm 2023-09-29T21:51:21.345 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.338+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.345 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.346 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T21:51:21.346 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.346 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.346 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.349 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.349 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.342+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.349 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.346+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.349 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.346+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.350 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.346+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.350 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.346+0000 7f9c13ce4f40 10 Creating default zonegroup 2023-09-29T21:51:21.351 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.346+0000 7f9c13ce4f40 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-09-29T21:51:21.366 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.362+0000 7f9c13ce4f40 10 WARNING: driver->list_zones() returned r=-2 2023-09-29T21:51:21.366 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.362+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.366 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.362+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.370 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.366+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.370 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.366+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.370 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.366+0000 7f9c13ce4f40 10 could not read realm id: (2) No such file or directory 2023-09-29T21:51:21.370 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.366+0000 7f9c13ce4f40 10 WARNING: failed to set zone as default, r=-22 2023-09-29T21:51:21.370 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.366+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.371 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.366+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.373 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.374 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T21:51:21.374 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.375 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T21:51:21.375 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 10 Using default name default 2023-09-29T21:51:21.375 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.376 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T21:51:21.376 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.376 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados_obj.operate() r=0 bl.length=874 2023-09-29T21:51:21.376 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 zone default found 2023-09-29T21:51:21.376 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 4 Realm: () 2023-09-29T21:51:21.376 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 4 ZoneGroup: default (07036c1a-fdd0-45df-a127-4f0270f8261a) 2023-09-29T21:51:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 4 Zone: default (75601555-ba08-46f6-b70b-01bac92eeb6b) 2023-09-29T21:51:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T21:51:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 zonegroup default 2023-09-29T21:51:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:21.377 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:21.370+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:23.362 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:23.358+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:23.362 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:23.358+0000 7f9c13ce4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:23.362 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:23.358+0000 7f9c13ce4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:23.362 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:23.358+0000 7f9c13ce4f40 20 started sync module instance, tier type = 2023-09-29T21:51:23.363 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:23.358+0000 7f9c13ce4f40 20 started zone id=75601555-ba08-46f6-b70b-01bac92eeb6b (name=default) with tier type = 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=0 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=1 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=2 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=3 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=4 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=5 2023-09-29T21:51:25.415 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=6 2023-09-29T21:51:25.417 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 20 add_watcher() i=7 2023-09-29T21:51:25.417 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.410+0000 7f9c13ce4f40 2 all 8 watchers are set, enabling cache 2023-09-29T21:51:25.417 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-09-29T21:51:25.417 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-09-29T21:51:25.418 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:25.418 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-09-29T21:51:25.418 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-09-29T21:51:25.418 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:25.419 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-09-29T21:51:25.420 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.414+0000 7f9c13ce4f40 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-09-29T21:51:25.421 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.418+0000 7f9c13ce4f40 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-09-29T21:51:25.428 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.422+0000 7f9bf946b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:25.428 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.422+0000 7f9c13ce4f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:25.429 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:25.422+0000 7f9c13ce4f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:28.392 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.386+0000 7f9c13ce4f40 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-09-29T21:51:28.392 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.386+0000 7f9c13ce4f40 5 note: GC not initialized 2023-09-29T21:51:28.400 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.394+0000 7f9bf4c62700 20 reqs_thread_entry: start 2023-09-29T21:51:28.400 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.394+0000 7f9c13ce4f40 20 init_complete bucket index max shards: 11 2023-09-29T21:51:28.400 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.394+0000 7f9c13ce4f40 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T21:51:28.400 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.394+0000 7f9bf2c5e700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 21:51:58 2023 2023-09-29T21:51:28.400 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-09-29T21:51:28.400 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.394+0000 7f9bf243c700 20 reqs_thread_entry: start 2023-09-29T21:51:28.427 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.422+0000 7f9bf946b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:28.431 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.426+0000 7f9c13ce4f40 20 remove_watcher() i=0 2023-09-29T21:51:28.431 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.426+0000 7f9c13ce4f40 2 removed watcher, disabling cache 2023-09-29T21:51:28.432 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.426+0000 7f9c13ce4f40 20 remove_watcher() i=1 2023-09-29T21:51:28.434 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.430+0000 7f9c13ce4f40 20 remove_watcher() i=2 2023-09-29T21:51:28.436 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.430+0000 7f9c13ce4f40 20 remove_watcher() i=3 2023-09-29T21:51:28.438 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.434+0000 7f9c13ce4f40 20 remove_watcher() i=4 2023-09-29T21:51:28.440 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.434+0000 7f9c13ce4f40 20 remove_watcher() i=5 2023-09-29T21:51:28.443 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.438+0000 7f9c13ce4f40 20 remove_watcher() i=6 2023-09-29T21:51:28.444 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.438+0000 7f9c13ce4f40 20 remove_watcher() i=7 2023-09-29T21:51:28.449 INFO:teuthology.orchestra.run.smithi131.stdout:[] 2023-09-29T21:51:28.453 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T21:51:28.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:28.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:28.453 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T21:51:28.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:28.500 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:28.517 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.510+0000 7f28aaadbf40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:28.517 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.510+0000 7f28aaadbf40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:28.523 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.518+0000 7f288ca5b700 20 reqs_thread_entry: start 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75601555-ba08-46f6-b70b-01bac92eeb6b", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "default.rgw.control", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "default.rgw.log", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T21:51:28.536 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "", 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "" 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2023-09-29T21:51:28.537 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2023-09-29T21:51:28.538 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2023-09-29T21:51:28.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:51:28.539 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:51:28.539 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:51:28.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "", 2023-09-29T21:51:28.539 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T21:51:28.539 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:51:28.540 DEBUG:tasks.util.rgw: json result: {'id': '75601555-ba08-46f6-b70b-01bac92eeb6b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T21:51:28.540 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2023-09-29T21:51:28.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2023-09-29T21:51:28.540 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2023-09-29T21:51:28.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:28.576 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:28.596 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.590+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 realm 2023-09-29T21:51:28.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-09-29T21:51:28.598 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T21:51:28.599 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 rados_obj.operate() r=0 bl.length=890 2023-09-29T21:51:28.600 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.594+0000 7f3e5b8f4f40 20 searching for the correct realm 2023-09-29T21:51:28.622 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 RGWRados::pool_iterate: got zonegroup_info.07036c1a-fdd0-45df-a127-4f0270f8261a 2023-09-29T21:51:28.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 RGWRados::pool_iterate: got zone_names.default 2023-09-29T21:51:28.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-09-29T21:51:28.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 RGWRados::pool_iterate: got zone_info.75601555-ba08-46f6-b70b-01bac92eeb6b 2023-09-29T21:51:28.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.623 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.624 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.624 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados_obj.operate() r=0 bl.length=46 2023-09-29T21:51:28.624 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.624 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados_obj.operate() r=0 bl.length=420 2023-09-29T21:51:28.624 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 zone default found 2023-09-29T21:51:28.625 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 4 Realm: () 2023-09-29T21:51:28.625 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 4 ZoneGroup: default (07036c1a-fdd0-45df-a127-4f0270f8261a) 2023-09-29T21:51:28.625 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 4 Zone: default (75601555-ba08-46f6-b70b-01bac92eeb6b) 2023-09-29T21:51:28.625 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 10 cannot find current period zonegroup using local zonegroup configuration 2023-09-29T21:51:28.625 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 zonegroup default 2023-09-29T21:51:28.625 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.618+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 rados->read ofs=0 len=0 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 rados_obj.operate() r=-2 bl.length=0 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 started sync module instance, tier type = 2023-09-29T21:51:28.626 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.622+0000 7f3e5b8f4f40 20 started zone id=75601555-ba08-46f6-b70b-01bac92eeb6b (name=default) with tier type = 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=0 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=1 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=2 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=3 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=4 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=5 2023-09-29T21:51:28.655 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.650+0000 7f3e5b8f4f40 20 add_watcher() i=6 2023-09-29T21:51:28.657 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.654+0000 7f3e5b8f4f40 20 add_watcher() i=7 2023-09-29T21:51:28.657 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.654+0000 7f3e5b8f4f40 2 all 8 watchers are set, enabling cache 2023-09-29T21:51:28.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.654+0000 7f3e4107b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:28.660 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.654+0000 7f3e5b8f4f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:28.661 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.654+0000 7f3e5b8f4f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:28.668 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.662+0000 7f3e5b8f4f40 5 note: GC not initialized 2023-09-29T21:51:28.672 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.666+0000 7f3e3c872700 20 reqs_thread_entry: start 2023-09-29T21:51:28.672 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.666+0000 7f3e5b8f4f40 20 init_complete bucket index max shards: 11 2023-09-29T21:51:28.672 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.666+0000 7f3e5b8f4f40 10 rgw notify: Started notification manager with: 1 workers 2023-09-29T21:51:28.672 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.666+0000 7f3e3a86e700 20 rgw notify: INFO: next queues processing will happen at: Fri Sep 29 21:51:58 2023 2023-09-29T21:51:28.672 INFO:teuthology.orchestra.run.smithi131.stderr: 2023-09-29T21:51:28.673 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.666+0000 7f3e3a04c700 20 reqs_thread_entry: start 2023-09-29T21:51:28.697 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.694+0000 7f3e4107b700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-09-29T21:51:28.702 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.698+0000 7f3e5b8f4f40 20 remove_watcher() i=0 2023-09-29T21:51:28.702 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.698+0000 7f3e5b8f4f40 2 removed watcher, disabling cache 2023-09-29T21:51:28.704 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.698+0000 7f3e5b8f4f40 20 remove_watcher() i=1 2023-09-29T21:51:28.707 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.702+0000 7f3e5b8f4f40 20 remove_watcher() i=2 2023-09-29T21:51:28.709 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.706+0000 7f3e5b8f4f40 20 remove_watcher() i=3 2023-09-29T21:51:28.711 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.706+0000 7f3e5b8f4f40 20 remove_watcher() i=4 2023-09-29T21:51:28.714 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.710+0000 7f3e5b8f4f40 20 remove_watcher() i=5 2023-09-29T21:51:28.716 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.710+0000 7f3e5b8f4f40 20 remove_watcher() i=6 2023-09-29T21:51:28.718 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.714+0000 7f3e5b8f4f40 20 remove_watcher() i=7 2023-09-29T21:51:28.723 INFO:teuthology.orchestra.run.smithi131.stdout:[] 2023-09-29T21:51:28.726 DEBUG:tasks.util.rgw: json result: [] 2023-09-29T21:51:28.726 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:28.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-09-29T21:51:28.727 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-09-29T21:51:28.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:28.815 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:28.832 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.826+0000 7f12921b7f40 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2023-09-29T21:51:28.832 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.826+0000 7f12921b7f40 20 check_secure_mon_conn(): mode 1 is insecure 2023-09-29T21:51:28.838 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:51:28.834+0000 7f1274137700 20 reqs_thread_entry: start 2023-09-29T21:51:28.850 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "75601555-ba08-46f6-b70b-01bac92eeb6b", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "default.rgw.meta:root", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "default.rgw.control", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "default.rgw.log:gc", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "default.rgw.log:lc", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "default.rgw.log", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "default.rgw.log:intent", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "default.rgw.log:usage", 2023-09-29T21:51:28.851 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "default.rgw.meta:roles", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "default.rgw.log:reshard", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "default.rgw.meta:users.email", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "default.rgw.otp", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "", 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "" 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:51:28.852 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "default.rgw.buckets.index", 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "default.rgw.buckets.data", 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2023-09-29T21:51:28.853 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:51:28.854 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:51:28.854 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:51:28.854 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "", 2023-09-29T21:51:28.854 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "default.rgw.log:notif" 2023-09-29T21:51:28.854 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:51:28.855 DEBUG:tasks.util.rgw: json result: {'id': '75601555-ba08-46f6-b70b-01bac92eeb6b', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-09-29T21:51:28.855 INFO:tasks.rgw:Starting rgw... 2023-09-29T21:51:28.855 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2023-09-29T21:51:28.855 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T21:51:28.855 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-09-29T21:51:28.856 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-09-29T21:51:28.858 INFO:tasks.rgw.c1.client.0:Started 2023-09-29T21:51:28.858 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2023-09-29T21:51:28.858 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T21:51:28.858 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T21:51:28.859 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T21:51:28.861 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T21:51:28.861 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2023-09-29T21:51:28.861 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T21:51:28.861 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-29T21:51:28.861 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-29T21:51:28.863 INFO:tasks.rgw.c2.client.0:Started 2023-09-29T21:51:28.863 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2023-09-29T21:51:28.864 INFO:tasks.rgw:Using beast as radosgw frontend 2023-09-29T21:51:28.864 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-09-29T21:51:28.864 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-09-29T21:51:28.866 INFO:tasks.rgw.c2.client.1:Started 2023-09-29T21:51:28.866 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8000/ 2023-09-29T21:51:28.866 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2023-09-29T21:51:28.898 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:28.898 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T21:51:28.901 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T21:51:28.901 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T21:51:29.903 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000/ 2023-09-29T21:51:29.917 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:29.917 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:29.920 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-29T21:51:29.921 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T21:51:29.921 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi112.front.sepia.ceph.com:8001/ 2023-09-29T21:51:29.921 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001/ 2023-09-29T21:51:29.933 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:29.934 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:29.935 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-29T21:51:29.936 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T21:51:29.936 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi131.front.sepia.ceph.com:8000/ 2023-09-29T21:51:29.937 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8000/ 2023-09-29T21:51:30.039 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:30.039 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:30.040 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T21:51:30.041 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2023-09-29T21:51:30.042 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi131.front.sepia.ceph.com:8001/ 2023-09-29T21:51:30.042 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8001/ 2023-09-29T21:51:30.057 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:30.057 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:30.059 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2023-09-29T21:51:30.060 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2023-09-29T21:51:30.061 INFO:teuthology.run_tasks:Running task rgw-multisite... 2023-09-29T21:51:32.028 INFO:tasks.rgw_multisite:creating realm.. 2023-09-29T21:51:32.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:32.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:32.028 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:32.069 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:32.069 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:32.116 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:32.117 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:32.117 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "test-realm", 2023-09-29T21:51:32.117 INFO:teuthology.orchestra.run.smithi112.stdout: "current_period": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T21:51:32.117 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T21:51:32.117 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:32.119 INFO:tasks.rgw_multisite:creating master zonegroup.. 2023-09-29T21:51:32.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:32.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:32.120 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup a --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:32.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:32.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:32.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:32.223+0000 7f694c19ff40 -1 old format 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:32.248 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "", 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [], 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [] 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:32.249 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:32.250 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:32.250 INFO:tasks.rgw_multisite:creating master zone.. 2023-09-29T21:51:32.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:32.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:32.251 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --default --master --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --endpoints http://smithi112.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:32.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:32.302 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:32.330 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=afcff763-1d9e-43af-a674-cef7f31c7813 2023-09-29T21:51:32.373 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T21:51:32.373 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a1.rgw.meta:root", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a1.rgw.control", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a1.rgw.log:gc", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a1.rgw.log:lc", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a1.rgw.log", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-09-29T21:51:32.374 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a1.rgw.otp", 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T21:51:32.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a1.rgw.buckets.index", 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a1.rgw.buckets.data" 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T21:51:32.376 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T21:51:32.377 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:32.377 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:32.377 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:32.377 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:32.377 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a1.rgw.log:notif" 2023-09-29T21:51:32.377 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:32.378 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create a1.rgw.buckets.data 64 64 --cluster c1 2023-09-29T21:51:32.976 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'a1.rgw.buckets.data' created 2023-09-29T21:51:32.990 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable a1.rgw.buckets.data rgw --cluster c1 2023-09-29T21:51:34.005 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'a1.rgw.buckets.data' 2023-09-29T21:51:34.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:34.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:34.024 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:34.064 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:34.065 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:34.096 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a1.rgw.meta:root", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a1.rgw.control", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a1.rgw.log:gc", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a1.rgw.log:lc", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a1.rgw.log", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a1.rgw.log:intent", 2023-09-29T21:51:34.097 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a1.rgw.log:usage", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a1.rgw.meta:roles", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a1.rgw.log:reshard", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a1.rgw.meta:users.keys", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a1.rgw.meta:users.email", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a1.rgw.meta:users.swift", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a1.rgw.meta:users.uid", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a1.rgw.otp", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:34.098 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a1.rgw.buckets.index", 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a1.rgw.buckets.data", 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:34.099 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a1.rgw.buckets.non-ec", 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a1.rgw.log:notif" 2023-09-29T21:51:34.100 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:34.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:34.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:34.101 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:34.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:34.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:34.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:34.207+0000 7f7401c67f40 0 period (cbc12b34-1e27-40f2-8eb8-fd649eede431 does not have zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f configured 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1, 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:51:41.103 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:41.104 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": false, 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T21:51:41.105 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T21:51:41.106 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T21:51:41.107 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T21:51:41.108 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:51:41.109 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T21:51:41.110 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T21:51:41.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T21:51:41.112 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:51:41.112 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:41.113 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-09-29T21:51:41.113 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T21:51:41.114 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-09-29T21:51:41.111+0000 7f1ac51be700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 33613) UID: 0 2023-09-29T21:51:41.114 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-09-29T21:51:41.111+0000 7f1ac8a1df80 -1 shutting down 2023-09-29T21:51:45.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-09-29T21:51:47.214 INFO:tasks.rgw.c1.client.0:Stopped 2023-09-29T21:51:47.214 INFO:tasks.rgw.c1.client.0:Restarting daemon 2023-09-29T21:51:47.215 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2023-09-29T21:51:47.217 INFO:tasks.rgw.c1.client.0:Started 2023-09-29T21:51:47.217 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2023-09-29T21:51:47.235 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:47.235 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:47.235 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T21:51:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T21:51:47.236 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T21:51:48.237 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8000 2023-09-29T21:51:48.252 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:51:48.252 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:51:48.276 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 9304 0 --:--:-- --:--:-- --:--:-- 9304 2023-09-29T21:51:48.277 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T21:51:48.277 INFO:tasks.rgw_multisite:creating admin user.. 2023-09-29T21:51:48.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:48.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:48.278 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:48.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:48.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:48.387 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:48.387 INFO:teuthology.orchestra.run.smithi112.stdout: "user_id": "realm-admin", 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "display_name": "Realm Admin", 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "email": "", 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "suspended": 0, 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "max_buckets": 1000, 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "subusers": [], 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "keys": [ 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "realm-admin", 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:51:48.388 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "swift_keys": [], 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": [], 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "op_mask": "read, write, delete", 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "system": true, 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "", 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "default_storage_class": "", 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_tags": [], 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:51:48.389 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "temp_url_keys": [], 2023-09-29T21:51:48.390 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "rgw", 2023-09-29T21:51:48.391 INFO:teuthology.orchestra.run.smithi112.stdout: "mfa_ids": [] 2023-09-29T21:51:48.391 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:48.391 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:51:48.413 INFO:tasks.rgw_multisite:creating zone.. 2023-09-29T21:51:48.413 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:48.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi112.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:48.413 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --endpoints http://smithi112.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:48.461 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:48.461 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:48.488 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=afcff763-1d9e-43af-a674-cef7f31c7813 2023-09-29T21:51:48.526 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:48.526 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:51:48.526 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T21:51:48.526 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-09-29T21:51:48.527 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T21:51:48.528 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data" 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:48.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:48.530 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:48.530 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:48.530 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-09-29T21:51:48.530 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:48.531 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool create a2.rgw.buckets.data 64 64 --cluster c1 2023-09-29T21:51:48.966 INFO:teuthology.orchestra.run.smithi112.stderr:pool 'a2.rgw.buckets.data' created 2023-09-29T21:51:48.982 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph osd pool application enable a2.rgw.buckets.data rgw --cluster c1 2023-09-29T21:51:49.986 INFO:teuthology.orchestra.run.smithi112.stderr:enabled application 'rgw' on pool 'a2.rgw.buckets.data' 2023-09-29T21:51:50.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:50.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:50.012 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:50.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:50.061 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:50.091 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-09-29T21:51:50.092 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:51:50.093 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-09-29T21:51:50.094 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-09-29T21:51:50.095 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:50.096 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:50.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:51:50.097 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:51:50.136 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:51:50.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:51:50.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:51:50.155+0000 7f1c6b7b6f40 0 period (b4e33480-bfb8-4364-b1d7-649c06bc8c56 does not have zone 272b60f1-3a4d-4009-a171-ab410395ba9a configured 2023-09-29T21:51:51.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-09-29T21:51:57.070 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T21:51:57.102 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:51:57.102 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:51:57.102 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2, 2023-09-29T21:51:57.102 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T21:51:57.102 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:57.103 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T21:51:57.104 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:51:57.106 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T21:51:57.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T21:51:57.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T21:51:57.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T21:51:57.110 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.111 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T21:51:57.112 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T21:51:57.123 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T21:51:57.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.123 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T21:51:57.123 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:51:57.124 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:51:57.126 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T21:51:57.126 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T21:51:57.127 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T21:51:57.123+0000 7f8fbf535700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 33614) UID: 0 2023-09-29T21:51:57.127 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T21:51:57.123+0000 7f8fc2d94f80 -1 shutting down 2023-09-29T21:52:02.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T21:52:03.228 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T21:52:03.228 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T21:52:03.229 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T21:52:03.232 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T21:52:03.232 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T21:52:03.250 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:52:03.251 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:52:03.251 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T21:52:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T21:52:03.252 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T21:52:04.253 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T21:52:04.271 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:52:04.271 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:52:04.275 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T21:52:04.275 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T21:52:04.275 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-09-29T21:52:04.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:04.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:04.276 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi112.front.sepia.ceph.com:8000 --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:04.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:04.323 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:04.392 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:04.392 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:04.392 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-realm", 2023-09-29T21:52:04.392 INFO:teuthology.orchestra.run.smithi131.stdout: "current_period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:04.392 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 2 2023-09-29T21:52:04.392 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:04.394 INFO:tasks.rgw_multisite:creating zonegroup.. 2023-09-29T21:52:04.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:04.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zonegroup', 'set', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:04.395 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zonegroup set --rgw-zonegroup b --rgw-realm test-realm --default --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:04.436 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:04.437 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:04.466 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:52:04.462+0000 7f1aabf77f40 -1 old format 2023-09-29T21:52:04.479 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b", 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b", 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": false, 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2023-09-29T21:52:04.480 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "", 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [], 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [] 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:04.481 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2023-09-29T21:52:04.482 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2023-09-29T21:52:04.482 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:04.482 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2023-09-29T21:52:04.482 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:04.482 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:04.482 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:04.483 INFO:tasks.rgw_multisite:creating zone.. 2023-09-29T21:52:04.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi131.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:04.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi131.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:04.483 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --default --master --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --endpoints http://smithi131.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:04.533 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:04.533 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:04.562 INFO:teuthology.orchestra.run.smithi131.stderr:NOTICE: set zone's realm_id=afcff763-1d9e-43af-a674-cef7f31c7813 2023-09-29T21:52:04.604 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:04.604 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:04.604 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b1", 2023-09-29T21:52:04.604 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "b1.rgw.meta:root", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "b1.rgw.control", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "b1.rgw.log:gc", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "b1.rgw.log:lc", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "b1.rgw.log", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-09-29T21:52:04.605 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "b1.rgw.otp", 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "b1.rgw.buckets.index", 2023-09-29T21:52:04.606 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "b1.rgw.buckets.data" 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:04.607 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "b1.rgw.log:notif" 2023-09-29T21:52:04.608 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:04.608 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create b1.rgw.buckets.data 64 64 --cluster c2 2023-09-29T21:52:05.360 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'b1.rgw.buckets.data' created 2023-09-29T21:52:05.374 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable b1.rgw.buckets.data rgw --cluster c2 2023-09-29T21:52:06.372 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'b1.rgw.buckets.data' 2023-09-29T21:52:06.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:06.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:06.401 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:06.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:06.439 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:06.470 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b1", 2023-09-29T21:52:06.470 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "b1.rgw.meta:root", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "b1.rgw.control", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "b1.rgw.log:gc", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "b1.rgw.log:lc", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "b1.rgw.log", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "b1.rgw.log:intent", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "b1.rgw.log:usage", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "b1.rgw.meta:roles", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "b1.rgw.log:reshard", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "b1.rgw.meta:users.keys", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "b1.rgw.meta:users.email", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "b1.rgw.meta:users.swift", 2023-09-29T21:52:06.471 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "b1.rgw.meta:users.uid", 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "b1.rgw.otp", 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "b1.rgw.buckets.index", 2023-09-29T21:52:06.472 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "b1.rgw.buckets.data", 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "b1.rgw.buckets.non-ec", 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:06.473 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:06.474 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "b1.rgw.log:notif" 2023-09-29T21:52:06.474 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:06.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:06.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:06.475 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:06.515 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:06.515 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:06.537 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:52:06.534+0000 7f31744e5f40 0 period (b4e33480-bfb8-4364-b1d7-649c06bc8c56 does not have zone 794315e0-2f8f-40cc-bb4b-5c0cda8a89b2 configured 2023-09-29T21:52:08.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T21:52:13.452 INFO:teuthology.orchestra.run.smithi131.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 3, 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": [], 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "period_map": { 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "zonegroups": [ 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.509 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "a", 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "a", 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": true, 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [ 2023-09-29T21:52:13.510 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "a2", 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:13.511 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "a1", 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:13.512 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:13.513 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [ 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD" 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.514 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.515 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b", 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b", 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": false, 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [ 2023-09-29T21:52:13.516 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b1", 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": false, 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:13.517 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:13.518 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:13.518 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:13.518 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:13.518 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:13.518 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2023-09-29T21:52:13.532 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [ 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD" 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2023-09-29T21:52:13.533 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "short_zone_ids": [ 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 1230604351 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.534 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 3868157510 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 3032081421 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zonegroup": "a", 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "period_config": { 2023-09-29T21:52:13.535 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_quota": { 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "user_quota": { 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2023-09-29T21:52:13.536 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "user_ratelimit": { 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_ratelimit": { 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2023-09-29T21:52:13.537 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "anonymous_ratelimit": { 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:13.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:13.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:13.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_name": "test-realm", 2023-09-29T21:52:13.539 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:52:13.539 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:13.540 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-29T21:52:13.540 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T21:52:13.589 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T21:52:13.586+0000 7f42aba25700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 33134) UID: 0 2023-09-29T21:52:13.589 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T21:52:13.586+0000 7f42af284f80 -1 shutting down 2023-09-29T21:52:18.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-29T21:52:19.642 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-29T21:52:19.642 INFO:tasks.rgw.c2.client.0:Restarting daemon 2023-09-29T21:52:19.642 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2023-09-29T21:52:19.645 INFO:tasks.rgw.c2.client.0:Started 2023-09-29T21:52:19.645 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8000 2023-09-29T21:52:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T21:52:19.663 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:52:19.664 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:52:19.664 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T21:52:19.664 INFO:teuthology.orchestra.run.smithi131.stderr:curl: (7) Failed to connect to smithi131.front.sepia.ceph.com port 8000: Connection refused 2023-09-29T21:52:20.665 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8000 2023-09-29T21:52:20.683 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:52:20.683 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:52:20.686 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T21:52:20.687 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2023-09-29T21:52:20.687 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2023-09-29T21:52:20.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:20.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--url', 'http://smithi112.front.sepia.ceph.com:8000', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:20.688 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi112.front.sepia.ceph.com:8000 --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --url http://smithi112.front.sepia.ceph.com:8000 --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:20.740 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:20.740 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:20.807 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:20.807 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:20.807 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "test-realm", 2023-09-29T21:52:20.807 INFO:teuthology.orchestra.run.smithi131.stdout: "current_period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:20.807 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 2 2023-09-29T21:52:20.808 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:20.810 INFO:tasks.rgw_multisite:creating zone.. 2023-09-29T21:52:20.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi131.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:20.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--endpoints', 'http://smithi131.front.sepia.ceph.com:8001', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:20.810 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --endpoints http://smithi131.front.sepia.ceph.com:8001 --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:20.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:20.859 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:20.884 INFO:teuthology.orchestra.run.smithi131.stderr:NOTICE: set zone's realm_id=afcff763-1d9e-43af-a674-cef7f31c7813 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b2", 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "b2.rgw.meta:root", 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "b2.rgw.control", 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "b2.rgw.log:gc", 2023-09-29T21:52:20.923 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "b2.rgw.log:lc", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "b2.rgw.log", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "b2.rgw.otp", 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2023-09-29T21:52:20.924 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "b2.rgw.buckets.index", 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "b2.rgw.buckets.data" 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:20.925 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "b2.rgw.log:notif" 2023-09-29T21:52:20.926 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:20.927 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool create b2.rgw.buckets.data 64 64 --cluster c2 2023-09-29T21:52:21.343 INFO:teuthology.orchestra.run.smithi131.stderr:pool 'b2.rgw.buckets.data' created 2023-09-29T21:52:21.360 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph osd pool application enable b2.rgw.buckets.data rgw --cluster c2 2023-09-29T21:52:22.376 INFO:teuthology.orchestra.run.smithi131.stderr:enabled application 'rgw' on pool 'b2.rgw.buckets.data' 2023-09-29T21:52:22.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:22.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:22.394 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:22.433 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:22.433 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b2", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "domain_root": "b2.rgw.meta:root", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "control_pool": "b2.rgw.control", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "gc_pool": "b2.rgw.log:gc", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "lc_pool": "b2.rgw.log:lc", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "log_pool": "b2.rgw.log", 2023-09-29T21:52:22.466 INFO:teuthology.orchestra.run.smithi131.stdout: "intent_log_pool": "b2.rgw.log:intent", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "usage_log_pool": "b2.rgw.log:usage", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "roles_pool": "b2.rgw.meta:roles", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "reshard_pool": "b2.rgw.log:reshard", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "user_keys_pool": "b2.rgw.meta:users.keys", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "user_email_pool": "b2.rgw.meta:users.email", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "user_swift_pool": "b2.rgw.meta:users.swift", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "user_uid_pool": "b2.rgw.meta:users.uid", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "otp_pool": "b2.rgw.otp", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "system_key": { 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T21:52:22.467 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_pools": [ 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "default-placement", 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "index_pool": "b2.rgw.buckets.index", 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": { 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD": { 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "data_pool": "b2.rgw.buckets.data", 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "compression_type": "random" 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:22.468 INFO:teuthology.orchestra.run.smithi131.stdout: "data_extra_pool": "b2.rgw.buckets.non-ec", 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: "index_type": 0, 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: "inline_data": true 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout: "notif_pool": "b2.rgw.log:notif" 2023-09-29T21:52:22.469 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:22.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:22.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:22.470 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:22.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:22.509 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:22.530 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:52:22.526+0000 7f5f7a65ef40 0 period (b4e33480-bfb8-4364-b1d7-649c06bc8c56 does not have zone 558b41d2-5c02-4685-b471-4770c7b56dfc configured 2023-09-29T21:52:23.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-29T21:52:29.455 INFO:teuthology.orchestra.run.smithi131.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T21:52:29.517 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:52:29.517 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:29.517 INFO:teuthology.orchestra.run.smithi131.stdout: "epoch": 4, 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": [], 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "period_map": { 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "zonegroups": [ 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "a", 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "a", 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": true, 2023-09-29T21:52:29.518 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [ 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "a2", 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T21:52:29.519 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:29.520 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "a1", 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:29.521 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2023-09-29T21:52:29.522 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [ 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD" 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.523 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "b", 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b", 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "api_name": "", 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "is_master": false, 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.524 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames": [], 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "hostnames_s3website": [], 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "zones": [ 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b2", 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2023-09-29T21:52:29.525 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:29.526 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:29.538 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:29.538 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.538 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.538 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.538 INFO:teuthology.orchestra.run.smithi131.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "b1", 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "endpoints": [ 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "log_meta": false, 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "log_data": true, 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_index_max_shards": 11, 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "read_only": false, 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "tier_type": "", 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from_all": true, 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_from": [], 2023-09-29T21:52:29.539 INFO:teuthology.orchestra.run.smithi131.stdout: "redirect_zone": "", 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "supported_features": [ 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "compress-encrypted", 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "placement_targets": [ 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "name": "default-placement", 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "tags": [], 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "storage_classes": [ 2023-09-29T21:52:29.540 INFO:teuthology.orchestra.run.smithi131.stdout: "STANDARD" 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "default_placement": "default-placement", 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_policy": { 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "groups": [] 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled_features": [ 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: "resharding" 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.541 INFO:teuthology.orchestra.run.smithi131.stdout: ], 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "short_zone_ids": [ 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 1230604351 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 192990612 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T21:52:29.542 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 3868157510 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "val": 3032081421 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zonegroup": "a", 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "period_config": { 2023-09-29T21:52:29.543 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_quota": { 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "user_quota": { 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "check_on_raw": false, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size": -1, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "max_size_kb": 0, 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: "max_objects": -1 2023-09-29T21:52:29.544 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "user_ratelimit": { 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "bucket_ratelimit": { 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2023-09-29T21:52:29.545 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "anonymous_ratelimit": { 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_ops": 0, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_ops": 0, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "max_read_bytes": 0, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "max_write_bytes": 0, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "enabled": false 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T21:52:29.546 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_name": "test-realm", 2023-09-29T21:52:29.547 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:52:29.547 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:52:29.548 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-09-29T21:52:29.548 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T21:52:29.589 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T21:52:29.585+0000 7fe3d8ae3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 33136) UID: 0 2023-09-29T21:52:29.589 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T21:52:29.585+0000 7fe3dc342f80 -1 shutting down 2023-09-29T21:52:35.288 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-09-29T21:52:35.649 INFO:tasks.rgw.c2.client.1:Stopped 2023-09-29T21:52:35.649 INFO:tasks.rgw.c2.client.1:Restarting daemon 2023-09-29T21:52:35.650 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2023-09-29T21:52:35.652 INFO:tasks.rgw.c2.client.1:Started 2023-09-29T21:52:35.653 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8001 2023-09-29T21:52:35.668 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:52:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T21:52:35.673 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:52:35.673 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T21:52:35.673 INFO:teuthology.orchestra.run.smithi131.stderr:curl: (7) Failed to connect to smithi131.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T21:52:36.674 DEBUG:teuthology.orchestra.run.smithi131:> curl http://smithi131.front.sepia.ceph.com:8001 2023-09-29T21:52:36.689 INFO:teuthology.orchestra.run.smithi131.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T21:52:36.689 INFO:teuthology.orchestra.run.smithi131.stderr: Dload Upload Total Spent Left Speed 2023-09-29T21:52:36.692 INFO:teuthology.orchestra.run.smithi131.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T21:52:36.693 INFO:teuthology.orchestra.run.smithi131.stdout:anonymous 2023-09-29T21:52:36.693 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2023-09-29T21:52:36.693 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2023-09-29T21:52:36.707 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi 2023-09-29T21:52:36.793 INFO:tasks.rgw_multisite_tests:creating test user.. 2023-09-29T21:52:36.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:36.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--caps', 'roles=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T21:52:36.794 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --caps 'roles=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T21:52:36.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:36.848 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "user_id": "rgw-multisite-test-user", 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "display_name": "Multisite Test User", 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "email": "", 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "suspended": 0, 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "max_buckets": 1000, 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "subusers": [], 2023-09-29T21:52:36.915 INFO:teuthology.orchestra.run.smithi112.stdout: "keys": [ 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "rgw-multisite-test-user", 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "P5GVQP2RXA6372D10DSL", 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "T7rvg96MrhIAdbRs2dAAq1neroxau532r1jaRxjx" 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "swift_keys": [], 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "caps": [ 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "roles", 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: "perm": "*" 2023-09-29T21:52:36.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "op_mask": "read, write, delete", 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "", 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "default_storage_class": "", 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_tags": [], 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:52:36.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "temp_url_keys": [], 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "rgw", 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout: "mfa_ids": [] 2023-09-29T21:52:36.918 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:52:36.919 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T21:52:36.939 INFO:rgw_multi.tests:meta checkpoint 2023-09-29T21:52:36.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:36.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:36.940 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:52:36.983 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:36.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:37.036 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:52:37.061 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:52:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:52:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:52:36.910908Z" 2023-09-29T21:52:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:52:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:37.063 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:52:37.063 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:52:37.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:37.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:37.064 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:52:37.107 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:37.107 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:37.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:37.153+0000 7efe1bc0ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:37.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:37.157+0000 7efe1bc0ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:37.186 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:52:37.186 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:52:37.186 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:52:37.186 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:52:37.186 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.187 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:37.188 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.189 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:37.190 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:37.191 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:37.192 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:52:37Z" 2023-09-29T21:52:37.193 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:52:37.194 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:52:40.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-09-29T21:52:42.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:42.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:42.195 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:52:42.241 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:42.242 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:42.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:42.289+0000 7f7e4aad1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:42.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:42.289+0000 7f7e4aad1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:42.320 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:52:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:52:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:52:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:52:42.320 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:52:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:52:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:42.321 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:52:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:52:42.322 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:42.323 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:42.324 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:52:42.325 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:42.326 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:42.327 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:42.328 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:52:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:52:42.329 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:52:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:52:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:42.330 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:52:42Z" 2023-09-29T21:52:42.330 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:52:42.331 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:52:47.332 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:47.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:47.333 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:52:47.378 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:47.378 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:47.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:47.425+0000 7fb310cb6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:47.433 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:47.429+0000 7fb310cb6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:52:47.456 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:47.457 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:47.458 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:52:47.459 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:52:47.460 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:47.461 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:52:47Z" 2023-09-29T21:52:47.462 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:52:47.463 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:52:52.464 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:52.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:52.465 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:52:52.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:52.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:52.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:52.565+0000 7f1b3ed16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:52.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:52.565+0000 7f1b3ed16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:52.590 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:52:52.590 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:52:52.590 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:52:52.590 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:52.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:52.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:52.593 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:52.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:52:52.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:52:52.596 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:52:52.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:52.596 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:52:52Z" 2023-09-29T21:52:52.596 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:52:52.597 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:52:57.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:57.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:52:57.598 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:52:57.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:52:57.644 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:52:57.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:57.697+0000 7fb833efbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:57.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:52:57.701+0000 7fb833efbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:57.724 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:57.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:57.726 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:57.727 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:52:57.728 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:52:57Z" 2023-09-29T21:52:57.729 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:52:57.730 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:02.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:02.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:02.732 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:02.777 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:02.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:02.831 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:02.829+0000 7efc7d5fcf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:02.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:02.829+0000 7efc7d5fcf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:02.856 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:02.856 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:02.857 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:02.858 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:02.859 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:02.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:02.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:02Z" 2023-09-29T21:53:02.862 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:02.863 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:07.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:07.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:07.865 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:07.911 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:07.911 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:07.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:07.969+0000 7f6c17fb8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:07.971 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:07.969+0000 7f6c17fb8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:07.996 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:07.997 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:07.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:07.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:08.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:08.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:07Z" 2023-09-29T21:53:08.002 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:08.003 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:13.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:13.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:13.005 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:13.051 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:13.052 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:13.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:13.105+0000 7f8f29166f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:13.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:13.105+0000 7f8f29166f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:13.132 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:13.132 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:13.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:13.134 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:13.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:13.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:13.137 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:13.138 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:13.139 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:13.140 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:13.141 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:13.142 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:13.143 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:13Z" 2023-09-29T21:53:13.144 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:13.145 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:18.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:18.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:18.146 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:18.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:18.193 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:18.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:18.245+0000 7f76616c3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:18.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:18.245+0000 7f76616c3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:18.269 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:18.270 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:18.271 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:18.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:18.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:18.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:18Z" 2023-09-29T21:53:18.275 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:18.276 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:23.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:23.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:23.277 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:23.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:23.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:23.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:23.376+0000 7f58fa5dbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:23.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:23.376+0000 7f58fa5dbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:23.402 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:23.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:23.404 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:23.405 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:23.406 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:23.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:23Z" 2023-09-29T21:53:23.408 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:23.409 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:28.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:28.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:28.411 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:28.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:28.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:28.512+0000 7fe6790bbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:28.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:28.512+0000 7fe6790bbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:28.537 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:28.538 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:28.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:28.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:28.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:28.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:28Z" 2023-09-29T21:53:28.543 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:28.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:33.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:33.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:33.545 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:33.591 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:33.591 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:33.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:33.644+0000 7fdc4f37bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:33.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:33.644+0000 7fdc4f37bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:33.672 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:33.672 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:33.672 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:33.672 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:33.673 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:33.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:33.675 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:33.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:33.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:33Z" 2023-09-29T21:53:33.678 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:33.679 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:38.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:38.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:38.681 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:38.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:38.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:38.776+0000 7fa59e164f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:38.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:38.780+0000 7fa59e164f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:38.805 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:38.806 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:38.807 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:38.808 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:38.809 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:38Z" 2023-09-29T21:53:38.810 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:38.811 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:43.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:43.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:43.813 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:43.859 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:43.859 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:43.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:43.912+0000 7f639cc0bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:43.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:43.912+0000 7f639cc0bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:43.939 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:43.941 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:43.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:43.943 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:43.944 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:43.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:43.945 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:43Z" 2023-09-29T21:53:43.945 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:43.946 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:48.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:48.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:48.947 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:48.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:48.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:49.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:49.040+0000 7f46f9239f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:49.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:49.040+0000 7f46f9239f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:49.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:49Z" 2023-09-29T21:53:49.076 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:49.077 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:54.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:54.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:54.078 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:54.125 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:54.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:54.176+0000 7f2ea88bff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:54.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:54.176+0000 7f2ea88bff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:54.201 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:54.202 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:54.203 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:54.204 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:54.205 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:54.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:54Z" 2023-09-29T21:53:54.207 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:54.208 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:53:59.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:59.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:53:59.210 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:53:59.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:53:59.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:53:59.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:59.304+0000 7f1b1fb50f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:59.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:53:59.308+0000 7f1b1fb50f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:59.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:59.333 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:59.334 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:59.335 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:53:59.336 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:53:59.337 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:53:59.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:53:59.338 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:53:59Z" 2023-09-29T21:53:59.338 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:53:59.339 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:04.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:04.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:04.340 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:04.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:04.388 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:04.436 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:04.431+0000 7f76fc20cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:04.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:04.435+0000 7f76fc20cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:04.461 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:04.461 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:04.461 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:04.461 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:04.462 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:04.463 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:04.464 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:04.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:04.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:04Z" 2023-09-29T21:54:04.467 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:04.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:09.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:09.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:09.470 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:09.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:09.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:09.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:09.563+0000 7f10174a2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:09.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:09.567+0000 7f10174a2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:09.593 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:09.593 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:09.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:09.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:09.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:09.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:09Z" 2023-09-29T21:54:09.599 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:09.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:14.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:14.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:14.601 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:14.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:14.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:14.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:14.695+0000 7f5572dddf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:14.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:14.699+0000 7f5572dddf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:14.724 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:14.725 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:14.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:14.727 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:14.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:14.729 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:14.730 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:14Z" 2023-09-29T21:54:14.731 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:14.732 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:19.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:19.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:19.734 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:19.781 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:19.781 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:19.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:19.831+0000 7f05c35a4f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:19.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:19.835+0000 7f05c35a4f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:19.860 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:19.861 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:19.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:19.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:19.864 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:19.865 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:19Z" 2023-09-29T21:54:19.866 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:19.867 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:24.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:24.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:24.869 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:24.914 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:24.914 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:24.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:24.963+0000 7fe177b80f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:24.969 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:24.967+0000 7fe177b80f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:24.992 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:24.993 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:24.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:24.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:24.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:24.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:24Z" 2023-09-29T21:54:24.998 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:24.999 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:30.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:30.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:30.000 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:30.051 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:30.051 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:30.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:30.099+0000 7f29d5c8ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:30.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:30.099+0000 7f29d5c8ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:30.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:30.129 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:30.130 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:30.131 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:30.132 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:30.133 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:30Z" 2023-09-29T21:54:30.134 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:30.135 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024356.910908_37.1 target= 2023-09-29T21:54:35.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.136 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:35.182 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:35.183 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:35.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:35.231+0000 7f7a9e935f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:35.235+0000 7f7a9e935f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.261 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:35.261 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:35.261 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:35.262 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:52:36.910908Z", 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.264 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.265 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.266 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:35.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.268 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:35.268 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:35.268 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:35.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.268 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:35Z" 2023-09-29T21:54:35.268 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:35.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:35.269 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-09-29T21:54:35.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.270 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:35.316 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:35.316 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:35.372 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:35.367+0000 7fe271c56f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.373 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:35.371+0000 7fe271c56f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.394 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2023-09-29T21:52:36.910908Z", 2023-09-29T21:54:35.395 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.396 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.397 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.398 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2023-09-29T21:54:35.399 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2023-09-29T21:54:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2023-09-29T21:54:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.400 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2023-09-29T21:54:35Z" 2023-09-29T21:54:35.400 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:54:35.401 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-09-29T21:54:35.401 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-09-29T21:54:35.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.401 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:35.446 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:35.447 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:35.501 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:35.499+0000 7fa1d71a8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.503 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:35.499+0000 7fa1d71a8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.526 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:54:35.526 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2023-09-29T21:54:35.526 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.527 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2023-09-29T21:52:36.910908Z", 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.528 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.529 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.530 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2023-09-29T21:54:35.531 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.532 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2023-09-29T21:54:35Z" 2023-09-29T21:54:35.532 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:54:35.533 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-09-29T21:54:35.533 INFO:rgw_multi.tests:meta checkpoint 2023-09-29T21:54:35.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.533 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:35.577 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:35.578 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:35.632 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:35.656 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:35.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.656 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:52:36.910908Z" 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.658 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:35.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.658 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:35.659 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:35.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.659 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:35.704 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:35.705 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:35.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:35.755+0000 7f0470ee6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:35.759+0000 7f0470ee6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:35.784 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:52:36.910908Z", 2023-09-29T21:54:35.785 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.786 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:35.788 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:35.789 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:35.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:35.790 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:35Z" 2023-09-29T21:54:35.790 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:35.791 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:35.791 INFO:rgw_multi.tests:starting meta checkpoint for zone=b1 2023-09-29T21:54:35.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b1', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.791 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:35.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:35.836 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:35.888 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:35.883+0000 7f221e632f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.889 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:35.887+0000 7f221e632f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:35.913 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.914 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2023-09-29T21:52:36.910908Z", 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.915 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.916 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:35.917 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:35.918 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2023-09-29T21:54:35Z" 2023-09-29T21:54:35.919 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:54:35.919 INFO:rgw_multi.tests:finish meta checkpoint for zone=b1 2023-09-29T21:54:35.920 INFO:rgw_multi.tests:starting meta checkpoint for zone=b2 2023-09-29T21:54:35.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'b2', '--rgw-zonegroup', 'b', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:35.920 DEBUG:teuthology.orchestra.run.smithi131:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:36.006 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:36.007 INFO:teuthology.orchestra.run.smithi131.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:36.060 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:36.055+0000 7fc70875bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:36.062 INFO:teuthology.orchestra.run.smithi131.stderr:2023-09-29T21:54:36.059+0000 7fc70875bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout: "sync_status": { 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout: "info": { 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout: "status": "sync", 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout: "num_shards": 4, 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:36.086 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "markers": [ 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 0, 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "1_1696024356.910908_37.1", 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 1, 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:36.087 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "2023-09-29T21:52:36.910908Z", 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 1, 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:36.088 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 2, 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:36.089 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: { 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "key": 3, 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "val": { 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "state": 1, 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "marker": "", 2023-09-29T21:54:36.090 INFO:teuthology.orchestra.run.smithi131.stdout: "next_step_marker": "", 2023-09-29T21:54:36.096 INFO:teuthology.orchestra.run.smithi131.stdout: "total_entries": 0, 2023-09-29T21:54:36.096 INFO:teuthology.orchestra.run.smithi131.stdout: "pos": 0, 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: "timestamp": "0.000000", 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: "realm_epoch": 2 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: } 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: "full_sync": { 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: "total": 1, 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: "complete": 1 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: }, 2023-09-29T21:54:36.097 INFO:teuthology.orchestra.run.smithi131.stdout: "current_time": "2023-09-29T21:54:36Z" 2023-09-29T21:54:36.098 INFO:teuthology.orchestra.run.smithi131.stdout:} 2023-09-29T21:54:36.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=b2 2023-09-29T21:54:36.099 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi' with args=['tests.py'] 2023-09-29T21:54:36.500 DEBUG:boto:Using access key provided by client. 2023-09-29T21:54:36.500 DEBUG:boto:Using secret key provided by client. 2023-09-29T21:54:36.501 DEBUG:boto:Using access key provided by client. 2023-09-29T21:54:36.501 DEBUG:boto:Using secret key provided by client. 2023-09-29T21:54:36.501 DEBUG:boto:Using access key provided by client. 2023-09-29T21:54:36.502 DEBUG:boto:Using secret key provided by client. 2023-09-29T21:54:36.502 DEBUG:boto:Using access key provided by client. 2023-09-29T21:54:36.502 DEBUG:boto:Using secret key provided by client. 2023-09-29T21:54:36.503 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-1 2023-09-29T21:54:36.503 DEBUG:boto:path=/hcnphz-1/ 2023-09-29T21:54:36.503 DEBUG:boto:auth_path=/hcnphz-1/ 2023-09-29T21:54:36.503 DEBUG:boto:Method: PUT 2023-09-29T21:54:36.505 DEBUG:boto:Path: /hcnphz-1/ 2023-09-29T21:54:36.505 DEBUG:boto:Data: 2023-09-29T21:54:36.505 DEBUG:boto:Headers: {} 2023-09-29T21:54:36.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:36.505 DEBUG:boto:Port: 8000 2023-09-29T21:54:36.505 DEBUG:boto:Params: {} 2023-09-29T21:54:36.505 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T21:54:36.506 DEBUG:boto:Token: None 2023-09-29T21:54:36.507 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:36 GMT /hcnphz-1/ 2023-09-29T21:54:36.507 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h4gXjvYr0MXMEOLmFO/B8Jkk8fI= 2023-09-29T21:54:36.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h4gXjvYr0MXMEOLmFO/B8Jkk8fI=', 'Content-Length': '0'} 2023-09-29T21:54:39.062 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000687747fa3c516fa9-006517479c-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:39.062 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-2 2023-09-29T21:54:39.062 DEBUG:boto:path=/hcnphz-2/ 2023-09-29T21:54:39.062 DEBUG:boto:auth_path=/hcnphz-2/ 2023-09-29T21:54:39.063 DEBUG:boto:Method: PUT 2023-09-29T21:54:39.063 DEBUG:boto:Path: /hcnphz-2/ 2023-09-29T21:54:39.063 DEBUG:boto:Data: 2023-09-29T21:54:39.063 DEBUG:boto:Headers: {} 2023-09-29T21:54:39.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:39.063 DEBUG:boto:Port: 8001 2023-09-29T21:54:39.063 DEBUG:boto:Params: {} 2023-09-29T21:54:39.063 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T21:54:39.064 DEBUG:boto:Token: None 2023-09-29T21:54:39.064 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:39 GMT /hcnphz-2/ 2023-09-29T21:54:39.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rRZaeF3rNPkOxUV7SrdQ201JsVw= 2023-09-29T21:54:39.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rRZaeF3rNPkOxUV7SrdQ201JsVw=', 'Content-Length': '0'} 2023-09-29T21:54:42.091 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000058b3e1b5b080c23-006517479f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:42.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:42.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:42.092 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:42.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:42.129 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:42.183 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:42.208 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024479.092081_45.1", 2023-09-29T21:54:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:54:39.092081Z" 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:42.210 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:42.211 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024479.092081_45.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:42.211 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:42.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:42.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:42.212 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:42.295 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:42.295 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:42.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:42.347+0000 7f9fe7fcff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:42.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:42.351+0000 7f9fe7fcff40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:42.377 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:42.377 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:42.377 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:42.377 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:42.378 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024479.044944_38.1", 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:39.044944Z", 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.379 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:42.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:42.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:42.384 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:42Z" 2023-09-29T21:54:42.384 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:42.385 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024479.044944_38.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:39.044944Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:42Z" } 2023-09-29T21:54:42.385 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024479.044944_38.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:39.044944Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:42.385 DEBUG:rgw_multi.tests:log_status={0: '1_1696024479.092081_45.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:42.385 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024479.044944_38.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:42.385 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696024479.092081_45.1 target=1_1696024479.044944_38.1 2023-09-29T21:54:47.386 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.387 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:47.433 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:47.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:47.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:47.482+0000 7f41deab8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:47.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:47.486+0000 7f41deab8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:47.514 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024479.092081_45.1", 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.515 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:39.092081Z", 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.516 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:47.517 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.518 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.519 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:47Z" 2023-09-29T21:54:47.520 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:47.522 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024479.092081_45.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:39.092081Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:47Z" } 2023-09-29T21:54:47.522 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024479.092081_45.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:39.092081Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:47.522 DEBUG:rgw_multi.tests:log_status={0: '1_1696024479.092081_45.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:47.522 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024479.092081_45.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:47.522 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:47.522 DEBUG:boto:path=/hcnphz-1/ 2023-09-29T21:54:47.523 DEBUG:boto:auth_path=/hcnphz-1/ 2023-09-29T21:54:47.523 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.523 DEBUG:boto:Path: /hcnphz-1/ 2023-09-29T21:54:47.523 DEBUG:boto:Data: 2023-09-29T21:54:47.523 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.523 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.523 DEBUG:boto:Params: {} 2023-09-29T21:54:47.524 DEBUG:boto:Token: None 2023-09-29T21:54:47.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-1/ 2023-09-29T21:54:47.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CTWpLh/Qxl2w78/oD5l2ZoGx9Oo= 2023-09-29T21:54:47.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CTWpLh/Qxl2w78/oD5l2ZoGx9Oo=', 'Content-Length': '0'} 2023-09-29T21:54:47.528 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006f204eb5a79f5b3-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.528 DEBUG:boto:path=/hcnphz-2/ 2023-09-29T21:54:47.528 DEBUG:boto:auth_path=/hcnphz-2/ 2023-09-29T21:54:47.528 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.528 DEBUG:boto:Path: /hcnphz-2/ 2023-09-29T21:54:47.528 DEBUG:boto:Data: 2023-09-29T21:54:47.529 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.529 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.529 DEBUG:boto:Params: {} 2023-09-29T21:54:47.529 DEBUG:boto:Token: None 2023-09-29T21:54:47.529 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-2/ 2023-09-29T21:54:47.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nebJO3Anuonq1mz1K1sr+f/nPV0= 2023-09-29T21:54:47.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nebJO3Anuonq1mz1K1sr+f/nPV0=', 'Content-Length': '0'} 2023-09-29T21:54:47.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ccfeef99b661601-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.534 DEBUG:boto:path=/hcnphz-1/ 2023-09-29T21:54:47.534 DEBUG:boto:auth_path=/hcnphz-1/ 2023-09-29T21:54:47.534 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.534 DEBUG:boto:Path: /hcnphz-1/ 2023-09-29T21:54:47.534 DEBUG:boto:Data: 2023-09-29T21:54:47.534 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.534 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.534 DEBUG:boto:Params: {} 2023-09-29T21:54:47.535 DEBUG:boto:Token: None 2023-09-29T21:54:47.535 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-1/ 2023-09-29T21:54:47.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CTWpLh/Qxl2w78/oD5l2ZoGx9Oo= 2023-09-29T21:54:47.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CTWpLh/Qxl2w78/oD5l2ZoGx9Oo=', 'Content-Length': '0'} 2023-09-29T21:54:47.538 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f9e13e007de027a-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.538 DEBUG:boto:path=/hcnphz-2/ 2023-09-29T21:54:47.538 DEBUG:boto:auth_path=/hcnphz-2/ 2023-09-29T21:54:47.538 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.539 DEBUG:boto:Path: /hcnphz-2/ 2023-09-29T21:54:47.539 DEBUG:boto:Data: 2023-09-29T21:54:47.539 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.539 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.539 DEBUG:boto:Params: {} 2023-09-29T21:54:47.539 DEBUG:boto:Token: None 2023-09-29T21:54:47.539 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-2/ 2023-09-29T21:54:47.539 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nebJO3Anuonq1mz1K1sr+f/nPV0= 2023-09-29T21:54:47.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nebJO3Anuonq1mz1K1sr+f/nPV0=', 'Content-Length': '0'} 2023-09-29T21:54:47.542 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000255408f8cae26aea-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.543 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2023-09-29T21:54:47.543 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-3 2023-09-29T21:54:47.544 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.544 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.544 DEBUG:boto:Method: PUT 2023-09-29T21:54:47.544 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.544 DEBUG:boto:Data: 2023-09-29T21:54:47.544 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.544 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.544 DEBUG:boto:Params: {} 2023-09-29T21:54:47.545 DEBUG:boto:Token: None 2023-09-29T21:54:47.545 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DfXLjameVacUOsskOWeLWRKBu60= 2023-09-29T21:54:47.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DfXLjameVacUOsskOWeLWRKBu60=', 'Content-Length': '0'} 2023-09-29T21:54:47.567 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000adb839cdfcd7deed-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.568 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-4 2023-09-29T21:54:47.568 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.568 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.568 DEBUG:boto:Method: PUT 2023-09-29T21:54:47.568 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.568 DEBUG:boto:Data: 2023-09-29T21:54:47.568 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.568 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.569 DEBUG:boto:Params: {} 2023-09-29T21:54:47.569 DEBUG:boto:Token: None 2023-09-29T21:54:47.569 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dc+MWQKAChj0fGtVJ2vCySO3auo= 2023-09-29T21:54:47.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dc+MWQKAChj0fGtVJ2vCySO3auo=', 'Content-Length': '0'} 2023-09-29T21:54:47.606 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002ee9fae04b0419d0-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.607 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:47.648 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:47.648 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:47.703 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:47.728 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:47.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024487.591809_53.1", 2023-09-29T21:54:47.728 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:54:47.591809Z" 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.730 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.730 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:47.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.730 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:47.731 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024487.591809_53.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:47.731 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:47.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.732 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:47.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:47.810 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:47.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:47.858+0000 7fa8256c1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:47.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:47.858+0000 7fa8256c1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:47.886 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.887 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024487.591809_53.1", 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:47.591809Z", 2023-09-29T21:54:47.888 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.889 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.890 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:47.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:47Z" 2023-09-29T21:54:47.893 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:47.894 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024487.591809_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:47.591809Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:47Z" } 2023-09-29T21:54:47.895 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024487.591809_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:47.591809Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:47.895 DEBUG:rgw_multi.tests:log_status={0: '1_1696024487.591809_53.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:47.895 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024487.591809_53.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:47.895 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:47.895 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.895 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.895 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.895 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.896 DEBUG:boto:Data: 2023-09-29T21:54:47.896 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.896 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.896 DEBUG:boto:Params: {} 2023-09-29T21:54:47.896 DEBUG:boto:Token: None 2023-09-29T21:54:47.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48= 2023-09-29T21:54:47.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48=', 'Content-Length': '0'} 2023-09-29T21:54:47.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000766c0f1d06f9f718-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.900 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.900 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.900 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.900 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.901 DEBUG:boto:Data: 2023-09-29T21:54:47.901 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.901 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.901 DEBUG:boto:Params: {} 2023-09-29T21:54:47.901 DEBUG:boto:Token: None 2023-09-29T21:54:47.901 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI= 2023-09-29T21:54:47.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI=', 'Content-Length': '0'} 2023-09-29T21:54:47.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071abea13d47ba29f-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.905 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.905 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.905 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.906 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.906 DEBUG:boto:Data: 2023-09-29T21:54:47.906 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.906 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.906 DEBUG:boto:Params: {} 2023-09-29T21:54:47.907 DEBUG:boto:Token: None 2023-09-29T21:54:47.907 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.907 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48= 2023-09-29T21:54:47.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48=', 'Content-Length': '0'} 2023-09-29T21:54:47.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004476951e78970323-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.911 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.911 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.912 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.912 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.912 DEBUG:boto:Data: 2023-09-29T21:54:47.912 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.913 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.913 DEBUG:boto:Params: {} 2023-09-29T21:54:47.913 DEBUG:boto:Token: None 2023-09-29T21:54:47.913 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI= 2023-09-29T21:54:47.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI=', 'Content-Length': '0'} 2023-09-29T21:54:47.917 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab8c5dd56a9a1d5e-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.918 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.918 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.918 DEBUG:boto:Method: PUT 2023-09-29T21:54:47.918 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.918 DEBUG:boto:Data: 2023-09-29T21:54:47.918 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.919 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.919 DEBUG:boto:Params: {} 2023-09-29T21:54:47.919 DEBUG:boto:Token: None 2023-09-29T21:54:47.919 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.919 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DfXLjameVacUOsskOWeLWRKBu60= 2023-09-29T21:54:47.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DfXLjameVacUOsskOWeLWRKBu60=', 'Content-Length': '0'} 2023-09-29T21:54:47.925 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef548d3f72bc1a7b-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.925 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.926 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.926 DEBUG:boto:Method: PUT 2023-09-29T21:54:47.926 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.926 DEBUG:boto:Data: 2023-09-29T21:54:47.926 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.926 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.926 DEBUG:boto:Params: {} 2023-09-29T21:54:47.927 DEBUG:boto:Token: None 2023-09-29T21:54:47.927 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dc+MWQKAChj0fGtVJ2vCySO3auo= 2023-09-29T21:54:47.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dc+MWQKAChj0fGtVJ2vCySO3auo=', 'Content-Length': '0'} 2023-09-29T21:54:47.932 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000673858d42bbfdb66-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.932 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.932 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.933 DEBUG:boto:Method: PUT 2023-09-29T21:54:47.933 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.933 DEBUG:boto:Data: 2023-09-29T21:54:47.933 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.934 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.934 DEBUG:boto:Params: {} 2023-09-29T21:54:47.934 DEBUG:boto:Token: None 2023-09-29T21:54:47.934 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.935 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DfXLjameVacUOsskOWeLWRKBu60= 2023-09-29T21:54:47.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DfXLjameVacUOsskOWeLWRKBu60=', 'Content-Length': '0'} 2023-09-29T21:54:47.946 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003097135fda966ee3-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.946 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.946 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.947 DEBUG:boto:Method: PUT 2023-09-29T21:54:47.947 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.947 DEBUG:boto:Data: 2023-09-29T21:54:47.947 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.947 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.947 DEBUG:boto:Params: {} 2023-09-29T21:54:47.947 DEBUG:boto:Token: None 2023-09-29T21:54:47.947 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dc+MWQKAChj0fGtVJ2vCySO3auo= 2023-09-29T21:54:47.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dc+MWQKAChj0fGtVJ2vCySO3auo=', 'Content-Length': '0'} 2023-09-29T21:54:47.959 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001f51afab5a81ebd9-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.960 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.960 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.960 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.960 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.960 DEBUG:boto:Data: 2023-09-29T21:54:47.960 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.960 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.961 DEBUG:boto:Params: {} 2023-09-29T21:54:47.961 DEBUG:boto:Token: None 2023-09-29T21:54:47.961 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48= 2023-09-29T21:54:47.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48=', 'Content-Length': '0'} 2023-09-29T21:54:47.964 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082e0a9314b874cf3-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.964 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.965 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.965 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.965 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.965 DEBUG:boto:Data: 2023-09-29T21:54:47.965 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:47.965 DEBUG:boto:Port: 8000 2023-09-29T21:54:47.965 DEBUG:boto:Params: {} 2023-09-29T21:54:47.966 DEBUG:boto:Token: None 2023-09-29T21:54:47.966 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.966 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI= 2023-09-29T21:54:47.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI=', 'Content-Length': '0'} 2023-09-29T21:54:47.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004c062c03a24fd816-00651747a7-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.969 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:47.969 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:47.969 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.969 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:47.970 DEBUG:boto:Data: 2023-09-29T21:54:47.970 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.970 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.970 DEBUG:boto:Params: {} 2023-09-29T21:54:47.970 DEBUG:boto:Token: None 2023-09-29T21:54:47.970 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-3/ 2023-09-29T21:54:47.970 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48= 2023-09-29T21:54:47.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iSkpyvCqAku6/Cj2cnHDEiQQK48=', 'Content-Length': '0'} 2023-09-29T21:54:47.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000d321f1df669a7db-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.974 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:47.975 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:47.975 DEBUG:boto:Method: HEAD 2023-09-29T21:54:47.975 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:47.975 DEBUG:boto:Data: 2023-09-29T21:54:47.975 DEBUG:boto:Headers: {} 2023-09-29T21:54:47.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:47.975 DEBUG:boto:Port: 8001 2023-09-29T21:54:47.975 DEBUG:boto:Params: {} 2023-09-29T21:54:47.975 DEBUG:boto:Token: None 2023-09-29T21:54:47.976 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:47 GMT /hcnphz-4/ 2023-09-29T21:54:47.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI= 2023-09-29T21:54:47.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gEpiVg7s4TqtknIGM8ggTR2MbhI=', 'Content-Length': '0'} 2023-09-29T21:54:47.978 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e779110adcbb0c19-00651747a7-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:47.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:47.979 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:48.021 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:48.022 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:48.069 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:48.093 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:48.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.093 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024487.591809_53.1", 2023-09-29T21:54:48.093 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:54:47.591809Z" 2023-09-29T21:54:48.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.094 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.094 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.095 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.095 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.096 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.096 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.096 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:48.098 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024487.591809_53.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.098 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:48.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.099 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:48.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:48.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:48.194+0000 7f5e51c89f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:48.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:48.194+0000 7f5e51c89f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:48.224 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024487.591809_53.1", 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:48.225 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:47.591809Z", 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:48.227 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.228 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:48.229 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:48.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.231 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:48Z" 2023-09-29T21:54:48.231 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:48.232 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024487.591809_53.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:47.591809Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:48Z" } 2023-09-29T21:54:48.232 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024487.591809_53.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:47.591809Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:48.232 DEBUG:rgw_multi.tests:log_status={0: '1_1696024487.591809_53.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.232 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024487.591809_53.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.232 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:48.232 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:48.233 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:48.233 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.233 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:48.233 DEBUG:boto:Data: 2023-09-29T21:54:48.233 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:48.233 DEBUG:boto:Port: 8000 2023-09-29T21:54:48.233 DEBUG:boto:Params: {} 2023-09-29T21:54:48.233 DEBUG:boto:Token: None 2023-09-29T21:54:48.234 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-3/ 2023-09-29T21:54:48.234 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3SkLxdjVU0p37nnCoUt3TlfC0Ak= 2023-09-29T21:54:48.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3SkLxdjVU0p37nnCoUt3TlfC0Ak=', 'Content-Length': '0'} 2023-09-29T21:54:48.237 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ca2351a4810482e-00651747a8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.238 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:48.238 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:48.238 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.238 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:48.238 DEBUG:boto:Data: 2023-09-29T21:54:48.238 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:48.239 DEBUG:boto:Port: 8000 2023-09-29T21:54:48.239 DEBUG:boto:Params: {} 2023-09-29T21:54:48.239 DEBUG:boto:Token: None 2023-09-29T21:54:48.239 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-4/ 2023-09-29T21:54:48.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HLNqB2pLOSVLY2EpmTjUJ6Un4dY= 2023-09-29T21:54:48.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HLNqB2pLOSVLY2EpmTjUJ6Un4dY=', 'Content-Length': '0'} 2023-09-29T21:54:48.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c5a8bdb27bdc8855-00651747a8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.242 DEBUG:boto:path=/hcnphz-3/ 2023-09-29T21:54:48.243 DEBUG:boto:auth_path=/hcnphz-3/ 2023-09-29T21:54:48.243 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.243 DEBUG:boto:Path: /hcnphz-3/ 2023-09-29T21:54:48.243 DEBUG:boto:Data: 2023-09-29T21:54:48.243 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:48.243 DEBUG:boto:Port: 8001 2023-09-29T21:54:48.243 DEBUG:boto:Params: {} 2023-09-29T21:54:48.243 DEBUG:boto:Token: None 2023-09-29T21:54:48.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-3/ 2023-09-29T21:54:48.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3SkLxdjVU0p37nnCoUt3TlfC0Ak= 2023-09-29T21:54:48.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3SkLxdjVU0p37nnCoUt3TlfC0Ak=', 'Content-Length': '0'} 2023-09-29T21:54:48.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ea58a9291c3a894-00651747a8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.248 DEBUG:boto:path=/hcnphz-4/ 2023-09-29T21:54:48.248 DEBUG:boto:auth_path=/hcnphz-4/ 2023-09-29T21:54:48.248 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.248 DEBUG:boto:Path: /hcnphz-4/ 2023-09-29T21:54:48.248 DEBUG:boto:Data: 2023-09-29T21:54:48.248 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:48.248 DEBUG:boto:Port: 8001 2023-09-29T21:54:48.249 DEBUG:boto:Params: {} 2023-09-29T21:54:48.249 DEBUG:boto:Token: None 2023-09-29T21:54:48.249 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-4/ 2023-09-29T21:54:48.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HLNqB2pLOSVLY2EpmTjUJ6Un4dY= 2023-09-29T21:54:48.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HLNqB2pLOSVLY2EpmTjUJ6Un4dY=', 'Content-Length': '0'} 2023-09-29T21:54:48.252 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae3fbff5fdab1f08-00651747a8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.252 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2023-09-29T21:54:48.253 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-5 2023-09-29T21:54:48.253 DEBUG:boto:path=/hcnphz-5/ 2023-09-29T21:54:48.253 DEBUG:boto:auth_path=/hcnphz-5/ 2023-09-29T21:54:48.253 DEBUG:boto:Method: PUT 2023-09-29T21:54:48.253 DEBUG:boto:Path: /hcnphz-5/ 2023-09-29T21:54:48.253 DEBUG:boto:Data: 2023-09-29T21:54:48.254 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:48.254 DEBUG:boto:Port: 8000 2023-09-29T21:54:48.254 DEBUG:boto:Params: {} 2023-09-29T21:54:48.254 DEBUG:boto:Token: None 2023-09-29T21:54:48.254 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-5/ 2023-09-29T21:54:48.254 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mcZbQnOxSD6dXHy2Dj55BfeMJs0= 2023-09-29T21:54:48.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mcZbQnOxSD6dXHy2Dj55BfeMJs0=', 'Content-Length': '0'} 2023-09-29T21:54:48.282 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d0f7838b0fdf8e03-00651747a8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.282 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-6 2023-09-29T21:54:48.282 DEBUG:boto:path=/hcnphz-6/ 2023-09-29T21:54:48.282 DEBUG:boto:auth_path=/hcnphz-6/ 2023-09-29T21:54:48.283 DEBUG:boto:Method: PUT 2023-09-29T21:54:48.283 DEBUG:boto:Path: /hcnphz-6/ 2023-09-29T21:54:48.283 DEBUG:boto:Data: 2023-09-29T21:54:48.283 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:48.283 DEBUG:boto:Port: 8001 2023-09-29T21:54:48.283 DEBUG:boto:Params: {} 2023-09-29T21:54:48.283 DEBUG:boto:Token: None 2023-09-29T21:54:48.284 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-6/ 2023-09-29T21:54:48.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c02ywbF0TTvSLz5iFRTsT+NEw+c= 2023-09-29T21:54:48.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c02ywbF0TTvSLz5iFRTsT+NEw+c=', 'Content-Length': '0'} 2023-09-29T21:54:48.325 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ee4c6086777c80f2-00651747a8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.325 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:48.374 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:48.374 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:48.418 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:48.443 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:48.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024488.308397_61.1", 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:54:48.308397Z" 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.444 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.445 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:48.446 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024488.308397_61.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.446 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:48.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.447 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:48.487 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:48.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:48.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:48.538+0000 7f385b42cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:48.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:48.538+0000 7f385b42cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:48.567 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024488.308397_61.1", 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.569 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:48.308397Z", 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.570 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.571 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:48.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.573 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:48Z" 2023-09-29T21:54:48.574 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:48.575 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024488.308397_61.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:48.308397Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:48Z" } 2023-09-29T21:54:48.575 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024488.308397_61.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:48.308397Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:48.576 DEBUG:rgw_multi.tests:log_status={0: '1_1696024488.308397_61.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.576 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024488.308397_61.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.576 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:48.576 DEBUG:boto:path=/hcnphz-5/ 2023-09-29T21:54:48.576 DEBUG:boto:auth_path=/hcnphz-5/ 2023-09-29T21:54:48.576 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.576 DEBUG:boto:Path: /hcnphz-5/ 2023-09-29T21:54:48.576 DEBUG:boto:Data: 2023-09-29T21:54:48.576 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:48.577 DEBUG:boto:Port: 8000 2023-09-29T21:54:48.577 DEBUG:boto:Params: {} 2023-09-29T21:54:48.577 DEBUG:boto:Token: None 2023-09-29T21:54:48.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-5/ 2023-09-29T21:54:48.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BRyF3Lg7dTO0ze37sL7nGMqcBBI= 2023-09-29T21:54:48.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BRyF3Lg7dTO0ze37sL7nGMqcBBI=', 'Content-Length': '0'} 2023-09-29T21:54:48.581 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c02ddbaa04acd2c1-00651747a8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.581 DEBUG:boto:path=/hcnphz-6/ 2023-09-29T21:54:48.581 DEBUG:boto:auth_path=/hcnphz-6/ 2023-09-29T21:54:48.581 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.581 DEBUG:boto:Path: /hcnphz-6/ 2023-09-29T21:54:48.581 DEBUG:boto:Data: 2023-09-29T21:54:48.581 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:48.582 DEBUG:boto:Port: 8000 2023-09-29T21:54:48.582 DEBUG:boto:Params: {} 2023-09-29T21:54:48.582 DEBUG:boto:Token: None 2023-09-29T21:54:48.582 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-6/ 2023-09-29T21:54:48.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utPiMATpH+fuIavkHiqAhRE6SX0= 2023-09-29T21:54:48.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utPiMATpH+fuIavkHiqAhRE6SX0=', 'Content-Length': '0'} 2023-09-29T21:54:48.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064f012cd07060609-00651747a8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.585 DEBUG:boto:path=/hcnphz-5/ 2023-09-29T21:54:48.585 DEBUG:boto:auth_path=/hcnphz-5/ 2023-09-29T21:54:48.585 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.585 DEBUG:boto:Path: /hcnphz-5/ 2023-09-29T21:54:48.586 DEBUG:boto:Data: 2023-09-29T21:54:48.586 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:48.586 DEBUG:boto:Port: 8001 2023-09-29T21:54:48.586 DEBUG:boto:Params: {} 2023-09-29T21:54:48.586 DEBUG:boto:Token: None 2023-09-29T21:54:48.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-5/ 2023-09-29T21:54:48.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BRyF3Lg7dTO0ze37sL7nGMqcBBI= 2023-09-29T21:54:48.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BRyF3Lg7dTO0ze37sL7nGMqcBBI=', 'Content-Length': '0'} 2023-09-29T21:54:48.590 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000023b84ebf23932825-00651747a8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.590 DEBUG:boto:path=/hcnphz-6/ 2023-09-29T21:54:48.590 DEBUG:boto:auth_path=/hcnphz-6/ 2023-09-29T21:54:48.590 DEBUG:boto:Method: HEAD 2023-09-29T21:54:48.590 DEBUG:boto:Path: /hcnphz-6/ 2023-09-29T21:54:48.591 DEBUG:boto:Data: 2023-09-29T21:54:48.591 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:48.591 DEBUG:boto:Port: 8001 2023-09-29T21:54:48.591 DEBUG:boto:Params: {} 2023-09-29T21:54:48.591 DEBUG:boto:Token: None 2023-09-29T21:54:48.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-6/ 2023-09-29T21:54:48.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utPiMATpH+fuIavkHiqAhRE6SX0= 2023-09-29T21:54:48.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utPiMATpH+fuIavkHiqAhRE6SX0=', 'Content-Length': '0'} 2023-09-29T21:54:48.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f9784a2faabfd3e7-00651747a8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.596 DEBUG:boto:path=/hcnphz-5/ 2023-09-29T21:54:48.596 DEBUG:boto:auth_path=/hcnphz-5/ 2023-09-29T21:54:48.596 DEBUG:boto:Method: DELETE 2023-09-29T21:54:48.596 DEBUG:boto:Path: /hcnphz-5/ 2023-09-29T21:54:48.596 DEBUG:boto:Data: 2023-09-29T21:54:48.596 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:48.596 DEBUG:boto:Port: 8000 2023-09-29T21:54:48.596 DEBUG:boto:Params: {} 2023-09-29T21:54:48.597 DEBUG:boto:Token: None 2023-09-29T21:54:48.597 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-5/ 2023-09-29T21:54:48.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2I8ZC5Qu0LnepHGDleViqXDJV10= 2023-09-29T21:54:48.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2I8ZC5Qu0LnepHGDleViqXDJV10=', 'Content-Length': '0'} 2023-09-29T21:54:48.657 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bbcd33c74ad2dbfe-00651747a8-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.657 DEBUG:boto:path=/hcnphz-6/ 2023-09-29T21:54:48.657 DEBUG:boto:auth_path=/hcnphz-6/ 2023-09-29T21:54:48.657 DEBUG:boto:Method: DELETE 2023-09-29T21:54:48.657 DEBUG:boto:Path: /hcnphz-6/ 2023-09-29T21:54:48.657 DEBUG:boto:Data: 2023-09-29T21:54:48.657 DEBUG:boto:Headers: {} 2023-09-29T21:54:48.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:48.658 DEBUG:boto:Port: 8001 2023-09-29T21:54:48.658 DEBUG:boto:Params: {} 2023-09-29T21:54:48.658 DEBUG:boto:Token: None 2023-09-29T21:54:48.658 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:54:48 GMT /hcnphz-6/ 2023-09-29T21:54:48.658 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zv+AilYBM+ggOdu7FtDV/x5RkRU= 2023-09-29T21:54:48.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zv+AilYBM+ggOdu7FtDV/x5RkRU=', 'Content-Length': '0'} 2023-09-29T21:54:48.788 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007152291b4ccab0ee-00651747a8-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:54:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:48.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.789 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:48.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:48.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:48.880 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:48.904 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:48.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.905 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024488.729853_69.1", 2023-09-29T21:54:48.905 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:54:48.729853Z" 2023-09-29T21:54:48.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:48.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:48.907 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:48.907 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:48.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:48.907 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:48.908 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024488.729853_69.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:48.908 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:48.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:48.908 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:48.995 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:48.996 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:49.050 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:49.046+0000 7fa270c73f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:49.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:49.050+0000 7fa270c73f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024488.729853_69.1", 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:48.729853Z", 2023-09-29T21:54:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.079 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.080 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.081 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:49Z" 2023-09-29T21:54:49.082 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:49.083 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024488.729853_69.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:48.729853Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:49Z" } 2023-09-29T21:54:49.083 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024488.729853_69.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:48.729853Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:49.083 DEBUG:rgw_multi.tests:log_status={0: '1_1696024488.729853_69.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:49.084 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024488.729853_69.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:49.084 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:49.084 DEBUG:boto:path=/hcnphz-5/ 2023-09-29T21:54:49.084 DEBUG:boto:auth_path=/hcnphz-5/ 2023-09-29T21:54:49.084 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.084 DEBUG:boto:Path: /hcnphz-5/ 2023-09-29T21:54:49.084 DEBUG:boto:Data: 2023-09-29T21:54:49.084 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.085 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.085 DEBUG:boto:Params: {} 2023-09-29T21:54:49.085 DEBUG:boto:Token: None 2023-09-29T21:54:49.085 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-5/ 2023-09-29T21:54:49.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/YWkJ+Yi1+XBsX6b24sTlpM06XY= 2023-09-29T21:54:49.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/YWkJ+Yi1+XBsX6b24sTlpM06XY=', 'Content-Length': '0'} 2023-09-29T21:54:49.086 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000ce649dd125daf59b-00651747a9-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.087 DEBUG:boto:path=/hcnphz-6/ 2023-09-29T21:54:49.087 DEBUG:boto:auth_path=/hcnphz-6/ 2023-09-29T21:54:49.087 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.087 DEBUG:boto:Path: /hcnphz-6/ 2023-09-29T21:54:49.087 DEBUG:boto:Data: 2023-09-29T21:54:49.087 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.087 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.087 DEBUG:boto:Params: {} 2023-09-29T21:54:49.088 DEBUG:boto:Token: None 2023-09-29T21:54:49.088 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-6/ 2023-09-29T21:54:49.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0GboLT6cHj07FrzH/m4iQZuPc4= 2023-09-29T21:54:49.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0GboLT6cHj07FrzH/m4iQZuPc4=', 'Content-Length': '0'} 2023-09-29T21:54:49.089 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000c61233ed11d8a52a-00651747a9-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.089 DEBUG:boto:path=/hcnphz-5/ 2023-09-29T21:54:49.089 DEBUG:boto:auth_path=/hcnphz-5/ 2023-09-29T21:54:49.089 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.090 DEBUG:boto:Path: /hcnphz-5/ 2023-09-29T21:54:49.090 DEBUG:boto:Data: 2023-09-29T21:54:49.090 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.090 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.090 DEBUG:boto:Params: {} 2023-09-29T21:54:49.090 DEBUG:boto:Token: None 2023-09-29T21:54:49.090 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-5/ 2023-09-29T21:54:49.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/YWkJ+Yi1+XBsX6b24sTlpM06XY= 2023-09-29T21:54:49.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/YWkJ+Yi1+XBsX6b24sTlpM06XY=', 'Content-Length': '0'} 2023-09-29T21:54:49.092 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000578faf28d62bfd43-00651747a9-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.093 DEBUG:boto:path=/hcnphz-6/ 2023-09-29T21:54:49.093 DEBUG:boto:auth_path=/hcnphz-6/ 2023-09-29T21:54:49.093 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.093 DEBUG:boto:Path: /hcnphz-6/ 2023-09-29T21:54:49.093 DEBUG:boto:Data: 2023-09-29T21:54:49.093 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.093 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.094 DEBUG:boto:Params: {} 2023-09-29T21:54:49.094 DEBUG:boto:Token: None 2023-09-29T21:54:49.094 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-6/ 2023-09-29T21:54:49.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0GboLT6cHj07FrzH/m4iQZuPc4= 2023-09-29T21:54:49.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0GboLT6cHj07FrzH/m4iQZuPc4=', 'Content-Length': '0'} 2023-09-29T21:54:49.095 DEBUG:boto:Response headers: [('Content-Length', '221'), ('x-amz-request-id', 'tx00000ff5700dcd49b2eee-00651747a9-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.095 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2023-09-29T21:54:49.096 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-7 2023-09-29T21:54:49.096 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:54:49.096 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:54:49.096 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.096 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:54:49.096 DEBUG:boto:Data: 2023-09-29T21:54:49.097 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.097 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.097 DEBUG:boto:Params: {} 2023-09-29T21:54:49.097 DEBUG:boto:Token: None 2023-09-29T21:54:49.097 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/ 2023-09-29T21:54:49.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D2WnX0Yt0NjWNFnjBuX3mYZUf68= 2023-09-29T21:54:49.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D2WnX0Yt0NjWNFnjBuX3mYZUf68=', 'Content-Length': '0'} 2023-09-29T21:54:49.122 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a9c48549a305df12-00651747a9-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.122 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-8 2023-09-29T21:54:49.122 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:54:49.122 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:54:49.122 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.122 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:54:49.122 DEBUG:boto:Data: 2023-09-29T21:54:49.123 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.123 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.123 DEBUG:boto:Params: {} 2023-09-29T21:54:49.123 DEBUG:boto:Token: None 2023-09-29T21:54:49.123 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/ 2023-09-29T21:54:49.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oUudSjNRtmgfsGi4fzHe2U8Xw/I= 2023-09-29T21:54:49.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oUudSjNRtmgfsGi4fzHe2U8Xw/I=', 'Content-Length': '0'} 2023-09-29T21:54:49.170 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f84d1632b8a85492-00651747a9-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.170 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:54:49.170 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:54:49.170 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.170 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:54:49.170 DEBUG:boto:Data: 2023-09-29T21:54:49.171 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.171 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.171 DEBUG:boto:Params: {} 2023-09-29T21:54:49.171 DEBUG:boto:Token: None 2023-09-29T21:54:49.171 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/ 2023-09-29T21:54:49.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8= 2023-09-29T21:54:49.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8=', 'Content-Length': '0'} 2023-09-29T21:54:49.174 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a96f2401d98fe7ea-00651747a9-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.174 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:54:49.175 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:54:49.175 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.175 DEBUG:boto:Path: /hcnphz-7/myobj 2023-09-29T21:54:49.175 DEBUG:boto:Data: 2023-09-29T21:54:49.175 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.175 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.175 DEBUG:boto:Params: {} 2023-09-29T21:54:49.176 DEBUG:boto:Token: None 2023-09-29T21:54:49.176 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/myobj 2023-09-29T21:54:49.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Oba/w6Tv05AWwCRFVhmD5coYCnI= 2023-09-29T21:54:49.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Oba/w6Tv05AWwCRFVhmD5coYCnI='} 2023-09-29T21:54:49.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000017942880a18dd07a-00651747a9-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.190 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:54:49.190 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:54:49.191 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.191 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:54:49.191 DEBUG:boto:Data: 2023-09-29T21:54:49.191 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.191 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.191 DEBUG:boto:Params: {} 2023-09-29T21:54:49.191 DEBUG:boto:Token: None 2023-09-29T21:54:49.191 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/ 2023-09-29T21:54:49.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8= 2023-09-29T21:54:49.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8=', 'Content-Length': '0'} 2023-09-29T21:54:49.194 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cd26400cdfd88dad-00651747a9-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.195 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:54:49.195 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:54:49.195 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.195 DEBUG:boto:Path: /hcnphz-7/_myobj 2023-09-29T21:54:49.195 DEBUG:boto:Data: 2023-09-29T21:54:49.195 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.195 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.195 DEBUG:boto:Params: {} 2023-09-29T21:54:49.196 DEBUG:boto:Token: None 2023-09-29T21:54:49.196 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/_myobj 2023-09-29T21:54:49.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h7Nv9krAclWWTDRAzkZrKHh9gDE= 2023-09-29T21:54:49.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h7Nv9krAclWWTDRAzkZrKHh9gDE='} 2023-09-29T21:54:49.238 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000074ccfd92a90503e5-00651747a9-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.239 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:54:49.239 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:54:49.239 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.239 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:54:49.239 DEBUG:boto:Data: 2023-09-29T21:54:49.239 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.240 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.240 DEBUG:boto:Params: {} 2023-09-29T21:54:49.240 DEBUG:boto:Token: None 2023-09-29T21:54:49.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/ 2023-09-29T21:54:49.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8= 2023-09-29T21:54:49.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8=', 'Content-Length': '0'} 2023-09-29T21:54:49.245 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab72126d6e6a1933-00651747a9-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.245 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:54:49.245 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:54:49.245 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.245 DEBUG:boto:Path: /hcnphz-7/%3A 2023-09-29T21:54:49.246 DEBUG:boto:Data: 2023-09-29T21:54:49.246 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.246 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.246 DEBUG:boto:Params: {} 2023-09-29T21:54:49.246 DEBUG:boto:Token: None 2023-09-29T21:54:49.246 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/%3A 2023-09-29T21:54:49.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GviEZU2RoO4DFER/5zAXOhPbPrA= 2023-09-29T21:54:49.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GviEZU2RoO4DFER/5zAXOhPbPrA='} 2023-09-29T21:54:49.265 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cedfe0685c8fbc65-00651747a9-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.265 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:54:49.265 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:54:49.266 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.266 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:54:49.266 DEBUG:boto:Data: 2023-09-29T21:54:49.266 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.266 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.266 DEBUG:boto:Params: {} 2023-09-29T21:54:49.267 DEBUG:boto:Token: None 2023-09-29T21:54:49.267 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/ 2023-09-29T21:54:49.267 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8= 2023-09-29T21:54:49.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fLvFsrC/bB2yojfQUviNXaNeyI8=', 'Content-Length': '0'} 2023-09-29T21:54:49.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004249b2fd5fe67eb8-00651747a9-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.271 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:54:49.271 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:54:49.271 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.271 DEBUG:boto:Path: /hcnphz-7/%26 2023-09-29T21:54:49.271 DEBUG:boto:Data: 2023-09-29T21:54:49.271 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:54:49.272 DEBUG:boto:Port: 8000 2023-09-29T21:54:49.272 DEBUG:boto:Params: {} 2023-09-29T21:54:49.272 DEBUG:boto:Token: None 2023-09-29T21:54:49.272 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-7/%26 2023-09-29T21:54:49.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VfpYH2vTpqZhHzD8RQBxS1OBusI= 2023-09-29T21:54:49.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VfpYH2vTpqZhHzD8RQBxS1OBusI='} 2023-09-29T21:54:49.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007633211ed4bc3172-00651747a9-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.315 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:54:49.315 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:54:49.315 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.315 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:54:49.315 DEBUG:boto:Data: 2023-09-29T21:54:49.315 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.315 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.316 DEBUG:boto:Params: {} 2023-09-29T21:54:49.316 DEBUG:boto:Token: None 2023-09-29T21:54:49.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/ 2023-09-29T21:54:49.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo= 2023-09-29T21:54:49.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo=', 'Content-Length': '0'} 2023-09-29T21:54:49.321 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de626146a04e656-00651747a9-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.321 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:54:49.321 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:54:49.321 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.322 DEBUG:boto:Path: /hcnphz-8/myobj 2023-09-29T21:54:49.322 DEBUG:boto:Data: 2023-09-29T21:54:49.322 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.322 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.322 DEBUG:boto:Params: {} 2023-09-29T21:54:49.322 DEBUG:boto:Token: None 2023-09-29T21:54:49.322 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/myobj 2023-09-29T21:54:49.323 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TLxa+GFGv/Li2KzJcoNKHnubw44= 2023-09-29T21:54:49.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TLxa+GFGv/Li2KzJcoNKHnubw44='} 2023-09-29T21:54:49.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016ad39aeb297927f-00651747a9-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.367 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:54:49.367 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:54:49.367 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.367 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:54:49.367 DEBUG:boto:Data: 2023-09-29T21:54:49.367 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.367 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.367 DEBUG:boto:Params: {} 2023-09-29T21:54:49.368 DEBUG:boto:Token: None 2023-09-29T21:54:49.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/ 2023-09-29T21:54:49.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo= 2023-09-29T21:54:49.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo=', 'Content-Length': '0'} 2023-09-29T21:54:49.372 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed7a6146452ab5f5-00651747a9-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.372 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:54:49.372 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:54:49.372 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.372 DEBUG:boto:Path: /hcnphz-8/_myobj 2023-09-29T21:54:49.372 DEBUG:boto:Data: 2023-09-29T21:54:49.373 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.373 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.373 DEBUG:boto:Params: {} 2023-09-29T21:54:49.373 DEBUG:boto:Token: None 2023-09-29T21:54:49.373 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/_myobj 2023-09-29T21:54:49.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EX49SlNarDjMrR+SYMIMiLJLoVA= 2023-09-29T21:54:49.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EX49SlNarDjMrR+SYMIMiLJLoVA='} 2023-09-29T21:54:49.397 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c851c8c5416435e8-00651747a9-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.398 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:54:49.398 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:54:49.398 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.398 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:54:49.398 DEBUG:boto:Data: 2023-09-29T21:54:49.398 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.399 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.399 DEBUG:boto:Params: {} 2023-09-29T21:54:49.399 DEBUG:boto:Token: None 2023-09-29T21:54:49.399 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/ 2023-09-29T21:54:49.399 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo= 2023-09-29T21:54:49.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo=', 'Content-Length': '0'} 2023-09-29T21:54:49.403 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c6c3720f7802e1e-00651747a9-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.403 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:54:49.403 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:54:49.403 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.403 DEBUG:boto:Path: /hcnphz-8/%3A 2023-09-29T21:54:49.404 DEBUG:boto:Data: 2023-09-29T21:54:49.404 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.404 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.404 DEBUG:boto:Params: {} 2023-09-29T21:54:49.404 DEBUG:boto:Token: None 2023-09-29T21:54:49.404 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/%3A 2023-09-29T21:54:49.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3kmo6YvtFDF/E1o1RWD0QX2MFtM= 2023-09-29T21:54:49.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3kmo6YvtFDF/E1o1RWD0QX2MFtM='} 2023-09-29T21:54:49.451 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b0027b0025ab4fba-00651747a9-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.451 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:54:49.451 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:54:49.451 DEBUG:boto:Method: HEAD 2023-09-29T21:54:49.451 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:54:49.451 DEBUG:boto:Data: 2023-09-29T21:54:49.452 DEBUG:boto:Headers: {} 2023-09-29T21:54:49.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.452 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.452 DEBUG:boto:Params: {} 2023-09-29T21:54:49.452 DEBUG:boto:Token: None 2023-09-29T21:54:49.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/ 2023-09-29T21:54:49.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo= 2023-09-29T21:54:49.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DdRgQqlBkl5FiGsDfWCZUxhnGwo=', 'Content-Length': '0'} 2023-09-29T21:54:49.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e1f4c4cd9192e75-00651747a9-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.457 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:54:49.457 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:54:49.457 DEBUG:boto:Method: PUT 2023-09-29T21:54:49.457 DEBUG:boto:Path: /hcnphz-8/%26 2023-09-29T21:54:49.457 DEBUG:boto:Data: 2023-09-29T21:54:49.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:54:49.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:54:49.458 DEBUG:boto:Port: 8001 2023-09-29T21:54:49.458 DEBUG:boto:Params: {} 2023-09-29T21:54:49.458 DEBUG:boto:Token: None 2023-09-29T21:54:49.459 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:54:49 GMT /hcnphz-8/%26 2023-09-29T21:54:49.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IpJzIXd4SaCLb+dDMym1uxonmHo= 2023-09-29T21:54:49.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:54:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IpJzIXd4SaCLb+dDMym1uxonmHo='} 2023-09-29T21:54:49.473 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e0e2325ac135def-00651747a9-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:54:49.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:49.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:49.474 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:49.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:49.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:49.574 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:54:49.601 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:54:49.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024489.145960_77.1", 2023-09-29T21:54:49.601 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:54:49.145960Z" 2023-09-29T21:54:49.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.602 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:49.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.603 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.603 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:54:49.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.603 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:54:49.604 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024489.145960_77.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:49.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:54:49.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:49.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:54:49.605 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:49.649 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:49.702 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:49.698+0000 7fd847995f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:49.704 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:49.698+0000 7fd847995f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:54:49.726 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024489.145960_77.1", 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:54:49.145960Z", 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:49.730 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:54:49.731 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:54:49.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:54:49Z" 2023-09-29T21:54:49.733 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:54:49.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024489.145960_77.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:54:49.145960Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:54:49Z" } 2023-09-29T21:54:49.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024489.145960_77.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:54:49.145960Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:54:49.734 DEBUG:rgw_multi.tests:log_status={0: '1_1696024489.145960_77.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:49.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024489.145960_77.1', 1: '', 2: '', 3: ''} 2023-09-29T21:54:49.734 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:54:49.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:54:49.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-7', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:54:49.735 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-7 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:54:49.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:54:49.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:54:49.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:49.886+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:54:54.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:54.886+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:54:59.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:54:59.890+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:04.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:04.890+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:09.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:09.890+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:14.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:14.890+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:19.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:19.890+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:24.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:24.894+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:29.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:29.897+0000 7f8bd66b8f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:34.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:34.897+0000 7f8bd66b8f40 1 bucket sync caught up with source: 2023-09-29T21:55:34.901 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.7.6, , 00000000001.9.6, , , , 00000000001.5.6, 00000000001.6.6, , ] 2023-09-29T21:55:34.901 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.7.6, , 00000000001.9.6, , , , 00000000001.5.6, 00000000001.6.6, , ] 2023-09-29T21:55:34.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:34.897+0000 7f8bd66b8f40 0 bucket checkpoint complete 2023-09-29T21:55:34.930 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-7 zones={a2, a1} 2023-09-29T21:55:34.930 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:34.930 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:34.930 DEBUG:boto:Method: HEAD 2023-09-29T21:55:34.930 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:55:34.931 DEBUG:boto:Data: 2023-09-29T21:55:34.931 DEBUG:boto:Headers: {} 2023-09-29T21:55:34.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:34.931 DEBUG:boto:Port: 8001 2023-09-29T21:55:34.931 DEBUG:boto:Params: {} 2023-09-29T21:55:34.931 DEBUG:boto:Token: None 2023-09-29T21:55:34.931 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:34 GMT /hcnphz-7/ 2023-09-29T21:55:34.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e1jfEwJTiRx4Ah9O8+G+sFyS6W8= 2023-09-29T21:55:34.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e1jfEwJTiRx4Ah9O8+G+sFyS6W8=', 'Content-Length': '0'} 2023-09-29T21:55:34.936 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb39969c96baad9e-00651747d6-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:55:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:34.936 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:34.936 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:34.937 DEBUG:boto:Method: HEAD 2023-09-29T21:55:34.937 DEBUG:boto:Path: /hcnphz-7/ 2023-09-29T21:55:34.937 DEBUG:boto:Data: 2023-09-29T21:55:34.937 DEBUG:boto:Headers: {} 2023-09-29T21:55:34.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:34.937 DEBUG:boto:Port: 8000 2023-09-29T21:55:34.937 DEBUG:boto:Params: {} 2023-09-29T21:55:34.937 DEBUG:boto:Token: None 2023-09-29T21:55:34.937 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:34 GMT /hcnphz-7/ 2023-09-29T21:55:34.938 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e1jfEwJTiRx4Ah9O8+G+sFyS6W8= 2023-09-29T21:55:34.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e1jfEwJTiRx4Ah9O8+G+sFyS6W8=', 'Content-Length': '0'} 2023-09-29T21:55:34.941 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005db1c8cb0f259407-00651747d6-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:55:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:34.942 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T21:55:34.942 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:34.942 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:34.942 DEBUG:boto:path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:34.942 DEBUG:boto:auth_path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:34.942 DEBUG:boto:Method: GET 2023-09-29T21:55:34.942 DEBUG:boto:Path: /hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:34.942 DEBUG:boto:Data: 2023-09-29T21:55:34.943 DEBUG:boto:Headers: {} 2023-09-29T21:55:34.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:34.943 DEBUG:boto:Port: 8001 2023-09-29T21:55:34.943 DEBUG:boto:Params: {} 2023-09-29T21:55:34.943 DEBUG:boto:Token: None 2023-09-29T21:55:34.943 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:34 GMT /hcnphz-7/?versions 2023-09-29T21:55:34.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pyRMC5uIRGJfMpoprIqKGqiX9cM= 2023-09-29T21:55:34.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pyRMC5uIRGJfMpoprIqKGqiX9cM=', 'Content-Length': '0'} 2023-09-29T21:55:34.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020965fa5de69fb0f-00651747d6-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:34.948 DEBUG:boto:b'hcnphz-7999false&nulltrue2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.080 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T21:55:35.080 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T21:55:35.080 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T21:55:35.080 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:55:35.081 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T21:55:35.081 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.081 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.081 DEBUG:boto:path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.081 DEBUG:boto:auth_path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.081 DEBUG:boto:Method: GET 2023-09-29T21:55:35.081 DEBUG:boto:Path: /hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.081 DEBUG:boto:Data: 2023-09-29T21:55:35.082 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.082 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.082 DEBUG:boto:Params: {} 2023-09-29T21:55:35.082 DEBUG:boto:Token: None 2023-09-29T21:55:35.082 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/?versions 2023-09-29T21:55:35.082 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qutV94SuC0N9jNizQf8gFqZaW+A= 2023-09-29T21:55:35.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qutV94SuC0N9jNizQf8gFqZaW+A=', 'Content-Length': '0'} 2023-09-29T21:55:35.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd4043f0978e92f-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.088 DEBUG:boto:b'hcnphz-7999false&nulltrue2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.088 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T21:55:35.089 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T21:55:35.089 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T21:55:35.089 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:55:35.089 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.089 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.089 DEBUG:boto:path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.089 DEBUG:boto:auth_path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.089 DEBUG:boto:Method: GET 2023-09-29T21:55:35.090 DEBUG:boto:Path: /hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.090 DEBUG:boto:Data: 2023-09-29T21:55:35.090 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.090 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.090 DEBUG:boto:Params: {} 2023-09-29T21:55:35.090 DEBUG:boto:Token: None 2023-09-29T21:55:35.090 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/?versions 2023-09-29T21:55:35.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qutV94SuC0N9jNizQf8gFqZaW+A= 2023-09-29T21:55:35.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qutV94SuC0N9jNizQf8gFqZaW+A=', 'Content-Length': '0'} 2023-09-29T21:55:35.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a05b9076b567176d-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.096 DEBUG:boto:b'hcnphz-7999false&nulltrue2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.097 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.097 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.097 DEBUG:boto:path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.097 DEBUG:boto:auth_path=/hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.097 DEBUG:boto:Method: GET 2023-09-29T21:55:35.097 DEBUG:boto:Path: /hcnphz-7/?versions&max-keys=999 2023-09-29T21:55:35.097 DEBUG:boto:Data: 2023-09-29T21:55:35.097 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.098 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.098 DEBUG:boto:Params: {} 2023-09-29T21:55:35.098 DEBUG:boto:Token: None 2023-09-29T21:55:35.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/?versions 2023-09-29T21:55:35.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qutV94SuC0N9jNizQf8gFqZaW+A= 2023-09-29T21:55:35.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qutV94SuC0N9jNizQf8gFqZaW+A=', 'Content-Length': '0'} 2023-09-29T21:55:35.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de3efc55795825ab-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.102 DEBUG:boto:b'hcnphz-7999false&nulltrue2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T21:55:35.103 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.103 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.103 DEBUG:boto:path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.103 DEBUG:boto:auth_path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.104 DEBUG:boto:Method: GET 2023-09-29T21:55:35.104 DEBUG:boto:Path: /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.104 DEBUG:boto:Data: 2023-09-29T21:55:35.104 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.104 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.104 DEBUG:boto:Params: {} 2023-09-29T21:55:35.104 DEBUG:boto:Token: None 2023-09-29T21:55:35.105 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs= 2023-09-29T21:55:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs=', 'Content-Length': '0'} 2023-09-29T21:55:35.107 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a514245b86700b77-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.108 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.108 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.108 DEBUG:boto:path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.108 DEBUG:boto:auth_path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.108 DEBUG:boto:Method: GET 2023-09-29T21:55:35.108 DEBUG:boto:Path: /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.109 DEBUG:boto:Data: 2023-09-29T21:55:35.109 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.109 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.109 DEBUG:boto:Params: {} 2023-09-29T21:55:35.109 DEBUG:boto:Token: None 2023-09-29T21:55:35.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs= 2023-09-29T21:55:35.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs=', 'Content-Length': '0'} 2023-09-29T21:55:35.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000044a1a0720bd403f2-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.112 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.113 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.113 DEBUG:boto:path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.113 DEBUG:boto:auth_path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.113 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.113 DEBUG:boto:Path: /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.113 DEBUG:boto:Data: 2023-09-29T21:55:35.113 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.114 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.114 DEBUG:boto:Params: {} 2023-09-29T21:55:35.114 DEBUG:boto:Token: None 2023-09-29T21:55:35.114 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SmmLJHusBsHAnov2WKE+MRDHJ30= 2023-09-29T21:55:35.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SmmLJHusBsHAnov2WKE+MRDHJ30=', 'Content-Length': '0'} 2023-09-29T21:55:35.116 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000460ee3685ead4284-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.117 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.117 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.117 DEBUG:boto:path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.117 DEBUG:boto:auth_path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.118 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.118 DEBUG:boto:Path: /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.118 DEBUG:boto:Data: 2023-09-29T21:55:35.118 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.118 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.118 DEBUG:boto:Params: {} 2023-09-29T21:55:35.119 DEBUG:boto:Token: None 2023-09-29T21:55:35.119 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.119 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SmmLJHusBsHAnov2WKE+MRDHJ30= 2023-09-29T21:55:35.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SmmLJHusBsHAnov2WKE+MRDHJ30=', 'Content-Length': '0'} 2023-09-29T21:55:35.121 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f84df5c8ab52b7ef-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T21:55:35.121 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.122 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.122 DEBUG:boto:path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.122 DEBUG:boto:auth_path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.122 DEBUG:boto:Method: GET 2023-09-29T21:55:35.122 DEBUG:boto:Path: /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.122 DEBUG:boto:Data: 2023-09-29T21:55:35.122 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.122 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.123 DEBUG:boto:Params: {} 2023-09-29T21:55:35.123 DEBUG:boto:Token: None 2023-09-29T21:55:35.123 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs= 2023-09-29T21:55:35.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs=', 'Content-Length': '0'} 2023-09-29T21:55:35.126 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec7a063bcc7a0465-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.126 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.126 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.126 DEBUG:boto:path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.126 DEBUG:boto:auth_path=/hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.126 DEBUG:boto:Method: GET 2023-09-29T21:55:35.127 DEBUG:boto:Path: /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.127 DEBUG:boto:Data: 2023-09-29T21:55:35.127 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.127 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.127 DEBUG:boto:Params: {} 2023-09-29T21:55:35.127 DEBUG:boto:Token: None 2023-09-29T21:55:35.127 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26?versionId=null 2023-09-29T21:55:35.127 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs= 2023-09-29T21:55:35.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:viX0kOzFCEL1j3oq3wtTAuHDxMs=', 'Content-Length': '0'} 2023-09-29T21:55:35.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b98ce6d8bcde913e-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.170 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.171 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.171 DEBUG:boto:path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.171 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.171 DEBUG:boto:Method: GET 2023-09-29T21:55:35.171 DEBUG:boto:Path: /hcnphz-7/?prefix=%26 2023-09-29T21:55:35.171 DEBUG:boto:Data: 2023-09-29T21:55:35.171 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.172 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.172 DEBUG:boto:Params: {} 2023-09-29T21:55:35.172 DEBUG:boto:Token: None 2023-09-29T21:55:35.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023cdba1e1fb5ceb1-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.178 DEBUG:boto:b'hcnphz-7&1000false&2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.179 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.179 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.179 DEBUG:boto:path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.179 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.179 DEBUG:boto:Method: GET 2023-09-29T21:55:35.179 DEBUG:boto:Path: /hcnphz-7/?prefix=%26 2023-09-29T21:55:35.179 DEBUG:boto:Data: 2023-09-29T21:55:35.179 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.180 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.180 DEBUG:boto:Params: {} 2023-09-29T21:55:35.180 DEBUG:boto:Token: None 2023-09-29T21:55:35.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023439e92cbb17550-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.185 DEBUG:boto:b'hcnphz-7&1000false&2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.186 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.186 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.186 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.186 DEBUG:boto:Path: /hcnphz-7/%26 2023-09-29T21:55:35.186 DEBUG:boto:Data: 2023-09-29T21:55:35.187 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.187 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.187 DEBUG:boto:Params: {} 2023-09-29T21:55:35.187 DEBUG:boto:Token: None 2023-09-29T21:55:35.187 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26 2023-09-29T21:55:35.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yf2KtrunRnFNxXVy9FGBhQVULxA= 2023-09-29T21:55:35.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yf2KtrunRnFNxXVy9FGBhQVULxA=', 'Content-Length': '0'} 2023-09-29T21:55:35.190 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0eefc80504e35db-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.190 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.190 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.190 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.191 DEBUG:boto:Path: /hcnphz-7/%26 2023-09-29T21:55:35.191 DEBUG:boto:Data: 2023-09-29T21:55:35.191 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.191 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.191 DEBUG:boto:Params: {} 2023-09-29T21:55:35.191 DEBUG:boto:Token: None 2023-09-29T21:55:35.191 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26 2023-09-29T21:55:35.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yf2KtrunRnFNxXVy9FGBhQVULxA= 2023-09-29T21:55:35.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yf2KtrunRnFNxXVy9FGBhQVULxA=', 'Content-Length': '0'} 2023-09-29T21:55:35.194 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b6af5e088be4859-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T21:55:35.194 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.195 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.195 DEBUG:boto:Method: GET 2023-09-29T21:55:35.195 DEBUG:boto:Path: /hcnphz-7/%26 2023-09-29T21:55:35.195 DEBUG:boto:Data: 2023-09-29T21:55:35.195 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.195 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.195 DEBUG:boto:Params: {} 2023-09-29T21:55:35.195 DEBUG:boto:Token: None 2023-09-29T21:55:35.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26 2023-09-29T21:55:35.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rZhBMkwR6hCQJ+JHx+49eEN3k2s= 2023-09-29T21:55:35.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rZhBMkwR6hCQJ+JHx+49eEN3k2s=', 'Content-Length': '0'} 2023-09-29T21:55:35.198 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000792191793126fe1d-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.242 DEBUG:boto:path=/hcnphz-7/%26 2023-09-29T21:55:35.243 DEBUG:boto:auth_path=/hcnphz-7/%26 2023-09-29T21:55:35.243 DEBUG:boto:Method: GET 2023-09-29T21:55:35.243 DEBUG:boto:Path: /hcnphz-7/%26 2023-09-29T21:55:35.243 DEBUG:boto:Data: 2023-09-29T21:55:35.243 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.243 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.243 DEBUG:boto:Params: {} 2023-09-29T21:55:35.244 DEBUG:boto:Token: None 2023-09-29T21:55:35.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%26 2023-09-29T21:55:35.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rZhBMkwR6hCQJ+JHx+49eEN3k2s= 2023-09-29T21:55:35.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rZhBMkwR6hCQJ+JHx+49eEN3k2s=', 'Content-Length': '0'} 2023-09-29T21:55:35.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f24b030fec43dd4a-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.248 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.248 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.248 DEBUG:boto:path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.248 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.248 DEBUG:boto:Method: GET 2023-09-29T21:55:35.248 DEBUG:boto:Path: /hcnphz-7/?prefix=%26 2023-09-29T21:55:35.248 DEBUG:boto:Data: 2023-09-29T21:55:35.248 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.249 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.249 DEBUG:boto:Params: {} 2023-09-29T21:55:35.249 DEBUG:boto:Token: None 2023-09-29T21:55:35.249 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027cd2639fbdd66e3-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.256 DEBUG:boto:b'hcnphz-7&1000false&2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.256 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.256 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.256 DEBUG:boto:path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.257 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%26 2023-09-29T21:55:35.257 DEBUG:boto:Method: GET 2023-09-29T21:55:35.257 DEBUG:boto:Path: /hcnphz-7/?prefix=%26 2023-09-29T21:55:35.257 DEBUG:boto:Data: 2023-09-29T21:55:35.257 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.257 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.257 DEBUG:boto:Params: {} 2023-09-29T21:55:35.257 DEBUG:boto:Token: None 2023-09-29T21:55:35.258 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.258 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1c1e196187ec179-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.306 DEBUG:boto:b'hcnphz-7&1000false&2023-09-29T21:54:49.275Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T21:55:35.307 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.307 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.307 DEBUG:boto:path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.307 DEBUG:boto:auth_path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.308 DEBUG:boto:Method: GET 2023-09-29T21:55:35.308 DEBUG:boto:Path: /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.308 DEBUG:boto:Data: 2023-09-29T21:55:35.308 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.308 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.308 DEBUG:boto:Params: {} 2023-09-29T21:55:35.308 DEBUG:boto:Token: None 2023-09-29T21:55:35.309 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.309 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg= 2023-09-29T21:55:35.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg=', 'Content-Length': '0'} 2023-09-29T21:55:35.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5217c1b9e334d24-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.312 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.312 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.313 DEBUG:boto:path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.313 DEBUG:boto:auth_path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.313 DEBUG:boto:Method: GET 2023-09-29T21:55:35.313 DEBUG:boto:Path: /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.313 DEBUG:boto:Data: 2023-09-29T21:55:35.313 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.313 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.313 DEBUG:boto:Params: {} 2023-09-29T21:55:35.314 DEBUG:boto:Token: None 2023-09-29T21:55:35.314 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.314 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg= 2023-09-29T21:55:35.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg=', 'Content-Length': '0'} 2023-09-29T21:55:35.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005faaf9eca551ee3c-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.317 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.317 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.317 DEBUG:boto:path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.317 DEBUG:boto:auth_path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.317 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.317 DEBUG:boto:Path: /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.318 DEBUG:boto:Data: 2023-09-29T21:55:35.318 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.318 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.318 DEBUG:boto:Params: {} 2023-09-29T21:55:35.318 DEBUG:boto:Token: None 2023-09-29T21:55:35.318 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.318 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ewU5QjQ7VUBCPGswkQYl3HCkM0E= 2023-09-29T21:55:35.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ewU5QjQ7VUBCPGswkQYl3HCkM0E=', 'Content-Length': '0'} 2023-09-29T21:55:35.321 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dee446643e053bab-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.322 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.322 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.322 DEBUG:boto:path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.322 DEBUG:boto:auth_path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.322 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.323 DEBUG:boto:Path: /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.323 DEBUG:boto:Data: 2023-09-29T21:55:35.323 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.323 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.323 DEBUG:boto:Params: {} 2023-09-29T21:55:35.323 DEBUG:boto:Token: None 2023-09-29T21:55:35.324 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ewU5QjQ7VUBCPGswkQYl3HCkM0E= 2023-09-29T21:55:35.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ewU5QjQ7VUBCPGswkQYl3HCkM0E=', 'Content-Length': '0'} 2023-09-29T21:55:35.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000efd24602ec9f8dff-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T21:55:35.327 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.327 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.327 DEBUG:boto:path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.327 DEBUG:boto:auth_path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.327 DEBUG:boto:Method: GET 2023-09-29T21:55:35.327 DEBUG:boto:Path: /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.327 DEBUG:boto:Data: 2023-09-29T21:55:35.327 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.328 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.328 DEBUG:boto:Params: {} 2023-09-29T21:55:35.328 DEBUG:boto:Token: None 2023-09-29T21:55:35.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg= 2023-09-29T21:55:35.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg=', 'Content-Length': '0'} 2023-09-29T21:55:35.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000093bffe9374c7743b-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.331 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.331 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.332 DEBUG:boto:path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.332 DEBUG:boto:auth_path=/hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.332 DEBUG:boto:Method: GET 2023-09-29T21:55:35.332 DEBUG:boto:Path: /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.332 DEBUG:boto:Data: 2023-09-29T21:55:35.332 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.332 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.332 DEBUG:boto:Params: {} 2023-09-29T21:55:35.332 DEBUG:boto:Token: None 2023-09-29T21:55:35.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A?versionId=null 2023-09-29T21:55:35.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg= 2023-09-29T21:55:35.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UUefjy+LHipFQuI0RAM9uSYHOJg=', 'Content-Length': '0'} 2023-09-29T21:55:35.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004571b92f44a3e269-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.336 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.336 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.336 DEBUG:boto:path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.337 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.337 DEBUG:boto:Method: GET 2023-09-29T21:55:35.337 DEBUG:boto:Path: /hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.337 DEBUG:boto:Data: 2023-09-29T21:55:35.337 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.337 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.337 DEBUG:boto:Params: {} 2023-09-29T21:55:35.337 DEBUG:boto:Token: None 2023-09-29T21:55:35.338 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ce9cf53c0aeac0f-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.390 DEBUG:boto:b'hcnphz-7:1000false:2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.391 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.391 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.391 DEBUG:boto:path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.391 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.391 DEBUG:boto:Method: GET 2023-09-29T21:55:35.391 DEBUG:boto:Path: /hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.391 DEBUG:boto:Data: 2023-09-29T21:55:35.392 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.392 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.392 DEBUG:boto:Params: {} 2023-09-29T21:55:35.392 DEBUG:boto:Token: None 2023-09-29T21:55:35.392 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000366047fefcad07c8-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.398 DEBUG:boto:b'hcnphz-7:1000false:2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.398 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.399 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.399 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.399 DEBUG:boto:Path: /hcnphz-7/%3A 2023-09-29T21:55:35.399 DEBUG:boto:Data: 2023-09-29T21:55:35.399 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.399 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.399 DEBUG:boto:Params: {} 2023-09-29T21:55:35.399 DEBUG:boto:Token: None 2023-09-29T21:55:35.400 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A 2023-09-29T21:55:35.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gA+CFxw1vsKLEbhlPC/kBQdGkPc= 2023-09-29T21:55:35.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gA+CFxw1vsKLEbhlPC/kBQdGkPc=', 'Content-Length': '0'} 2023-09-29T21:55:35.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1f96d9cb5341c31-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.403 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.403 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.404 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.404 DEBUG:boto:Path: /hcnphz-7/%3A 2023-09-29T21:55:35.404 DEBUG:boto:Data: 2023-09-29T21:55:35.404 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.405 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.405 DEBUG:boto:Params: {} 2023-09-29T21:55:35.405 DEBUG:boto:Token: None 2023-09-29T21:55:35.405 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A 2023-09-29T21:55:35.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gA+CFxw1vsKLEbhlPC/kBQdGkPc= 2023-09-29T21:55:35.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gA+CFxw1vsKLEbhlPC/kBQdGkPc=', 'Content-Length': '0'} 2023-09-29T21:55:35.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095828a22046db8ba-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T21:55:35.409 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.409 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.409 DEBUG:boto:Method: GET 2023-09-29T21:55:35.410 DEBUG:boto:Path: /hcnphz-7/%3A 2023-09-29T21:55:35.410 DEBUG:boto:Data: 2023-09-29T21:55:35.410 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.410 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.411 DEBUG:boto:Params: {} 2023-09-29T21:55:35.411 DEBUG:boto:Token: None 2023-09-29T21:55:35.411 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A 2023-09-29T21:55:35.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:58+Z5CpdObcszZJdHVyt7eC8oV8= 2023-09-29T21:55:35.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:58+Z5CpdObcszZJdHVyt7eC8oV8=', 'Content-Length': '0'} 2023-09-29T21:55:35.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af7a4982ccfa0a0d-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.415 DEBUG:boto:path=/hcnphz-7/%3A 2023-09-29T21:55:35.415 DEBUG:boto:auth_path=/hcnphz-7/%3A 2023-09-29T21:55:35.415 DEBUG:boto:Method: GET 2023-09-29T21:55:35.415 DEBUG:boto:Path: /hcnphz-7/%3A 2023-09-29T21:55:35.416 DEBUG:boto:Data: 2023-09-29T21:55:35.416 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.416 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.416 DEBUG:boto:Params: {} 2023-09-29T21:55:35.416 DEBUG:boto:Token: None 2023-09-29T21:55:35.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/%3A 2023-09-29T21:55:35.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:58+Z5CpdObcszZJdHVyt7eC8oV8= 2023-09-29T21:55:35.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:58+Z5CpdObcszZJdHVyt7eC8oV8=', 'Content-Length': '0'} 2023-09-29T21:55:35.419 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000374bbd113da55793-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.462 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.463 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.463 DEBUG:boto:path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.463 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.463 DEBUG:boto:Method: GET 2023-09-29T21:55:35.463 DEBUG:boto:Path: /hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.463 DEBUG:boto:Data: 2023-09-29T21:55:35.463 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.463 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.464 DEBUG:boto:Params: {} 2023-09-29T21:55:35.464 DEBUG:boto:Token: None 2023-09-29T21:55:35.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fd71811257f0067-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.470 DEBUG:boto:b'hcnphz-7:1000false:2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.470 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.471 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.471 DEBUG:boto:path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.471 DEBUG:boto:auth_path=/hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.471 DEBUG:boto:Method: GET 2023-09-29T21:55:35.471 DEBUG:boto:Path: /hcnphz-7/?prefix=%3A 2023-09-29T21:55:35.471 DEBUG:boto:Data: 2023-09-29T21:55:35.471 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.471 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.471 DEBUG:boto:Params: {} 2023-09-29T21:55:35.472 DEBUG:boto:Token: None 2023-09-29T21:55:35.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a28122788dfa5435-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.477 DEBUG:boto:b'hcnphz-7:1000false:2023-09-29T21:54:49.250Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T21:55:35.478 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.478 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.478 DEBUG:boto:path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.478 DEBUG:boto:auth_path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.478 DEBUG:boto:Method: GET 2023-09-29T21:55:35.478 DEBUG:boto:Path: /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.478 DEBUG:boto:Data: 2023-09-29T21:55:35.478 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.479 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.479 DEBUG:boto:Params: {} 2023-09-29T21:55:35.479 DEBUG:boto:Token: None 2023-09-29T21:55:35.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg= 2023-09-29T21:55:35.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg=', 'Content-Length': '0'} 2023-09-29T21:55:35.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d095af68640a9426-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.522 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.523 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.523 DEBUG:boto:path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.523 DEBUG:boto:auth_path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.523 DEBUG:boto:Method: GET 2023-09-29T21:55:35.523 DEBUG:boto:Path: /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.523 DEBUG:boto:Data: 2023-09-29T21:55:35.523 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.523 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.523 DEBUG:boto:Params: {} 2023-09-29T21:55:35.524 DEBUG:boto:Token: None 2023-09-29T21:55:35.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg= 2023-09-29T21:55:35.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg=', 'Content-Length': '0'} 2023-09-29T21:55:35.527 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbac2c105a53fbf6-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.528 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.528 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.528 DEBUG:boto:path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.528 DEBUG:boto:auth_path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.528 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.528 DEBUG:boto:Path: /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.528 DEBUG:boto:Data: 2023-09-29T21:55:35.529 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.529 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.529 DEBUG:boto:Params: {} 2023-09-29T21:55:35.529 DEBUG:boto:Token: None 2023-09-29T21:55:35.529 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Luf3XAI8HzgUDyGMp5qnCcpO0Hw= 2023-09-29T21:55:35.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Luf3XAI8HzgUDyGMp5qnCcpO0Hw=', 'Content-Length': '0'} 2023-09-29T21:55:35.532 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d69c60538e81ff42-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.533 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.533 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.533 DEBUG:boto:path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.533 DEBUG:boto:auth_path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.533 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.533 DEBUG:boto:Path: /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.533 DEBUG:boto:Data: 2023-09-29T21:55:35.533 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.534 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.534 DEBUG:boto:Params: {} 2023-09-29T21:55:35.534 DEBUG:boto:Token: None 2023-09-29T21:55:35.534 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.534 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Luf3XAI8HzgUDyGMp5qnCcpO0Hw= 2023-09-29T21:55:35.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Luf3XAI8HzgUDyGMp5qnCcpO0Hw=', 'Content-Length': '0'} 2023-09-29T21:55:35.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003459f2c1a939e96d-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T21:55:35.537 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.537 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.537 DEBUG:boto:path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.537 DEBUG:boto:auth_path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.537 DEBUG:boto:Method: GET 2023-09-29T21:55:35.538 DEBUG:boto:Path: /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.538 DEBUG:boto:Data: 2023-09-29T21:55:35.538 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.538 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.538 DEBUG:boto:Params: {} 2023-09-29T21:55:35.538 DEBUG:boto:Token: None 2023-09-29T21:55:35.538 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.538 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg= 2023-09-29T21:55:35.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg=', 'Content-Length': '0'} 2023-09-29T21:55:35.541 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b4d359666bee540-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.541 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.541 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.541 DEBUG:boto:path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.541 DEBUG:boto:auth_path=/hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.542 DEBUG:boto:Method: GET 2023-09-29T21:55:35.542 DEBUG:boto:Path: /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.542 DEBUG:boto:Data: 2023-09-29T21:55:35.542 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.542 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.542 DEBUG:boto:Params: {} 2023-09-29T21:55:35.542 DEBUG:boto:Token: None 2023-09-29T21:55:35.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj?versionId=null 2023-09-29T21:55:35.543 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg= 2023-09-29T21:55:35.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MeQtZ8TDiSLu3+GxyjBHo4g5Bgg=', 'Content-Length': '0'} 2023-09-29T21:55:35.545 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4d310a8468cd778-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.545 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.545 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.546 DEBUG:boto:path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.546 DEBUG:boto:auth_path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.546 DEBUG:boto:Method: GET 2023-09-29T21:55:35.546 DEBUG:boto:Path: /hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.546 DEBUG:boto:Data: 2023-09-29T21:55:35.546 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.546 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.546 DEBUG:boto:Params: {} 2023-09-29T21:55:35.546 DEBUG:boto:Token: None 2023-09-29T21:55:35.547 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.547 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e20c0bfa01a01e44-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.551 DEBUG:boto:b'hcnphz-7_myobj1000false_myobj2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.552 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.552 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.552 DEBUG:boto:path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.552 DEBUG:boto:auth_path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.552 DEBUG:boto:Method: GET 2023-09-29T21:55:35.552 DEBUG:boto:Path: /hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.552 DEBUG:boto:Data: 2023-09-29T21:55:35.552 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.553 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.553 DEBUG:boto:Params: {} 2023-09-29T21:55:35.553 DEBUG:boto:Token: None 2023-09-29T21:55:35.553 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000040c9540b4035353a-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.558 DEBUG:boto:b'hcnphz-7_myobj1000false_myobj2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.558 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.559 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.559 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.559 DEBUG:boto:Path: /hcnphz-7/_myobj 2023-09-29T21:55:35.559 DEBUG:boto:Data: 2023-09-29T21:55:35.559 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.559 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.559 DEBUG:boto:Params: {} 2023-09-29T21:55:35.559 DEBUG:boto:Token: None 2023-09-29T21:55:35.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj 2023-09-29T21:55:35.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8WxBzWn4ovhZyeDMFNDfsQrweDI= 2023-09-29T21:55:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8WxBzWn4ovhZyeDMFNDfsQrweDI=', 'Content-Length': '0'} 2023-09-29T21:55:35.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e9b59ed57edeeca-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.562 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.563 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.563 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.563 DEBUG:boto:Path: /hcnphz-7/_myobj 2023-09-29T21:55:35.563 DEBUG:boto:Data: 2023-09-29T21:55:35.563 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.563 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.563 DEBUG:boto:Params: {} 2023-09-29T21:55:35.563 DEBUG:boto:Token: None 2023-09-29T21:55:35.564 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj 2023-09-29T21:55:35.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8WxBzWn4ovhZyeDMFNDfsQrweDI= 2023-09-29T21:55:35.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8WxBzWn4ovhZyeDMFNDfsQrweDI=', 'Content-Length': '0'} 2023-09-29T21:55:35.566 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001ddd9e9d0a3ad250-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T21:55:35.566 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.567 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.567 DEBUG:boto:Method: GET 2023-09-29T21:55:35.567 DEBUG:boto:Path: /hcnphz-7/_myobj 2023-09-29T21:55:35.567 DEBUG:boto:Data: 2023-09-29T21:55:35.567 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.567 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.567 DEBUG:boto:Params: {} 2023-09-29T21:55:35.567 DEBUG:boto:Token: None 2023-09-29T21:55:35.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj 2023-09-29T21:55:35.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gWVAvFT2MuINSDpVhMGmTUDyfa8= 2023-09-29T21:55:35.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gWVAvFT2MuINSDpVhMGmTUDyfa8=', 'Content-Length': '0'} 2023-09-29T21:55:35.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f52576630892618-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.618 DEBUG:boto:path=/hcnphz-7/_myobj 2023-09-29T21:55:35.619 DEBUG:boto:auth_path=/hcnphz-7/_myobj 2023-09-29T21:55:35.619 DEBUG:boto:Method: GET 2023-09-29T21:55:35.619 DEBUG:boto:Path: /hcnphz-7/_myobj 2023-09-29T21:55:35.619 DEBUG:boto:Data: 2023-09-29T21:55:35.619 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.619 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.619 DEBUG:boto:Params: {} 2023-09-29T21:55:35.620 DEBUG:boto:Token: None 2023-09-29T21:55:35.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/_myobj 2023-09-29T21:55:35.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gWVAvFT2MuINSDpVhMGmTUDyfa8= 2023-09-29T21:55:35.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gWVAvFT2MuINSDpVhMGmTUDyfa8=', 'Content-Length': '0'} 2023-09-29T21:55:35.623 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f600034f1dcf937-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.623 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.623 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.623 DEBUG:boto:path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.624 DEBUG:boto:auth_path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.624 DEBUG:boto:Method: GET 2023-09-29T21:55:35.624 DEBUG:boto:Path: /hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.624 DEBUG:boto:Data: 2023-09-29T21:55:35.624 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.624 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.624 DEBUG:boto:Params: {} 2023-09-29T21:55:35.624 DEBUG:boto:Token: None 2023-09-29T21:55:35.625 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.625 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006abf6fa8517af44c-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.630 DEBUG:boto:b'hcnphz-7_myobj1000false_myobj2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.631 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.631 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.631 DEBUG:boto:path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.631 DEBUG:boto:auth_path=/hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.631 DEBUG:boto:Method: GET 2023-09-29T21:55:35.631 DEBUG:boto:Path: /hcnphz-7/?prefix=_myobj 2023-09-29T21:55:35.632 DEBUG:boto:Data: 2023-09-29T21:55:35.632 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.632 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.632 DEBUG:boto:Params: {} 2023-09-29T21:55:35.632 DEBUG:boto:Token: None 2023-09-29T21:55:35.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e91127a4a6426440-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.638 DEBUG:boto:b'hcnphz-7_myobj1000false_myobj2023-09-29T21:54:49.199Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:55:35.638 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.638 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.638 DEBUG:boto:path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.639 DEBUG:boto:auth_path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.639 DEBUG:boto:Method: GET 2023-09-29T21:55:35.639 DEBUG:boto:Path: /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.639 DEBUG:boto:Data: 2023-09-29T21:55:35.639 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.639 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.639 DEBUG:boto:Params: {} 2023-09-29T21:55:35.639 DEBUG:boto:Token: None 2023-09-29T21:55:35.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y= 2023-09-29T21:55:35.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y=', 'Content-Length': '0'} 2023-09-29T21:55:35.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a43f413258a8241d-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.643 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.643 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.643 DEBUG:boto:path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.643 DEBUG:boto:auth_path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.643 DEBUG:boto:Method: GET 2023-09-29T21:55:35.644 DEBUG:boto:Path: /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.644 DEBUG:boto:Data: 2023-09-29T21:55:35.644 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.644 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.644 DEBUG:boto:Params: {} 2023-09-29T21:55:35.644 DEBUG:boto:Token: None 2023-09-29T21:55:35.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y= 2023-09-29T21:55:35.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y=', 'Content-Length': '0'} 2023-09-29T21:55:35.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067053041ef336afe-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.647 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.647 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.647 DEBUG:boto:path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.647 DEBUG:boto:auth_path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.647 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.648 DEBUG:boto:Path: /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.648 DEBUG:boto:Data: 2023-09-29T21:55:35.648 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.648 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.648 DEBUG:boto:Params: {} 2023-09-29T21:55:35.648 DEBUG:boto:Token: None 2023-09-29T21:55:35.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FB6VvyJSHadWaxtbZwj9lknKAb8= 2023-09-29T21:55:35.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FB6VvyJSHadWaxtbZwj9lknKAb8=', 'Content-Length': '0'} 2023-09-29T21:55:35.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003de072e1bdab9e18-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.651 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.651 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.651 DEBUG:boto:path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.651 DEBUG:boto:auth_path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.652 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.652 DEBUG:boto:Path: /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.652 DEBUG:boto:Data: 2023-09-29T21:55:35.652 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.652 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.652 DEBUG:boto:Params: {} 2023-09-29T21:55:35.652 DEBUG:boto:Token: None 2023-09-29T21:55:35.652 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FB6VvyJSHadWaxtbZwj9lknKAb8= 2023-09-29T21:55:35.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FB6VvyJSHadWaxtbZwj9lknKAb8=', 'Content-Length': '0'} 2023-09-29T21:55:35.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043f9cae8f90b2eac-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:55:35.656 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.656 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.656 DEBUG:boto:path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.656 DEBUG:boto:auth_path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.656 DEBUG:boto:Method: GET 2023-09-29T21:55:35.656 DEBUG:boto:Path: /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.656 DEBUG:boto:Data: 2023-09-29T21:55:35.657 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.657 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.657 DEBUG:boto:Params: {} 2023-09-29T21:55:35.657 DEBUG:boto:Token: None 2023-09-29T21:55:35.657 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y= 2023-09-29T21:55:35.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y=', 'Content-Length': '0'} 2023-09-29T21:55:35.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000914177e0fd472a93-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.706 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.707 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.707 DEBUG:boto:path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.707 DEBUG:boto:auth_path=/hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.707 DEBUG:boto:Method: GET 2023-09-29T21:55:35.707 DEBUG:boto:Path: /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.707 DEBUG:boto:Data: 2023-09-29T21:55:35.707 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.708 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.708 DEBUG:boto:Params: {} 2023-09-29T21:55:35.708 DEBUG:boto:Token: None 2023-09-29T21:55:35.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj?versionId=null 2023-09-29T21:55:35.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y= 2023-09-29T21:55:35.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HCFse4E5FN/TPwi42aPH8K5RX/Y=', 'Content-Length': '0'} 2023-09-29T21:55:35.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bf6bcbef7ea0332-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.711 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.711 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.711 DEBUG:boto:path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.712 DEBUG:boto:auth_path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.712 DEBUG:boto:Method: GET 2023-09-29T21:55:35.712 DEBUG:boto:Path: /hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.712 DEBUG:boto:Data: 2023-09-29T21:55:35.712 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.712 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.712 DEBUG:boto:Params: {} 2023-09-29T21:55:35.712 DEBUG:boto:Token: None 2023-09-29T21:55:35.713 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005a6082b582ca0c43-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.719 DEBUG:boto:b'hcnphz-7myobj1000falsemyobj2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.719 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.720 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.720 DEBUG:boto:path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.720 DEBUG:boto:auth_path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.720 DEBUG:boto:Method: GET 2023-09-29T21:55:35.720 DEBUG:boto:Path: /hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.720 DEBUG:boto:Data: 2023-09-29T21:55:35.720 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.720 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.720 DEBUG:boto:Params: {} 2023-09-29T21:55:35.720 DEBUG:boto:Token: None 2023-09-29T21:55:35.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aecec5ea57264bc-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.725 DEBUG:boto:b'hcnphz-7myobj1000falsemyobj2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.726 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.726 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.726 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.726 DEBUG:boto:Path: /hcnphz-7/myobj 2023-09-29T21:55:35.726 DEBUG:boto:Data: 2023-09-29T21:55:35.726 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.727 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.727 DEBUG:boto:Params: {} 2023-09-29T21:55:35.727 DEBUG:boto:Token: None 2023-09-29T21:55:35.727 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj 2023-09-29T21:55:35.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kOvbyf5CSxJqW/By7tb+duQSqMU= 2023-09-29T21:55:35.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kOvbyf5CSxJqW/By7tb+duQSqMU=', 'Content-Length': '0'} 2023-09-29T21:55:35.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c25ca0f29525f573-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.730 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.730 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.730 DEBUG:boto:Method: HEAD 2023-09-29T21:55:35.730 DEBUG:boto:Path: /hcnphz-7/myobj 2023-09-29T21:55:35.730 DEBUG:boto:Data: 2023-09-29T21:55:35.731 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.731 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.731 DEBUG:boto:Params: {} 2023-09-29T21:55:35.731 DEBUG:boto:Token: None 2023-09-29T21:55:35.731 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj 2023-09-29T21:55:35.731 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kOvbyf5CSxJqW/By7tb+duQSqMU= 2023-09-29T21:55:35.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kOvbyf5CSxJqW/By7tb+duQSqMU=', 'Content-Length': '0'} 2023-09-29T21:55:35.734 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bda573ee7c42c15-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:55:35.734 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.734 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.734 DEBUG:boto:Method: GET 2023-09-29T21:55:35.734 DEBUG:boto:Path: /hcnphz-7/myobj 2023-09-29T21:55:35.735 DEBUG:boto:Data: 2023-09-29T21:55:35.735 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.735 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.735 DEBUG:boto:Params: {} 2023-09-29T21:55:35.735 DEBUG:boto:Token: None 2023-09-29T21:55:35.735 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj 2023-09-29T21:55:35.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6Y8g499t6Jgom6dPbpkwAkLBwCQ= 2023-09-29T21:55:35.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6Y8g499t6Jgom6dPbpkwAkLBwCQ=', 'Content-Length': '0'} 2023-09-29T21:55:35.738 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c78492a2836f92c-00651747d7-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.738 DEBUG:boto:path=/hcnphz-7/myobj 2023-09-29T21:55:35.739 DEBUG:boto:auth_path=/hcnphz-7/myobj 2023-09-29T21:55:35.739 DEBUG:boto:Method: GET 2023-09-29T21:55:35.739 DEBUG:boto:Path: /hcnphz-7/myobj 2023-09-29T21:55:35.739 DEBUG:boto:Data: 2023-09-29T21:55:35.739 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.739 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.739 DEBUG:boto:Params: {} 2023-09-29T21:55:35.739 DEBUG:boto:Token: None 2023-09-29T21:55:35.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/myobj 2023-09-29T21:55:35.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6Y8g499t6Jgom6dPbpkwAkLBwCQ= 2023-09-29T21:55:35.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6Y8g499t6Jgom6dPbpkwAkLBwCQ=', 'Content-Length': '0'} 2023-09-29T21:55:35.742 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ef5cffd1c7ef450-00651747d7-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.743 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.743 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.743 DEBUG:boto:path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.743 DEBUG:boto:auth_path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.743 DEBUG:boto:Method: GET 2023-09-29T21:55:35.743 DEBUG:boto:Path: /hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.743 DEBUG:boto:Data: 2023-09-29T21:55:35.744 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:55:35.744 DEBUG:boto:Port: 8001 2023-09-29T21:55:35.744 DEBUG:boto:Params: {} 2023-09-29T21:55:35.744 DEBUG:boto:Token: None 2023-09-29T21:55:35.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6221371f5cfac8d-00651747d7-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.790 DEBUG:boto:b'hcnphz-7myobj1000falsemyobj2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.791 DEBUG:boto:path=/hcnphz-7/ 2023-09-29T21:55:35.791 DEBUG:boto:auth_path=/hcnphz-7/ 2023-09-29T21:55:35.791 DEBUG:boto:path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.791 DEBUG:boto:auth_path=/hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.791 DEBUG:boto:Method: GET 2023-09-29T21:55:35.791 DEBUG:boto:Path: /hcnphz-7/?prefix=myobj 2023-09-29T21:55:35.792 DEBUG:boto:Data: 2023-09-29T21:55:35.792 DEBUG:boto:Headers: {} 2023-09-29T21:55:35.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:55:35.792 DEBUG:boto:Port: 8000 2023-09-29T21:55:35.792 DEBUG:boto:Params: {} 2023-09-29T21:55:35.792 DEBUG:boto:Token: None 2023-09-29T21:55:35.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:55:35 GMT /hcnphz-7/ 2023-09-29T21:55:35.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc= 2023-09-29T21:55:35.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:55:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpEAdSGMVQpbhOgHF21MrOD24oc=', 'Content-Length': '0'} 2023-09-29T21:55:35.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdfae8314f5c4976-00651747d7-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:55:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:55:35.798 DEBUG:boto:b'hcnphz-7myobj1000falsemyobj2023-09-29T21:54:49.181Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:55:35.799 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-7 zones={a2, a1} 2023-09-29T21:55:35.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:55:35.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-8', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:55:35.799 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-8 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:55:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:55:35.850 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:55:35.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:35.909+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:40.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:40.909+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:45.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:45.913+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:50.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:50.917+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:55:55.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:55:55.917+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:00.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:00.917+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:05.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:05.917+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:10.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:10.920+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:15.921 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:15.920+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:20.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:20.920+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:25.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:25.920+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:30.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:30.924+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:35.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:35.924+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:40.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:40.924+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:45.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:45.928+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:50.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:50.928+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:56:55.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:56:55.927+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:00.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:00.927+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:05.931 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:05.927+0000 7f35d9032f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:10.935 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:10.931+0000 7f35d9032f40 1 bucket sync caught up with source: 2023-09-29T21:57:10.935 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.8.6, , 00000000001.6.6, , , , 00000000001.12.6, 00000000001.7.6, , ] 2023-09-29T21:57:10.935 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.8.6, , 00000000001.6.6, , , , 00000000001.12.6, 00000000001.7.6, , ] 2023-09-29T21:57:10.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:10.931+0000 7f35d9032f40 0 bucket checkpoint complete 2023-09-29T21:57:10.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-8 zones={a1, a2} 2023-09-29T21:57:10.961 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:10.961 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:10.962 DEBUG:boto:Method: HEAD 2023-09-29T21:57:10.962 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:57:10.962 DEBUG:boto:Data: 2023-09-29T21:57:10.962 DEBUG:boto:Headers: {} 2023-09-29T21:57:10.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:10.962 DEBUG:boto:Port: 8000 2023-09-29T21:57:10.962 DEBUG:boto:Params: {} 2023-09-29T21:57:10.963 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T21:57:10.963 DEBUG:boto:Token: None 2023-09-29T21:57:10.963 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:10 GMT /hcnphz-8/ 2023-09-29T21:57:10.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qlMoXlXRuWXayWkeufiPSQtTWU= 2023-09-29T21:57:10.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qlMoXlXRuWXayWkeufiPSQtTWU=', 'Content-Length': '0'} 2023-09-29T21:57:10.969 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005e9bf465a7263f20-0065174836-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:10.969 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:10.970 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:10.970 DEBUG:boto:Method: HEAD 2023-09-29T21:57:10.970 DEBUG:boto:Path: /hcnphz-8/ 2023-09-29T21:57:10.970 DEBUG:boto:Data: 2023-09-29T21:57:10.970 DEBUG:boto:Headers: {} 2023-09-29T21:57:10.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:10.970 DEBUG:boto:Port: 8001 2023-09-29T21:57:10.971 DEBUG:boto:Params: {} 2023-09-29T21:57:10.971 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T21:57:10.971 DEBUG:boto:Token: None 2023-09-29T21:57:10.971 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:10 GMT /hcnphz-8/ 2023-09-29T21:57:10.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qlMoXlXRuWXayWkeufiPSQtTWU= 2023-09-29T21:57:10.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qlMoXlXRuWXayWkeufiPSQtTWU=', 'Content-Length': '0'} 2023-09-29T21:57:10.977 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068cf6958eedeb27a-0065174836-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:10.977 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T21:57:10.977 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:10.977 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:10.977 DEBUG:boto:path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.977 DEBUG:boto:auth_path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.978 DEBUG:boto:Method: GET 2023-09-29T21:57:10.978 DEBUG:boto:Path: /hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.978 DEBUG:boto:Data: 2023-09-29T21:57:10.978 DEBUG:boto:Headers: {} 2023-09-29T21:57:10.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:10.978 DEBUG:boto:Port: 8000 2023-09-29T21:57:10.978 DEBUG:boto:Params: {} 2023-09-29T21:57:10.979 DEBUG:boto:Token: None 2023-09-29T21:57:10.979 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:10 GMT /hcnphz-8/?versions 2023-09-29T21:57:10.979 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QLhExW8JefgM8TZR91UmGJ31rI4= 2023-09-29T21:57:10.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QLhExW8JefgM8TZR91UmGJ31rI4=', 'Content-Length': '0'} 2023-09-29T21:57:10.983 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea97fdc696cfb1a5-0065174836-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:10.984 DEBUG:boto:b'hcnphz-8999false&nulltrue2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:10.984 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T21:57:10.985 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T21:57:10.985 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T21:57:10.985 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:57:10.985 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T21:57:10.985 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:10.985 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:10.985 DEBUG:boto:path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.985 DEBUG:boto:auth_path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.986 DEBUG:boto:Method: GET 2023-09-29T21:57:10.986 DEBUG:boto:Path: /hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.986 DEBUG:boto:Data: 2023-09-29T21:57:10.986 DEBUG:boto:Headers: {} 2023-09-29T21:57:10.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:10.986 DEBUG:boto:Port: 8001 2023-09-29T21:57:10.986 DEBUG:boto:Params: {} 2023-09-29T21:57:10.986 DEBUG:boto:Token: None 2023-09-29T21:57:10.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:10 GMT /hcnphz-8/?versions 2023-09-29T21:57:10.987 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QLhExW8JefgM8TZR91UmGJ31rI4= 2023-09-29T21:57:10.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QLhExW8JefgM8TZR91UmGJ31rI4=', 'Content-Length': '0'} 2023-09-29T21:57:10.991 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cef5b6bf8fa6933-0065174836-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:10.991 DEBUG:boto:b'hcnphz-8999false&nulltrue2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:10.991 DEBUG:tasks.rgw_multi.zone_rados:o=& 2023-09-29T21:57:10.991 DEBUG:tasks.rgw_multi.zone_rados:o=: 2023-09-29T21:57:10.992 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2023-09-29T21:57:10.992 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:57:10.992 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:10.992 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:10.992 DEBUG:boto:path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.992 DEBUG:boto:auth_path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.992 DEBUG:boto:Method: GET 2023-09-29T21:57:10.992 DEBUG:boto:Path: /hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.993 DEBUG:boto:Data: 2023-09-29T21:57:10.993 DEBUG:boto:Headers: {} 2023-09-29T21:57:10.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:10.993 DEBUG:boto:Port: 8000 2023-09-29T21:57:10.993 DEBUG:boto:Params: {} 2023-09-29T21:57:10.993 DEBUG:boto:Token: None 2023-09-29T21:57:10.994 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:10 GMT /hcnphz-8/?versions 2023-09-29T21:57:10.994 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QLhExW8JefgM8TZR91UmGJ31rI4= 2023-09-29T21:57:10.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QLhExW8JefgM8TZR91UmGJ31rI4=', 'Content-Length': '0'} 2023-09-29T21:57:10.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d84c0a7d7007ba-0065174836-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:10.998 DEBUG:boto:b'hcnphz-8999false&nulltrue2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:10.999 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:10.999 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:10.999 DEBUG:boto:path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.999 DEBUG:boto:auth_path=/hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.999 DEBUG:boto:Method: GET 2023-09-29T21:57:10.999 DEBUG:boto:Path: /hcnphz-8/?versions&max-keys=999 2023-09-29T21:57:10.999 DEBUG:boto:Data: 2023-09-29T21:57:10.999 DEBUG:boto:Headers: {} 2023-09-29T21:57:10.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.000 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.000 DEBUG:boto:Params: {} 2023-09-29T21:57:11.000 DEBUG:boto:Token: None 2023-09-29T21:57:11.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/?versions 2023-09-29T21:57:11.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tu6JjgYIHZwMmhxJl6U/hBt/doc= 2023-09-29T21:57:11.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tu6JjgYIHZwMmhxJl6U/hBt/doc=', 'Content-Length': '0'} 2023-09-29T21:57:11.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007daaeaa04979bd0a-0065174836-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.004 DEBUG:boto:b'hcnphz-8999false&nulltrue2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal:nulltrue2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal_myobjnulltrue2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalmyobjnulltrue2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.005 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T21:57:11.005 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.005 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.005 DEBUG:boto:path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.005 DEBUG:boto:auth_path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.005 DEBUG:boto:Method: GET 2023-09-29T21:57:11.005 DEBUG:boto:Path: /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.006 DEBUG:boto:Data: 2023-09-29T21:57:11.006 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.006 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.006 DEBUG:boto:Params: {} 2023-09-29T21:57:11.006 DEBUG:boto:Token: None 2023-09-29T21:57:11.006 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg= 2023-09-29T21:57:11.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg=', 'Content-Length': '0'} 2023-09-29T21:57:11.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fa153450a3ea6512-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.009 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.010 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.010 DEBUG:boto:path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.010 DEBUG:boto:auth_path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.010 DEBUG:boto:Method: GET 2023-09-29T21:57:11.010 DEBUG:boto:Path: /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.010 DEBUG:boto:Data: 2023-09-29T21:57:11.010 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.011 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.011 DEBUG:boto:Params: {} 2023-09-29T21:57:11.011 DEBUG:boto:Token: None 2023-09-29T21:57:11.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg= 2023-09-29T21:57:11.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg=', 'Content-Length': '0'} 2023-09-29T21:57:11.014 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d75a380a6ac4ffeb-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.054 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.055 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.055 DEBUG:boto:path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.055 DEBUG:boto:auth_path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.055 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.055 DEBUG:boto:Path: /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.055 DEBUG:boto:Data: 2023-09-29T21:57:11.055 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.056 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.056 DEBUG:boto:Params: {} 2023-09-29T21:57:11.056 DEBUG:boto:Token: None 2023-09-29T21:57:11.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l3ich5dhr8jlvByexYuH+QHPDA0= 2023-09-29T21:57:11.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l3ich5dhr8jlvByexYuH+QHPDA0=', 'Content-Length': '0'} 2023-09-29T21:57:11.059 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062a7f5bbf74de9c8-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.059 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.060 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.060 DEBUG:boto:path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.060 DEBUG:boto:auth_path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.060 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.060 DEBUG:boto:Path: /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.060 DEBUG:boto:Data: 2023-09-29T21:57:11.060 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.060 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.061 DEBUG:boto:Params: {} 2023-09-29T21:57:11.061 DEBUG:boto:Token: None 2023-09-29T21:57:11.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l3ich5dhr8jlvByexYuH+QHPDA0= 2023-09-29T21:57:11.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l3ich5dhr8jlvByexYuH+QHPDA0=', 'Content-Length': '0'} 2023-09-29T21:57:11.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000722599963c782655-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T21:57:11.064 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.064 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.064 DEBUG:boto:path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.065 DEBUG:boto:auth_path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.065 DEBUG:boto:Method: GET 2023-09-29T21:57:11.065 DEBUG:boto:Path: /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.065 DEBUG:boto:Data: 2023-09-29T21:57:11.065 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.065 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.065 DEBUG:boto:Params: {} 2023-09-29T21:57:11.065 DEBUG:boto:Token: None 2023-09-29T21:57:11.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg= 2023-09-29T21:57:11.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg=', 'Content-Length': '0'} 2023-09-29T21:57:11.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b802259e49f500e-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.069 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.069 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.069 DEBUG:boto:path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.069 DEBUG:boto:auth_path=/hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.069 DEBUG:boto:Method: GET 2023-09-29T21:57:11.069 DEBUG:boto:Path: /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.069 DEBUG:boto:Data: 2023-09-29T21:57:11.069 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.070 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.070 DEBUG:boto:Params: {} 2023-09-29T21:57:11.070 DEBUG:boto:Token: None 2023-09-29T21:57:11.070 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26?versionId=null 2023-09-29T21:57:11.070 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg= 2023-09-29T21:57:11.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gkj3z4OCvqf25XXUsDb7AsbfBTg=', 'Content-Length': '0'} 2023-09-29T21:57:11.073 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c9119e05a2f7dcd2-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.073 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.074 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.074 DEBUG:boto:path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.074 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.074 DEBUG:boto:Method: GET 2023-09-29T21:57:11.074 DEBUG:boto:Path: /hcnphz-8/?prefix=%26 2023-09-29T21:57:11.074 DEBUG:boto:Data: 2023-09-29T21:57:11.074 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.074 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.075 DEBUG:boto:Params: {} 2023-09-29T21:57:11.075 DEBUG:boto:Token: None 2023-09-29T21:57:11.075 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.075 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.079 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0a66eca3ae6b7d-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.122 DEBUG:boto:b'hcnphz-8&1000false&2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.123 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.123 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.123 DEBUG:boto:path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.123 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.123 DEBUG:boto:Method: GET 2023-09-29T21:57:11.124 DEBUG:boto:Path: /hcnphz-8/?prefix=%26 2023-09-29T21:57:11.124 DEBUG:boto:Data: 2023-09-29T21:57:11.124 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.124 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.124 DEBUG:boto:Params: {} 2023-09-29T21:57:11.124 DEBUG:boto:Token: None 2023-09-29T21:57:11.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.125 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e6efa6e163b504f-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.130 DEBUG:boto:b'hcnphz-8&1000false&2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.130 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.131 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.131 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.131 DEBUG:boto:Path: /hcnphz-8/%26 2023-09-29T21:57:11.131 DEBUG:boto:Data: 2023-09-29T21:57:11.131 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.132 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.132 DEBUG:boto:Params: {} 2023-09-29T21:57:11.132 DEBUG:boto:Token: None 2023-09-29T21:57:11.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26 2023-09-29T21:57:11.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y1SzelDSntJSKEbbxUF07EVAPjs= 2023-09-29T21:57:11.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y1SzelDSntJSKEbbxUF07EVAPjs=', 'Content-Length': '0'} 2023-09-29T21:57:11.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035d5af53687d2327-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.135 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.135 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.135 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.135 DEBUG:boto:Path: /hcnphz-8/%26 2023-09-29T21:57:11.136 DEBUG:boto:Data: 2023-09-29T21:57:11.136 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.136 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.136 DEBUG:boto:Params: {} 2023-09-29T21:57:11.136 DEBUG:boto:Token: None 2023-09-29T21:57:11.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26 2023-09-29T21:57:11.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y1SzelDSntJSKEbbxUF07EVAPjs= 2023-09-29T21:57:11.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y1SzelDSntJSKEbbxUF07EVAPjs=', 'Content-Length': '0'} 2023-09-29T21:57:11.139 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaafe734da2bb097-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2023-09-29T21:57:11.139 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.140 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.140 DEBUG:boto:Method: GET 2023-09-29T21:57:11.140 DEBUG:boto:Path: /hcnphz-8/%26 2023-09-29T21:57:11.140 DEBUG:boto:Data: 2023-09-29T21:57:11.140 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.140 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.140 DEBUG:boto:Params: {} 2023-09-29T21:57:11.140 DEBUG:boto:Token: None 2023-09-29T21:57:11.141 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26 2023-09-29T21:57:11.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mVKBIQ7UUV9DSWNLCj3MPMV1774= 2023-09-29T21:57:11.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mVKBIQ7UUV9DSWNLCj3MPMV1774=', 'Content-Length': '0'} 2023-09-29T21:57:11.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2d1f6bb41f65b60-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.144 DEBUG:boto:path=/hcnphz-8/%26 2023-09-29T21:57:11.144 DEBUG:boto:auth_path=/hcnphz-8/%26 2023-09-29T21:57:11.144 DEBUG:boto:Method: GET 2023-09-29T21:57:11.145 DEBUG:boto:Path: /hcnphz-8/%26 2023-09-29T21:57:11.145 DEBUG:boto:Data: 2023-09-29T21:57:11.145 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.145 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.145 DEBUG:boto:Params: {} 2023-09-29T21:57:11.145 DEBUG:boto:Token: None 2023-09-29T21:57:11.145 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%26 2023-09-29T21:57:11.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mVKBIQ7UUV9DSWNLCj3MPMV1774= 2023-09-29T21:57:11.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mVKBIQ7UUV9DSWNLCj3MPMV1774=', 'Content-Length': '0'} 2023-09-29T21:57:11.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004893e96bab34cc9d-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.194 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.195 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.195 DEBUG:boto:path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.195 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.195 DEBUG:boto:Method: GET 2023-09-29T21:57:11.195 DEBUG:boto:Path: /hcnphz-8/?prefix=%26 2023-09-29T21:57:11.195 DEBUG:boto:Data: 2023-09-29T21:57:11.195 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.195 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.196 DEBUG:boto:Params: {} 2023-09-29T21:57:11.196 DEBUG:boto:Token: None 2023-09-29T21:57:11.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda8ea5422c07e31-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.201 DEBUG:boto:b'hcnphz-8&1000false&2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.201 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.201 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.202 DEBUG:boto:path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.202 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%26 2023-09-29T21:57:11.202 DEBUG:boto:Method: GET 2023-09-29T21:57:11.202 DEBUG:boto:Path: /hcnphz-8/?prefix=%26 2023-09-29T21:57:11.202 DEBUG:boto:Data: 2023-09-29T21:57:11.202 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.202 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.202 DEBUG:boto:Params: {} 2023-09-29T21:57:11.203 DEBUG:boto:Token: None 2023-09-29T21:57:11.203 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.203 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.207 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdf98f261e67eb47-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.208 DEBUG:boto:b'hcnphz-8&1000false&2023-09-29T21:54:49.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T21:57:11.208 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.208 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.208 DEBUG:boto:path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.208 DEBUG:boto:auth_path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.209 DEBUG:boto:Method: GET 2023-09-29T21:57:11.209 DEBUG:boto:Path: /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.209 DEBUG:boto:Data: 2023-09-29T21:57:11.209 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.209 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.209 DEBUG:boto:Params: {} 2023-09-29T21:57:11.209 DEBUG:boto:Token: None 2023-09-29T21:57:11.210 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.210 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k= 2023-09-29T21:57:11.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k=', 'Content-Length': '0'} 2023-09-29T21:57:11.212 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab96a66730b58bf7-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.254 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.255 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.255 DEBUG:boto:path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.255 DEBUG:boto:auth_path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.255 DEBUG:boto:Method: GET 2023-09-29T21:57:11.255 DEBUG:boto:Path: /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.255 DEBUG:boto:Data: 2023-09-29T21:57:11.255 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.255 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.256 DEBUG:boto:Params: {} 2023-09-29T21:57:11.256 DEBUG:boto:Token: None 2023-09-29T21:57:11.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k= 2023-09-29T21:57:11.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k=', 'Content-Length': '0'} 2023-09-29T21:57:11.260 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffde1a301d9e3c1c-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.260 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.260 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.260 DEBUG:boto:path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.260 DEBUG:boto:auth_path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.261 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.261 DEBUG:boto:Path: /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.261 DEBUG:boto:Data: 2023-09-29T21:57:11.261 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.261 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.261 DEBUG:boto:Params: {} 2023-09-29T21:57:11.261 DEBUG:boto:Token: None 2023-09-29T21:57:11.261 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hlSejBozao6TtSqIyL4lCJQtR8w= 2023-09-29T21:57:11.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hlSejBozao6TtSqIyL4lCJQtR8w=', 'Content-Length': '0'} 2023-09-29T21:57:11.264 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7135760515172f7-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.264 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.265 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.265 DEBUG:boto:path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.265 DEBUG:boto:auth_path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.265 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.265 DEBUG:boto:Path: /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.265 DEBUG:boto:Data: 2023-09-29T21:57:11.265 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.265 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.266 DEBUG:boto:Params: {} 2023-09-29T21:57:11.266 DEBUG:boto:Token: None 2023-09-29T21:57:11.266 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.266 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hlSejBozao6TtSqIyL4lCJQtR8w= 2023-09-29T21:57:11.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hlSejBozao6TtSqIyL4lCJQtR8w=', 'Content-Length': '0'} 2023-09-29T21:57:11.269 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd606a9b94707b21-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T21:57:11.269 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.269 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.269 DEBUG:boto:path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.269 DEBUG:boto:auth_path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.270 DEBUG:boto:Method: GET 2023-09-29T21:57:11.270 DEBUG:boto:Path: /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.270 DEBUG:boto:Data: 2023-09-29T21:57:11.270 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.270 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.270 DEBUG:boto:Params: {} 2023-09-29T21:57:11.270 DEBUG:boto:Token: None 2023-09-29T21:57:11.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k= 2023-09-29T21:57:11.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k=', 'Content-Length': '0'} 2023-09-29T21:57:11.274 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bd7cc4b89ebe5808-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.274 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.274 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.274 DEBUG:boto:path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.274 DEBUG:boto:auth_path=/hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.274 DEBUG:boto:Method: GET 2023-09-29T21:57:11.274 DEBUG:boto:Path: /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.275 DEBUG:boto:Data: 2023-09-29T21:57:11.275 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.275 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.275 DEBUG:boto:Params: {} 2023-09-29T21:57:11.275 DEBUG:boto:Token: None 2023-09-29T21:57:11.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A?versionId=null 2023-09-29T21:57:11.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k= 2023-09-29T21:57:11.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:82yYZs/AAP+FbZ3EMKe8flxGz8k=', 'Content-Length': '0'} 2023-09-29T21:57:11.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000daa99b837f9eaf4a-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.279 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.279 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.279 DEBUG:boto:path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.279 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.279 DEBUG:boto:Method: GET 2023-09-29T21:57:11.279 DEBUG:boto:Path: /hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.279 DEBUG:boto:Data: 2023-09-29T21:57:11.279 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.280 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.280 DEBUG:boto:Params: {} 2023-09-29T21:57:11.280 DEBUG:boto:Token: None 2023-09-29T21:57:11.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000338a4a3629f3cdc5-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.285 DEBUG:boto:b'hcnphz-8:1000false:2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.286 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.286 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.286 DEBUG:boto:path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.286 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.286 DEBUG:boto:Method: GET 2023-09-29T21:57:11.286 DEBUG:boto:Path: /hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.287 DEBUG:boto:Data: 2023-09-29T21:57:11.287 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.287 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.287 DEBUG:boto:Params: {} 2023-09-29T21:57:11.287 DEBUG:boto:Token: None 2023-09-29T21:57:11.287 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbb5476a45f59623-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.338 DEBUG:boto:b'hcnphz-8:1000false:2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.339 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.339 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.339 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.339 DEBUG:boto:Path: /hcnphz-8/%3A 2023-09-29T21:57:11.339 DEBUG:boto:Data: 2023-09-29T21:57:11.340 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.340 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.340 DEBUG:boto:Params: {} 2023-09-29T21:57:11.340 DEBUG:boto:Token: None 2023-09-29T21:57:11.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A 2023-09-29T21:57:11.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x4+UOammLHg2jUnsVYvQkhBvMBg= 2023-09-29T21:57:11.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x4+UOammLHg2jUnsVYvQkhBvMBg=', 'Content-Length': '0'} 2023-09-29T21:57:11.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb62a1ce23523e97-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.343 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.344 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.344 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.344 DEBUG:boto:Path: /hcnphz-8/%3A 2023-09-29T21:57:11.344 DEBUG:boto:Data: 2023-09-29T21:57:11.344 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.344 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.344 DEBUG:boto:Params: {} 2023-09-29T21:57:11.344 DEBUG:boto:Token: None 2023-09-29T21:57:11.345 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A 2023-09-29T21:57:11.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x4+UOammLHg2jUnsVYvQkhBvMBg= 2023-09-29T21:57:11.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x4+UOammLHg2jUnsVYvQkhBvMBg=', 'Content-Length': '0'} 2023-09-29T21:57:11.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b11e308ca594b0fa-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2023-09-29T21:57:11.348 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.348 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.348 DEBUG:boto:Method: GET 2023-09-29T21:57:11.348 DEBUG:boto:Path: /hcnphz-8/%3A 2023-09-29T21:57:11.348 DEBUG:boto:Data: 2023-09-29T21:57:11.348 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.349 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.349 DEBUG:boto:Params: {} 2023-09-29T21:57:11.349 DEBUG:boto:Token: None 2023-09-29T21:57:11.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A 2023-09-29T21:57:11.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J4gkher1Tf+d7GT7m40jNcUrdG0= 2023-09-29T21:57:11.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J4gkher1Tf+d7GT7m40jNcUrdG0=', 'Content-Length': '0'} 2023-09-29T21:57:11.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077078d0345594fd9-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.352 DEBUG:boto:path=/hcnphz-8/%3A 2023-09-29T21:57:11.352 DEBUG:boto:auth_path=/hcnphz-8/%3A 2023-09-29T21:57:11.352 DEBUG:boto:Method: GET 2023-09-29T21:57:11.352 DEBUG:boto:Path: /hcnphz-8/%3A 2023-09-29T21:57:11.353 DEBUG:boto:Data: 2023-09-29T21:57:11.353 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.353 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.353 DEBUG:boto:Params: {} 2023-09-29T21:57:11.353 DEBUG:boto:Token: None 2023-09-29T21:57:11.353 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/%3A 2023-09-29T21:57:11.353 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J4gkher1Tf+d7GT7m40jNcUrdG0= 2023-09-29T21:57:11.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J4gkher1Tf+d7GT7m40jNcUrdG0=', 'Content-Length': '0'} 2023-09-29T21:57:11.356 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a3026a3b5b3811b-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.356 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.357 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.357 DEBUG:boto:path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.357 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.357 DEBUG:boto:Method: GET 2023-09-29T21:57:11.357 DEBUG:boto:Path: /hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.357 DEBUG:boto:Data: 2023-09-29T21:57:11.357 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.357 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.358 DEBUG:boto:Params: {} 2023-09-29T21:57:11.358 DEBUG:boto:Token: None 2023-09-29T21:57:11.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f30b668e16ca1a7b-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.410 DEBUG:boto:b'hcnphz-8:1000false:2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.411 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.411 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.411 DEBUG:boto:path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.411 DEBUG:boto:auth_path=/hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.411 DEBUG:boto:Method: GET 2023-09-29T21:57:11.412 DEBUG:boto:Path: /hcnphz-8/?prefix=%3A 2023-09-29T21:57:11.412 DEBUG:boto:Data: 2023-09-29T21:57:11.412 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.412 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.412 DEBUG:boto:Params: {} 2023-09-29T21:57:11.412 DEBUG:boto:Token: None 2023-09-29T21:57:11.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001fba9d3aaa670640-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.418 DEBUG:boto:b'hcnphz-8:1000false:2023-09-29T21:54:49.408Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T21:57:11.418 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.418 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.419 DEBUG:boto:path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.419 DEBUG:boto:auth_path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.419 DEBUG:boto:Method: GET 2023-09-29T21:57:11.419 DEBUG:boto:Path: /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.419 DEBUG:boto:Data: 2023-09-29T21:57:11.419 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.419 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.420 DEBUG:boto:Params: {} 2023-09-29T21:57:11.420 DEBUG:boto:Token: None 2023-09-29T21:57:11.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4= 2023-09-29T21:57:11.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4=', 'Content-Length': '0'} 2023-09-29T21:57:11.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000055ea620fc62852-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.423 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.424 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.424 DEBUG:boto:path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.424 DEBUG:boto:auth_path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.424 DEBUG:boto:Method: GET 2023-09-29T21:57:11.424 DEBUG:boto:Path: /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.424 DEBUG:boto:Data: 2023-09-29T21:57:11.425 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.425 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.425 DEBUG:boto:Params: {} 2023-09-29T21:57:11.425 DEBUG:boto:Token: None 2023-09-29T21:57:11.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4= 2023-09-29T21:57:11.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4=', 'Content-Length': '0'} 2023-09-29T21:57:11.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000707b17cf80c5cc19-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.470 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.471 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.471 DEBUG:boto:path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.471 DEBUG:boto:auth_path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.471 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.471 DEBUG:boto:Path: /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.471 DEBUG:boto:Data: 2023-09-29T21:57:11.471 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.472 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.472 DEBUG:boto:Params: {} 2023-09-29T21:57:11.472 DEBUG:boto:Token: None 2023-09-29T21:57:11.472 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pw2J+jpL3JYEEQE8/3RGdKPIJfk= 2023-09-29T21:57:11.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pw2J+jpL3JYEEQE8/3RGdKPIJfk=', 'Content-Length': '0'} 2023-09-29T21:57:11.475 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acafa524cc4a285e-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.476 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.476 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.476 DEBUG:boto:path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.476 DEBUG:boto:auth_path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.476 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.476 DEBUG:boto:Path: /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.476 DEBUG:boto:Data: 2023-09-29T21:57:11.477 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.477 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.477 DEBUG:boto:Params: {} 2023-09-29T21:57:11.477 DEBUG:boto:Token: None 2023-09-29T21:57:11.477 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pw2J+jpL3JYEEQE8/3RGdKPIJfk= 2023-09-29T21:57:11.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pw2J+jpL3JYEEQE8/3RGdKPIJfk=', 'Content-Length': '0'} 2023-09-29T21:57:11.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a584d4b651415593-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T21:57:11.481 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.481 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.481 DEBUG:boto:path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.481 DEBUG:boto:auth_path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.482 DEBUG:boto:Method: GET 2023-09-29T21:57:11.482 DEBUG:boto:Path: /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.482 DEBUG:boto:Data: 2023-09-29T21:57:11.482 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.482 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.482 DEBUG:boto:Params: {} 2023-09-29T21:57:11.482 DEBUG:boto:Token: None 2023-09-29T21:57:11.482 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.483 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4= 2023-09-29T21:57:11.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4=', 'Content-Length': '0'} 2023-09-29T21:57:11.485 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008379f5dcea1339a3-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.486 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.486 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.486 DEBUG:boto:path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.486 DEBUG:boto:auth_path=/hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.486 DEBUG:boto:Method: GET 2023-09-29T21:57:11.486 DEBUG:boto:Path: /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.486 DEBUG:boto:Data: 2023-09-29T21:57:11.486 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.487 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.487 DEBUG:boto:Params: {} 2023-09-29T21:57:11.487 DEBUG:boto:Token: None 2023-09-29T21:57:11.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj?versionId=null 2023-09-29T21:57:11.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4= 2023-09-29T21:57:11.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cs8kLySGon9ZqB4Z2vvh1wjqmO4=', 'Content-Length': '0'} 2023-09-29T21:57:11.490 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000303d6d3de0a902b3-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.490 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.491 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.491 DEBUG:boto:path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.491 DEBUG:boto:auth_path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.491 DEBUG:boto:Method: GET 2023-09-29T21:57:11.491 DEBUG:boto:Path: /hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.491 DEBUG:boto:Data: 2023-09-29T21:57:11.491 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.491 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.492 DEBUG:boto:Params: {} 2023-09-29T21:57:11.492 DEBUG:boto:Token: None 2023-09-29T21:57:11.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dc2b5815d2a6e2b-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.497 DEBUG:boto:b'hcnphz-8_myobj1000false_myobj2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.498 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.498 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.498 DEBUG:boto:path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.498 DEBUG:boto:auth_path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.498 DEBUG:boto:Method: GET 2023-09-29T21:57:11.498 DEBUG:boto:Path: /hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.498 DEBUG:boto:Data: 2023-09-29T21:57:11.498 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.499 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.499 DEBUG:boto:Params: {} 2023-09-29T21:57:11.499 DEBUG:boto:Token: None 2023-09-29T21:57:11.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61ba670ca2cb7a6-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.546 DEBUG:boto:b'hcnphz-8_myobj1000false_myobj2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.547 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.547 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.547 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.547 DEBUG:boto:Path: /hcnphz-8/_myobj 2023-09-29T21:57:11.548 DEBUG:boto:Data: 2023-09-29T21:57:11.548 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.548 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.548 DEBUG:boto:Params: {} 2023-09-29T21:57:11.548 DEBUG:boto:Token: None 2023-09-29T21:57:11.548 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj 2023-09-29T21:57:11.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Nu9214RYmXzAt0lTHu9x1HwVhk= 2023-09-29T21:57:11.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Nu9214RYmXzAt0lTHu9x1HwVhk=', 'Content-Length': '0'} 2023-09-29T21:57:11.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000709e7398a2f62f57-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.552 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.552 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.552 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.552 DEBUG:boto:Path: /hcnphz-8/_myobj 2023-09-29T21:57:11.552 DEBUG:boto:Data: 2023-09-29T21:57:11.552 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.552 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.552 DEBUG:boto:Params: {} 2023-09-29T21:57:11.553 DEBUG:boto:Token: None 2023-09-29T21:57:11.553 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj 2023-09-29T21:57:11.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Nu9214RYmXzAt0lTHu9x1HwVhk= 2023-09-29T21:57:11.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Nu9214RYmXzAt0lTHu9x1HwVhk=', 'Content-Length': '0'} 2023-09-29T21:57:11.555 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8658d5164aeec1a-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2023-09-29T21:57:11.556 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.556 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.556 DEBUG:boto:Method: GET 2023-09-29T21:57:11.556 DEBUG:boto:Path: /hcnphz-8/_myobj 2023-09-29T21:57:11.556 DEBUG:boto:Data: 2023-09-29T21:57:11.556 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.557 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.557 DEBUG:boto:Params: {} 2023-09-29T21:57:11.557 DEBUG:boto:Token: None 2023-09-29T21:57:11.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj 2023-09-29T21:57:11.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R+TVuBwDaAqyiSE2VMa1bePQZlA= 2023-09-29T21:57:11.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R+TVuBwDaAqyiSE2VMa1bePQZlA=', 'Content-Length': '0'} 2023-09-29T21:57:11.560 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a17e30e441a774b-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.602 DEBUG:boto:path=/hcnphz-8/_myobj 2023-09-29T21:57:11.603 DEBUG:boto:auth_path=/hcnphz-8/_myobj 2023-09-29T21:57:11.603 DEBUG:boto:Method: GET 2023-09-29T21:57:11.603 DEBUG:boto:Path: /hcnphz-8/_myobj 2023-09-29T21:57:11.603 DEBUG:boto:Data: 2023-09-29T21:57:11.603 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.603 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.603 DEBUG:boto:Params: {} 2023-09-29T21:57:11.604 DEBUG:boto:Token: None 2023-09-29T21:57:11.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/_myobj 2023-09-29T21:57:11.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R+TVuBwDaAqyiSE2VMa1bePQZlA= 2023-09-29T21:57:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R+TVuBwDaAqyiSE2VMa1bePQZlA=', 'Content-Length': '0'} 2023-09-29T21:57:11.607 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a879d7ef0cddd59-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.607 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.607 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.607 DEBUG:boto:path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.608 DEBUG:boto:auth_path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.608 DEBUG:boto:Method: GET 2023-09-29T21:57:11.608 DEBUG:boto:Path: /hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.608 DEBUG:boto:Data: 2023-09-29T21:57:11.608 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.608 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.608 DEBUG:boto:Params: {} 2023-09-29T21:57:11.608 DEBUG:boto:Token: None 2023-09-29T21:57:11.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.613 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0020009a69c4518-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.613 DEBUG:boto:b'hcnphz-8_myobj1000false_myobj2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.614 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.614 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.614 DEBUG:boto:path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.614 DEBUG:boto:auth_path=/hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.614 DEBUG:boto:Method: GET 2023-09-29T21:57:11.614 DEBUG:boto:Path: /hcnphz-8/?prefix=_myobj 2023-09-29T21:57:11.614 DEBUG:boto:Data: 2023-09-29T21:57:11.615 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.615 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.615 DEBUG:boto:Params: {} 2023-09-29T21:57:11.615 DEBUG:boto:Token: None 2023-09-29T21:57:11.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000871c5d79920a87c0-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.620 DEBUG:boto:b'hcnphz-8_myobj1000false_myobj2023-09-29T21:54:49.379Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:11.621 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.621 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.621 DEBUG:boto:path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.621 DEBUG:boto:auth_path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.621 DEBUG:boto:Method: GET 2023-09-29T21:57:11.621 DEBUG:boto:Path: /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.621 DEBUG:boto:Data: 2023-09-29T21:57:11.621 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.621 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.622 DEBUG:boto:Params: {} 2023-09-29T21:57:11.622 DEBUG:boto:Token: None 2023-09-29T21:57:11.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04= 2023-09-29T21:57:11.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04=', 'Content-Length': '0'} 2023-09-29T21:57:11.624 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9ba6f0776b7a9f4-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.625 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.625 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.625 DEBUG:boto:path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.625 DEBUG:boto:auth_path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.625 DEBUG:boto:Method: GET 2023-09-29T21:57:11.625 DEBUG:boto:Path: /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.625 DEBUG:boto:Data: 2023-09-29T21:57:11.626 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.626 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.626 DEBUG:boto:Params: {} 2023-09-29T21:57:11.626 DEBUG:boto:Token: None 2023-09-29T21:57:11.626 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.626 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04= 2023-09-29T21:57:11.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04=', 'Content-Length': '0'} 2023-09-29T21:57:11.629 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029a85d68b27827bb-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.670 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.671 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.671 DEBUG:boto:path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.671 DEBUG:boto:auth_path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.671 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.671 DEBUG:boto:Path: /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.671 DEBUG:boto:Data: 2023-09-29T21:57:11.671 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.671 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.671 DEBUG:boto:Params: {} 2023-09-29T21:57:11.672 DEBUG:boto:Token: None 2023-09-29T21:57:11.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mbYy8kkB1DlFNbAKqthcF62ztZw= 2023-09-29T21:57:11.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mbYy8kkB1DlFNbAKqthcF62ztZw=', 'Content-Length': '0'} 2023-09-29T21:57:11.675 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042f3c4ee183e0211-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.675 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.675 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.675 DEBUG:boto:path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.675 DEBUG:boto:auth_path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.676 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.676 DEBUG:boto:Path: /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.676 DEBUG:boto:Data: 2023-09-29T21:57:11.676 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.676 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.676 DEBUG:boto:Params: {} 2023-09-29T21:57:11.676 DEBUG:boto:Token: None 2023-09-29T21:57:11.676 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mbYy8kkB1DlFNbAKqthcF62ztZw= 2023-09-29T21:57:11.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mbYy8kkB1DlFNbAKqthcF62ztZw=', 'Content-Length': '0'} 2023-09-29T21:57:11.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f0f48c184a64d1e0-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:11.680 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.680 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.680 DEBUG:boto:path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.680 DEBUG:boto:auth_path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.680 DEBUG:boto:Method: GET 2023-09-29T21:57:11.680 DEBUG:boto:Path: /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.680 DEBUG:boto:Data: 2023-09-29T21:57:11.681 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.681 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.681 DEBUG:boto:Params: {} 2023-09-29T21:57:11.681 DEBUG:boto:Token: None 2023-09-29T21:57:11.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04= 2023-09-29T21:57:11.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04=', 'Content-Length': '0'} 2023-09-29T21:57:11.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070f0ae0e4ea577c8-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.726 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.727 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.727 DEBUG:boto:path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.727 DEBUG:boto:auth_path=/hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.727 DEBUG:boto:Method: GET 2023-09-29T21:57:11.727 DEBUG:boto:Path: /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.727 DEBUG:boto:Data: 2023-09-29T21:57:11.727 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.728 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.728 DEBUG:boto:Params: {} 2023-09-29T21:57:11.728 DEBUG:boto:Token: None 2023-09-29T21:57:11.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj?versionId=null 2023-09-29T21:57:11.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04= 2023-09-29T21:57:11.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EeitOPd5c7jeg3G6fU6wmtxdl04=', 'Content-Length': '0'} 2023-09-29T21:57:11.731 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea5d3d13f538a650-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.732 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.732 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.732 DEBUG:boto:path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.732 DEBUG:boto:auth_path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.732 DEBUG:boto:Method: GET 2023-09-29T21:57:11.732 DEBUG:boto:Path: /hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.732 DEBUG:boto:Data: 2023-09-29T21:57:11.733 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.733 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.733 DEBUG:boto:Params: {} 2023-09-29T21:57:11.733 DEBUG:boto:Token: None 2023-09-29T21:57:11.733 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083d18126cfa85b61-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.738 DEBUG:boto:b'hcnphz-8myobj1000falsemyobj2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.739 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.739 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.739 DEBUG:boto:path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.739 DEBUG:boto:auth_path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.739 DEBUG:boto:Method: GET 2023-09-29T21:57:11.739 DEBUG:boto:Path: /hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.740 DEBUG:boto:Data: 2023-09-29T21:57:11.740 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.740 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.740 DEBUG:boto:Params: {} 2023-09-29T21:57:11.740 DEBUG:boto:Token: None 2023-09-29T21:57:11.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cc6c00005b5ec5a-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.745 DEBUG:boto:b'hcnphz-8myobj1000falsemyobj2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.745 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.745 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.745 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.746 DEBUG:boto:Path: /hcnphz-8/myobj 2023-09-29T21:57:11.746 DEBUG:boto:Data: 2023-09-29T21:57:11.746 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.746 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.746 DEBUG:boto:Params: {} 2023-09-29T21:57:11.746 DEBUG:boto:Token: None 2023-09-29T21:57:11.747 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj 2023-09-29T21:57:11.747 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bADY6nM5k78zmAIS6LJSevSbyRw= 2023-09-29T21:57:11.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bADY6nM5k78zmAIS6LJSevSbyRw=', 'Content-Length': '0'} 2023-09-29T21:57:11.749 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000186a7a360b51504d-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.749 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.750 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.750 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.750 DEBUG:boto:Path: /hcnphz-8/myobj 2023-09-29T21:57:11.750 DEBUG:boto:Data: 2023-09-29T21:57:11.750 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.750 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.750 DEBUG:boto:Params: {} 2023-09-29T21:57:11.750 DEBUG:boto:Token: None 2023-09-29T21:57:11.751 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj 2023-09-29T21:57:11.751 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bADY6nM5k78zmAIS6LJSevSbyRw= 2023-09-29T21:57:11.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bADY6nM5k78zmAIS6LJSevSbyRw=', 'Content-Length': '0'} 2023-09-29T21:57:11.753 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004288dfe64624b553-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:11.753 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.754 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.754 DEBUG:boto:Method: GET 2023-09-29T21:57:11.754 DEBUG:boto:Path: /hcnphz-8/myobj 2023-09-29T21:57:11.754 DEBUG:boto:Data: 2023-09-29T21:57:11.754 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.754 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.754 DEBUG:boto:Params: {} 2023-09-29T21:57:11.755 DEBUG:boto:Token: None 2023-09-29T21:57:11.755 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj 2023-09-29T21:57:11.755 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KWkZOD3MCpGJV47wqB4zr3TbIME= 2023-09-29T21:57:11.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KWkZOD3MCpGJV47wqB4zr3TbIME=', 'Content-Length': '0'} 2023-09-29T21:57:11.757 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0d22614863f382b-0065174837-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.757 DEBUG:boto:path=/hcnphz-8/myobj 2023-09-29T21:57:11.758 DEBUG:boto:auth_path=/hcnphz-8/myobj 2023-09-29T21:57:11.758 DEBUG:boto:Method: GET 2023-09-29T21:57:11.758 DEBUG:boto:Path: /hcnphz-8/myobj 2023-09-29T21:57:11.758 DEBUG:boto:Data: 2023-09-29T21:57:11.758 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.758 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.759 DEBUG:boto:Params: {} 2023-09-29T21:57:11.760 DEBUG:boto:Token: None 2023-09-29T21:57:11.761 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/myobj 2023-09-29T21:57:11.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KWkZOD3MCpGJV47wqB4zr3TbIME= 2023-09-29T21:57:11.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KWkZOD3MCpGJV47wqB4zr3TbIME=', 'Content-Length': '0'} 2023-09-29T21:57:11.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:54:49 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008339dad989b8e184-0065174837-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.806 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.807 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.807 DEBUG:boto:path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.807 DEBUG:boto:auth_path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.807 DEBUG:boto:Method: GET 2023-09-29T21:57:11.807 DEBUG:boto:Path: /hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.807 DEBUG:boto:Data: 2023-09-29T21:57:11.807 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.808 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.808 DEBUG:boto:Params: {} 2023-09-29T21:57:11.808 DEBUG:boto:Token: None 2023-09-29T21:57:11.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000285c2893d367e1d4-0065174837-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.813 DEBUG:boto:b'hcnphz-8myobj1000falsemyobj2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.814 DEBUG:boto:path=/hcnphz-8/ 2023-09-29T21:57:11.814 DEBUG:boto:auth_path=/hcnphz-8/ 2023-09-29T21:57:11.814 DEBUG:boto:path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.814 DEBUG:boto:auth_path=/hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.814 DEBUG:boto:Method: GET 2023-09-29T21:57:11.815 DEBUG:boto:Path: /hcnphz-8/?prefix=myobj 2023-09-29T21:57:11.815 DEBUG:boto:Data: 2023-09-29T21:57:11.815 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.815 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.815 DEBUG:boto:Params: {} 2023-09-29T21:57:11.815 DEBUG:boto:Token: None 2023-09-29T21:57:11.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-8/ 2023-09-29T21:57:11.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y= 2023-09-29T21:57:11.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kVmMnsMH9GWBIZnRwyI4slmS+Y=', 'Content-Length': '0'} 2023-09-29T21:57:11.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041094a7f8dd010a9-0065174837-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.820 DEBUG:boto:b'hcnphz-8myobj1000falsemyobj2023-09-29T21:54:49.330Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:11.821 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-8 zones={a1, a2} 2023-09-29T21:57:11.821 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2023-09-29T21:57:11.822 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-9 2023-09-29T21:57:11.822 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:11.822 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:11.823 DEBUG:boto:Method: PUT 2023-09-29T21:57:11.823 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:11.823 DEBUG:boto:Data: 2023-09-29T21:57:11.823 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.823 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.823 DEBUG:boto:Params: {} 2023-09-29T21:57:11.823 DEBUG:boto:Token: None 2023-09-29T21:57:11.824 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-9/ 2023-09-29T21:57:11.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CNp1J7G3Cx6/XWxs2SCZ75+Oll8= 2023-09-29T21:57:11.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CNp1J7G3Cx6/XWxs2SCZ75+Oll8=', 'Content-Length': '0'} 2023-09-29T21:57:11.849 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000be6f79f817a63bdc-0065174837-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.850 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-10 2023-09-29T21:57:11.850 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:11.850 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:11.850 DEBUG:boto:Method: PUT 2023-09-29T21:57:11.851 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:57:11.851 DEBUG:boto:Data: 2023-09-29T21:57:11.851 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.851 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.851 DEBUG:boto:Params: {} 2023-09-29T21:57:11.851 DEBUG:boto:Token: None 2023-09-29T21:57:11.852 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-10/ 2023-09-29T21:57:11.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AXL/NIwmXddh4TImnYKUupm7QQw= 2023-09-29T21:57:11.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AXL/NIwmXddh4TImnYKUupm7QQw=', 'Content-Length': '0'} 2023-09-29T21:57:11.896 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000093fe11d3ca9306a0-0065174837-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.896 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:11.896 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:11.896 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.896 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:11.897 DEBUG:boto:Data: 2023-09-29T21:57:11.897 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.897 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.897 DEBUG:boto:Params: {} 2023-09-29T21:57:11.897 DEBUG:boto:Token: None 2023-09-29T21:57:11.897 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-9/ 2023-09-29T21:57:11.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p6lgziSoisyDLvBwuFU6qWV3kwU= 2023-09-29T21:57:11.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p6lgziSoisyDLvBwuFU6qWV3kwU=', 'Content-Length': '0'} 2023-09-29T21:57:11.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009bcee1ca675a4f65-0065174837-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.901 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:11.901 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:11.901 DEBUG:boto:Method: PUT 2023-09-29T21:57:11.901 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:11.901 DEBUG:boto:Data: 2023-09-29T21:57:11.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:57:11.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:11.902 DEBUG:boto:Port: 8000 2023-09-29T21:57:11.902 DEBUG:boto:Params: {} 2023-09-29T21:57:11.902 DEBUG:boto:Token: None 2023-09-29T21:57:11.902 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-9/myobj 2023-09-29T21:57:11.902 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RyIZb1bhqnuQGYSqoP2iK5oIQHU= 2023-09-29T21:57:11.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RyIZb1bhqnuQGYSqoP2iK5oIQHU='} 2023-09-29T21:57:11.913 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b2f87ce863c1d979-0065174837-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.913 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:11.913 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:11.913 DEBUG:boto:Method: HEAD 2023-09-29T21:57:11.913 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:57:11.913 DEBUG:boto:Data: 2023-09-29T21:57:11.914 DEBUG:boto:Headers: {} 2023-09-29T21:57:11.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.914 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.914 DEBUG:boto:Params: {} 2023-09-29T21:57:11.914 DEBUG:boto:Token: None 2023-09-29T21:57:11.914 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-10/ 2023-09-29T21:57:11.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ogZr0Dq13lwq+VKtEjUOZdxzHOo= 2023-09-29T21:57:11.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ogZr0Dq13lwq+VKtEjUOZdxzHOo=', 'Content-Length': '0'} 2023-09-29T21:57:11.918 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000040cf3360433042d2-0065174837-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.918 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:11.918 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:11.918 DEBUG:boto:Method: PUT 2023-09-29T21:57:11.918 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:11.919 DEBUG:boto:Data: 2023-09-29T21:57:11.919 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T21:57:11.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:11.919 DEBUG:boto:Port: 8001 2023-09-29T21:57:11.919 DEBUG:boto:Params: {} 2023-09-29T21:57:11.919 DEBUG:boto:Token: None 2023-09-29T21:57:11.919 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 21:57:11 GMT /hcnphz-10/myobj 2023-09-29T21:57:11.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sw8mEzXD5+tEmjQ6jiNcSW7ORKs= 2023-09-29T21:57:11.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:57:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sw8mEzXD5+tEmjQ6jiNcSW7ORKs='} 2023-09-29T21:57:11.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c29133ca8b7cf019-0065174837-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:11.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:57:11.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:57:11.963 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:57:12.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:57:12.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:57:12.076 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:57:12.099 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:57:12.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.100 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T21:57:12.100 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T21:57:12.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.879079_47.1", 2023-09-29T21:57:12.101 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.879079Z" 2023-09-29T21:57:12.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:57:12.102 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:57:12.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.102 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:57:12.102 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:57:12.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:57:12.103 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:57:12.105 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696024631.879079_47.1', 2: '', 3: ''} 2023-09-29T21:57:12.105 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:57:12.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:57:12.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:57:12.106 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:57:12.150 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:57:12.151 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:57:12.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:12.203+0000 7fcf078a0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:57:12.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:12.203+0000 7fcf078a0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:57:12.230 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.231 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T21:57:12.232 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.879079_47.1", 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:57:12.233 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.879079Z", 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:57:12.234 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.235 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:57:12.236 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:57:12Z" 2023-09-29T21:57:12.237 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:57:12.238 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696024631.879079_47.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:57:11.879079Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:57:12Z" } 2023-09-29T21:57:12.239 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696024631.879079_47.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.879079Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:57:12.239 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696024631.879079_47.1', 2: '', 3: ''} 2023-09-29T21:57:12.239 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696024631.879079_47.1', 2: '', 3: ''} 2023-09-29T21:57:12.239 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:57:12.239 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:12.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:12.240 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:57:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:57:12.291 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:57:12.353 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:12.351+0000 7fe2592f4f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:17.354 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:17.351+0000 7fe2592f4f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:22.355 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:22.355+0000 7fe2592f4f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:27.356 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:27.351+0000 7fe2592f4f40 1 waiting to reach incremental sync.. 2023-09-29T21:57:32.359 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:32.355+0000 7fe2592f4f40 1 bucket sync caught up with source: 2023-09-29T21:57:32.359 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2023-09-29T21:57:32.359 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.5.6, , , , , , , , , ] 2023-09-29T21:57:32.360 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:32.355+0000 7fe2592f4f40 0 bucket checkpoint complete 2023-09-29T21:57:32.386 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-9 zones={a2, a1} 2023-09-29T21:57:32.386 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.386 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.387 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.387 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:32.387 DEBUG:boto:Data: 2023-09-29T21:57:32.387 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.387 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.387 DEBUG:boto:Params: {} 2023-09-29T21:57:32.387 DEBUG:boto:Token: None 2023-09-29T21:57:32.388 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/ 2023-09-29T21:57:32.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g/DTLYZSos3+GQq0EUHhPlLwebg= 2023-09-29T21:57:32.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g/DTLYZSos3+GQq0EUHhPlLwebg=', 'Content-Length': '0'} 2023-09-29T21:57:32.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d57a7bee62d649e7-006517484c-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.391 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.391 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.391 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.392 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:32.392 DEBUG:boto:Data: 2023-09-29T21:57:32.392 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.392 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.392 DEBUG:boto:Params: {} 2023-09-29T21:57:32.392 DEBUG:boto:Token: None 2023-09-29T21:57:32.392 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/ 2023-09-29T21:57:32.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g/DTLYZSos3+GQq0EUHhPlLwebg= 2023-09-29T21:57:32.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g/DTLYZSos3+GQq0EUHhPlLwebg=', 'Content-Length': '0'} 2023-09-29T21:57:32.396 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a742c42d03b274c3-006517484c-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.396 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T21:57:32.396 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.396 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.396 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.397 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.397 DEBUG:boto:Method: GET 2023-09-29T21:57:32.397 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.397 DEBUG:boto:Data: 2023-09-29T21:57:32.397 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.397 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.397 DEBUG:boto:Params: {} 2023-09-29T21:57:32.398 DEBUG:boto:Token: None 2023-09-29T21:57:32.398 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/?versions 2023-09-29T21:57:32.398 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg= 2023-09-29T21:57:32.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg=', 'Content-Length': '0'} 2023-09-29T21:57:32.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db05608f0ffa877b-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.402 DEBUG:boto:b'hcnphz-9999falsemyobjnulltrue2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.402 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:57:32.403 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T21:57:32.403 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.403 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.403 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.403 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.403 DEBUG:boto:Method: GET 2023-09-29T21:57:32.403 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.403 DEBUG:boto:Data: 2023-09-29T21:57:32.404 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.404 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.404 DEBUG:boto:Params: {} 2023-09-29T21:57:32.404 DEBUG:boto:Token: None 2023-09-29T21:57:32.404 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/?versions 2023-09-29T21:57:32.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg= 2023-09-29T21:57:32.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg=', 'Content-Length': '0'} 2023-09-29T21:57:32.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000561e690ef48e6a44-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.408 DEBUG:boto:b'hcnphz-9999falsemyobjnulltrue2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.409 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:57:32.409 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.409 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.409 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.409 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.409 DEBUG:boto:Method: GET 2023-09-29T21:57:32.409 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.409 DEBUG:boto:Data: 2023-09-29T21:57:32.410 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.410 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.410 DEBUG:boto:Params: {} 2023-09-29T21:57:32.410 DEBUG:boto:Token: None 2023-09-29T21:57:32.410 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/?versions 2023-09-29T21:57:32.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg= 2023-09-29T21:57:32.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg=', 'Content-Length': '0'} 2023-09-29T21:57:32.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066a5d24406894530-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.415 DEBUG:boto:b'hcnphz-9999falsemyobjnulltrue2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.415 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.415 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.415 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.415 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.415 DEBUG:boto:Method: GET 2023-09-29T21:57:32.415 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:32.416 DEBUG:boto:Data: 2023-09-29T21:57:32.416 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.416 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.416 DEBUG:boto:Params: {} 2023-09-29T21:57:32.416 DEBUG:boto:Token: None 2023-09-29T21:57:32.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/?versions 2023-09-29T21:57:32.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg= 2023-09-29T21:57:32.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xlq2fEenvLmAJWobwAxv781H3Jg=', 'Content-Length': '0'} 2023-09-29T21:57:32.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067a033756ea5b300-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.420 DEBUG:boto:b'hcnphz-9999falsemyobjnulltrue2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:32.421 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.421 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.421 DEBUG:boto:path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.421 DEBUG:boto:auth_path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.421 DEBUG:boto:Method: GET 2023-09-29T21:57:32.421 DEBUG:boto:Path: /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.421 DEBUG:boto:Data: 2023-09-29T21:57:32.421 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.422 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.422 DEBUG:boto:Params: {} 2023-09-29T21:57:32.422 DEBUG:boto:Token: None 2023-09-29T21:57:32.422 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM= 2023-09-29T21:57:32.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM=', 'Content-Length': '0'} 2023-09-29T21:57:32.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a9600a671b3cdbc-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.425 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.425 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.425 DEBUG:boto:path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.425 DEBUG:boto:auth_path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.425 DEBUG:boto:Method: GET 2023-09-29T21:57:32.425 DEBUG:boto:Path: /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.425 DEBUG:boto:Data: 2023-09-29T21:57:32.426 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.426 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.426 DEBUG:boto:Params: {} 2023-09-29T21:57:32.426 DEBUG:boto:Token: None 2023-09-29T21:57:32.426 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM= 2023-09-29T21:57:32.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM=', 'Content-Length': '0'} 2023-09-29T21:57:32.429 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b5d6e3b386cba9b-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.474 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.475 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.475 DEBUG:boto:path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.475 DEBUG:boto:auth_path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.475 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.475 DEBUG:boto:Path: /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.475 DEBUG:boto:Data: 2023-09-29T21:57:32.475 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.475 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.476 DEBUG:boto:Params: {} 2023-09-29T21:57:32.476 DEBUG:boto:Token: None 2023-09-29T21:57:32.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tHVlcVNCaaWnITF/RjwJm0K38Y0= 2023-09-29T21:57:32.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tHVlcVNCaaWnITF/RjwJm0K38Y0=', 'Content-Length': '0'} 2023-09-29T21:57:32.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a55513ca53bbd3c-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.479 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.479 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.479 DEBUG:boto:path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.480 DEBUG:boto:auth_path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.480 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.480 DEBUG:boto:Path: /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.480 DEBUG:boto:Data: 2023-09-29T21:57:32.480 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.480 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.480 DEBUG:boto:Params: {} 2023-09-29T21:57:32.480 DEBUG:boto:Token: None 2023-09-29T21:57:32.481 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tHVlcVNCaaWnITF/RjwJm0K38Y0= 2023-09-29T21:57:32.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tHVlcVNCaaWnITF/RjwJm0K38Y0=', 'Content-Length': '0'} 2023-09-29T21:57:32.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd3ef2266a1f8887-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:32.484 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.484 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.484 DEBUG:boto:path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.484 DEBUG:boto:auth_path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.485 DEBUG:boto:Method: GET 2023-09-29T21:57:32.485 DEBUG:boto:Path: /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.485 DEBUG:boto:Data: 2023-09-29T21:57:32.485 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.485 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.485 DEBUG:boto:Params: {} 2023-09-29T21:57:32.485 DEBUG:boto:Token: None 2023-09-29T21:57:32.486 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.486 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM= 2023-09-29T21:57:32.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM=', 'Content-Length': '0'} 2023-09-29T21:57:32.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009b2f48590db2df72-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.530 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.531 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.531 DEBUG:boto:path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.531 DEBUG:boto:auth_path=/hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.531 DEBUG:boto:Method: GET 2023-09-29T21:57:32.531 DEBUG:boto:Path: /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.531 DEBUG:boto:Data: 2023-09-29T21:57:32.531 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.531 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.532 DEBUG:boto:Params: {} 2023-09-29T21:57:32.532 DEBUG:boto:Token: None 2023-09-29T21:57:32.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj?versionId=null 2023-09-29T21:57:32.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM= 2023-09-29T21:57:32.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b5MnsRS9Tgwt5SsQ9svhXxXk1JM=', 'Content-Length': '0'} 2023-09-29T21:57:32.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3d6efb558cea528-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.535 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.536 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.536 DEBUG:boto:path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.536 DEBUG:boto:auth_path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.536 DEBUG:boto:Method: GET 2023-09-29T21:57:32.536 DEBUG:boto:Path: /hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.536 DEBUG:boto:Data: 2023-09-29T21:57:32.536 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.536 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.536 DEBUG:boto:Params: {} 2023-09-29T21:57:32.537 DEBUG:boto:Token: None 2023-09-29T21:57:32.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/ 2023-09-29T21:57:32.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI= 2023-09-29T21:57:32.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI=', 'Content-Length': '0'} 2023-09-29T21:57:32.542 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1a4363b413a910b-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.542 DEBUG:boto:b'hcnphz-9myobj1000falsemyobj2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.542 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.543 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.545 DEBUG:boto:path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.545 DEBUG:boto:auth_path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.545 DEBUG:boto:Method: GET 2023-09-29T21:57:32.545 DEBUG:boto:Path: /hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.545 DEBUG:boto:Data: 2023-09-29T21:57:32.545 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.546 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.546 DEBUG:boto:Params: {} 2023-09-29T21:57:32.546 DEBUG:boto:Token: None 2023-09-29T21:57:32.546 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/ 2023-09-29T21:57:32.546 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI= 2023-09-29T21:57:32.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI=', 'Content-Length': '0'} 2023-09-29T21:57:32.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009995341bbece0ed-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.550 DEBUG:boto:b'hcnphz-9myobj1000falsemyobj2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.551 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.551 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.551 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.551 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:32.551 DEBUG:boto:Data: 2023-09-29T21:57:32.551 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.552 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.552 DEBUG:boto:Params: {} 2023-09-29T21:57:32.552 DEBUG:boto:Token: None 2023-09-29T21:57:32.552 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj 2023-09-29T21:57:32.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+OMVXcGevd8C96YQoObF5vaYDQc= 2023-09-29T21:57:32.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+OMVXcGevd8C96YQoObF5vaYDQc=', 'Content-Length': '0'} 2023-09-29T21:57:32.554 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000027385fa52f3ec6d-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.555 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.555 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.555 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.555 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:32.555 DEBUG:boto:Data: 2023-09-29T21:57:32.555 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.555 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.556 DEBUG:boto:Params: {} 2023-09-29T21:57:32.556 DEBUG:boto:Token: None 2023-09-29T21:57:32.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj 2023-09-29T21:57:32.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+OMVXcGevd8C96YQoObF5vaYDQc= 2023-09-29T21:57:32.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+OMVXcGevd8C96YQoObF5vaYDQc=', 'Content-Length': '0'} 2023-09-29T21:57:32.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000067eaf089c3019714-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:32.559 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.559 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.559 DEBUG:boto:Method: GET 2023-09-29T21:57:32.559 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:32.559 DEBUG:boto:Data: 2023-09-29T21:57:32.559 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.559 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.559 DEBUG:boto:Params: {} 2023-09-29T21:57:32.560 DEBUG:boto:Token: None 2023-09-29T21:57:32.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj 2023-09-29T21:57:32.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lv21XKJF8gW7NC7MGn8FyeN2VRQ= 2023-09-29T21:57:32.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lv21XKJF8gW7NC7MGn8FyeN2VRQ=', 'Content-Length': '0'} 2023-09-29T21:57:32.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025ddc9a4f04c62d5-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.606 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:32.606 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:32.607 DEBUG:boto:Method: GET 2023-09-29T21:57:32.607 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:32.607 DEBUG:boto:Data: 2023-09-29T21:57:32.607 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.607 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.607 DEBUG:boto:Params: {} 2023-09-29T21:57:32.607 DEBUG:boto:Token: None 2023-09-29T21:57:32.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/myobj 2023-09-29T21:57:32.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lv21XKJF8gW7NC7MGn8FyeN2VRQ= 2023-09-29T21:57:32.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lv21XKJF8gW7NC7MGn8FyeN2VRQ=', 'Content-Length': '0'} 2023-09-29T21:57:32.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e44b754c1ad98846-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.611 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.611 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.611 DEBUG:boto:path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.611 DEBUG:boto:auth_path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.611 DEBUG:boto:Method: GET 2023-09-29T21:57:32.611 DEBUG:boto:Path: /hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.611 DEBUG:boto:Data: 2023-09-29T21:57:32.611 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.612 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.612 DEBUG:boto:Params: {} 2023-09-29T21:57:32.612 DEBUG:boto:Token: None 2023-09-29T21:57:32.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/ 2023-09-29T21:57:32.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI= 2023-09-29T21:57:32.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI=', 'Content-Length': '0'} 2023-09-29T21:57:32.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae3dfff6181ef5e-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.617 DEBUG:boto:b'hcnphz-9myobj1000falsemyobj2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.617 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:32.617 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:32.617 DEBUG:boto:path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.618 DEBUG:boto:auth_path=/hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.618 DEBUG:boto:Method: GET 2023-09-29T21:57:32.618 DEBUG:boto:Path: /hcnphz-9/?prefix=myobj 2023-09-29T21:57:32.618 DEBUG:boto:Data: 2023-09-29T21:57:32.618 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.618 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.618 DEBUG:boto:Params: {} 2023-09-29T21:57:32.618 DEBUG:boto:Token: None 2023-09-29T21:57:32.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-9/ 2023-09-29T21:57:32.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI= 2023-09-29T21:57:32.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YOwQ+kDQPgiY0abkn97nj0v/WYI=', 'Content-Length': '0'} 2023-09-29T21:57:32.622 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b61c134b81fffe4f-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.622 DEBUG:boto:b'hcnphz-9myobj1000falsemyobj2023-09-29T21:57:11.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.623 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-9 zones={a2, a1} 2023-09-29T21:57:32.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:32.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:32.623 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:57:32.674 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:57:32.675 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:57:32.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:32.739+0000 7f02a4cd3f40 1 bucket sync caught up with source: 2023-09-29T21:57:32.742 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-09-29T21:57:32.742 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.11.6, , , , , , , , , ] 2023-09-29T21:57:32.743 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:32.739+0000 7f02a4cd3f40 0 bucket checkpoint complete 2023-09-29T21:57:32.769 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-10 zones={a1, a2} 2023-09-29T21:57:32.769 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.769 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.769 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.770 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:57:32.770 DEBUG:boto:Data: 2023-09-29T21:57:32.770 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.770 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.770 DEBUG:boto:Params: {} 2023-09-29T21:57:32.771 DEBUG:boto:Token: None 2023-09-29T21:57:32.771 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/ 2023-09-29T21:57:32.771 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NDOn+SfpSSCMo9kNn2jBeWf2iOg= 2023-09-29T21:57:32.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NDOn+SfpSSCMo9kNn2jBeWf2iOg=', 'Content-Length': '0'} 2023-09-29T21:57:32.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005754b3d496831919-006517484c-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.775 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.775 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.775 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.775 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:57:32.775 DEBUG:boto:Data: 2023-09-29T21:57:32.776 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.776 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.776 DEBUG:boto:Params: {} 2023-09-29T21:57:32.776 DEBUG:boto:Token: None 2023-09-29T21:57:32.776 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/ 2023-09-29T21:57:32.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NDOn+SfpSSCMo9kNn2jBeWf2iOg= 2023-09-29T21:57:32.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NDOn+SfpSSCMo9kNn2jBeWf2iOg=', 'Content-Length': '0'} 2023-09-29T21:57:32.780 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000568e9df8801132b6-006517484c-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T21:57:32.781 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.781 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.781 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.781 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.782 DEBUG:boto:Method: GET 2023-09-29T21:57:32.782 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.782 DEBUG:boto:Data: 2023-09-29T21:57:32.782 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.782 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.782 DEBUG:boto:Params: {} 2023-09-29T21:57:32.782 DEBUG:boto:Token: None 2023-09-29T21:57:32.783 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/?versions 2023-09-29T21:57:32.783 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU= 2023-09-29T21:57:32.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU=', 'Content-Length': '0'} 2023-09-29T21:57:32.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b92fd5b10231ad3-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.787 DEBUG:boto:b'hcnphz-10999falsemyobjnulltrue2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.787 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:57:32.787 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T21:57:32.788 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.788 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.788 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.788 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.788 DEBUG:boto:Method: GET 2023-09-29T21:57:32.788 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.789 DEBUG:boto:Data: 2023-09-29T21:57:32.789 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.789 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.789 DEBUG:boto:Params: {} 2023-09-29T21:57:32.789 DEBUG:boto:Token: None 2023-09-29T21:57:32.789 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/?versions 2023-09-29T21:57:32.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU= 2023-09-29T21:57:32.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU=', 'Content-Length': '0'} 2023-09-29T21:57:32.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000823136c4d142b362-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.794 DEBUG:boto:b'hcnphz-10999falsemyobjnulltrue2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.794 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2023-09-29T21:57:32.795 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.795 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.795 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.795 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.796 DEBUG:boto:Method: GET 2023-09-29T21:57:32.796 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.796 DEBUG:boto:Data: 2023-09-29T21:57:32.796 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.797 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.797 DEBUG:boto:Params: {} 2023-09-29T21:57:32.797 DEBUG:boto:Token: None 2023-09-29T21:57:32.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/?versions 2023-09-29T21:57:32.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU= 2023-09-29T21:57:32.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU=', 'Content-Length': '0'} 2023-09-29T21:57:32.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe8c7917f696202c-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.842 DEBUG:boto:b'hcnphz-10999falsemyobjnulltrue2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.843 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.843 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.844 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.844 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.844 DEBUG:boto:Method: GET 2023-09-29T21:57:32.844 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:57:32.844 DEBUG:boto:Data: 2023-09-29T21:57:32.844 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.845 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.845 DEBUG:boto:Params: {} 2023-09-29T21:57:32.845 DEBUG:boto:Token: None 2023-09-29T21:57:32.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/?versions 2023-09-29T21:57:32.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU= 2023-09-29T21:57:32.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G2waizKsQx4IQf63uvyzuSGx7XU=', 'Content-Length': '0'} 2023-09-29T21:57:32.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ff8e813bcb5c4a3-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.850 DEBUG:boto:b'hcnphz-10999falsemyobjnulltrue2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:32.850 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.851 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.851 DEBUG:boto:path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.851 DEBUG:boto:auth_path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.851 DEBUG:boto:Method: GET 2023-09-29T21:57:32.851 DEBUG:boto:Path: /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.851 DEBUG:boto:Data: 2023-09-29T21:57:32.851 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.851 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.851 DEBUG:boto:Params: {} 2023-09-29T21:57:32.852 DEBUG:boto:Token: None 2023-09-29T21:57:32.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0= 2023-09-29T21:57:32.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0=', 'Content-Length': '0'} 2023-09-29T21:57:32.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000519f1cd903c2bc0b-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.855 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.855 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.856 DEBUG:boto:path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.856 DEBUG:boto:auth_path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.856 DEBUG:boto:Method: GET 2023-09-29T21:57:32.856 DEBUG:boto:Path: /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.857 DEBUG:boto:Data: 2023-09-29T21:57:32.857 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.857 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.857 DEBUG:boto:Params: {} 2023-09-29T21:57:32.858 DEBUG:boto:Token: None 2023-09-29T21:57:32.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.858 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0= 2023-09-29T21:57:32.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0=', 'Content-Length': '0'} 2023-09-29T21:57:32.861 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce0465e84d6ed284-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.902 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.903 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.903 DEBUG:boto:path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.903 DEBUG:boto:auth_path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.903 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.903 DEBUG:boto:Path: /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.904 DEBUG:boto:Data: 2023-09-29T21:57:32.904 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.904 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.904 DEBUG:boto:Params: {} 2023-09-29T21:57:32.905 DEBUG:boto:Token: None 2023-09-29T21:57:32.905 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dk+G7G01BB0LuG0gtsw18Attyhw= 2023-09-29T21:57:32.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dk+G7G01BB0LuG0gtsw18Attyhw=', 'Content-Length': '0'} 2023-09-29T21:57:32.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbbb90b25f468c90-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.909 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.909 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.909 DEBUG:boto:path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.909 DEBUG:boto:auth_path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.910 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.910 DEBUG:boto:Path: /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.910 DEBUG:boto:Data: 2023-09-29T21:57:32.910 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.911 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.911 DEBUG:boto:Params: {} 2023-09-29T21:57:32.911 DEBUG:boto:Token: None 2023-09-29T21:57:32.911 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.911 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dk+G7G01BB0LuG0gtsw18Attyhw= 2023-09-29T21:57:32.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dk+G7G01BB0LuG0gtsw18Attyhw=', 'Content-Length': '0'} 2023-09-29T21:57:32.914 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d16bb52aa6e0ee4-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:32.915 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.915 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.915 DEBUG:boto:path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.915 DEBUG:boto:auth_path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.915 DEBUG:boto:Method: GET 2023-09-29T21:57:32.915 DEBUG:boto:Path: /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.916 DEBUG:boto:Data: 2023-09-29T21:57:32.916 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.916 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.916 DEBUG:boto:Params: {} 2023-09-29T21:57:32.916 DEBUG:boto:Token: None 2023-09-29T21:57:32.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.916 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0= 2023-09-29T21:57:32.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0=', 'Content-Length': '0'} 2023-09-29T21:57:32.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cd80743e22b4b7c-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.920 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.920 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.920 DEBUG:boto:path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.920 DEBUG:boto:auth_path=/hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.920 DEBUG:boto:Method: GET 2023-09-29T21:57:32.920 DEBUG:boto:Path: /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.921 DEBUG:boto:Data: 2023-09-29T21:57:32.921 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.921 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.921 DEBUG:boto:Params: {} 2023-09-29T21:57:32.921 DEBUG:boto:Token: None 2023-09-29T21:57:32.921 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj?versionId=null 2023-09-29T21:57:32.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0= 2023-09-29T21:57:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B0YPcFwRuWYlyS95KS8xDlcL6v0=', 'Content-Length': '0'} 2023-09-29T21:57:32.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009a1d468b0791eecd-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.925 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.925 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.925 DEBUG:boto:path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:32.925 DEBUG:boto:auth_path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:32.925 DEBUG:boto:Method: GET 2023-09-29T21:57:32.925 DEBUG:boto:Path: /hcnphz-10/?prefix=myobj 2023-09-29T21:57:32.925 DEBUG:boto:Data: 2023-09-29T21:57:32.926 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.926 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.926 DEBUG:boto:Params: {} 2023-09-29T21:57:32.926 DEBUG:boto:Token: None 2023-09-29T21:57:32.926 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/ 2023-09-29T21:57:32.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vs92glzMBPjbL70dh9ClqFV62fY= 2023-09-29T21:57:32.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vs92glzMBPjbL70dh9ClqFV62fY=', 'Content-Length': '0'} 2023-09-29T21:57:32.931 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc4bc2c83296bc62-006517484c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.974 DEBUG:boto:b'hcnphz-10myobj1000falsemyobj2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.975 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:32.975 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:32.976 DEBUG:boto:path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:32.976 DEBUG:boto:auth_path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:32.976 DEBUG:boto:Method: GET 2023-09-29T21:57:32.976 DEBUG:boto:Path: /hcnphz-10/?prefix=myobj 2023-09-29T21:57:32.976 DEBUG:boto:Data: 2023-09-29T21:57:32.977 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.977 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.977 DEBUG:boto:Params: {} 2023-09-29T21:57:32.977 DEBUG:boto:Token: None 2023-09-29T21:57:32.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/ 2023-09-29T21:57:32.978 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vs92glzMBPjbL70dh9ClqFV62fY= 2023-09-29T21:57:32.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vs92glzMBPjbL70dh9ClqFV62fY=', 'Content-Length': '0'} 2023-09-29T21:57:32.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000040d3d9572579385-006517484c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.982 DEBUG:boto:b'hcnphz-10myobj1000falsemyobj2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:32.983 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.983 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.983 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.983 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:32.983 DEBUG:boto:Data: 2023-09-29T21:57:32.984 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.984 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.984 DEBUG:boto:Params: {} 2023-09-29T21:57:32.984 DEBUG:boto:Token: None 2023-09-29T21:57:32.984 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj 2023-09-29T21:57:32.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zOfp9aD8dJYAnCfc3eieaJXyGjI= 2023-09-29T21:57:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zOfp9aD8dJYAnCfc3eieaJXyGjI=', 'Content-Length': '0'} 2023-09-29T21:57:32.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd7861225e84919a-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.987 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.987 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.988 DEBUG:boto:Method: HEAD 2023-09-29T21:57:32.988 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:32.988 DEBUG:boto:Data: 2023-09-29T21:57:32.988 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.988 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.988 DEBUG:boto:Params: {} 2023-09-29T21:57:32.988 DEBUG:boto:Token: None 2023-09-29T21:57:32.989 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj 2023-09-29T21:57:32.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zOfp9aD8dJYAnCfc3eieaJXyGjI= 2023-09-29T21:57:32.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zOfp9aD8dJYAnCfc3eieaJXyGjI=', 'Content-Length': '0'} 2023-09-29T21:57:32.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ed2b51a1b97058e-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2023-09-29T21:57:32.992 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.992 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.992 DEBUG:boto:Method: GET 2023-09-29T21:57:32.992 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:32.992 DEBUG:boto:Data: 2023-09-29T21:57:32.992 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:32.993 DEBUG:boto:Port: 8000 2023-09-29T21:57:32.993 DEBUG:boto:Params: {} 2023-09-29T21:57:32.993 DEBUG:boto:Token: None 2023-09-29T21:57:32.993 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj 2023-09-29T21:57:32.993 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E4saPy3bOIfk/5dRRjYnl/i5ens= 2023-09-29T21:57:32.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E4saPy3bOIfk/5dRRjYnl/i5ens=', 'Content-Length': '0'} 2023-09-29T21:57:32.996 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b0c7f2b76b0c4f0-006517484c-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:32.996 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:32.996 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:32.996 DEBUG:boto:Method: GET 2023-09-29T21:57:32.997 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:32.997 DEBUG:boto:Data: 2023-09-29T21:57:32.997 DEBUG:boto:Headers: {} 2023-09-29T21:57:32.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:32.997 DEBUG:boto:Port: 8001 2023-09-29T21:57:32.997 DEBUG:boto:Params: {} 2023-09-29T21:57:32.997 DEBUG:boto:Token: None 2023-09-29T21:57:32.998 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:32 GMT /hcnphz-10/myobj 2023-09-29T21:57:32.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E4saPy3bOIfk/5dRRjYnl/i5ens= 2023-09-29T21:57:32.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E4saPy3bOIfk/5dRRjYnl/i5ens=', 'Content-Length': '0'} 2023-09-29T21:57:33.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed703bee111fe27b-006517484c-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:33.042 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:33.043 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:33.043 DEBUG:boto:path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:33.043 DEBUG:boto:auth_path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:33.043 DEBUG:boto:Method: GET 2023-09-29T21:57:33.043 DEBUG:boto:Path: /hcnphz-10/?prefix=myobj 2023-09-29T21:57:33.043 DEBUG:boto:Data: 2023-09-29T21:57:33.043 DEBUG:boto:Headers: {} 2023-09-29T21:57:33.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:33.044 DEBUG:boto:Port: 8000 2023-09-29T21:57:33.044 DEBUG:boto:Params: {} 2023-09-29T21:57:33.044 DEBUG:boto:Token: None 2023-09-29T21:57:33.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:33 GMT /hcnphz-10/ 2023-09-29T21:57:33.045 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IdJykmOc5dd2B5CaBr4ojUgvoBQ= 2023-09-29T21:57:33.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IdJykmOc5dd2B5CaBr4ojUgvoBQ=', 'Content-Length': '0'} 2023-09-29T21:57:33.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2c28efb7b6b6fdf-006517484d-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:33.049 DEBUG:boto:b'hcnphz-10myobj1000falsemyobj2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:33.050 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:33.050 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:33.050 DEBUG:boto:path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:33.050 DEBUG:boto:auth_path=/hcnphz-10/?prefix=myobj 2023-09-29T21:57:33.051 DEBUG:boto:Method: GET 2023-09-29T21:57:33.051 DEBUG:boto:Path: /hcnphz-10/?prefix=myobj 2023-09-29T21:57:33.051 DEBUG:boto:Data: 2023-09-29T21:57:33.051 DEBUG:boto:Headers: {} 2023-09-29T21:57:33.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:33.051 DEBUG:boto:Port: 8001 2023-09-29T21:57:33.051 DEBUG:boto:Params: {} 2023-09-29T21:57:33.051 DEBUG:boto:Token: None 2023-09-29T21:57:33.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:33 GMT /hcnphz-10/ 2023-09-29T21:57:33.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IdJykmOc5dd2B5CaBr4ojUgvoBQ= 2023-09-29T21:57:33.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IdJykmOc5dd2B5CaBr4ojUgvoBQ=', 'Content-Length': '0'} 2023-09-29T21:57:33.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8ca14a932ff2e2c-006517484d-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:33.056 DEBUG:boto:b'hcnphz-10myobj1000falsemyobj2023-09-29T21:57:11.924Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T21:57:33.057 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-10 zones={a1, a2} 2023-09-29T21:57:33.057 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:33.057 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:33.058 DEBUG:boto:Method: HEAD 2023-09-29T21:57:33.058 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:33.058 DEBUG:boto:Data: 2023-09-29T21:57:33.058 DEBUG:boto:Headers: {} 2023-09-29T21:57:33.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:33.059 DEBUG:boto:Port: 8000 2023-09-29T21:57:33.059 DEBUG:boto:Params: {} 2023-09-29T21:57:33.059 DEBUG:boto:Token: None 2023-09-29T21:57:33.059 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:33 GMT /hcnphz-9/ 2023-09-29T21:57:33.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T7hJnaNwXPAxDxYkiOsJUnhi7MQ= 2023-09-29T21:57:33.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T7hJnaNwXPAxDxYkiOsJUnhi7MQ=', 'Content-Length': '0'} 2023-09-29T21:57:33.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044bb0fb8171f0048-006517484d-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:33.064 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:33.064 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:33.064 DEBUG:boto:Method: HEAD 2023-09-29T21:57:33.064 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:33.064 DEBUG:boto:Data: 2023-09-29T21:57:33.065 DEBUG:boto:Headers: {} 2023-09-29T21:57:33.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:33.065 DEBUG:boto:Port: 8000 2023-09-29T21:57:33.065 DEBUG:boto:Params: {} 2023-09-29T21:57:33.065 DEBUG:boto:Token: None 2023-09-29T21:57:33.065 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:33 GMT /hcnphz-9/myobj 2023-09-29T21:57:33.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SSiCL0NSFSfC7lv35OMcNp3MOOM= 2023-09-29T21:57:33.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SSiCL0NSFSfC7lv35OMcNp3MOOM=', 'Content-Length': '0'} 2023-09-29T21:57:33.068 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002e6688a8e192aebe-006517484d-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:33.068 DEBUG:boto:path=/hcnphz-9/myobj 2023-09-29T21:57:33.068 DEBUG:boto:auth_path=/hcnphz-9/myobj 2023-09-29T21:57:33.068 DEBUG:boto:Method: DELETE 2023-09-29T21:57:33.068 DEBUG:boto:Path: /hcnphz-9/myobj 2023-09-29T21:57:33.068 DEBUG:boto:Data: 2023-09-29T21:57:33.069 DEBUG:boto:Headers: {} 2023-09-29T21:57:33.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:33.069 DEBUG:boto:Port: 8000 2023-09-29T21:57:33.069 DEBUG:boto:Params: {} 2023-09-29T21:57:33.069 DEBUG:boto:Token: None 2023-09-29T21:57:33.069 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:57:33 GMT /hcnphz-9/myobj 2023-09-29T21:57:33.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9OVJ8oQVQonUGXcrqaeWMIJJCLA= 2023-09-29T21:57:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9OVJ8oQVQonUGXcrqaeWMIJJCLA=', 'Content-Length': '0'} 2023-09-29T21:57:33.078 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000428c8cf4a8c32708-006517484d-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:57:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:33.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:33.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-9', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:33.078 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-9 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:57:33.130 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:57:33.130 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:57:33.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:33.195+0000 7f3c6f590f40 1 waiting for incremental sync to catch up: 2023-09-29T21:57:33.198 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2023-09-29T21:57:33.198 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-09-29T21:57:38.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:38.198+0000 7f3c6f590f40 1 waiting for incremental sync to catch up: 2023-09-29T21:57:38.201 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2023-09-29T21:57:38.201 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-09-29T21:57:43.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:43.198+0000 7f3c6f590f40 1 waiting for incremental sync to catch up: 2023-09-29T21:57:43.203 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2023-09-29T21:57:43.203 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-09-29T21:57:48.206 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:48.202+0000 7f3c6f590f40 1 waiting for incremental sync to catch up: 2023-09-29T21:57:48.206 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.5.6, , , , , , , , , ] 2023-09-29T21:57:48.206 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-09-29T21:57:53.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:53.206+0000 7f3c6f590f40 1 bucket sync caught up with source: 2023-09-29T21:57:53.208 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.7.6, , , , , , , , , ] 2023-09-29T21:57:53.208 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.7.6, , , , , , , , , ] 2023-09-29T21:57:53.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:53.206+0000 7f3c6f590f40 0 bucket checkpoint complete 2023-09-29T21:57:53.237 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-9 zones={a2, a1} 2023-09-29T21:57:53.237 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:53.237 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:53.237 DEBUG:boto:Method: HEAD 2023-09-29T21:57:53.237 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:53.237 DEBUG:boto:Data: 2023-09-29T21:57:53.238 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:53.238 DEBUG:boto:Port: 8001 2023-09-29T21:57:53.238 DEBUG:boto:Params: {} 2023-09-29T21:57:53.238 DEBUG:boto:Token: None 2023-09-29T21:57:53.238 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-9/ 2023-09-29T21:57:53.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MLrPl3uwHZxEjBnc+zWL/fIFDZc= 2023-09-29T21:57:53.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MLrPl3uwHZxEjBnc+zWL/fIFDZc=', 'Content-Length': '0'} 2023-09-29T21:57:53.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000226e9ad88d3632b2-0065174861-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.242 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:53.242 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:53.242 DEBUG:boto:Method: HEAD 2023-09-29T21:57:53.242 DEBUG:boto:Path: /hcnphz-9/ 2023-09-29T21:57:53.242 DEBUG:boto:Data: 2023-09-29T21:57:53.242 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:53.243 DEBUG:boto:Port: 8000 2023-09-29T21:57:53.243 DEBUG:boto:Params: {} 2023-09-29T21:57:53.243 DEBUG:boto:Token: None 2023-09-29T21:57:53.243 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-9/ 2023-09-29T21:57:53.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MLrPl3uwHZxEjBnc+zWL/fIFDZc= 2023-09-29T21:57:53.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MLrPl3uwHZxEjBnc+zWL/fIFDZc=', 'Content-Length': '0'} 2023-09-29T21:57:53.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cd22675be6e93db-0065174861-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.247 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T21:57:53.247 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:53.247 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:53.247 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.247 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.247 DEBUG:boto:Method: GET 2023-09-29T21:57:53.248 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.248 DEBUG:boto:Data: 2023-09-29T21:57:53.248 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:53.248 DEBUG:boto:Port: 8001 2023-09-29T21:57:53.248 DEBUG:boto:Params: {} 2023-09-29T21:57:53.248 DEBUG:boto:Token: None 2023-09-29T21:57:53.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-9/?versions 2023-09-29T21:57:53.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs= 2023-09-29T21:57:53.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs=', 'Content-Length': '0'} 2023-09-29T21:57:53.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000069a07683a50a47b-0065174861-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.252 DEBUG:boto:b'hcnphz-9999false' 2023-09-29T21:57:53.253 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T21:57:53.253 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:53.253 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:53.253 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.253 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.253 DEBUG:boto:Method: GET 2023-09-29T21:57:53.254 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.254 DEBUG:boto:Data: 2023-09-29T21:57:53.254 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:53.254 DEBUG:boto:Port: 8000 2023-09-29T21:57:53.254 DEBUG:boto:Params: {} 2023-09-29T21:57:53.254 DEBUG:boto:Token: None 2023-09-29T21:57:53.254 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-9/?versions 2023-09-29T21:57:53.255 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs= 2023-09-29T21:57:53.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs=', 'Content-Length': '0'} 2023-09-29T21:57:53.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d03d7c9da8b2afd-0065174861-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.258 DEBUG:boto:b'hcnphz-9999false' 2023-09-29T21:57:53.258 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:53.258 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:53.258 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.258 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.259 DEBUG:boto:Method: GET 2023-09-29T21:57:53.259 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.259 DEBUG:boto:Data: 2023-09-29T21:57:53.259 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:53.259 DEBUG:boto:Port: 8001 2023-09-29T21:57:53.259 DEBUG:boto:Params: {} 2023-09-29T21:57:53.259 DEBUG:boto:Token: None 2023-09-29T21:57:53.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-9/?versions 2023-09-29T21:57:53.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs= 2023-09-29T21:57:53.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs=', 'Content-Length': '0'} 2023-09-29T21:57:53.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d7bc9bade056e22-0065174861-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.263 DEBUG:boto:b'hcnphz-9999false' 2023-09-29T21:57:53.264 DEBUG:boto:path=/hcnphz-9/ 2023-09-29T21:57:53.264 DEBUG:boto:auth_path=/hcnphz-9/ 2023-09-29T21:57:53.264 DEBUG:boto:path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.264 DEBUG:boto:auth_path=/hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.264 DEBUG:boto:Method: GET 2023-09-29T21:57:53.264 DEBUG:boto:Path: /hcnphz-9/?versions&max-keys=999 2023-09-29T21:57:53.264 DEBUG:boto:Data: 2023-09-29T21:57:53.264 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:57:53.265 DEBUG:boto:Port: 8000 2023-09-29T21:57:53.265 DEBUG:boto:Params: {} 2023-09-29T21:57:53.265 DEBUG:boto:Token: None 2023-09-29T21:57:53.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-9/?versions 2023-09-29T21:57:53.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs= 2023-09-29T21:57:53.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q1KlpMRieDaRaJDaLzuEEjF+kHs=', 'Content-Length': '0'} 2023-09-29T21:57:53.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be6876cf7f0156fa-0065174861-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.269 DEBUG:boto:b'hcnphz-9999false' 2023-09-29T21:57:53.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-9 zones={a2, a1} 2023-09-29T21:57:53.270 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:57:53.270 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:57:53.270 DEBUG:boto:Method: HEAD 2023-09-29T21:57:53.270 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:57:53.270 DEBUG:boto:Data: 2023-09-29T21:57:53.270 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:53.270 DEBUG:boto:Port: 8001 2023-09-29T21:57:53.270 DEBUG:boto:Params: {} 2023-09-29T21:57:53.271 DEBUG:boto:Token: None 2023-09-29T21:57:53.271 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-10/ 2023-09-29T21:57:53.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+jhUrValwI9BSlfTlQw4i9OoBYQ= 2023-09-29T21:57:53.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+jhUrValwI9BSlfTlQw4i9OoBYQ=', 'Content-Length': '0'} 2023-09-29T21:57:53.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061c98a8a6628566f-0065174861-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.274 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:53.275 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:53.275 DEBUG:boto:Method: HEAD 2023-09-29T21:57:53.275 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:53.275 DEBUG:boto:Data: 2023-09-29T21:57:53.275 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:53.275 DEBUG:boto:Port: 8001 2023-09-29T21:57:53.275 DEBUG:boto:Params: {} 2023-09-29T21:57:53.275 DEBUG:boto:Token: None 2023-09-29T21:57:53.276 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-10/myobj 2023-09-29T21:57:53.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F1NF0nVDgl1KjVcweBx4NTZZw5w= 2023-09-29T21:57:53.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F1NF0nVDgl1KjVcweBx4NTZZw5w=', 'Content-Length': '0'} 2023-09-29T21:57:53.278 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:57:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058a94f84c4a36cfc-0065174861-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.278 DEBUG:boto:path=/hcnphz-10/myobj 2023-09-29T21:57:53.278 DEBUG:boto:auth_path=/hcnphz-10/myobj 2023-09-29T21:57:53.278 DEBUG:boto:Method: DELETE 2023-09-29T21:57:53.279 DEBUG:boto:Path: /hcnphz-10/myobj 2023-09-29T21:57:53.279 DEBUG:boto:Data: 2023-09-29T21:57:53.279 DEBUG:boto:Headers: {} 2023-09-29T21:57:53.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:57:53.279 DEBUG:boto:Port: 8001 2023-09-29T21:57:53.279 DEBUG:boto:Params: {} 2023-09-29T21:57:53.279 DEBUG:boto:Token: None 2023-09-29T21:57:53.279 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:57:53 GMT /hcnphz-10/myobj 2023-09-29T21:57:53.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FaZoZDZZze/tHOBvTSgligEsGH8= 2023-09-29T21:57:53.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:57:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FaZoZDZZze/tHOBvTSgligEsGH8=', 'Content-Length': '0'} 2023-09-29T21:57:53.290 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000617bf65d9ce8bbc3-0065174861-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:57:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:57:53.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:53.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-10', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:57:53.291 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-10 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:57:53.343 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:57:53.343 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:57:53.405 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:53.402+0000 7f30ef4f2f40 1 waiting for incremental sync to catch up: 2023-09-29T21:57:53.405 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-09-29T21:57:53.405 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-09-29T21:57:58.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:57:58.402+0000 7f30ef4f2f40 1 waiting for incremental sync to catch up: 2023-09-29T21:57:58.408 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-09-29T21:57:58.408 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-09-29T21:58:03.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:03.406+0000 7f30ef4f2f40 1 waiting for incremental sync to catch up: 2023-09-29T21:58:03.410 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-09-29T21:58:03.410 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-09-29T21:58:08.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:08.410+0000 7f30ef4f2f40 1 waiting for incremental sync to catch up: 2023-09-29T21:58:08.413 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.11.6, , , , , , , , , ] 2023-09-29T21:58:08.413 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-09-29T21:58:13.415 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:13.410+0000 7f30ef4f2f40 1 bucket sync caught up with source: 2023-09-29T21:58:13.415 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.15.6, , , , , , , , , ] 2023-09-29T21:58:13.416 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.15.6, , , , , , , , , ] 2023-09-29T21:58:13.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:13.410+0000 7f30ef4f2f40 0 bucket checkpoint complete 2023-09-29T21:58:13.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-10 zones={a1, a2} 2023-09-29T21:58:13.443 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:58:13.443 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:58:13.443 DEBUG:boto:Method: HEAD 2023-09-29T21:58:13.443 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:58:13.444 DEBUG:boto:Data: 2023-09-29T21:58:13.444 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:13.444 DEBUG:boto:Port: 8000 2023-09-29T21:58:13.444 DEBUG:boto:Params: {} 2023-09-29T21:58:13.444 DEBUG:boto:Token: None 2023-09-29T21:58:13.444 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-10/ 2023-09-29T21:58:13.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:03ecCdMj0osiJABIiU8PaK+p04U= 2023-09-29T21:58:13.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:03ecCdMj0osiJABIiU8PaK+p04U=', 'Content-Length': '0'} 2023-09-29T21:58:13.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000664f6da838172de-0065174875-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.449 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:58:13.449 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:58:13.449 DEBUG:boto:Method: HEAD 2023-09-29T21:58:13.449 DEBUG:boto:Path: /hcnphz-10/ 2023-09-29T21:58:13.449 DEBUG:boto:Data: 2023-09-29T21:58:13.449 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:13.450 DEBUG:boto:Port: 8001 2023-09-29T21:58:13.450 DEBUG:boto:Params: {} 2023-09-29T21:58:13.450 DEBUG:boto:Token: None 2023-09-29T21:58:13.450 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-10/ 2023-09-29T21:58:13.450 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:03ecCdMj0osiJABIiU8PaK+p04U= 2023-09-29T21:58:13.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:03ecCdMj0osiJABIiU8PaK+p04U=', 'Content-Length': '0'} 2023-09-29T21:58:13.454 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb30abc163f96dd0-0065174875-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.454 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T21:58:13.454 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:58:13.454 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:58:13.454 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.455 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.455 DEBUG:boto:Method: GET 2023-09-29T21:58:13.455 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.455 DEBUG:boto:Data: 2023-09-29T21:58:13.455 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:13.455 DEBUG:boto:Port: 8000 2023-09-29T21:58:13.455 DEBUG:boto:Params: {} 2023-09-29T21:58:13.455 DEBUG:boto:Token: None 2023-09-29T21:58:13.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-10/?versions 2023-09-29T21:58:13.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ= 2023-09-29T21:58:13.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ=', 'Content-Length': '0'} 2023-09-29T21:58:13.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb5756cef8ae65e6-0065174875-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.459 DEBUG:boto:b'hcnphz-10999false' 2023-09-29T21:58:13.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T21:58:13.460 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:58:13.460 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:58:13.460 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.460 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.460 DEBUG:boto:Method: GET 2023-09-29T21:58:13.461 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.461 DEBUG:boto:Data: 2023-09-29T21:58:13.461 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:13.461 DEBUG:boto:Port: 8001 2023-09-29T21:58:13.461 DEBUG:boto:Params: {} 2023-09-29T21:58:13.461 DEBUG:boto:Token: None 2023-09-29T21:58:13.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-10/?versions 2023-09-29T21:58:13.462 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ= 2023-09-29T21:58:13.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ=', 'Content-Length': '0'} 2023-09-29T21:58:13.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054f1614f92932d34-0065174875-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.466 DEBUG:boto:b'hcnphz-10999false' 2023-09-29T21:58:13.466 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:58:13.466 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:58:13.466 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.467 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.467 DEBUG:boto:Method: GET 2023-09-29T21:58:13.467 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.467 DEBUG:boto:Data: 2023-09-29T21:58:13.467 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:13.467 DEBUG:boto:Port: 8000 2023-09-29T21:58:13.467 DEBUG:boto:Params: {} 2023-09-29T21:58:13.467 DEBUG:boto:Token: None 2023-09-29T21:58:13.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-10/?versions 2023-09-29T21:58:13.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ= 2023-09-29T21:58:13.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ=', 'Content-Length': '0'} 2023-09-29T21:58:13.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a393b8f3e06b710-0065174875-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.514 DEBUG:boto:b'hcnphz-10999false' 2023-09-29T21:58:13.515 DEBUG:boto:path=/hcnphz-10/ 2023-09-29T21:58:13.515 DEBUG:boto:auth_path=/hcnphz-10/ 2023-09-29T21:58:13.515 DEBUG:boto:path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.515 DEBUG:boto:auth_path=/hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.515 DEBUG:boto:Method: GET 2023-09-29T21:58:13.515 DEBUG:boto:Path: /hcnphz-10/?versions&max-keys=999 2023-09-29T21:58:13.516 DEBUG:boto:Data: 2023-09-29T21:58:13.516 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:13.516 DEBUG:boto:Port: 8001 2023-09-29T21:58:13.516 DEBUG:boto:Params: {} 2023-09-29T21:58:13.516 DEBUG:boto:Token: None 2023-09-29T21:58:13.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-10/?versions 2023-09-29T21:58:13.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ= 2023-09-29T21:58:13.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zW7sXmWvQ1UjMzwEOgqUxUiT/fQ=', 'Content-Length': '0'} 2023-09-29T21:58:13.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3475556534dba2e-0065174875-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.522 DEBUG:boto:b'hcnphz-10999false' 2023-09-29T21:58:13.522 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-10 zones={a1, a2} 2023-09-29T21:58:13.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2023-09-29T21:58:13.523 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-11 2023-09-29T21:58:13.524 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T21:58:13.524 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T21:58:13.524 DEBUG:boto:Method: PUT 2023-09-29T21:58:13.524 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T21:58:13.524 DEBUG:boto:Data: 2023-09-29T21:58:13.525 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:13.525 DEBUG:boto:Port: 8000 2023-09-29T21:58:13.525 DEBUG:boto:Params: {} 2023-09-29T21:58:13.525 DEBUG:boto:Token: None 2023-09-29T21:58:13.525 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-11/ 2023-09-29T21:58:13.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DBmaBV+P5uLDBoDew+EAgfo9jJA= 2023-09-29T21:58:13.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DBmaBV+P5uLDBoDew+EAgfo9jJA=', 'Content-Length': '0'} 2023-09-29T21:58:13.553 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000003c52d7c9d45f7f90-0065174875-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.553 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-12 2023-09-29T21:58:13.553 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T21:58:13.553 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T21:58:13.553 DEBUG:boto:Method: PUT 2023-09-29T21:58:13.554 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T21:58:13.554 DEBUG:boto:Data: 2023-09-29T21:58:13.554 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:13.554 DEBUG:boto:Port: 8001 2023-09-29T21:58:13.554 DEBUG:boto:Params: {} 2023-09-29T21:58:13.554 DEBUG:boto:Token: None 2023-09-29T21:58:13.554 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-12/ 2023-09-29T21:58:13.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hp3R5JkDcRIzwAkTTiA8z8ep7xM= 2023-09-29T21:58:13.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hp3R5JkDcRIzwAkTTiA8z8ep7xM=', 'Content-Length': '0'} 2023-09-29T21:58:13.594 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004d78a77422921997-0065174875-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.594 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T21:58:13.594 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T21:58:13.594 DEBUG:boto:path=/hcnphz-11/?versioning 2023-09-29T21:58:13.594 DEBUG:boto:auth_path=/hcnphz-11/?versioning 2023-09-29T21:58:13.594 DEBUG:boto:Method: PUT 2023-09-29T21:58:13.595 DEBUG:boto:Path: /hcnphz-11/?versioning 2023-09-29T21:58:13.595 DEBUG:boto:Data: Enabled Disabled 2023-09-29T21:58:13.595 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:13.595 DEBUG:boto:Port: 8000 2023-09-29T21:58:13.595 DEBUG:boto:Params: {} 2023-09-29T21:58:13.595 DEBUG:boto:Token: None 2023-09-29T21:58:13.595 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-11/?versioning 2023-09-29T21:58:13.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VnNj/kX1n1nWXVLieLk6M2+rOb0= 2023-09-29T21:58:13.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VnNj/kX1n1nWXVLieLk6M2+rOb0=', 'Content-Length': '228'} 2023-09-29T21:58:13.603 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a6447d569bf2363d-0065174875-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.603 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T21:58:13.603 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T21:58:13.603 DEBUG:boto:path=/hcnphz-12/?versioning 2023-09-29T21:58:13.604 DEBUG:boto:auth_path=/hcnphz-12/?versioning 2023-09-29T21:58:13.604 DEBUG:boto:Method: PUT 2023-09-29T21:58:13.604 DEBUG:boto:Path: /hcnphz-12/?versioning 2023-09-29T21:58:13.604 DEBUG:boto:Data: Enabled Disabled 2023-09-29T21:58:13.604 DEBUG:boto:Headers: {} 2023-09-29T21:58:13.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:13.604 DEBUG:boto:Port: 8001 2023-09-29T21:58:13.604 DEBUG:boto:Params: {} 2023-09-29T21:58:13.604 DEBUG:boto:Token: None 2023-09-29T21:58:13.605 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 21:58:13 GMT /hcnphz-12/?versioning 2023-09-29T21:58:13.605 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r0fyYstTbF9YAPD8x90A4838vwA= 2023-09-29T21:58:13.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r0fyYstTbF9YAPD8x90A4838vwA=', 'Content-Length': '228'} 2023-09-29T21:58:13.618 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000023a5973c8a4254d9-0065174875-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:13.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:58:13.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:58:13.619 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:58:13.662 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:58:13.662 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:58:13.716 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024693.614963_60.1", 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:58:13.614963Z" 2023-09-29T21:58:13.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:13.746 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T21:58:13.747 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696024693.614963_60.1', 2: '', 3: ''} 2023-09-29T21:58:13.748 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T21:58:13.748 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:58:13.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:58:13.748 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:58:13.792 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:58:13.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:58:13.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:13.846+0000 7fe92cf88f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:58:13.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:13.846+0000 7fe92cf88f40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:58:13.873 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:58:13.873 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:58:13.873 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:58:13.873 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:58:13.873 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:58:13.873 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T21:58:13.874 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:58:13.875 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024693.542657_49.1", 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:58:13.542657Z", 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:13.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:58:13.878 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:58:13.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.880 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:58:13.881 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:58:13.881 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:58:13.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:13.881 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:58:13Z" 2023-09-29T21:58:13.881 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:58:13.882 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696024693.542657_49.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:58:13.542657Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:58:13Z" } 2023-09-29T21:58:13.883 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696024693.542657_49.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:58:13.542657Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:58:13.883 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696024693.614963_60.1', 2: '', 3: ''} 2023-09-29T21:58:13.883 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696024693.542657_49.1', 2: '', 3: ''} 2023-09-29T21:58:13.883 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696024693.614963_60.1 target=1_1696024693.542657_49.1 2023-09-29T21:58:18.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:58:18.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T21:58:18.884 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T21:58:18.932 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:58:18.932 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:58:18.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:18.985+0000 7f5b7b26cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:58:18.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:18.985+0000 7f5b7b26cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T21:58:19.014 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T21:58:19.014 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T21:58:19.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:19.016 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:19.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T21:58:19.016 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:19.016 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T21:58:19.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:19.016 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024693.614963_60.1", 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:19.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:58:13.614963Z", 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:19.018 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T21:58:19.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T21:58:18Z" 2023-09-29T21:58:19.021 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T21:58:19.022 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696024693.614963_60.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:58:13.614963Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T21:58:18Z" } 2023-09-29T21:58:19.022 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696024693.614963_60.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:58:13.614963Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T21:58:19.023 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696024693.614963_60.1', 2: '', 3: ''} 2023-09-29T21:58:19.023 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696024693.614963_60.1', 2: '', 3: ''} 2023-09-29T21:58:19.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T21:58:19.023 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T21:58:19.023 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T21:58:19.023 DEBUG:boto:Method: HEAD 2023-09-29T21:58:19.023 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T21:58:19.023 DEBUG:boto:Data: 2023-09-29T21:58:19.024 DEBUG:boto:Headers: {} 2023-09-29T21:58:19.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:19.024 DEBUG:boto:Port: 8000 2023-09-29T21:58:19.024 DEBUG:boto:Params: {} 2023-09-29T21:58:19.024 DEBUG:boto:Token: None 2023-09-29T21:58:19.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:19 GMT /hcnphz-11/ 2023-09-29T21:58:19.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsA/W5gvaDQauZhc1VXl4YPI1IU= 2023-09-29T21:58:19.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsA/W5gvaDQauZhc1VXl4YPI1IU=', 'Content-Length': '0'} 2023-09-29T21:58:19.028 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000282e425dc955b409-006517487b-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:19.029 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T21:58:19.029 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T21:58:19.029 DEBUG:boto:Method: PUT 2023-09-29T21:58:19.029 DEBUG:boto:Path: /hcnphz-11/dummy 2023-09-29T21:58:19.029 DEBUG:boto:Data: 2023-09-29T21:58:19.029 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T21:58:19.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:19.029 DEBUG:boto:Port: 8000 2023-09-29T21:58:19.030 DEBUG:boto:Params: {} 2023-09-29T21:58:19.030 DEBUG:boto:Token: None 2023-09-29T21:58:19.030 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 21:58:19 GMT /hcnphz-11/dummy 2023-09-29T21:58:19.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eSlx9LNGxkoG0PcsSpX6fCDkxCI= 2023-09-29T21:58:19.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eSlx9LNGxkoG0PcsSpX6fCDkxCI='} 2023-09-29T21:58:19.074 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-amz-request-id', 'tx00000ca6a4b5ae74cb96b-006517487b-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:19.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:58:19.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:58:19.075 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:58:19.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:58:19.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:58:19.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:19.189+0000 7f6ad38a8f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:24.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:24.189+0000 7f6ad38a8f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:29.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:29.193+0000 7f6ad38a8f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:34.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:34.197+0000 7f6ad38a8f40 1 bucket sync caught up with source: 2023-09-29T21:58:34.199 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-29T21:58:34.199 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.8.11, , , , , ] 2023-09-29T21:58:34.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:34.197+0000 7f6ad38a8f40 0 bucket checkpoint complete 2023-09-29T21:58:34.228 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T21:58:34.229 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T21:58:34.229 DEBUG:boto:Method: HEAD 2023-09-29T21:58:34.229 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T21:58:34.229 DEBUG:boto:Data: 2023-09-29T21:58:34.229 DEBUG:boto:Headers: {} 2023-09-29T21:58:34.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:34.229 DEBUG:boto:Port: 8001 2023-09-29T21:58:34.230 DEBUG:boto:Params: {} 2023-09-29T21:58:34.230 DEBUG:boto:Token: None 2023-09-29T21:58:34.230 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:34 GMT /hcnphz-12/ 2023-09-29T21:58:34.230 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:agxOu+DwMCCaWktUeYNpWKzRNBo= 2023-09-29T21:58:34.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:agxOu+DwMCCaWktUeYNpWKzRNBo=', 'Content-Length': '0'} 2023-09-29T21:58:34.234 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000010056b2de769d761-006517488a-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:34.235 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T21:58:34.235 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T21:58:34.235 DEBUG:boto:Method: PUT 2023-09-29T21:58:34.235 DEBUG:boto:Path: /hcnphz-12/dummy 2023-09-29T21:58:34.236 DEBUG:boto:Data: 2023-09-29T21:58:34.236 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T21:58:34.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:34.236 DEBUG:boto:Port: 8001 2023-09-29T21:58:34.236 DEBUG:boto:Params: {} 2023-09-29T21:58:34.237 DEBUG:boto:Token: None 2023-09-29T21:58:34.237 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 21:58:34 GMT /hcnphz-12/dummy 2023-09-29T21:58:34.237 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2FjVPpdNwITyC2XBIu9t8c1DXk= 2023-09-29T21:58:34.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2FjVPpdNwITyC2XBIu9t8c1DXk='} 2023-09-29T21:58:34.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-amz-request-id', 'tx000006e104763253e04c1-006517488a-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:34.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:58:34.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:58:34.264 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:58:34.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:58:34.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:58:34.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:34.377+0000 7fe52cb64f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:39.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:39.377+0000 7fe52cb64f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:44.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:44.377+0000 7fe52cb64f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:49.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:49.377+0000 7fe52cb64f40 1 waiting to reach incremental sync.. 2023-09-29T21:58:54.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:54.381+0000 7fe52cb64f40 1 bucket sync caught up with source: 2023-09-29T21:58:54.383 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.11.11, , , , , ] 2023-09-29T21:58:54.383 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.11.11, , , , , ] 2023-09-29T21:58:54.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:54.381+0000 7fe52cb64f40 0 bucket checkpoint complete 2023-09-29T21:58:54.411 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T21:58:54.412 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T21:58:54.412 DEBUG:boto:Method: HEAD 2023-09-29T21:58:54.412 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T21:58:54.412 DEBUG:boto:Data: 2023-09-29T21:58:54.412 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.412 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.412 DEBUG:boto:Params: {} 2023-09-29T21:58:54.413 DEBUG:boto:Token: None 2023-09-29T21:58:54.413 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/ 2023-09-29T21:58:54.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pp3gRdKMtbRh0PnsGg3t4rX1TpA= 2023-09-29T21:58:54.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pp3gRdKMtbRh0PnsGg3t4rX1TpA=', 'Content-Length': '0'} 2023-09-29T21:58:54.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d1c70b7d7e812e1-006517489e-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.417 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.417 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.418 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.418 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T21:58:54.418 DEBUG:boto:Data: 2023-09-29T21:58:54.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.418 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.418 DEBUG:boto:Params: {} 2023-09-29T21:58:54.418 DEBUG:boto:Token: None 2023-09-29T21:58:54.419 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a1 2023-09-29T21:58:54.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KaKQ2HFNWKCUn6yQGfF/kUMEp7k= 2023-09-29T21:58:54.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KaKQ2HFNWKCUn6yQGfF/kUMEp7k='} 2023-09-29T21:58:54.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-amz-request-id', 'tx000002fdc3f3c0a06afbb-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.442 DEBUG:rgw_multi.tests:version1 id=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T21:58:54.442 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.443 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.443 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.443 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T21:58:54.443 DEBUG:boto:Data: 2023-09-29T21:58:54.443 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.443 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.443 DEBUG:boto:Params: {} 2023-09-29T21:58:54.443 DEBUG:boto:Token: None 2023-09-29T21:58:54.444 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a1 2023-09-29T21:58:54.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V05n9huXH/Uh08MoDeOOC5bA0dE= 2023-09-29T21:58:54.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V05n9huXH/Uh08MoDeOOC5bA0dE='} 2023-09-29T21:58:54.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh'), ('x-amz-request-id', 'tx0000034919e73362da11b-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.463 DEBUG:rgw_multi.tests:version2 id=ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh 2023-09-29T21:58:54.463 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.463 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.463 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh 2023-09-29T21:58:54.464 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh 2023-09-29T21:58:54.464 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.464 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh 2023-09-29T21:58:54.464 DEBUG:boto:Data: 2023-09-29T21:58:54.464 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.464 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.464 DEBUG:boto:Params: {} 2023-09-29T21:58:54.465 DEBUG:boto:Token: None 2023-09-29T21:58:54.465 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a1?versionId=ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh 2023-09-29T21:58:54.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aupSNSsmXpE8cX+bZl+3koz6Cek= 2023-09-29T21:58:54.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aupSNSsmXpE8cX+bZl+3koz6Cek=', 'Content-Length': '0'} 2023-09-29T21:58:54.484 DEBUG:boto:Response headers: [('x-amz-version-id', 'ifRvrQMoPgb20i3GIpY6hLqpgMlSlLh'), ('x-amz-request-id', 'tx00000fc68f2a8aa8674e5-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.485 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.485 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.485 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.485 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T21:58:54.485 DEBUG:boto:Data: 2023-09-29T21:58:54.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.486 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.486 DEBUG:boto:Params: {} 2023-09-29T21:58:54.486 DEBUG:boto:Token: None 2023-09-29T21:58:54.486 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a1 2023-09-29T21:58:54.486 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0i1gTdo10GTnGvQWs63sfqINRz8= 2023-09-29T21:58:54.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0i1gTdo10GTnGvQWs63sfqINRz8='} 2023-09-29T21:58:54.530 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PW3H3aTDd.DRSgHjF2ev8NWopjccLtG'), ('x-amz-request-id', 'tx0000059733e5254359db2-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.531 DEBUG:rgw_multi.tests:version3 id=PW3H3aTDd.DRSgHjF2ev8NWopjccLtG 2023-09-29T21:58:54.531 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.531 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T21:58:54.531 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=PW3H3aTDd.DRSgHjF2ev8NWopjccLtG 2023-09-29T21:58:54.531 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=PW3H3aTDd.DRSgHjF2ev8NWopjccLtG 2023-09-29T21:58:54.532 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.532 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=PW3H3aTDd.DRSgHjF2ev8NWopjccLtG 2023-09-29T21:58:54.532 DEBUG:boto:Data: 2023-09-29T21:58:54.532 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.532 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.532 DEBUG:boto:Params: {} 2023-09-29T21:58:54.532 DEBUG:boto:Token: None 2023-09-29T21:58:54.533 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a1?versionId=PW3H3aTDd.DRSgHjF2ev8NWopjccLtG 2023-09-29T21:58:54.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8tF3Pg/2G08TIK7T6c9kxZhLGBo= 2023-09-29T21:58:54.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8tF3Pg/2G08TIK7T6c9kxZhLGBo=', 'Content-Length': '0'} 2023-09-29T21:58:54.553 DEBUG:boto:Response headers: [('x-amz-version-id', 'PW3H3aTDd.DRSgHjF2ev8NWopjccLtG'), ('x-amz-request-id', 'tx000003b4dab42e3178a7e-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.554 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T21:58:54.554 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T21:58:54.554 DEBUG:boto:Method: HEAD 2023-09-29T21:58:54.554 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T21:58:54.554 DEBUG:boto:Data: 2023-09-29T21:58:54.554 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.555 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.555 DEBUG:boto:Params: {} 2023-09-29T21:58:54.555 DEBUG:boto:Token: None 2023-09-29T21:58:54.555 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/ 2023-09-29T21:58:54.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pp3gRdKMtbRh0PnsGg3t4rX1TpA= 2023-09-29T21:58:54.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pp3gRdKMtbRh0PnsGg3t4rX1TpA=', 'Content-Length': '0'} 2023-09-29T21:58:54.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e3b52d37889827a-006517489e-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.559 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.559 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.559 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.559 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T21:58:54.559 DEBUG:boto:Data: 2023-09-29T21:58:54.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.560 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.560 DEBUG:boto:Params: {} 2023-09-29T21:58:54.560 DEBUG:boto:Token: None 2023-09-29T21:58:54.560 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a2 2023-09-29T21:58:54.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eg0wjwfHCgpOA/sd4DkxElSgrH0= 2023-09-29T21:58:54.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eg0wjwfHCgpOA/sd4DkxElSgrH0='} 2023-09-29T21:58:54.606 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-amz-request-id', 'tx000007a55c38d3fddb521-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.607 DEBUG:rgw_multi.tests:version1 id=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T21:58:54.607 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.607 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.607 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.607 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T21:58:54.608 DEBUG:boto:Data: 2023-09-29T21:58:54.608 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.608 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.608 DEBUG:boto:Params: {} 2023-09-29T21:58:54.608 DEBUG:boto:Token: None 2023-09-29T21:58:54.608 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a2 2023-09-29T21:58:54.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ORHGLOb8Zq5/AXS59g/9d1V/5JU= 2023-09-29T21:58:54.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ORHGLOb8Zq5/AXS59g/9d1V/5JU='} 2023-09-29T21:58:54.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D'), ('x-amz-request-id', 'tx00000160f5363381a5d8a-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.629 DEBUG:rgw_multi.tests:version2 id=2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D 2023-09-29T21:58:54.629 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.629 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.629 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D 2023-09-29T21:58:54.629 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D 2023-09-29T21:58:54.630 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.630 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D 2023-09-29T21:58:54.630 DEBUG:boto:Data: 2023-09-29T21:58:54.630 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.630 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.630 DEBUG:boto:Params: {} 2023-09-29T21:58:54.630 DEBUG:boto:Token: None 2023-09-29T21:58:54.631 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a2?versionId=2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D 2023-09-29T21:58:54.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sz+Xw7+qTDT89WtObpZDXkVrAGM= 2023-09-29T21:58:54.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sz+Xw7+qTDT89WtObpZDXkVrAGM=', 'Content-Length': '0'} 2023-09-29T21:58:54.651 DEBUG:boto:Response headers: [('x-amz-version-id', '2iIeBA2nMMCkpHmXt08MGHFbn8u8G-D'), ('x-amz-request-id', 'tx000003bbe30b397e6f09c-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.651 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.652 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.652 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.652 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T21:58:54.652 DEBUG:boto:Data: 2023-09-29T21:58:54.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.652 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.652 DEBUG:boto:Params: {} 2023-09-29T21:58:54.652 DEBUG:boto:Token: None 2023-09-29T21:58:54.653 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a2 2023-09-29T21:58:54.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T3/1ZdFb3Kf3YBAXoNNGfcULfP0= 2023-09-29T21:58:54.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T3/1ZdFb3Kf3YBAXoNNGfcULfP0='} 2023-09-29T21:58:54.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL'), ('x-amz-request-id', 'tx0000032999b8c3b009350-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.699 DEBUG:rgw_multi.tests:version3 id=WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL 2023-09-29T21:58:54.699 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.699 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T21:58:54.699 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL 2023-09-29T21:58:54.699 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL 2023-09-29T21:58:54.699 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.700 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL 2023-09-29T21:58:54.700 DEBUG:boto:Data: 2023-09-29T21:58:54.700 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.700 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.700 DEBUG:boto:Params: {} 2023-09-29T21:58:54.700 DEBUG:boto:Token: None 2023-09-29T21:58:54.700 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-11/obj-a2?versionId=WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL 2023-09-29T21:58:54.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XJFsoT0P92Z3g5imsV5PvBvikSo= 2023-09-29T21:58:54.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XJFsoT0P92Z3g5imsV5PvBvikSo=', 'Content-Length': '0'} 2023-09-29T21:58:54.722 DEBUG:boto:Response headers: [('x-amz-version-id', 'WaFChDsXZ9JtCh4Bh2AnSCCemcHglsL'), ('x-amz-request-id', 'tx00000c3d6051f207fb153-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.723 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T21:58:54.723 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T21:58:54.723 DEBUG:boto:Method: HEAD 2023-09-29T21:58:54.723 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T21:58:54.723 DEBUG:boto:Data: 2023-09-29T21:58:54.723 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.723 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.723 DEBUG:boto:Params: {} 2023-09-29T21:58:54.724 DEBUG:boto:Token: None 2023-09-29T21:58:54.724 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/ 2023-09-29T21:58:54.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zYH/CmAQ5dmg5cNGJKZhKO3d8s= 2023-09-29T21:58:54.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zYH/CmAQ5dmg5cNGJKZhKO3d8s=', 'Content-Length': '0'} 2023-09-29T21:58:54.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009960c4ec5ac05abb-006517489e-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.728 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.728 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.728 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.728 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T21:58:54.728 DEBUG:boto:Data: 2023-09-29T21:58:54.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.728 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.729 DEBUG:boto:Params: {} 2023-09-29T21:58:54.729 DEBUG:boto:Token: None 2023-09-29T21:58:54.729 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a1 2023-09-29T21:58:54.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:StOtEpJAWKlY+uRlIvhMDWpPQlg= 2023-09-29T21:58:54.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:StOtEpJAWKlY+uRlIvhMDWpPQlg='} 2023-09-29T21:58:54.756 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-amz-request-id', 'tx00000aa6c3c2526130db1-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.756 DEBUG:rgw_multi.tests:version1 id=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T21:58:54.756 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.757 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.757 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.757 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T21:58:54.757 DEBUG:boto:Data: 2023-09-29T21:58:54.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.757 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.757 DEBUG:boto:Params: {} 2023-09-29T21:58:54.757 DEBUG:boto:Token: None 2023-09-29T21:58:54.758 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a1 2023-09-29T21:58:54.758 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o0d5Nb+9V5DR6+wnbu7gfGK1jMs= 2023-09-29T21:58:54.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o0d5Nb+9V5DR6+wnbu7gfGK1jMs='} 2023-09-29T21:58:54.803 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'WzElwhPTMrrh3tKhzn7gxXWtOErwASv'), ('x-amz-request-id', 'tx000000fc556d60a7aedaf-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.803 DEBUG:rgw_multi.tests:version2 id=WzElwhPTMrrh3tKhzn7gxXWtOErwASv 2023-09-29T21:58:54.803 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.803 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.803 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=WzElwhPTMrrh3tKhzn7gxXWtOErwASv 2023-09-29T21:58:54.803 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=WzElwhPTMrrh3tKhzn7gxXWtOErwASv 2023-09-29T21:58:54.803 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.804 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=WzElwhPTMrrh3tKhzn7gxXWtOErwASv 2023-09-29T21:58:54.804 DEBUG:boto:Data: 2023-09-29T21:58:54.804 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.804 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.804 DEBUG:boto:Params: {} 2023-09-29T21:58:54.804 DEBUG:boto:Token: None 2023-09-29T21:58:54.804 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a1?versionId=WzElwhPTMrrh3tKhzn7gxXWtOErwASv 2023-09-29T21:58:54.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:em9/leS5aBtJVRh2OhSYEjN5kZ4= 2023-09-29T21:58:54.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:em9/leS5aBtJVRh2OhSYEjN5kZ4=', 'Content-Length': '0'} 2023-09-29T21:58:54.827 DEBUG:boto:Response headers: [('x-amz-version-id', 'WzElwhPTMrrh3tKhzn7gxXWtOErwASv'), ('x-amz-request-id', 'tx000002224ad0fa3299b4c-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.828 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.828 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.828 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.828 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T21:58:54.829 DEBUG:boto:Data: 2023-09-29T21:58:54.829 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.829 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.829 DEBUG:boto:Params: {} 2023-09-29T21:58:54.829 DEBUG:boto:Token: None 2023-09-29T21:58:54.829 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a1 2023-09-29T21:58:54.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nlC6cHzX41MWbzbQfGi6VdWQk/0= 2023-09-29T21:58:54.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nlC6cHzX41MWbzbQfGi6VdWQk/0='} 2023-09-29T21:58:54.848 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Yp9xwA-hHYZoinTKhztITI6rAUkzvLy'), ('x-amz-request-id', 'tx00000ee071fcb7358199d-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.848 DEBUG:rgw_multi.tests:version3 id=Yp9xwA-hHYZoinTKhztITI6rAUkzvLy 2023-09-29T21:58:54.848 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.848 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T21:58:54.848 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=Yp9xwA-hHYZoinTKhztITI6rAUkzvLy 2023-09-29T21:58:54.848 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=Yp9xwA-hHYZoinTKhztITI6rAUkzvLy 2023-09-29T21:58:54.849 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.849 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=Yp9xwA-hHYZoinTKhztITI6rAUkzvLy 2023-09-29T21:58:54.849 DEBUG:boto:Data: 2023-09-29T21:58:54.849 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T21:58:54.849 DEBUG:boto:Port: 8000 2023-09-29T21:58:54.849 DEBUG:boto:Params: {} 2023-09-29T21:58:54.849 DEBUG:boto:Token: None 2023-09-29T21:58:54.849 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a1?versionId=Yp9xwA-hHYZoinTKhztITI6rAUkzvLy 2023-09-29T21:58:54.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yUtwODUHLUMguGkIRh3Df4vybJ4= 2023-09-29T21:58:54.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUtwODUHLUMguGkIRh3Df4vybJ4=', 'Content-Length': '0'} 2023-09-29T21:58:54.870 DEBUG:boto:Response headers: [('x-amz-version-id', 'Yp9xwA-hHYZoinTKhztITI6rAUkzvLy'), ('x-amz-request-id', 'tx000005a5edca22a5451c4-006517489e-1122-a1'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.871 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T21:58:54.871 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T21:58:54.871 DEBUG:boto:Method: HEAD 2023-09-29T21:58:54.871 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T21:58:54.871 DEBUG:boto:Data: 2023-09-29T21:58:54.871 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.871 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.872 DEBUG:boto:Params: {} 2023-09-29T21:58:54.872 DEBUG:boto:Token: None 2023-09-29T21:58:54.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/ 2023-09-29T21:58:54.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zYH/CmAQ5dmg5cNGJKZhKO3d8s= 2023-09-29T21:58:54.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zYH/CmAQ5dmg5cNGJKZhKO3d8s=', 'Content-Length': '0'} 2023-09-29T21:58:54.875 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087c8d64205b40be6-006517489e-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.876 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.876 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.876 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.876 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T21:58:54.876 DEBUG:boto:Data: 2023-09-29T21:58:54.876 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.877 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.877 DEBUG:boto:Params: {} 2023-09-29T21:58:54.877 DEBUG:boto:Token: None 2023-09-29T21:58:54.877 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a2 2023-09-29T21:58:54.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:im6N7NqThUPFjaH9TxYsUBajh4U= 2023-09-29T21:58:54.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:im6N7NqThUPFjaH9TxYsUBajh4U='} 2023-09-29T21:58:54.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-amz-request-id', 'tx0000055c63fa9bde3c3b8-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.896 DEBUG:rgw_multi.tests:version1 id=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T21:58:54.896 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.897 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.897 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.897 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T21:58:54.897 DEBUG:boto:Data: 2023-09-29T21:58:54.897 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.897 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.897 DEBUG:boto:Params: {} 2023-09-29T21:58:54.898 DEBUG:boto:Token: None 2023-09-29T21:58:54.898 DEBUG:boto:StringToSign: PUT Lg6VKF8IoH3qF+fuERshyA== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a2 2023-09-29T21:58:54.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8Hx5KqMGb/CJc9kkvmJuJIroJwA= 2023-09-29T21:58:54.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'Lg6VKF8IoH3qF+fuERshyA==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8Hx5KqMGb/CJc9kkvmJuJIroJwA='} 2023-09-29T21:58:54.943 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"2e0e95285f08a07dea17e7ee111b21c8"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI'), ('x-amz-request-id', 'tx00000ea4fb1b482fc7e27-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.943 DEBUG:rgw_multi.tests:version2 id=DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI 2023-09-29T21:58:54.943 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.943 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.943 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI 2023-09-29T21:58:54.943 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI 2023-09-29T21:58:54.944 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.944 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI 2023-09-29T21:58:54.944 DEBUG:boto:Data: 2023-09-29T21:58:54.944 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.944 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.944 DEBUG:boto:Params: {} 2023-09-29T21:58:54.945 DEBUG:boto:Token: None 2023-09-29T21:58:54.945 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a2?versionId=DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI 2023-09-29T21:58:54.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:al7czhwxnacCanFZVNuOg57praw= 2023-09-29T21:58:54.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:al7czhwxnacCanFZVNuOg57praw=', 'Content-Length': '0'} 2023-09-29T21:58:54.967 DEBUG:boto:Response headers: [('x-amz-version-id', 'DL1uRKVdJ5jnSs448YFDFfzQsc0xaxI'), ('x-amz-request-id', 'tx00000b730cdcbed6daa10-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.968 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.968 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.968 DEBUG:boto:Method: PUT 2023-09-29T21:58:54.968 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T21:58:54.968 DEBUG:boto:Data: 2023-09-29T21:58:54.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T21:58:54.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.969 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.969 DEBUG:boto:Params: {} 2023-09-29T21:58:54.969 DEBUG:boto:Token: None 2023-09-29T21:58:54.969 DEBUG:boto:StringToSign: PUT HiNA5/HU7CwjRIhUZJ8d7Q== application/octet-stream Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a2 2023-09-29T21:58:54.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xOUnKRjCW1F5wS6iOdgczzfMLPI= 2023-09-29T21:58:54.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'HiNA5/HU7CwjRIhUZJ8d7Q==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xOUnKRjCW1F5wS6iOdgczzfMLPI='} 2023-09-29T21:58:54.989 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"1e2340e7f1d4ec2c23448854649f1ded"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Sjvyy6TgXogtlsyADwGBFbMglYchCxj'), ('x-amz-request-id', 'tx0000018b9bb14b06c2e15-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:54.990 DEBUG:rgw_multi.tests:version3 id=Sjvyy6TgXogtlsyADwGBFbMglYchCxj 2023-09-29T21:58:54.990 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.990 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T21:58:54.990 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=Sjvyy6TgXogtlsyADwGBFbMglYchCxj 2023-09-29T21:58:54.990 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=Sjvyy6TgXogtlsyADwGBFbMglYchCxj 2023-09-29T21:58:54.990 DEBUG:boto:Method: DELETE 2023-09-29T21:58:54.990 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=Sjvyy6TgXogtlsyADwGBFbMglYchCxj 2023-09-29T21:58:54.991 DEBUG:boto:Data: 2023-09-29T21:58:54.991 DEBUG:boto:Headers: {} 2023-09-29T21:58:54.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T21:58:54.991 DEBUG:boto:Port: 8001 2023-09-29T21:58:54.991 DEBUG:boto:Params: {} 2023-09-29T21:58:54.991 DEBUG:boto:Token: None 2023-09-29T21:58:54.991 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 21:58:54 GMT /hcnphz-12/obj-a2?versionId=Sjvyy6TgXogtlsyADwGBFbMglYchCxj 2023-09-29T21:58:54.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:24qovGjyGJAJ5BaQ2/WNR3F9ZmM= 2023-09-29T21:58:54.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 21:58:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:24qovGjyGJAJ5BaQ2/WNR3F9ZmM=', 'Content-Length': '0'} 2023-09-29T21:58:55.012 DEBUG:boto:Response headers: [('x-amz-version-id', 'Sjvyy6TgXogtlsyADwGBFbMglYchCxj'), ('x-amz-request-id', 'tx000009d93cd1a2eb04656-006517489e-1126-a2'), ('Date', 'Fri, 29 Sep 2023 21:58:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T21:58:55.013 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-11 2023-09-29T21:58:55.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:58:55.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T21:58:55.014 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T21:58:55.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T21:58:55.067 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T21:58:55.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:58:55.133+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:58:55.134 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-29T21:58:55.134 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:00.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:00.136+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:00.137 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-29T21:59:00.137 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:05.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:05.136+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:05.139 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-29T21:59:05.139 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:10.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:10.140+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:10.142 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.8.11, , , , , ] 2023-09-29T21:59:10.142 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:15.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:15.144+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:15.144 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , , 00000000001.13.6, ] 2023-09-29T21:59:15.144 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:20.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:20.144+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:20.147 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , , 00000000001.13.6, ] 2023-09-29T21:59:20.147 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:25.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:25.148+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:25.149 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , , 00000000001.13.6, ] 2023-09-29T21:59:25.149 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:30.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:30.152+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:30.152 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , , 00000000001.13.6, ] 2023-09-29T21:59:30.152 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:35.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:35.152+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:35.155 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T21:59:35.155 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:40.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:40.155+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:40.157 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T21:59:40.157 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:45.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:45.155+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:45.159 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T21:59:45.160 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:50.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:50.159+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:50.162 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T21:59:50.162 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T21:59:55.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T21:59:55.163+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T21:59:55.165 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T21:59:55.165 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T22:00:00.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:00.163+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:00.167 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T22:00:00.167 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T22:00:05.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:05.167+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:05.170 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T22:00:05.170 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T22:00:10.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:10.171+0000 7fc17cc10f40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:10.172 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000001.13.6, ] 2023-09-29T22:00:10.172 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T22:00:15.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:15.171+0000 7fc17cc10f40 1 bucket sync caught up with source: 2023-09-29T22:00:15.175 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000010.30.6, ] 2023-09-29T22:00:15.175 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , , 00000000010.30.6, ] 2023-09-29T22:00:15.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:15.171+0000 7fc17cc10f40 0 bucket checkpoint complete 2023-09-29T22:00:15.203 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-11 2023-09-29T22:00:15.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:15.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:15.204 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:15.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:15.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:15.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:15.323+0000 7f4e2729cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:15.324 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000010.37.6, , ] 2023-09-29T22:00:15.324 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000010.37.6, 00000000002.18.11, ] 2023-09-29T22:00:20.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:20.322+0000 7f4e2729cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:20.327 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000010.37.6, , ] 2023-09-29T22:00:20.327 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000010.37.6, 00000000002.18.11, ] 2023-09-29T22:00:25.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:25.326+0000 7f4e2729cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:25.329 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000010.37.6, , ] 2023-09-29T22:00:25.329 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000010.37.6, 00000000002.18.11, ] 2023-09-29T22:00:30.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:30.330+0000 7f4e2729cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:30.331 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000010.37.6, , ] 2023-09-29T22:00:30.332 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000010.37.6, 00000000002.18.11, ] 2023-09-29T22:00:35.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:35.330+0000 7f4e2729cf40 1 bucket sync caught up with source: 2023-09-29T22:00:35.334 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000010.37.6, 00000000002.18.11, ] 2023-09-29T22:00:35.335 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000010.37.6, 00000000002.18.11, ] 2023-09-29T22:00:35.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:35.330+0000 7f4e2729cf40 0 bucket checkpoint complete 2023-09-29T22:00:35.361 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-11 zones={a2, a1} 2023-09-29T22:00:35.362 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.362 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.362 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.362 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T22:00:35.362 DEBUG:boto:Data: 2023-09-29T22:00:35.362 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.363 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.363 DEBUG:boto:Params: {} 2023-09-29T22:00:35.363 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:00:35.363 DEBUG:boto:Token: None 2023-09-29T22:00:35.364 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OYHLFgLPUYBhKvmYtup2JPaJhx0= 2023-09-29T22:00:35.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OYHLFgLPUYBhKvmYtup2JPaJhx0=', 'Content-Length': '0'} 2023-09-29T22:00:35.370 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000042c498de670f1540-0065174903-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.370 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.370 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.371 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.371 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T22:00:35.371 DEBUG:boto:Data: 2023-09-29T22:00:35.371 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.371 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.371 DEBUG:boto:Params: {} 2023-09-29T22:00:35.371 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:00:35.372 DEBUG:boto:Token: None 2023-09-29T22:00:35.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OYHLFgLPUYBhKvmYtup2JPaJhx0= 2023-09-29T22:00:35.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OYHLFgLPUYBhKvmYtup2JPaJhx0=', 'Content-Length': '0'} 2023-09-29T22:00:35.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009142a7d49928a177-0065174903-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:00:35.378 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.378 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.378 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.378 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.378 DEBUG:boto:Method: GET 2023-09-29T22:00:35.378 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.378 DEBUG:boto:Data: 2023-09-29T22:00:35.378 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.379 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.379 DEBUG:boto:Params: {} 2023-09-29T22:00:35.379 DEBUG:boto:Token: None 2023-09-29T22:00:35.379 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/?versions 2023-09-29T22:00:35.379 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0= 2023-09-29T22:00:35.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0=', 'Content-Length': '0'} 2023-09-29T22:00:35.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce215c40a8109b6-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.384 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.385 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:35.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:35.385 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:35.385 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:00:35.385 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.386 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.386 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.386 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.386 DEBUG:boto:Method: GET 2023-09-29T22:00:35.386 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.386 DEBUG:boto:Data: 2023-09-29T22:00:35.386 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.386 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.387 DEBUG:boto:Params: {} 2023-09-29T22:00:35.387 DEBUG:boto:Token: None 2023-09-29T22:00:35.387 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/?versions 2023-09-29T22:00:35.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0= 2023-09-29T22:00:35.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0=', 'Content-Length': '0'} 2023-09-29T22:00:35.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c896b61f1ae8c06-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.391 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.392 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:35.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:35.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:35.392 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.392 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.393 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.393 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.393 DEBUG:boto:Method: GET 2023-09-29T22:00:35.393 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.393 DEBUG:boto:Data: 2023-09-29T22:00:35.393 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.393 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.393 DEBUG:boto:Params: {} 2023-09-29T22:00:35.393 DEBUG:boto:Token: None 2023-09-29T22:00:35.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/?versions 2023-09-29T22:00:35.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0= 2023-09-29T22:00:35.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0=', 'Content-Length': '0'} 2023-09-29T22:00:35.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001823aa113a5677f8-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.398 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.401 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.402 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.402 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.402 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.402 DEBUG:boto:Method: GET 2023-09-29T22:00:35.402 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:35.402 DEBUG:boto:Data: 2023-09-29T22:00:35.402 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.403 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.403 DEBUG:boto:Params: {} 2023-09-29T22:00:35.403 DEBUG:boto:Token: None 2023-09-29T22:00:35.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/?versions 2023-09-29T22:00:35.403 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0= 2023-09-29T22:00:35.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rQ91n1ReltMC3KISvY2+CwIs2A0=', 'Content-Length': '0'} 2023-09-29T22:00:35.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1d99043bdb94346-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.408 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:35.409 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.409 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.409 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.409 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.409 DEBUG:boto:Method: GET 2023-09-29T22:00:35.409 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.410 DEBUG:boto:Data: 2023-09-29T22:00:35.410 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.410 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.410 DEBUG:boto:Params: {} 2023-09-29T22:00:35.410 DEBUG:boto:Token: None 2023-09-29T22:00:35.410 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc= 2023-09-29T22:00:35.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc=', 'Content-Length': '0'} 2023-09-29T22:00:35.413 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000048781de56a0665c5-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.413 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.413 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.413 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.414 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.414 DEBUG:boto:Method: GET 2023-09-29T22:00:35.414 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.414 DEBUG:boto:Data: 2023-09-29T22:00:35.414 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.414 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.414 DEBUG:boto:Params: {} 2023-09-29T22:00:35.414 DEBUG:boto:Token: None 2023-09-29T22:00:35.415 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc= 2023-09-29T22:00:35.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc=', 'Content-Length': '0'} 2023-09-29T22:00:35.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bdf11ed6ca7dbf2b-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.417 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.417 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.418 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.418 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.418 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.418 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.418 DEBUG:boto:Data: 2023-09-29T22:00:35.418 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.418 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.418 DEBUG:boto:Params: {} 2023-09-29T22:00:35.418 DEBUG:boto:Token: None 2023-09-29T22:00:35.419 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0VMq2QtsUiyVutyKwFYemYptDhE= 2023-09-29T22:00:35.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0VMq2QtsUiyVutyKwFYemYptDhE=', 'Content-Length': '0'} 2023-09-29T22:00:35.421 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f04d620d0f8ec761-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.421 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.422 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.422 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.422 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.422 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.422 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.422 DEBUG:boto:Data: 2023-09-29T22:00:35.422 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.422 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.422 DEBUG:boto:Params: {} 2023-09-29T22:00:35.423 DEBUG:boto:Token: None 2023-09-29T22:00:35.423 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.423 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0VMq2QtsUiyVutyKwFYemYptDhE= 2023-09-29T22:00:35.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0VMq2QtsUiyVutyKwFYemYptDhE=', 'Content-Length': '0'} 2023-09-29T22:00:35.425 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d7e17d024a29d826-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:35.426 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.426 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.426 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.426 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.426 DEBUG:boto:Method: GET 2023-09-29T22:00:35.426 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.426 DEBUG:boto:Data: 2023-09-29T22:00:35.426 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.427 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.427 DEBUG:boto:Params: {} 2023-09-29T22:00:35.427 DEBUG:boto:Token: None 2023-09-29T22:00:35.427 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc= 2023-09-29T22:00:35.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc=', 'Content-Length': '0'} 2023-09-29T22:00:35.429 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001c619c1da71ad96f-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.430 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.430 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.430 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.430 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.430 DEBUG:boto:Method: GET 2023-09-29T22:00:35.430 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.430 DEBUG:boto:Data: 2023-09-29T22:00:35.431 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.431 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.431 DEBUG:boto:Params: {} 2023-09-29T22:00:35.431 DEBUG:boto:Token: None 2023-09-29T22:00:35.431 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.431 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc= 2023-09-29T22:00:35.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc=', 'Content-Length': '0'} 2023-09-29T22:00:35.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e792e5b09c037630-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.434 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.434 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.435 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.435 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.435 DEBUG:boto:Method: GET 2023-09-29T22:00:35.435 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.435 DEBUG:boto:Data: 2023-09-29T22:00:35.435 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.435 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.435 DEBUG:boto:Params: {} 2023-09-29T22:00:35.435 DEBUG:boto:Token: None 2023-09-29T22:00:35.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.440 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080963b4b0bc6d41e-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.440 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.441 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.441 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.441 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.441 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.441 DEBUG:boto:Method: GET 2023-09-29T22:00:35.441 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.442 DEBUG:boto:Data: 2023-09-29T22:00:35.442 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.442 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.442 DEBUG:boto:Params: {} 2023-09-29T22:00:35.442 DEBUG:boto:Token: None 2023-09-29T22:00:35.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6d78bffa6f64755-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.490 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.491 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.491 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.491 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.491 DEBUG:boto:Path: /hcnphz-11/dummy 2023-09-29T22:00:35.491 DEBUG:boto:Data: 2023-09-29T22:00:35.491 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.492 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.492 DEBUG:boto:Params: {} 2023-09-29T22:00:35.492 DEBUG:boto:Token: None 2023-09-29T22:00:35.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy 2023-09-29T22:00:35.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EjJju6ktinaU2RCQwSkfBKBM9HQ= 2023-09-29T22:00:35.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EjJju6ktinaU2RCQwSkfBKBM9HQ=', 'Content-Length': '0'} 2023-09-29T22:00:35.496 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000dde99e8cb1ad6fdf-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.497 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.497 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.497 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.497 DEBUG:boto:Path: /hcnphz-11/dummy 2023-09-29T22:00:35.497 DEBUG:boto:Data: 2023-09-29T22:00:35.497 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.497 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.497 DEBUG:boto:Params: {} 2023-09-29T22:00:35.498 DEBUG:boto:Token: None 2023-09-29T22:00:35.498 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy 2023-09-29T22:00:35.498 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EjJju6ktinaU2RCQwSkfBKBM9HQ= 2023-09-29T22:00:35.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EjJju6ktinaU2RCQwSkfBKBM9HQ=', 'Content-Length': '0'} 2023-09-29T22:00:35.501 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002328a0faf41fa0ea-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:35.502 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.502 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.502 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.502 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.502 DEBUG:boto:Method: GET 2023-09-29T22:00:35.502 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.502 DEBUG:boto:Data: 2023-09-29T22:00:35.503 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.503 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.503 DEBUG:boto:Params: {} 2023-09-29T22:00:35.503 DEBUG:boto:Token: None 2023-09-29T22:00:35.503 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc= 2023-09-29T22:00:35.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc=', 'Content-Length': '0'} 2023-09-29T22:00:35.506 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000429307317b2fb3b1-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.506 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:35.507 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:35.507 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.507 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.507 DEBUG:boto:Method: GET 2023-09-29T22:00:35.507 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.507 DEBUG:boto:Data: 2023-09-29T22:00:35.507 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.507 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.507 DEBUG:boto:Params: {} 2023-09-29T22:00:35.508 DEBUG:boto:Token: None 2023-09-29T22:00:35.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:35.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc= 2023-09-29T22:00:35.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d92OW2QD4KCdHzI9WuIdx9XeEzc=', 'Content-Length': '0'} 2023-09-29T22:00:35.511 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000010a338c2d53a8415-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.511 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.511 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.511 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.511 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.512 DEBUG:boto:Method: GET 2023-09-29T22:00:35.512 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.512 DEBUG:boto:Data: 2023-09-29T22:00:35.512 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.512 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.512 DEBUG:boto:Params: {} 2023-09-29T22:00:35.512 DEBUG:boto:Token: None 2023-09-29T22:00:35.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094f59878bc066250-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.558 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.559 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.559 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.559 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.559 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.559 DEBUG:boto:Method: GET 2023-09-29T22:00:35.559 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:35.559 DEBUG:boto:Data: 2023-09-29T22:00:35.560 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.560 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.560 DEBUG:boto:Params: {} 2023-09-29T22:00:35.560 DEBUG:boto:Token: None 2023-09-29T22:00:35.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000128557f5968e621c-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.566 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:35.566 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.567 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.567 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.567 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.567 DEBUG:boto:Method: GET 2023-09-29T22:00:35.567 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.567 DEBUG:boto:Data: 2023-09-29T22:00:35.567 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.567 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.568 DEBUG:boto:Params: {} 2023-09-29T22:00:35.568 DEBUG:boto:Token: None 2023-09-29T22:00:35.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8= 2023-09-29T22:00:35.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8=', 'Content-Length': '0'} 2023-09-29T22:00:35.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004b0fd5524cbdf34a-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.571 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.572 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.572 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.572 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.572 DEBUG:boto:Method: GET 2023-09-29T22:00:35.572 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.572 DEBUG:boto:Data: 2023-09-29T22:00:35.572 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.572 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.573 DEBUG:boto:Params: {} 2023-09-29T22:00:35.573 DEBUG:boto:Token: None 2023-09-29T22:00:35.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8= 2023-09-29T22:00:35.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8=', 'Content-Length': '0'} 2023-09-29T22:00:35.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0623de32214c4e9-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.576 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.576 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.576 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.577 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.577 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.577 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.577 DEBUG:boto:Data: 2023-09-29T22:00:35.577 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.577 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.577 DEBUG:boto:Params: {} 2023-09-29T22:00:35.577 DEBUG:boto:Token: None 2023-09-29T22:00:35.578 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.578 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dvHGwEn3jHPVPfrsTpHNC3zIq/0= 2023-09-29T22:00:35.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dvHGwEn3jHPVPfrsTpHNC3zIq/0=', 'Content-Length': '0'} 2023-09-29T22:00:35.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c552e82b76c8b9f7-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.581 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.581 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.581 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.581 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.581 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.581 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.581 DEBUG:boto:Data: 2023-09-29T22:00:35.581 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.582 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.582 DEBUG:boto:Params: {} 2023-09-29T22:00:35.582 DEBUG:boto:Token: None 2023-09-29T22:00:35.582 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dvHGwEn3jHPVPfrsTpHNC3zIq/0= 2023-09-29T22:00:35.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dvHGwEn3jHPVPfrsTpHNC3zIq/0=', 'Content-Length': '0'} 2023-09-29T22:00:35.585 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fb67663a1537506c-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:35.585 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.585 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.585 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.585 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.585 DEBUG:boto:Method: GET 2023-09-29T22:00:35.586 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.586 DEBUG:boto:Data: 2023-09-29T22:00:35.586 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.586 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.586 DEBUG:boto:Params: {} 2023-09-29T22:00:35.586 DEBUG:boto:Token: None 2023-09-29T22:00:35.586 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8= 2023-09-29T22:00:35.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8=', 'Content-Length': '0'} 2023-09-29T22:00:35.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000da50fbaf8c36ffa7-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.630 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.631 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.631 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.631 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.631 DEBUG:boto:Method: GET 2023-09-29T22:00:35.631 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.631 DEBUG:boto:Data: 2023-09-29T22:00:35.631 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.631 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.632 DEBUG:boto:Params: {} 2023-09-29T22:00:35.632 DEBUG:boto:Token: None 2023-09-29T22:00:35.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8= 2023-09-29T22:00:35.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8=', 'Content-Length': '0'} 2023-09-29T22:00:35.635 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006c490fa0301f1b90-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.635 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.635 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.635 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.636 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.636 DEBUG:boto:Method: GET 2023-09-29T22:00:35.636 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.636 DEBUG:boto:Data: 2023-09-29T22:00:35.636 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.636 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.636 DEBUG:boto:Params: {} 2023-09-29T22:00:35.636 DEBUG:boto:Token: None 2023-09-29T22:00:35.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d9a7edf7d955ab5-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.686 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.687 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.687 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.687 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.687 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.687 DEBUG:boto:Method: GET 2023-09-29T22:00:35.687 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.688 DEBUG:boto:Data: 2023-09-29T22:00:35.688 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.688 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.688 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.688 DEBUG:boto:Params: {} 2023-09-29T22:00:35.688 DEBUG:boto:Token: None 2023-09-29T22:00:35.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a1386984786b70b-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.694 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.694 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.694 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.694 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.694 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T22:00:35.694 DEBUG:boto:Data: 2023-09-29T22:00:35.694 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.695 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.695 DEBUG:boto:Params: {} 2023-09-29T22:00:35.695 DEBUG:boto:Token: None 2023-09-29T22:00:35.695 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1 2023-09-29T22:00:35.695 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IBRjkG/o6JZWRBYICxCryca0UWU= 2023-09-29T22:00:35.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBRjkG/o6JZWRBYICxCryca0UWU=', 'Content-Length': '0'} 2023-09-29T22:00:35.699 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fab21d5f9c90e76-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.699 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.699 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.699 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.699 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T22:00:35.700 DEBUG:boto:Data: 2023-09-29T22:00:35.700 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.700 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.700 DEBUG:boto:Params: {} 2023-09-29T22:00:35.700 DEBUG:boto:Token: None 2023-09-29T22:00:35.700 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1 2023-09-29T22:00:35.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IBRjkG/o6JZWRBYICxCryca0UWU= 2023-09-29T22:00:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBRjkG/o6JZWRBYICxCryca0UWU=', 'Content-Length': '0'} 2023-09-29T22:00:35.704 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025d2c4e149acb621-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:35.704 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.705 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.705 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.705 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.705 DEBUG:boto:Method: GET 2023-09-29T22:00:35.705 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.705 DEBUG:boto:Data: 2023-09-29T22:00:35.705 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.705 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.705 DEBUG:boto:Params: {} 2023-09-29T22:00:35.706 DEBUG:boto:Token: None 2023-09-29T22:00:35.706 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.706 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8= 2023-09-29T22:00:35.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8=', 'Content-Length': '0'} 2023-09-29T22:00:35.708 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c3a53013955d394e-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.754 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.755 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:35.755 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.755 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.755 DEBUG:boto:Method: GET 2023-09-29T22:00:35.755 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.755 DEBUG:boto:Data: 2023-09-29T22:00:35.755 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.755 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.755 DEBUG:boto:Params: {} 2023-09-29T22:00:35.756 DEBUG:boto:Token: None 2023-09-29T22:00:35.756 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:35.756 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8= 2023-09-29T22:00:35.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rgOY7xaRda2DafEj8iu9pzZGiG8=', 'Content-Length': '0'} 2023-09-29T22:00:35.759 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fae2041e2d8d3e0-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.759 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.759 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.759 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.760 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.760 DEBUG:boto:Method: GET 2023-09-29T22:00:35.760 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.760 DEBUG:boto:Data: 2023-09-29T22:00:35.760 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.760 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.760 DEBUG:boto:Params: {} 2023-09-29T22:00:35.760 DEBUG:boto:Token: None 2023-09-29T22:00:35.761 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004b87fdf76c46641d-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.809 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.810 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.810 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.810 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.810 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.811 DEBUG:boto:Method: GET 2023-09-29T22:00:35.811 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:35.811 DEBUG:boto:Data: 2023-09-29T22:00:35.811 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.811 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.811 DEBUG:boto:Params: {} 2023-09-29T22:00:35.811 DEBUG:boto:Token: None 2023-09-29T22:00:35.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.817 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a380020fc4113be0-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.818 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:35.818 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.818 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.818 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.818 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.819 DEBUG:boto:Method: GET 2023-09-29T22:00:35.819 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.819 DEBUG:boto:Data: 2023-09-29T22:00:35.819 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.819 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.819 DEBUG:boto:Params: {} 2023-09-29T22:00:35.819 DEBUG:boto:Token: None 2023-09-29T22:00:35.819 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE= 2023-09-29T22:00:35.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE=', 'Content-Length': '0'} 2023-09-29T22:00:35.823 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001e9b83bb6d5d67be-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.823 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.823 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.823 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.824 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.824 DEBUG:boto:Method: GET 2023-09-29T22:00:35.824 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.824 DEBUG:boto:Data: 2023-09-29T22:00:35.824 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.824 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.824 DEBUG:boto:Params: {} 2023-09-29T22:00:35.824 DEBUG:boto:Token: None 2023-09-29T22:00:35.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE= 2023-09-29T22:00:35.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE=', 'Content-Length': '0'} 2023-09-29T22:00:35.827 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f22dd98777950237-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.828 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.828 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.828 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.828 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.828 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.828 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.828 DEBUG:boto:Data: 2023-09-29T22:00:35.828 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.829 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.829 DEBUG:boto:Params: {} 2023-09-29T22:00:35.829 DEBUG:boto:Token: None 2023-09-29T22:00:35.829 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4CFMCba5RY/6+WYK9xoM6lSfu4g= 2023-09-29T22:00:35.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4CFMCba5RY/6+WYK9xoM6lSfu4g=', 'Content-Length': '0'} 2023-09-29T22:00:35.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000192d15ccd9308fbb-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.832 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.832 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.832 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.832 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.832 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.832 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.832 DEBUG:boto:Data: 2023-09-29T22:00:35.832 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.833 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.833 DEBUG:boto:Params: {} 2023-09-29T22:00:35.833 DEBUG:boto:Token: None 2023-09-29T22:00:35.833 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4CFMCba5RY/6+WYK9xoM6lSfu4g= 2023-09-29T22:00:35.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4CFMCba5RY/6+WYK9xoM6lSfu4g=', 'Content-Length': '0'} 2023-09-29T22:00:35.835 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009196d6efc5995241-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:35.835 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.836 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.836 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.836 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.836 DEBUG:boto:Method: GET 2023-09-29T22:00:35.836 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.836 DEBUG:boto:Data: 2023-09-29T22:00:35.836 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.836 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.837 DEBUG:boto:Params: {} 2023-09-29T22:00:35.837 DEBUG:boto:Token: None 2023-09-29T22:00:35.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE= 2023-09-29T22:00:35.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE=', 'Content-Length': '0'} 2023-09-29T22:00:35.839 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ac34f4cb6095a327-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.882 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.883 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.883 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.883 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.883 DEBUG:boto:Method: GET 2023-09-29T22:00:35.883 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.883 DEBUG:boto:Data: 2023-09-29T22:00:35.883 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.884 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.884 DEBUG:boto:Params: {} 2023-09-29T22:00:35.884 DEBUG:boto:Token: None 2023-09-29T22:00:35.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE= 2023-09-29T22:00:35.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE=', 'Content-Length': '0'} 2023-09-29T22:00:35.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1ba3f3b8e35735f-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.930 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.931 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.931 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:35.931 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:35.931 DEBUG:boto:Method: GET 2023-09-29T22:00:35.931 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:35.931 DEBUG:boto:Data: 2023-09-29T22:00:35.931 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.932 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.932 DEBUG:boto:Params: {} 2023-09-29T22:00:35.932 DEBUG:boto:Token: None 2023-09-29T22:00:35.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da2904770d279fa9-0065174903-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.938 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.939 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:35.939 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:35.939 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:35.939 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:35.939 DEBUG:boto:Method: GET 2023-09-29T22:00:35.939 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:35.940 DEBUG:boto:Data: 2023-09-29T22:00:35.940 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.940 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.940 DEBUG:boto:Params: {} 2023-09-29T22:00:35.940 DEBUG:boto:Token: None 2023-09-29T22:00:35.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/ 2023-09-29T22:00:35.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk= 2023-09-29T22:00:35.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DDJb6kt3ZalvV1z4CNWfQVHUtgk=', 'Content-Length': '0'} 2023-09-29T22:00:35.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3d4182543ad951c-0065174903-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.945 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:35.946 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.946 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.946 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.946 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T22:00:35.946 DEBUG:boto:Data: 2023-09-29T22:00:35.946 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.946 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.947 DEBUG:boto:Params: {} 2023-09-29T22:00:35.947 DEBUG:boto:Token: None 2023-09-29T22:00:35.947 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2 2023-09-29T22:00:35.947 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UpYi/i1n7bxPsBXXKqSf9WCqLgY= 2023-09-29T22:00:35.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UpYi/i1n7bxPsBXXKqSf9WCqLgY=', 'Content-Length': '0'} 2023-09-29T22:00:35.950 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004679a0151c1453f4-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.950 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.950 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.951 DEBUG:boto:Method: HEAD 2023-09-29T22:00:35.951 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T22:00:35.951 DEBUG:boto:Data: 2023-09-29T22:00:35.951 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.951 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.951 DEBUG:boto:Params: {} 2023-09-29T22:00:35.951 DEBUG:boto:Token: None 2023-09-29T22:00:35.951 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2 2023-09-29T22:00:35.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UpYi/i1n7bxPsBXXKqSf9WCqLgY= 2023-09-29T22:00:35.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UpYi/i1n7bxPsBXXKqSf9WCqLgY=', 'Content-Length': '0'} 2023-09-29T22:00:35.954 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bf6a5258bce22a42-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:35.955 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.955 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.955 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.955 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.955 DEBUG:boto:Method: GET 2023-09-29T22:00:35.955 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.955 DEBUG:boto:Data: 2023-09-29T22:00:35.956 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:35.956 DEBUG:boto:Port: 8001 2023-09-29T22:00:35.956 DEBUG:boto:Params: {} 2023-09-29T22:00:35.956 DEBUG:boto:Token: None 2023-09-29T22:00:35.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE= 2023-09-29T22:00:35.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE=', 'Content-Length': '0'} 2023-09-29T22:00:35.959 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b4f9af4694e084b3-0065174903-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:35.959 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.959 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:35.960 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.960 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.960 DEBUG:boto:Method: GET 2023-09-29T22:00:35.960 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.960 DEBUG:boto:Data: 2023-09-29T22:00:35.960 DEBUG:boto:Headers: {} 2023-09-29T22:00:35.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:35.960 DEBUG:boto:Port: 8000 2023-09-29T22:00:35.960 DEBUG:boto:Params: {} 2023-09-29T22:00:35.960 DEBUG:boto:Token: None 2023-09-29T22:00:35.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:35 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:35.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE= 2023-09-29T22:00:35.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i6yXnnlhice+NVyKDtomr/GsBlE=', 'Content-Length': '0'} 2023-09-29T22:00:35.963 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ae5c15f785e7367-0065174903-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.010 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:36.010 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:36.011 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:36.011 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:36.013 DEBUG:boto:Method: GET 2023-09-29T22:00:36.013 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:36.013 DEBUG:boto:Data: 2023-09-29T22:00:36.013 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.014 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.014 DEBUG:boto:Params: {} 2023-09-29T22:00:36.014 DEBUG:boto:Token: None 2023-09-29T22:00:36.014 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-11/ 2023-09-29T22:00:36.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aPP/XG6FdxqRNhrTUkMTinB8/14= 2023-09-29T22:00:36.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aPP/XG6FdxqRNhrTUkMTinB8/14=', 'Content-Length': '0'} 2023-09-29T22:00:36.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000060a8c3824cd7808-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.021 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.021 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:36.022 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:36.022 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:36.022 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:36.022 DEBUG:boto:Method: GET 2023-09-29T22:00:36.022 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:36.022 DEBUG:boto:Data: 2023-09-29T22:00:36.022 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.023 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.023 DEBUG:boto:Params: {} 2023-09-29T22:00:36.023 DEBUG:boto:Token: None 2023-09-29T22:00:36.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-11/ 2023-09-29T22:00:36.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aPP/XG6FdxqRNhrTUkMTinB8/14= 2023-09-29T22:00:36.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aPP/XG6FdxqRNhrTUkMTinB8/14=', 'Content-Length': '0'} 2023-09-29T22:00:36.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3740bbd01908504-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.027 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.028 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-11 zones={a2, a1} 2023-09-29T22:00:36.028 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-12 2023-09-29T22:00:36.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:36.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:36.028 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:36.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:36.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:36.134+0000 7f9c39d29f40 1 bucket sync caught up with source: 2023-09-29T22:00:36.136 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.6.11, , , 00000000002.3.11, 00000000010.29.6, ] 2023-09-29T22:00:36.136 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.6.11, , , 00000000002.3.11, 00000000010.29.6, ] 2023-09-29T22:00:36.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:36.134+0000 7f9c39d29f40 0 bucket checkpoint complete 2023-09-29T22:00:36.160 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-12 2023-09-29T22:00:36.160 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:36.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:36.160 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:36.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:36.207 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:36.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:36.270+0000 7f16a0775f40 1 bucket sync caught up with source: 2023-09-29T22:00:36.273 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.13.10, , , 00000000010.22.6, 00000000003.24.10, ] 2023-09-29T22:00:36.273 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000010.22.6, 00000000003.24.10, ] 2023-09-29T22:00:36.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:36.270+0000 7f16a0775f40 0 bucket checkpoint complete 2023-09-29T22:00:36.298 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-12 zones={a2, a1} 2023-09-29T22:00:36.299 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.299 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.299 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.299 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T22:00:36.299 DEBUG:boto:Data: 2023-09-29T22:00:36.299 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.300 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.300 DEBUG:boto:Params: {} 2023-09-29T22:00:36.300 DEBUG:boto:Token: None 2023-09-29T22:00:36.300 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VJRH58qSD2BbFvzPbGPgKKiSub8= 2023-09-29T22:00:36.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VJRH58qSD2BbFvzPbGPgKKiSub8=', 'Content-Length': '0'} 2023-09-29T22:00:36.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c9fb9ce624086c27-0065174904-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.304 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.304 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.304 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.304 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T22:00:36.304 DEBUG:boto:Data: 2023-09-29T22:00:36.304 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.305 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.305 DEBUG:boto:Params: {} 2023-09-29T22:00:36.305 DEBUG:boto:Token: None 2023-09-29T22:00:36.305 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.305 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VJRH58qSD2BbFvzPbGPgKKiSub8= 2023-09-29T22:00:36.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VJRH58qSD2BbFvzPbGPgKKiSub8=', 'Content-Length': '0'} 2023-09-29T22:00:36.309 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000be9c3530886243df-0065174904-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.309 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:00:36.309 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.309 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.309 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.309 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.310 DEBUG:boto:Method: GET 2023-09-29T22:00:36.310 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.310 DEBUG:boto:Data: 2023-09-29T22:00:36.310 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.310 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.310 DEBUG:boto:Params: {} 2023-09-29T22:00:36.310 DEBUG:boto:Token: None 2023-09-29T22:00:36.311 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/?versions 2023-09-29T22:00:36.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0= 2023-09-29T22:00:36.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0=', 'Content-Length': '0'} 2023-09-29T22:00:36.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd87dd5ccf906a13-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.315 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.316 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:36.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:36.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:36.316 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:00:36.316 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.316 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.316 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.316 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.317 DEBUG:boto:Method: GET 2023-09-29T22:00:36.317 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.317 DEBUG:boto:Data: 2023-09-29T22:00:36.317 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.317 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.317 DEBUG:boto:Params: {} 2023-09-29T22:00:36.317 DEBUG:boto:Token: None 2023-09-29T22:00:36.317 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/?versions 2023-09-29T22:00:36.318 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0= 2023-09-29T22:00:36.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0=', 'Content-Length': '0'} 2023-09-29T22:00:36.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a59228b4a716d8b2-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.321 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:36.322 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.323 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.323 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.323 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.323 DEBUG:boto:Method: GET 2023-09-29T22:00:36.323 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.323 DEBUG:boto:Data: 2023-09-29T22:00:36.323 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.323 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.324 DEBUG:boto:Params: {} 2023-09-29T22:00:36.324 DEBUG:boto:Token: None 2023-09-29T22:00:36.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/?versions 2023-09-29T22:00:36.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0= 2023-09-29T22:00:36.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0=', 'Content-Length': '0'} 2023-09-29T22:00:36.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eed6771748f33a6a-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.328 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.328 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.328 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.329 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.329 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.329 DEBUG:boto:Method: GET 2023-09-29T22:00:36.329 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:36.329 DEBUG:boto:Data: 2023-09-29T22:00:36.329 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.329 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.329 DEBUG:boto:Params: {} 2023-09-29T22:00:36.329 DEBUG:boto:Token: None 2023-09-29T22:00:36.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/?versions 2023-09-29T22:00:36.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0= 2023-09-29T22:00:36.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TVy/UOPKLNtGCY0nXbWhvf/Pdu0=', 'Content-Length': '0'} 2023-09-29T22:00:36.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001492d84d17224402-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.333 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:36.334 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.334 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.335 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.335 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.335 DEBUG:boto:Method: GET 2023-09-29T22:00:36.335 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.335 DEBUG:boto:Data: 2023-09-29T22:00:36.336 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.336 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.336 DEBUG:boto:Params: {} 2023-09-29T22:00:36.336 DEBUG:boto:Token: None 2023-09-29T22:00:36.337 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI= 2023-09-29T22:00:36.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI=', 'Content-Length': '0'} 2023-09-29T22:00:36.340 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002867591510762bcf-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.340 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.340 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.340 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.341 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.341 DEBUG:boto:Method: GET 2023-09-29T22:00:36.341 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.341 DEBUG:boto:Data: 2023-09-29T22:00:36.341 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.342 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.342 DEBUG:boto:Params: {} 2023-09-29T22:00:36.342 DEBUG:boto:Token: None 2023-09-29T22:00:36.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI= 2023-09-29T22:00:36.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI=', 'Content-Length': '0'} 2023-09-29T22:00:36.345 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000850a3732af35201a-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.346 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.346 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.346 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.346 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.347 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.347 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.347 DEBUG:boto:Data: 2023-09-29T22:00:36.347 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.348 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.348 DEBUG:boto:Params: {} 2023-09-29T22:00:36.348 DEBUG:boto:Token: None 2023-09-29T22:00:36.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kPaOjaVMxkWmntjZdcm1vvqS1QU= 2023-09-29T22:00:36.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kPaOjaVMxkWmntjZdcm1vvqS1QU=', 'Content-Length': '0'} 2023-09-29T22:00:36.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b74fda0fd9ae01d8-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.352 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.352 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.352 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.352 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.352 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.353 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.353 DEBUG:boto:Data: 2023-09-29T22:00:36.353 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.353 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.354 DEBUG:boto:Params: {} 2023-09-29T22:00:36.354 DEBUG:boto:Token: None 2023-09-29T22:00:36.355 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kPaOjaVMxkWmntjZdcm1vvqS1QU= 2023-09-29T22:00:36.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kPaOjaVMxkWmntjZdcm1vvqS1QU=', 'Content-Length': '0'} 2023-09-29T22:00:36.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009224b139054dc6ec-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:36.358 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.358 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.359 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.359 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.359 DEBUG:boto:Method: GET 2023-09-29T22:00:36.359 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.359 DEBUG:boto:Data: 2023-09-29T22:00:36.359 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.359 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.359 DEBUG:boto:Params: {} 2023-09-29T22:00:36.360 DEBUG:boto:Token: None 2023-09-29T22:00:36.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI= 2023-09-29T22:00:36.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI=', 'Content-Length': '0'} 2023-09-29T22:00:36.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007756b5f9de314aeb-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.363 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.363 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.363 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.364 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.364 DEBUG:boto:Method: GET 2023-09-29T22:00:36.364 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.364 DEBUG:boto:Data: 2023-09-29T22:00:36.364 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.364 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.364 DEBUG:boto:Params: {} 2023-09-29T22:00:36.364 DEBUG:boto:Token: None 2023-09-29T22:00:36.365 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI= 2023-09-29T22:00:36.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI=', 'Content-Length': '0'} 2023-09-29T22:00:36.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007c00ae21314f419f-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.368 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.368 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.368 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.368 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.368 DEBUG:boto:Method: GET 2023-09-29T22:00:36.368 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.368 DEBUG:boto:Data: 2023-09-29T22:00:36.369 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.369 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.369 DEBUG:boto:Params: {} 2023-09-29T22:00:36.369 DEBUG:boto:Token: None 2023-09-29T22:00:36.369 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.370 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e23adb838376a76-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.418 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.419 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.419 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.419 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.420 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.420 DEBUG:boto:Method: GET 2023-09-29T22:00:36.420 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.420 DEBUG:boto:Data: 2023-09-29T22:00:36.420 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.421 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.421 DEBUG:boto:Params: {} 2023-09-29T22:00:36.421 DEBUG:boto:Token: None 2023-09-29T22:00:36.421 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.421 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae588992b4007bde-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.425 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.426 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.426 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.426 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.426 DEBUG:boto:Path: /hcnphz-12/dummy 2023-09-29T22:00:36.426 DEBUG:boto:Data: 2023-09-29T22:00:36.426 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.426 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.427 DEBUG:boto:Params: {} 2023-09-29T22:00:36.427 DEBUG:boto:Token: None 2023-09-29T22:00:36.427 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy 2023-09-29T22:00:36.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G5buWzQHuA5U+5bw8f+6yC3T/EE= 2023-09-29T22:00:36.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G5buWzQHuA5U+5bw8f+6yC3T/EE=', 'Content-Length': '0'} 2023-09-29T22:00:36.430 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fdab09295924b8d4-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.431 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.431 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.431 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.431 DEBUG:boto:Path: /hcnphz-12/dummy 2023-09-29T22:00:36.432 DEBUG:boto:Data: 2023-09-29T22:00:36.432 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.432 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.432 DEBUG:boto:Params: {} 2023-09-29T22:00:36.433 DEBUG:boto:Token: None 2023-09-29T22:00:36.433 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy 2023-09-29T22:00:36.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G5buWzQHuA5U+5bw8f+6yC3T/EE= 2023-09-29T22:00:36.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G5buWzQHuA5U+5bw8f+6yC3T/EE=', 'Content-Length': '0'} 2023-09-29T22:00:36.437 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000470307cc13def467-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:36.437 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.437 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.438 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.438 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.438 DEBUG:boto:Method: GET 2023-09-29T22:00:36.438 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.438 DEBUG:boto:Data: 2023-09-29T22:00:36.439 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.439 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.439 DEBUG:boto:Params: {} 2023-09-29T22:00:36.439 DEBUG:boto:Token: None 2023-09-29T22:00:36.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI= 2023-09-29T22:00:36.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI=', 'Content-Length': '0'} 2023-09-29T22:00:36.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a87d5c77da909b0e-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.443 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:36.443 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:36.443 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.444 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.444 DEBUG:boto:Method: GET 2023-09-29T22:00:36.444 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.444 DEBUG:boto:Data: 2023-09-29T22:00:36.444 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.445 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.445 DEBUG:boto:Params: {} 2023-09-29T22:00:36.445 DEBUG:boto:Token: None 2023-09-29T22:00:36.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:36.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI= 2023-09-29T22:00:36.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3/mvqtAwXXxl7HxdhTfACX+6oI=', 'Content-Length': '0'} 2023-09-29T22:00:36.448 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f17703c9335abcea-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.448 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.449 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.449 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.449 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.449 DEBUG:boto:Method: GET 2023-09-29T22:00:36.449 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.450 DEBUG:boto:Data: 2023-09-29T22:00:36.450 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.450 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.450 DEBUG:boto:Params: {} 2023-09-29T22:00:36.451 DEBUG:boto:Token: None 2023-09-29T22:00:36.451 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.451 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008518f3983ddf8600-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.502 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.503 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.503 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.503 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.503 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.503 DEBUG:boto:Method: GET 2023-09-29T22:00:36.504 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:36.504 DEBUG:boto:Data: 2023-09-29T22:00:36.504 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.504 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.504 DEBUG:boto:Params: {} 2023-09-29T22:00:36.504 DEBUG:boto:Token: None 2023-09-29T22:00:36.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a66f8c1cb0f3cda5-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.554 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:36.555 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.555 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.555 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.555 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.556 DEBUG:boto:Method: GET 2023-09-29T22:00:36.556 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.556 DEBUG:boto:Data: 2023-09-29T22:00:36.556 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.556 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.556 DEBUG:boto:Params: {} 2023-09-29T22:00:36.556 DEBUG:boto:Token: None 2023-09-29T22:00:36.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900= 2023-09-29T22:00:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900=', 'Content-Length': '0'} 2023-09-29T22:00:36.560 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a411721b5a718248-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.560 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.561 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.561 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.561 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.561 DEBUG:boto:Method: GET 2023-09-29T22:00:36.561 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.561 DEBUG:boto:Data: 2023-09-29T22:00:36.561 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.561 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.561 DEBUG:boto:Params: {} 2023-09-29T22:00:36.562 DEBUG:boto:Token: None 2023-09-29T22:00:36.562 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900= 2023-09-29T22:00:36.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900=', 'Content-Length': '0'} 2023-09-29T22:00:36.565 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2747e9b527941c0-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.565 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.565 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.565 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.566 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.566 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.566 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.566 DEBUG:boto:Data: 2023-09-29T22:00:36.566 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.566 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.566 DEBUG:boto:Params: {} 2023-09-29T22:00:36.566 DEBUG:boto:Token: None 2023-09-29T22:00:36.567 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xiFpLa6tonSTolkPQ3RUF6+A5lA= 2023-09-29T22:00:36.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xiFpLa6tonSTolkPQ3RUF6+A5lA=', 'Content-Length': '0'} 2023-09-29T22:00:36.570 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cd17896854219491-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.570 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.570 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.570 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.570 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.570 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.571 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.571 DEBUG:boto:Data: 2023-09-29T22:00:36.571 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.571 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.571 DEBUG:boto:Params: {} 2023-09-29T22:00:36.571 DEBUG:boto:Token: None 2023-09-29T22:00:36.571 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xiFpLa6tonSTolkPQ3RUF6+A5lA= 2023-09-29T22:00:36.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xiFpLa6tonSTolkPQ3RUF6+A5lA=', 'Content-Length': '0'} 2023-09-29T22:00:36.574 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000009ff34e6fbd7cbad-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:36.575 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.575 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.575 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.575 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.576 DEBUG:boto:Method: GET 2023-09-29T22:00:36.576 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.576 DEBUG:boto:Data: 2023-09-29T22:00:36.576 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.576 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.576 DEBUG:boto:Params: {} 2023-09-29T22:00:36.576 DEBUG:boto:Token: None 2023-09-29T22:00:36.577 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900= 2023-09-29T22:00:36.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900=', 'Content-Length': '0'} 2023-09-29T22:00:36.580 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000027119b124d71b492-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.622 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.623 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.623 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.623 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.623 DEBUG:boto:Method: GET 2023-09-29T22:00:36.623 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.623 DEBUG:boto:Data: 2023-09-29T22:00:36.623 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.623 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.624 DEBUG:boto:Params: {} 2023-09-29T22:00:36.624 DEBUG:boto:Token: None 2023-09-29T22:00:36.624 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.624 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900= 2023-09-29T22:00:36.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900=', 'Content-Length': '0'} 2023-09-29T22:00:36.627 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002ff18a39f58e573d-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.627 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.627 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.628 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.628 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.628 DEBUG:boto:Method: GET 2023-09-29T22:00:36.628 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.628 DEBUG:boto:Data: 2023-09-29T22:00:36.628 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.628 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.628 DEBUG:boto:Params: {} 2023-09-29T22:00:36.629 DEBUG:boto:Token: None 2023-09-29T22:00:36.629 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5e4ecfcf9756405-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.634 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.634 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.634 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.634 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.634 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.637 DEBUG:boto:Method: GET 2023-09-29T22:00:36.637 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.637 DEBUG:boto:Data: 2023-09-29T22:00:36.638 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.638 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.638 DEBUG:boto:Params: {} 2023-09-29T22:00:36.638 DEBUG:boto:Token: None 2023-09-29T22:00:36.638 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8d3f7d3e51af0a6-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.643 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.643 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.643 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.644 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.644 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T22:00:36.644 DEBUG:boto:Data: 2023-09-29T22:00:36.644 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.644 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.644 DEBUG:boto:Params: {} 2023-09-29T22:00:36.644 DEBUG:boto:Token: None 2023-09-29T22:00:36.645 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1 2023-09-29T22:00:36.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IeVHeYSsmOPsJQT24QXIQgT2JT8= 2023-09-29T22:00:36.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IeVHeYSsmOPsJQT24QXIQgT2JT8=', 'Content-Length': '0'} 2023-09-29T22:00:36.648 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061a609cfb105b0e2-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.648 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.648 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.648 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.648 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T22:00:36.649 DEBUG:boto:Data: 2023-09-29T22:00:36.649 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.649 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.649 DEBUG:boto:Params: {} 2023-09-29T22:00:36.649 DEBUG:boto:Token: None 2023-09-29T22:00:36.649 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1 2023-09-29T22:00:36.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IeVHeYSsmOPsJQT24QXIQgT2JT8= 2023-09-29T22:00:36.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IeVHeYSsmOPsJQT24QXIQgT2JT8=', 'Content-Length': '0'} 2023-09-29T22:00:36.653 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d6d6d2b787f03c66-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:36.654 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.654 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.654 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.655 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.655 DEBUG:boto:Method: GET 2023-09-29T22:00:36.655 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.655 DEBUG:boto:Data: 2023-09-29T22:00:36.655 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.656 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.656 DEBUG:boto:Params: {} 2023-09-29T22:00:36.656 DEBUG:boto:Token: None 2023-09-29T22:00:36.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900= 2023-09-29T22:00:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900=', 'Content-Length': '0'} 2023-09-29T22:00:36.660 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f5fc6d9f473679f-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.702 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.702 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:36.703 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.703 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.703 DEBUG:boto:Method: GET 2023-09-29T22:00:36.703 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.703 DEBUG:boto:Data: 2023-09-29T22:00:36.703 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.703 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.703 DEBUG:boto:Params: {} 2023-09-29T22:00:36.704 DEBUG:boto:Token: None 2023-09-29T22:00:36.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:36.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900= 2023-09-29T22:00:36.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sv1LEdReLIWU8n4SEogm0yEW900=', 'Content-Length': '0'} 2023-09-29T22:00:36.707 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003b8b16ff62fa24ba-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.707 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.707 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.707 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.707 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.707 DEBUG:boto:Method: GET 2023-09-29T22:00:36.708 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.708 DEBUG:boto:Data: 2023-09-29T22:00:36.708 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.708 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.708 DEBUG:boto:Params: {} 2023-09-29T22:00:36.708 DEBUG:boto:Token: None 2023-09-29T22:00:36.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dde3bab62c62d058-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.713 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.713 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.713 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.714 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.714 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.714 DEBUG:boto:Method: GET 2023-09-29T22:00:36.714 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:36.714 DEBUG:boto:Data: 2023-09-29T22:00:36.714 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.714 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.714 DEBUG:boto:Params: {} 2023-09-29T22:00:36.715 DEBUG:boto:Token: None 2023-09-29T22:00:36.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000533dfd3d48830842-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.720 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:36.720 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.720 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.720 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.720 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.721 DEBUG:boto:Method: GET 2023-09-29T22:00:36.721 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.721 DEBUG:boto:Data: 2023-09-29T22:00:36.721 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.721 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.721 DEBUG:boto:Params: {} 2023-09-29T22:00:36.721 DEBUG:boto:Token: None 2023-09-29T22:00:36.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w= 2023-09-29T22:00:36.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w=', 'Content-Length': '0'} 2023-09-29T22:00:36.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000027bfbe429bcc01a-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.724 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.725 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.725 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.725 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.725 DEBUG:boto:Method: GET 2023-09-29T22:00:36.725 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.725 DEBUG:boto:Data: 2023-09-29T22:00:36.725 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.725 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.726 DEBUG:boto:Params: {} 2023-09-29T22:00:36.726 DEBUG:boto:Token: None 2023-09-29T22:00:36.726 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w= 2023-09-29T22:00:36.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w=', 'Content-Length': '0'} 2023-09-29T22:00:36.728 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000782d6581f8146ca-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.729 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.729 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.729 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.729 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.729 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.729 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.729 DEBUG:boto:Data: 2023-09-29T22:00:36.729 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.730 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.730 DEBUG:boto:Params: {} 2023-09-29T22:00:36.730 DEBUG:boto:Token: None 2023-09-29T22:00:36.730 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zGB4VYIrG1Slqf0M2qqkn7mOxR4= 2023-09-29T22:00:36.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zGB4VYIrG1Slqf0M2qqkn7mOxR4=', 'Content-Length': '0'} 2023-09-29T22:00:36.732 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a90df6c303732e4-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.733 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.733 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.733 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.733 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.733 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.733 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.733 DEBUG:boto:Data: 2023-09-29T22:00:36.733 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.734 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.734 DEBUG:boto:Params: {} 2023-09-29T22:00:36.734 DEBUG:boto:Token: None 2023-09-29T22:00:36.734 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.734 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zGB4VYIrG1Slqf0M2qqkn7mOxR4= 2023-09-29T22:00:36.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zGB4VYIrG1Slqf0M2qqkn7mOxR4=', 'Content-Length': '0'} 2023-09-29T22:00:36.736 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000032347d33df3e83d3-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:36.737 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.737 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.737 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.737 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.737 DEBUG:boto:Method: GET 2023-09-29T22:00:36.737 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.737 DEBUG:boto:Data: 2023-09-29T22:00:36.738 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.738 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.738 DEBUG:boto:Params: {} 2023-09-29T22:00:36.738 DEBUG:boto:Token: None 2023-09-29T22:00:36.738 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w= 2023-09-29T22:00:36.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w=', 'Content-Length': '0'} 2023-09-29T22:00:36.741 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b63e063deb22500b-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.741 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.741 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.741 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.742 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.742 DEBUG:boto:Method: GET 2023-09-29T22:00:36.742 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.742 DEBUG:boto:Data: 2023-09-29T22:00:36.742 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.742 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.743 DEBUG:boto:Params: {} 2023-09-29T22:00:36.743 DEBUG:boto:Token: None 2023-09-29T22:00:36.743 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.743 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w= 2023-09-29T22:00:36.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w=', 'Content-Length': '0'} 2023-09-29T22:00:36.745 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b7198a8eff4e93d-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.790 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.791 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.791 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.791 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.791 DEBUG:boto:Method: GET 2023-09-29T22:00:36.791 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.791 DEBUG:boto:Data: 2023-09-29T22:00:36.791 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.791 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.792 DEBUG:boto:Params: {} 2023-09-29T22:00:36.792 DEBUG:boto:Token: None 2023-09-29T22:00:36.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000546c7814e8ced243-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.797 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.797 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.797 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.798 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.798 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.798 DEBUG:boto:Method: GET 2023-09-29T22:00:36.798 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.798 DEBUG:boto:Data: 2023-09-29T22:00:36.798 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.798 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.799 DEBUG:boto:Params: {} 2023-09-29T22:00:36.799 DEBUG:boto:Token: None 2023-09-29T22:00:36.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.799 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.803 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e97c7394ad22280e-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.850 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.851 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.851 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.851 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.851 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T22:00:36.851 DEBUG:boto:Data: 2023-09-29T22:00:36.852 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.852 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.852 DEBUG:boto:Params: {} 2023-09-29T22:00:36.852 DEBUG:boto:Token: None 2023-09-29T22:00:36.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2 2023-09-29T22:00:36.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t1pjnYtYEMkW+Qx9FwVuKvcwSCs= 2023-09-29T22:00:36.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t1pjnYtYEMkW+Qx9FwVuKvcwSCs=', 'Content-Length': '0'} 2023-09-29T22:00:36.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000099e3ad9c7da9ae21-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.859 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.859 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.859 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.859 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T22:00:36.859 DEBUG:boto:Data: 2023-09-29T22:00:36.859 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.860 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.860 DEBUG:boto:Params: {} 2023-09-29T22:00:36.860 DEBUG:boto:Token: None 2023-09-29T22:00:36.860 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2 2023-09-29T22:00:36.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t1pjnYtYEMkW+Qx9FwVuKvcwSCs= 2023-09-29T22:00:36.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t1pjnYtYEMkW+Qx9FwVuKvcwSCs=', 'Content-Length': '0'} 2023-09-29T22:00:36.864 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003ec958c62ea20094-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:36.864 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.864 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.864 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.865 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.865 DEBUG:boto:Method: GET 2023-09-29T22:00:36.865 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.865 DEBUG:boto:Data: 2023-09-29T22:00:36.865 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.865 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.865 DEBUG:boto:Params: {} 2023-09-29T22:00:36.865 DEBUG:boto:Token: None 2023-09-29T22:00:36.866 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.866 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w= 2023-09-29T22:00:36.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w=', 'Content-Length': '0'} 2023-09-29T22:00:36.869 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b261c18f96599c8-0065174904-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.914 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.915 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:36.915 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.915 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.915 DEBUG:boto:Method: GET 2023-09-29T22:00:36.915 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.915 DEBUG:boto:Data: 2023-09-29T22:00:36.915 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.916 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.916 DEBUG:boto:Params: {} 2023-09-29T22:00:36.916 DEBUG:boto:Token: None 2023-09-29T22:00:36.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:36.916 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w= 2023-09-29T22:00:36.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aNGOQ7gtMm52LEYtYhLLfL+z/7w=', 'Content-Length': '0'} 2023-09-29T22:00:36.919 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002c3ecaa76bdafb99-0065174904-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.962 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.963 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.963 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.964 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.964 DEBUG:boto:Method: GET 2023-09-29T22:00:36.964 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.964 DEBUG:boto:Data: 2023-09-29T22:00:36.964 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:36.964 DEBUG:boto:Port: 8001 2023-09-29T22:00:36.965 DEBUG:boto:Params: {} 2023-09-29T22:00:36.965 DEBUG:boto:Token: None 2023-09-29T22:00:36.965 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.965 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.969 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bdd0936e99d9631-0065174904-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.970 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.970 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:36.970 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:36.970 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.971 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.971 DEBUG:boto:Method: GET 2023-09-29T22:00:36.971 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:36.971 DEBUG:boto:Data: 2023-09-29T22:00:36.971 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.972 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.972 DEBUG:boto:Params: {} 2023-09-29T22:00:36.972 DEBUG:boto:Token: None 2023-09-29T22:00:36.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-12/ 2023-09-29T22:00:36.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q= 2023-09-29T22:00:36.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZpZzmVtevi5AW9uGFCucoc90B4Q=', 'Content-Length': '0'} 2023-09-29T22:00:36.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a7b97a82656d616-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.976 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:36.977 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-12 zones={a2, a1} 2023-09-29T22:00:36.977 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:36.977 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:36.977 DEBUG:boto:Method: HEAD 2023-09-29T22:00:36.977 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T22:00:36.977 DEBUG:boto:Data: 2023-09-29T22:00:36.977 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.978 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.978 DEBUG:boto:Params: {} 2023-09-29T22:00:36.978 DEBUG:boto:Token: None 2023-09-29T22:00:36.978 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-11/ 2023-09-29T22:00:36.978 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gHdx7/FzAqu6xgChJG8Am8k+dFM= 2023-09-29T22:00:36.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gHdx7/FzAqu6xgChJG8Am8k+dFM=', 'Content-Length': '0'} 2023-09-29T22:00:36.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001a0517e3a0b2f03f-0065174904-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:36.982 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:36.982 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:36.982 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999&prefix=obj-a1 2023-09-29T22:00:36.982 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999&prefix=obj-a1 2023-09-29T22:00:36.982 DEBUG:boto:Method: GET 2023-09-29T22:00:36.982 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999&prefix=obj-a1 2023-09-29T22:00:36.983 DEBUG:boto:Data: 2023-09-29T22:00:36.983 DEBUG:boto:Headers: {} 2023-09-29T22:00:36.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:36.983 DEBUG:boto:Port: 8000 2023-09-29T22:00:36.983 DEBUG:boto:Params: {} 2023-09-29T22:00:36.983 DEBUG:boto:Token: None 2023-09-29T22:00:36.983 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:36 GMT /hcnphz-11/?versions 2023-09-29T22:00:36.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dsqWq34UxdGcdRgAIR1hZ14ekRk= 2023-09-29T22:00:36.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dsqWq34UxdGcdRgAIR1hZ14ekRk=', 'Content-Length': '0'} 2023-09-29T22:00:36.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfb84b40f9cb79c1-0065174904-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.034 DEBUG:boto:b'hcnphz-11obj-a1999falseobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:37.035 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:37.035 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:37.035 DEBUG:boto:path=/hcnphz-11/obj-a1?acl 2023-09-29T22:00:37.035 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?acl 2023-09-29T22:00:37.036 DEBUG:boto:Method: PUT 2023-09-29T22:00:37.036 DEBUG:boto:Path: /hcnphz-11/obj-a1?acl 2023-09-29T22:00:37.036 DEBUG:boto:Data: 2023-09-29T22:00:37.036 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:00:37.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:37.036 DEBUG:boto:Port: 8000 2023-09-29T22:00:37.036 DEBUG:boto:Params: {} 2023-09-29T22:00:37.036 DEBUG:boto:Token: None 2023-09-29T22:00:37.037 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:00:37 GMT x-amz-acl:public-read /hcnphz-11/obj-a1?acl 2023-09-29T22:00:37.037 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pm0Orj0CNh2m0KpSsoSdROBI05U= 2023-09-29T22:00:37.037 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pm0Orj0CNh2m0KpSsoSdROBI05U=', 'Content-Length': '0'} 2023-09-29T22:00:37.050 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000018abd7cf111ba9ca-0065174905-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.051 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:37.051 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:37.051 DEBUG:boto:Method: HEAD 2023-09-29T22:00:37.051 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T22:00:37.051 DEBUG:boto:Data: 2023-09-29T22:00:37.051 DEBUG:boto:Headers: {} 2023-09-29T22:00:37.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:37.051 DEBUG:boto:Port: 8001 2023-09-29T22:00:37.052 DEBUG:boto:Params: {} 2023-09-29T22:00:37.052 DEBUG:boto:Token: None 2023-09-29T22:00:37.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:37 GMT /hcnphz-11/ 2023-09-29T22:00:37.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dHIhQNKsOD2mYLD3IyMyF33FlUc= 2023-09-29T22:00:37.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dHIhQNKsOD2mYLD3IyMyF33FlUc=', 'Content-Length': '0'} 2023-09-29T22:00:37.056 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a62e6f014c3c50ef-0065174905-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.056 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:37.056 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:37.056 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999&prefix=obj-a2 2023-09-29T22:00:37.057 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999&prefix=obj-a2 2023-09-29T22:00:37.057 DEBUG:boto:Method: GET 2023-09-29T22:00:37.057 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999&prefix=obj-a2 2023-09-29T22:00:37.057 DEBUG:boto:Data: 2023-09-29T22:00:37.057 DEBUG:boto:Headers: {} 2023-09-29T22:00:37.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:37.058 DEBUG:boto:Port: 8001 2023-09-29T22:00:37.058 DEBUG:boto:Params: {} 2023-09-29T22:00:37.058 DEBUG:boto:Token: None 2023-09-29T22:00:37.058 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:37 GMT /hcnphz-11/?versions 2023-09-29T22:00:37.058 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mLnMMULx7BOj5wAGBC/J1Tw+egQ= 2023-09-29T22:00:37.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mLnMMULx7BOj5wAGBC/J1Tw+egQ=', 'Content-Length': '0'} 2023-09-29T22:00:37.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c927773a576e8395-0065174905-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.110 DEBUG:boto:b'hcnphz-11obj-a2999falseobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:37.111 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:37.111 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:37.111 DEBUG:boto:path=/hcnphz-11/obj-a2?acl 2023-09-29T22:00:37.111 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?acl 2023-09-29T22:00:37.111 DEBUG:boto:Method: PUT 2023-09-29T22:00:37.111 DEBUG:boto:Path: /hcnphz-11/obj-a2?acl 2023-09-29T22:00:37.111 DEBUG:boto:Data: 2023-09-29T22:00:37.112 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:00:37.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:37.112 DEBUG:boto:Port: 8001 2023-09-29T22:00:37.112 DEBUG:boto:Params: {} 2023-09-29T22:00:37.112 DEBUG:boto:Token: None 2023-09-29T22:00:37.112 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:00:37 GMT x-amz-acl:public-read /hcnphz-11/obj-a2?acl 2023-09-29T22:00:37.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eqiij/Yw/Xi3GEql87WQWnzhVE4= 2023-09-29T22:00:37.113 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eqiij/Yw/Xi3GEql87WQWnzhVE4=', 'Content-Length': '0'} 2023-09-29T22:00:37.126 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d46fc49399e9b787-0065174905-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.126 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:37.126 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:37.126 DEBUG:boto:Method: HEAD 2023-09-29T22:00:37.126 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T22:00:37.126 DEBUG:boto:Data: 2023-09-29T22:00:37.127 DEBUG:boto:Headers: {} 2023-09-29T22:00:37.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:37.127 DEBUG:boto:Port: 8000 2023-09-29T22:00:37.127 DEBUG:boto:Params: {} 2023-09-29T22:00:37.127 DEBUG:boto:Token: None 2023-09-29T22:00:37.127 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:37 GMT /hcnphz-12/ 2023-09-29T22:00:37.127 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QIWWkMHxTqod+B0rkDRa/4Ez8sU= 2023-09-29T22:00:37.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QIWWkMHxTqod+B0rkDRa/4Ez8sU=', 'Content-Length': '0'} 2023-09-29T22:00:37.130 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000db50ac102d619fb8-0065174905-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.131 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:37.131 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:37.131 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999&prefix=obj-a1 2023-09-29T22:00:37.131 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999&prefix=obj-a1 2023-09-29T22:00:37.131 DEBUG:boto:Method: GET 2023-09-29T22:00:37.131 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999&prefix=obj-a1 2023-09-29T22:00:37.131 DEBUG:boto:Data: 2023-09-29T22:00:37.131 DEBUG:boto:Headers: {} 2023-09-29T22:00:37.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:37.132 DEBUG:boto:Port: 8000 2023-09-29T22:00:37.132 DEBUG:boto:Params: {} 2023-09-29T22:00:37.132 DEBUG:boto:Token: None 2023-09-29T22:00:37.132 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:37 GMT /hcnphz-12/?versions 2023-09-29T22:00:37.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UnKM7AEDoXOmxMdsFZb9KPpvaZ8= 2023-09-29T22:00:37.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UnKM7AEDoXOmxMdsFZb9KPpvaZ8=', 'Content-Length': '0'} 2023-09-29T22:00:37.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f9dd9f44a2b7d08-0065174905-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.136 DEBUG:boto:b'hcnphz-12obj-a1999falseobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:37.137 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:37.137 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:37.137 DEBUG:boto:path=/hcnphz-12/obj-a1?acl 2023-09-29T22:00:37.137 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?acl 2023-09-29T22:00:37.137 DEBUG:boto:Method: PUT 2023-09-29T22:00:37.137 DEBUG:boto:Path: /hcnphz-12/obj-a1?acl 2023-09-29T22:00:37.137 DEBUG:boto:Data: 2023-09-29T22:00:37.138 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:00:37.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:37.138 DEBUG:boto:Port: 8000 2023-09-29T22:00:37.138 DEBUG:boto:Params: {} 2023-09-29T22:00:37.138 DEBUG:boto:Token: None 2023-09-29T22:00:37.138 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:00:37 GMT x-amz-acl:public-read /hcnphz-12/obj-a1?acl 2023-09-29T22:00:37.139 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i1bTLrVj5wav8yht5zwtSzzsH1E= 2023-09-29T22:00:37.139 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i1bTLrVj5wav8yht5zwtSzzsH1E=', 'Content-Length': '0'} 2023-09-29T22:00:37.150 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000802b81a9cfef8681-0065174905-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.150 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:37.150 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:37.151 DEBUG:boto:Method: HEAD 2023-09-29T22:00:37.151 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T22:00:37.151 DEBUG:boto:Data: 2023-09-29T22:00:37.151 DEBUG:boto:Headers: {} 2023-09-29T22:00:37.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:37.151 DEBUG:boto:Port: 8001 2023-09-29T22:00:37.151 DEBUG:boto:Params: {} 2023-09-29T22:00:37.151 DEBUG:boto:Token: None 2023-09-29T22:00:37.152 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:37 GMT /hcnphz-12/ 2023-09-29T22:00:37.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QIWWkMHxTqod+B0rkDRa/4Ez8sU= 2023-09-29T22:00:37.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QIWWkMHxTqod+B0rkDRa/4Ez8sU=', 'Content-Length': '0'} 2023-09-29T22:00:37.155 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d3898566851b2b8-0065174905-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.155 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:37.156 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:37.156 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999&prefix=obj-a2 2023-09-29T22:00:37.156 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999&prefix=obj-a2 2023-09-29T22:00:37.156 DEBUG:boto:Method: GET 2023-09-29T22:00:37.156 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999&prefix=obj-a2 2023-09-29T22:00:37.156 DEBUG:boto:Data: 2023-09-29T22:00:37.156 DEBUG:boto:Headers: {} 2023-09-29T22:00:37.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:37.157 DEBUG:boto:Port: 8001 2023-09-29T22:00:37.157 DEBUG:boto:Params: {} 2023-09-29T22:00:37.157 DEBUG:boto:Token: None 2023-09-29T22:00:37.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:37 GMT /hcnphz-12/?versions 2023-09-29T22:00:37.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UnKM7AEDoXOmxMdsFZb9KPpvaZ8= 2023-09-29T22:00:37.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UnKM7AEDoXOmxMdsFZb9KPpvaZ8=', 'Content-Length': '0'} 2023-09-29T22:00:37.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc573dbc95d25e03-0065174905-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.202 DEBUG:boto:b'hcnphz-12obj-a2999falseobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:37.203 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:37.203 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:37.203 DEBUG:boto:path=/hcnphz-12/obj-a2?acl 2023-09-29T22:00:37.203 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?acl 2023-09-29T22:00:37.203 DEBUG:boto:Method: PUT 2023-09-29T22:00:37.204 DEBUG:boto:Path: /hcnphz-12/obj-a2?acl 2023-09-29T22:00:37.204 DEBUG:boto:Data: 2023-09-29T22:00:37.204 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:00:37.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:37.204 DEBUG:boto:Port: 8001 2023-09-29T22:00:37.204 DEBUG:boto:Params: {} 2023-09-29T22:00:37.204 DEBUG:boto:Token: None 2023-09-29T22:00:37.205 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:00:37 GMT x-amz-acl:public-read /hcnphz-12/obj-a2?acl 2023-09-29T22:00:37.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r6moXyiZ3QATdXRr5umzNTIUSS4= 2023-09-29T22:00:37.205 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r6moXyiZ3QATdXRr5umzNTIUSS4=', 'Content-Length': '0'} 2023-09-29T22:00:37.222 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1e896abf2585565-0065174905-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:37.222 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-11 2023-09-29T22:00:37.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:37.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:37.223 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-11 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:37.273 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:37.273 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:37.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:37.334+0000 7fbf467b5f40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:37.336 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000010.30.6, ] 2023-09-29T22:00:37.336 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000011.33.6, ] 2023-09-29T22:00:42.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:42.334+0000 7fbf467b5f40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:42.339 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000010.30.6, ] 2023-09-29T22:00:42.339 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000011.33.6, ] 2023-09-29T22:00:47.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:47.338+0000 7fbf467b5f40 1 waiting for incremental sync to catch up: 2023-09-29T22:00:47.341 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000010.30.6, ] 2023-09-29T22:00:47.341 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000011.33.6, ] 2023-09-29T22:00:52.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:52.342+0000 7fbf467b5f40 1 bucket sync caught up with source: 2023-09-29T22:00:52.344 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000011.33.6, ] 2023-09-29T22:00:52.344 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.10.10, , , 00000000002.10.11, 00000000011.33.6, ] 2023-09-29T22:00:52.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:52.342+0000 7fbf467b5f40 0 bucket checkpoint complete 2023-09-29T22:00:52.372 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-11 2023-09-29T22:00:52.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:52.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-11', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:52.373 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-11 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:52.423 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:52.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:52.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:52.486+0000 7fdc1cc03f40 1 bucket sync caught up with source: 2023-09-29T22:00:52.488 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.14.11, , , 00000000011.40.6, 00000000002.18.11, ] 2023-09-29T22:00:52.488 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.14.11, , , 00000000011.40.6, 00000000002.18.11, ] 2023-09-29T22:00:52.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:52.486+0000 7fdc1cc03f40 0 bucket checkpoint complete 2023-09-29T22:00:52.513 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-11 zones={a2, a1} 2023-09-29T22:00:52.513 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.513 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.513 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.513 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T22:00:52.514 DEBUG:boto:Data: 2023-09-29T22:00:52.514 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.514 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.514 DEBUG:boto:Params: {} 2023-09-29T22:00:52.514 DEBUG:boto:Token: None 2023-09-29T22:00:52.514 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.515 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pkzCXAyRMOJqK/zNUAcKZfNQDVg= 2023-09-29T22:00:52.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pkzCXAyRMOJqK/zNUAcKZfNQDVg=', 'Content-Length': '0'} 2023-09-29T22:00:52.518 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d01ba654ce1ef986-0065174914-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.519 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.519 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.519 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.519 DEBUG:boto:Path: /hcnphz-11/ 2023-09-29T22:00:52.519 DEBUG:boto:Data: 2023-09-29T22:00:52.519 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.519 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.519 DEBUG:boto:Params: {} 2023-09-29T22:00:52.520 DEBUG:boto:Token: None 2023-09-29T22:00:52.520 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pkzCXAyRMOJqK/zNUAcKZfNQDVg= 2023-09-29T22:00:52.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pkzCXAyRMOJqK/zNUAcKZfNQDVg=', 'Content-Length': '0'} 2023-09-29T22:00:52.523 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8db9cfe90d50bfe-0065174914-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.523 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:00:52.523 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.523 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.524 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.524 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.524 DEBUG:boto:Method: GET 2023-09-29T22:00:52.524 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.524 DEBUG:boto:Data: 2023-09-29T22:00:52.524 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.524 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.524 DEBUG:boto:Params: {} 2023-09-29T22:00:52.525 DEBUG:boto:Token: None 2023-09-29T22:00:52.525 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/?versions 2023-09-29T22:00:52.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs= 2023-09-29T22:00:52.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs=', 'Content-Length': '0'} 2023-09-29T22:00:52.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de45521ab9110149-0065174914-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.530 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.530 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:52.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:52.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:52.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:00:52.531 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.531 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.531 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.531 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.531 DEBUG:boto:Method: GET 2023-09-29T22:00:52.531 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.532 DEBUG:boto:Data: 2023-09-29T22:00:52.532 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.532 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.532 DEBUG:boto:Params: {} 2023-09-29T22:00:52.532 DEBUG:boto:Token: None 2023-09-29T22:00:52.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/?versions 2023-09-29T22:00:52.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs= 2023-09-29T22:00:52.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs=', 'Content-Length': '0'} 2023-09-29T22:00:52.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5dd1fb0b92583e6-0065174914-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.537 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.538 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:52.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:52.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:52.538 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.538 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.538 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.539 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.539 DEBUG:boto:Method: GET 2023-09-29T22:00:52.539 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.539 DEBUG:boto:Data: 2023-09-29T22:00:52.539 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.539 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.539 DEBUG:boto:Params: {} 2023-09-29T22:00:52.539 DEBUG:boto:Token: None 2023-09-29T22:00:52.539 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/?versions 2023-09-29T22:00:52.540 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs= 2023-09-29T22:00:52.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs=', 'Content-Length': '0'} 2023-09-29T22:00:52.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096714500f30fdb25-0065174914-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.543 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.544 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.544 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.544 DEBUG:boto:path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.544 DEBUG:boto:auth_path=/hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.544 DEBUG:boto:Method: GET 2023-09-29T22:00:52.544 DEBUG:boto:Path: /hcnphz-11/?versions&max-keys=999 2023-09-29T22:00:52.545 DEBUG:boto:Data: 2023-09-29T22:00:52.545 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.545 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.545 DEBUG:boto:Params: {} 2023-09-29T22:00:52.545 DEBUG:boto:Token: None 2023-09-29T22:00:52.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/?versions 2023-09-29T22:00:52.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs= 2023-09-29T22:00:52.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eQSLW4ZAzMAsQBEeUe7dNaqxzs=', 'Content-Length': '0'} 2023-09-29T22:00:52.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a084ab16c9e8a9d7-0065174914-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.550 DEBUG:boto:b'hcnphz-11999falsedummyC3zZXR31QQjCAHWK1lX6.eqnadzf-Xhtrue2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1FaeW-gBhkz9YM-wKPOiwbdq3dWizjTltrue2023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-true2023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:52.551 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.551 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.551 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.551 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.551 DEBUG:boto:Method: GET 2023-09-29T22:00:52.552 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.552 DEBUG:boto:Data: 2023-09-29T22:00:52.552 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.552 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.552 DEBUG:boto:Params: {} 2023-09-29T22:00:52.552 DEBUG:boto:Token: None 2023-09-29T22:00:52.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4= 2023-09-29T22:00:52.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4=', 'Content-Length': '0'} 2023-09-29T22:00:52.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004975502243ebe1c5-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.555 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.556 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.556 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.556 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.556 DEBUG:boto:Method: GET 2023-09-29T22:00:52.556 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.556 DEBUG:boto:Data: 2023-09-29T22:00:52.556 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.557 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.557 DEBUG:boto:Params: {} 2023-09-29T22:00:52.557 DEBUG:boto:Token: None 2023-09-29T22:00:52.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4= 2023-09-29T22:00:52.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4=', 'Content-Length': '0'} 2023-09-29T22:00:52.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000502fdc2cab81c335-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.560 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.560 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.560 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.560 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.560 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.561 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.561 DEBUG:boto:Data: 2023-09-29T22:00:52.561 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.561 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.561 DEBUG:boto:Params: {} 2023-09-29T22:00:52.561 DEBUG:boto:Token: None 2023-09-29T22:00:52.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8020f8GtkyXkOnSHm8FktFI1PDw= 2023-09-29T22:00:52.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8020f8GtkyXkOnSHm8FktFI1PDw=', 'Content-Length': '0'} 2023-09-29T22:00:52.564 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bd5660c3ee7f140e-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.564 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.564 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.565 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.565 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.565 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.565 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.565 DEBUG:boto:Data: 2023-09-29T22:00:52.565 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.565 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.565 DEBUG:boto:Params: {} 2023-09-29T22:00:52.565 DEBUG:boto:Token: None 2023-09-29T22:00:52.566 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.566 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8020f8GtkyXkOnSHm8FktFI1PDw= 2023-09-29T22:00:52.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8020f8GtkyXkOnSHm8FktFI1PDw=', 'Content-Length': '0'} 2023-09-29T22:00:52.568 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bef589c2d87b6c6e-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:52.569 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.569 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.569 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.569 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.569 DEBUG:boto:Method: GET 2023-09-29T22:00:52.569 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.569 DEBUG:boto:Data: 2023-09-29T22:00:52.569 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.570 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.570 DEBUG:boto:Params: {} 2023-09-29T22:00:52.570 DEBUG:boto:Token: None 2023-09-29T22:00:52.570 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.570 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4= 2023-09-29T22:00:52.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4=', 'Content-Length': '0'} 2023-09-29T22:00:52.573 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000004f02447bffb4977b-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.573 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.573 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.573 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.573 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.573 DEBUG:boto:Method: GET 2023-09-29T22:00:52.573 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.574 DEBUG:boto:Data: 2023-09-29T22:00:52.574 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.574 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.574 DEBUG:boto:Params: {} 2023-09-29T22:00:52.574 DEBUG:boto:Token: None 2023-09-29T22:00:52.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4= 2023-09-29T22:00:52.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4=', 'Content-Length': '0'} 2023-09-29T22:00:52.577 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000eb70e3815310cb92-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.577 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.577 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.577 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.578 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.578 DEBUG:boto:Method: GET 2023-09-29T22:00:52.578 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.578 DEBUG:boto:Data: 2023-09-29T22:00:52.578 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.578 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.578 DEBUG:boto:Params: {} 2023-09-29T22:00:52.578 DEBUG:boto:Token: None 2023-09-29T22:00:52.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc795dbcb60ff036-0065174914-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.630 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.631 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.631 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.631 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.631 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.631 DEBUG:boto:Method: GET 2023-09-29T22:00:52.632 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.632 DEBUG:boto:Data: 2023-09-29T22:00:52.632 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.632 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.632 DEBUG:boto:Params: {} 2023-09-29T22:00:52.632 DEBUG:boto:Token: None 2023-09-29T22:00:52.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad24fafa79217ab3-0065174914-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.638 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.638 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.638 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.638 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.638 DEBUG:boto:Path: /hcnphz-11/dummy 2023-09-29T22:00:52.638 DEBUG:boto:Data: 2023-09-29T22:00:52.639 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.639 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.639 DEBUG:boto:Params: {} 2023-09-29T22:00:52.639 DEBUG:boto:Token: None 2023-09-29T22:00:52.639 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy 2023-09-29T22:00:52.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dMMOyasaDNk8LE2CmoGroLKFT7Q= 2023-09-29T22:00:52.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dMMOyasaDNk8LE2CmoGroLKFT7Q=', 'Content-Length': '0'} 2023-09-29T22:00:52.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bcc3d0da447cfd79-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.644 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.644 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.644 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.644 DEBUG:boto:Path: /hcnphz-11/dummy 2023-09-29T22:00:52.644 DEBUG:boto:Data: 2023-09-29T22:00:52.644 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.644 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.644 DEBUG:boto:Params: {} 2023-09-29T22:00:52.645 DEBUG:boto:Token: None 2023-09-29T22:00:52.645 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy 2023-09-29T22:00:52.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dMMOyasaDNk8LE2CmoGroLKFT7Q= 2023-09-29T22:00:52.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dMMOyasaDNk8LE2CmoGroLKFT7Q=', 'Content-Length': '0'} 2023-09-29T22:00:52.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007fc21672bf992188-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:52.649 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.649 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.649 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.649 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.649 DEBUG:boto:Method: GET 2023-09-29T22:00:52.650 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.650 DEBUG:boto:Data: 2023-09-29T22:00:52.650 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.650 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.650 DEBUG:boto:Params: {} 2023-09-29T22:00:52.650 DEBUG:boto:Token: None 2023-09-29T22:00:52.650 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4= 2023-09-29T22:00:52.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4=', 'Content-Length': '0'} 2023-09-29T22:00:52.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000473921c152cad470-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.654 DEBUG:boto:path=/hcnphz-11/dummy 2023-09-29T22:00:52.654 DEBUG:boto:auth_path=/hcnphz-11/dummy 2023-09-29T22:00:52.654 DEBUG:boto:path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.654 DEBUG:boto:auth_path=/hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.654 DEBUG:boto:Method: GET 2023-09-29T22:00:52.654 DEBUG:boto:Path: /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.654 DEBUG:boto:Data: 2023-09-29T22:00:52.654 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.655 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.655 DEBUG:boto:Params: {} 2023-09-29T22:00:52.655 DEBUG:boto:Token: None 2023-09-29T22:00:52.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/dummy?versionId=C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh 2023-09-29T22:00:52.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4= 2023-09-29T22:00:52.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAuQrA1ZAtzq7fp/DJh/o1EQgb4=', 'Content-Length': '0'} 2023-09-29T22:00:52.658 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:19 GMT'), ('x-amz-version-id', 'C3zZXR31QQjCAHWK1lX6.eqnadzf-Xh'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f3d695d34ac11b1-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.658 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.658 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.658 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.658 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.658 DEBUG:boto:Method: GET 2023-09-29T22:00:52.659 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.659 DEBUG:boto:Data: 2023-09-29T22:00:52.659 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.659 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.659 DEBUG:boto:Params: {} 2023-09-29T22:00:52.659 DEBUG:boto:Token: None 2023-09-29T22:00:52.659 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000990c6ad918465641-0065174914-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.706 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.707 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.707 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.707 DEBUG:boto:path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.707 DEBUG:boto:auth_path=/hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.707 DEBUG:boto:Method: GET 2023-09-29T22:00:52.708 DEBUG:boto:Path: /hcnphz-11/?prefix=dummy 2023-09-29T22:00:52.708 DEBUG:boto:Data: 2023-09-29T22:00:52.708 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.708 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.708 DEBUG:boto:Params: {} 2023-09-29T22:00:52.709 DEBUG:boto:Token: None 2023-09-29T22:00:52.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ceb1c24ab5d1ad89-0065174914-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.715 DEBUG:boto:b'hcnphz-11dummy1000falsedummy2023-09-29T21:58:19.034Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:52.716 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.716 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.717 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.717 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.717 DEBUG:boto:Method: GET 2023-09-29T22:00:52.717 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.718 DEBUG:boto:Data: 2023-09-29T22:00:52.718 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.718 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.718 DEBUG:boto:Params: {} 2023-09-29T22:00:52.719 DEBUG:boto:Token: None 2023-09-29T22:00:52.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.719 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE= 2023-09-29T22:00:52.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.723 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000af780703244b510a-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.766 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.767 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.767 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.767 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.767 DEBUG:boto:Method: GET 2023-09-29T22:00:52.767 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.767 DEBUG:boto:Data: 2023-09-29T22:00:52.767 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.768 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.768 DEBUG:boto:Params: {} 2023-09-29T22:00:52.768 DEBUG:boto:Token: None 2023-09-29T22:00:52.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE= 2023-09-29T22:00:52.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054fa777a3b8b96ac-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.771 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.772 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.772 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.772 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.772 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.772 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.772 DEBUG:boto:Data: 2023-09-29T22:00:52.772 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.772 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.773 DEBUG:boto:Params: {} 2023-09-29T22:00:52.773 DEBUG:boto:Token: None 2023-09-29T22:00:52.773 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9nn6KN45tMFuBrteslSFkrd3SJs= 2023-09-29T22:00:52.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9nn6KN45tMFuBrteslSFkrd3SJs=', 'Content-Length': '0'} 2023-09-29T22:00:52.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef1421cf6922bbd9-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.776 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.776 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.776 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.776 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.777 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.777 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.777 DEBUG:boto:Data: 2023-09-29T22:00:52.777 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.777 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.777 DEBUG:boto:Params: {} 2023-09-29T22:00:52.777 DEBUG:boto:Token: None 2023-09-29T22:00:52.777 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.778 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9nn6KN45tMFuBrteslSFkrd3SJs= 2023-09-29T22:00:52.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9nn6KN45tMFuBrteslSFkrd3SJs=', 'Content-Length': '0'} 2023-09-29T22:00:52.780 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009da56b19b7a1b0c3-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:52.781 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.781 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.781 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.781 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.781 DEBUG:boto:Method: GET 2023-09-29T22:00:52.781 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.782 DEBUG:boto:Data: 2023-09-29T22:00:52.782 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.782 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.782 DEBUG:boto:Params: {} 2023-09-29T22:00:52.782 DEBUG:boto:Token: None 2023-09-29T22:00:52.782 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.782 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE= 2023-09-29T22:00:52.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.785 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078043a345b037406-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.826 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.827 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.827 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.827 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.827 DEBUG:boto:Method: GET 2023-09-29T22:00:52.827 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.827 DEBUG:boto:Data: 2023-09-29T22:00:52.827 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.828 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.828 DEBUG:boto:Params: {} 2023-09-29T22:00:52.828 DEBUG:boto:Token: None 2023-09-29T22:00:52.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.828 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE= 2023-09-29T22:00:52.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.831 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000055c3782150fb3cc0-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.832 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.832 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.832 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.832 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.832 DEBUG:boto:Method: GET 2023-09-29T22:00:52.832 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.832 DEBUG:boto:Data: 2023-09-29T22:00:52.832 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.833 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.833 DEBUG:boto:Params: {} 2023-09-29T22:00:52.833 DEBUG:boto:Token: None 2023-09-29T22:00:52.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.839 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca3bf44b2391c2a-0065174914-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.882 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.883 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.883 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.883 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.883 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.883 DEBUG:boto:Method: GET 2023-09-29T22:00:52.884 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.884 DEBUG:boto:Data: 2023-09-29T22:00:52.884 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.884 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.884 DEBUG:boto:Params: {} 2023-09-29T22:00:52.884 DEBUG:boto:Token: None 2023-09-29T22:00:52.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078f7864f214945ce-0065174914-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.890 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.890 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.890 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.891 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.891 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T22:00:52.891 DEBUG:boto:Data: 2023-09-29T22:00:52.891 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.891 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.891 DEBUG:boto:Params: {} 2023-09-29T22:00:52.892 DEBUG:boto:Token: None 2023-09-29T22:00:52.892 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1 2023-09-29T22:00:52.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VTxdd0mYQBv86Kb2Af4X/z8VwoE= 2023-09-29T22:00:52.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VTxdd0mYQBv86Kb2Af4X/z8VwoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.896 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ed7baabcee36199-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.896 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.897 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.897 DEBUG:boto:Method: HEAD 2023-09-29T22:00:52.897 DEBUG:boto:Path: /hcnphz-11/obj-a1 2023-09-29T22:00:52.897 DEBUG:boto:Data: 2023-09-29T22:00:52.897 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.897 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.897 DEBUG:boto:Params: {} 2023-09-29T22:00:52.898 DEBUG:boto:Token: None 2023-09-29T22:00:52.898 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1 2023-09-29T22:00:52.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VTxdd0mYQBv86Kb2Af4X/z8VwoE= 2023-09-29T22:00:52.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VTxdd0mYQBv86Kb2Af4X/z8VwoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.901 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000036bda655def0d24e-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:52.902 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.902 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.902 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.902 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.902 DEBUG:boto:Method: GET 2023-09-29T22:00:52.902 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.903 DEBUG:boto:Data: 2023-09-29T22:00:52.903 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.903 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.903 DEBUG:boto:Params: {} 2023-09-29T22:00:52.903 DEBUG:boto:Token: None 2023-09-29T22:00:52.903 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.903 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE= 2023-09-29T22:00:52.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.906 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009e486b2fcfcaf9c6-0065174914-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.946 DEBUG:boto:path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.947 DEBUG:boto:auth_path=/hcnphz-11/obj-a1 2023-09-29T22:00:52.947 DEBUG:boto:path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.947 DEBUG:boto:auth_path=/hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.947 DEBUG:boto:Method: GET 2023-09-29T22:00:52.947 DEBUG:boto:Path: /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.947 DEBUG:boto:Data: 2023-09-29T22:00:52.947 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:52.947 DEBUG:boto:Port: 8000 2023-09-29T22:00:52.947 DEBUG:boto:Params: {} 2023-09-29T22:00:52.948 DEBUG:boto:Token: None 2023-09-29T22:00:52.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/obj-a1?versionId=FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl 2023-09-29T22:00:52.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE= 2023-09-29T22:00:52.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VNhMHJ3GdzT13GTMn0c+nOrehoE=', 'Content-Length': '0'} 2023-09-29T22:00:52.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'FaeW-gBhkz9YM-wKPOiwbdq3dWizjTl'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008d9c73767e51608d-0065174914-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.951 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.951 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.951 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.952 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.952 DEBUG:boto:Method: GET 2023-09-29T22:00:52.952 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.952 DEBUG:boto:Data: 2023-09-29T22:00:52.952 DEBUG:boto:Headers: {} 2023-09-29T22:00:52.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:52.952 DEBUG:boto:Port: 8001 2023-09-29T22:00:52.952 DEBUG:boto:Params: {} 2023-09-29T22:00:52.953 DEBUG:boto:Token: None 2023-09-29T22:00:52.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:52 GMT /hcnphz-11/ 2023-09-29T22:00:52.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo= 2023-09-29T22:00:52.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcOTI24kLSEmuR3PtDEE97KBDpo=', 'Content-Length': '0'} 2023-09-29T22:00:52.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000becf61e9470f9ae8-0065174914-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:52.998 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:52.999 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:52.999 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:52.999 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.999 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.999 DEBUG:boto:Method: GET 2023-09-29T22:00:52.999 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a1 2023-09-29T22:00:52.999 DEBUG:boto:Data: 2023-09-29T22:00:53.001 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.001 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.001 DEBUG:boto:Params: {} 2023-09-29T22:00:53.001 DEBUG:boto:Token: None 2023-09-29T22:00:53.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/ 2023-09-29T22:00:53.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo= 2023-09-29T22:00:53.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo=', 'Content-Length': '0'} 2023-09-29T22:00:53.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e560eaece66398a-0065174914-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.009 DEBUG:boto:b'hcnphz-11obj-a11000falseobj-a12023-09-29T21:58:54.423Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:53.009 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.009 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.010 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.010 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.010 DEBUG:boto:Method: GET 2023-09-29T22:00:53.010 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.010 DEBUG:boto:Data: 2023-09-29T22:00:53.010 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.010 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.011 DEBUG:boto:Params: {} 2023-09-29T22:00:53.011 DEBUG:boto:Token: None 2023-09-29T22:00:53.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8= 2023-09-29T22:00:53.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8=', 'Content-Length': '0'} 2023-09-29T22:00:53.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005c2fe2a526d3997f-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.014 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.014 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.014 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.014 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.014 DEBUG:boto:Method: GET 2023-09-29T22:00:53.015 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.015 DEBUG:boto:Data: 2023-09-29T22:00:53.015 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.015 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.015 DEBUG:boto:Params: {} 2023-09-29T22:00:53.015 DEBUG:boto:Token: None 2023-09-29T22:00:53.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.015 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8= 2023-09-29T22:00:53.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8=', 'Content-Length': '0'} 2023-09-29T22:00:53.017 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000506acc89e58b16f9-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.018 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.018 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.018 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.018 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.018 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.018 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.019 DEBUG:boto:Data: 2023-09-29T22:00:53.019 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.019 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.019 DEBUG:boto:Params: {} 2023-09-29T22:00:53.019 DEBUG:boto:Token: None 2023-09-29T22:00:53.019 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lrf0RzBYQssSOH10PDVP09AA9PU= 2023-09-29T22:00:53.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lrf0RzBYQssSOH10PDVP09AA9PU=', 'Content-Length': '0'} 2023-09-29T22:00:53.021 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000050c60c51b18e9d81-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.022 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.022 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.022 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.022 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.022 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.022 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.023 DEBUG:boto:Data: 2023-09-29T22:00:53.023 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.023 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.023 DEBUG:boto:Params: {} 2023-09-29T22:00:53.023 DEBUG:boto:Token: None 2023-09-29T22:00:53.023 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lrf0RzBYQssSOH10PDVP09AA9PU= 2023-09-29T22:00:53.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lrf0RzBYQssSOH10PDVP09AA9PU=', 'Content-Length': '0'} 2023-09-29T22:00:53.026 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d1a9009f962f3426-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:53.026 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.026 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.026 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.026 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.027 DEBUG:boto:Method: GET 2023-09-29T22:00:53.027 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.027 DEBUG:boto:Data: 2023-09-29T22:00:53.027 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.027 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.027 DEBUG:boto:Params: {} 2023-09-29T22:00:53.027 DEBUG:boto:Token: None 2023-09-29T22:00:53.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.028 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8= 2023-09-29T22:00:53.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8=', 'Content-Length': '0'} 2023-09-29T22:00:53.030 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000086a868a19a3b38d9-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.030 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.030 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.031 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.031 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.031 DEBUG:boto:Method: GET 2023-09-29T22:00:53.031 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.031 DEBUG:boto:Data: 2023-09-29T22:00:53.031 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.031 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.031 DEBUG:boto:Params: {} 2023-09-29T22:00:53.031 DEBUG:boto:Token: None 2023-09-29T22:00:53.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8= 2023-09-29T22:00:53.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8=', 'Content-Length': '0'} 2023-09-29T22:00:53.034 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000910b1147170cb3d7-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.078 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:53.079 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:53.079 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.079 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.079 DEBUG:boto:Method: GET 2023-09-29T22:00:53.079 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.079 DEBUG:boto:Data: 2023-09-29T22:00:53.079 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.079 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.080 DEBUG:boto:Params: {} 2023-09-29T22:00:53.080 DEBUG:boto:Token: None 2023-09-29T22:00:53.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/ 2023-09-29T22:00:53.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo= 2023-09-29T22:00:53.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo=', 'Content-Length': '0'} 2023-09-29T22:00:53.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e00567db85b90fdf-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.085 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.086 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:53.086 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:53.086 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.086 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.086 DEBUG:boto:Method: GET 2023-09-29T22:00:53.086 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.086 DEBUG:boto:Data: 2023-09-29T22:00:53.086 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.087 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.087 DEBUG:boto:Params: {} 2023-09-29T22:00:53.087 DEBUG:boto:Token: None 2023-09-29T22:00:53.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/ 2023-09-29T22:00:53.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo= 2023-09-29T22:00:53.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo=', 'Content-Length': '0'} 2023-09-29T22:00:53.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f1d967c2291a43e-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.138 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.139 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.139 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.139 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.139 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T22:00:53.139 DEBUG:boto:Data: 2023-09-29T22:00:53.140 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.140 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.140 DEBUG:boto:Params: {} 2023-09-29T22:00:53.140 DEBUG:boto:Token: None 2023-09-29T22:00:53.140 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2 2023-09-29T22:00:53.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r1tNRbg3IzHfIX2+aH93f+wv48= 2023-09-29T22:00:53.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r1tNRbg3IzHfIX2+aH93f+wv48=', 'Content-Length': '0'} 2023-09-29T22:00:53.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3cc170868cbce2a-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.145 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.145 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.145 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.145 DEBUG:boto:Path: /hcnphz-11/obj-a2 2023-09-29T22:00:53.145 DEBUG:boto:Data: 2023-09-29T22:00:53.146 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.146 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.146 DEBUG:boto:Params: {} 2023-09-29T22:00:53.146 DEBUG:boto:Token: None 2023-09-29T22:00:53.146 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2 2023-09-29T22:00:53.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r1tNRbg3IzHfIX2+aH93f+wv48= 2023-09-29T22:00:53.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r1tNRbg3IzHfIX2+aH93f+wv48=', 'Content-Length': '0'} 2023-09-29T22:00:53.150 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b95a986cf032a1d2-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:53.150 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.150 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.150 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.151 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.151 DEBUG:boto:Method: GET 2023-09-29T22:00:53.151 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.151 DEBUG:boto:Data: 2023-09-29T22:00:53.151 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.151 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.151 DEBUG:boto:Params: {} 2023-09-29T22:00:53.151 DEBUG:boto:Token: None 2023-09-29T22:00:53.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8= 2023-09-29T22:00:53.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8=', 'Content-Length': '0'} 2023-09-29T22:00:53.155 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007b17a8ae3029d472-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.155 DEBUG:boto:path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.155 DEBUG:boto:auth_path=/hcnphz-11/obj-a2 2023-09-29T22:00:53.155 DEBUG:boto:path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.156 DEBUG:boto:auth_path=/hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.156 DEBUG:boto:Method: GET 2023-09-29T22:00:53.156 DEBUG:boto:Path: /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.156 DEBUG:boto:Data: 2023-09-29T22:00:53.156 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.156 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.156 DEBUG:boto:Params: {} 2023-09-29T22:00:53.157 DEBUG:boto:Token: None 2023-09-29T22:00:53.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/obj-a2?versionId=gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ- 2023-09-29T22:00:53.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8= 2023-09-29T22:00:53.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xRwicAm4hMutgsFDL60nD5f/ZQ8=', 'Content-Length': '0'} 2023-09-29T22:00:53.160 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'gXIdPPdDjfzBfuP1DnIgwqFNPWUXkQ-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006f1c0a44775dbf83-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.205 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:53.205 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:53.205 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.205 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.205 DEBUG:boto:Method: GET 2023-09-29T22:00:53.206 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.206 DEBUG:boto:Data: 2023-09-29T22:00:53.206 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.206 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.206 DEBUG:boto:Params: {} 2023-09-29T22:00:53.206 DEBUG:boto:Token: None 2023-09-29T22:00:53.206 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/ 2023-09-29T22:00:53.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo= 2023-09-29T22:00:53.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo=', 'Content-Length': '0'} 2023-09-29T22:00:53.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8d4cc0091eaa88b-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.213 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.213 DEBUG:boto:path=/hcnphz-11/ 2023-09-29T22:00:53.213 DEBUG:boto:auth_path=/hcnphz-11/ 2023-09-29T22:00:53.213 DEBUG:boto:path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.213 DEBUG:boto:auth_path=/hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.214 DEBUG:boto:Method: GET 2023-09-29T22:00:53.214 DEBUG:boto:Path: /hcnphz-11/?prefix=obj-a2 2023-09-29T22:00:53.214 DEBUG:boto:Data: 2023-09-29T22:00:53.214 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.214 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.214 DEBUG:boto:Params: {} 2023-09-29T22:00:53.214 DEBUG:boto:Token: None 2023-09-29T22:00:53.214 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-11/ 2023-09-29T22:00:53.215 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo= 2023-09-29T22:00:53.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Vx0cZ7Au2ACgKqiHZzeHtx7+mo=', 'Content-Length': '0'} 2023-09-29T22:00:53.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48587ea0329d497-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.220 DEBUG:boto:b'hcnphz-11obj-a21000falseobj-a22023-09-29T21:58:54.564Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.220 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-11 zones={a2, a1} 2023-09-29T22:00:53.220 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-12 2023-09-29T22:00:53.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:53.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:53.221 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-12 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:53.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:53.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:53.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:53.330+0000 7fa4847d3f40 1 bucket sync caught up with source: 2023-09-29T22:00:53.334 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.6.11, , , 00000000002.3.11, 00000000011.32.6, ] 2023-09-29T22:00:53.335 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.6.11, , , 00000000002.3.11, 00000000011.32.6, ] 2023-09-29T22:00:53.335 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:53.330+0000 7fa4847d3f40 0 bucket checkpoint complete 2023-09-29T22:00:53.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-12 2023-09-29T22:00:53.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:53.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-12', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:53.359 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-12 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:53.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:53.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:53.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:53.474+0000 7f245af8af40 1 bucket sync caught up with source: 2023-09-29T22:00:53.479 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.13.10, , , 00000000011.26.6, 00000000003.24.10, ] 2023-09-29T22:00:53.479 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.13.10, , , 00000000011.26.6, 00000000003.24.10, ] 2023-09-29T22:00:53.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:53.474+0000 7f245af8af40 0 bucket checkpoint complete 2023-09-29T22:00:53.502 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-12 zones={a2, a1} 2023-09-29T22:00:53.503 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.503 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.503 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.503 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T22:00:53.503 DEBUG:boto:Data: 2023-09-29T22:00:53.503 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.503 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.504 DEBUG:boto:Params: {} 2023-09-29T22:00:53.504 DEBUG:boto:Token: None 2023-09-29T22:00:53.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c1qheLz3qoUdT4ILLNxFqYpksME= 2023-09-29T22:00:53.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c1qheLz3qoUdT4ILLNxFqYpksME=', 'Content-Length': '0'} 2023-09-29T22:00:53.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000000071396a111eeae-0065174915-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.508 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.508 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.508 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.508 DEBUG:boto:Path: /hcnphz-12/ 2023-09-29T22:00:53.508 DEBUG:boto:Data: 2023-09-29T22:00:53.508 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.509 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.509 DEBUG:boto:Params: {} 2023-09-29T22:00:53.509 DEBUG:boto:Token: None 2023-09-29T22:00:53.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c1qheLz3qoUdT4ILLNxFqYpksME= 2023-09-29T22:00:53.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c1qheLz3qoUdT4ILLNxFqYpksME=', 'Content-Length': '0'} 2023-09-29T22:00:53.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e533632b8cbdfa0-0065174915-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.513 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:00:53.513 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.513 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.513 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.513 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.513 DEBUG:boto:Method: GET 2023-09-29T22:00:53.513 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.513 DEBUG:boto:Data: 2023-09-29T22:00:53.514 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.514 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.514 DEBUG:boto:Params: {} 2023-09-29T22:00:53.514 DEBUG:boto:Token: None 2023-09-29T22:00:53.514 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/?versions 2023-09-29T22:00:53.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A= 2023-09-29T22:00:53.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A=', 'Content-Length': '0'} 2023-09-29T22:00:53.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4a3f7837f11932c-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.518 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.519 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:53.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:53.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:53.519 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:00:53.520 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.520 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.520 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.520 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.520 DEBUG:boto:Method: GET 2023-09-29T22:00:53.520 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.520 DEBUG:boto:Data: 2023-09-29T22:00:53.520 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.521 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.521 DEBUG:boto:Params: {} 2023-09-29T22:00:53.521 DEBUG:boto:Token: None 2023-09-29T22:00:53.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/?versions 2023-09-29T22:00:53.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A= 2023-09-29T22:00:53.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A=', 'Content-Length': '0'} 2023-09-29T22:00:53.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb01bac511162caa-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.525 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.526 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:00:53.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a1 2023-09-29T22:00:53.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj-a2 2023-09-29T22:00:53.526 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.526 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.526 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.526 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.526 DEBUG:boto:Method: GET 2023-09-29T22:00:53.527 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.527 DEBUG:boto:Data: 2023-09-29T22:00:53.527 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.527 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.527 DEBUG:boto:Params: {} 2023-09-29T22:00:53.527 DEBUG:boto:Token: None 2023-09-29T22:00:53.527 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/?versions 2023-09-29T22:00:53.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A= 2023-09-29T22:00:53.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A=', 'Content-Length': '0'} 2023-09-29T22:00:53.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039be27be7f4eb717-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.531 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.531 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.532 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.532 DEBUG:boto:path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.532 DEBUG:boto:auth_path=/hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.532 DEBUG:boto:Method: GET 2023-09-29T22:00:53.532 DEBUG:boto:Path: /hcnphz-12/?versions&max-keys=999 2023-09-29T22:00:53.532 DEBUG:boto:Data: 2023-09-29T22:00:53.532 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.532 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.533 DEBUG:boto:Params: {} 2023-09-29T22:00:53.533 DEBUG:boto:Token: None 2023-09-29T22:00:53.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/?versions 2023-09-29T22:00:53.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A= 2023-09-29T22:00:53.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q+cGBjcGMp7urARnjeZ8lnMJN4A=', 'Content-Length': '0'} 2023-09-29T22:00:53.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a085e48aeacb395-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.537 DEBUG:boto:b'hcnphz-12999falsedummyOizGUzRwgLNMw1lqQo..6LMJ3ALbugptrue2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a1fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGwtrue2023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj-a2I72531cJulXMyB6yxvPBWj7fn2AYfOPtrue2023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:53.537 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.538 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.538 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.538 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.538 DEBUG:boto:Method: GET 2023-09-29T22:00:53.538 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.538 DEBUG:boto:Data: 2023-09-29T22:00:53.538 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.538 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.538 DEBUG:boto:Params: {} 2023-09-29T22:00:53.539 DEBUG:boto:Token: None 2023-09-29T22:00:53.539 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.539 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0= 2023-09-29T22:00:53.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0=', 'Content-Length': '0'} 2023-09-29T22:00:53.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000891d4f2021f8976f-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.542 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.542 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.542 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.542 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.543 DEBUG:boto:Method: GET 2023-09-29T22:00:53.543 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.543 DEBUG:boto:Data: 2023-09-29T22:00:53.543 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.543 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.543 DEBUG:boto:Params: {} 2023-09-29T22:00:53.543 DEBUG:boto:Token: None 2023-09-29T22:00:53.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0= 2023-09-29T22:00:53.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0=', 'Content-Length': '0'} 2023-09-29T22:00:53.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000bda1ba0012f0d559-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.547 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.547 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.547 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.547 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.547 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.548 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.548 DEBUG:boto:Data: 2023-09-29T22:00:53.548 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.548 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.549 DEBUG:boto:Params: {} 2023-09-29T22:00:53.549 DEBUG:boto:Token: None 2023-09-29T22:00:53.549 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oxq4v6HnJeHWHvcFVR5UGjzWS8Y= 2023-09-29T22:00:53.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oxq4v6HnJeHWHvcFVR5UGjzWS8Y=', 'Content-Length': '0'} 2023-09-29T22:00:53.552 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ab834736c166365e-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.553 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.553 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.553 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.553 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.554 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.554 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.554 DEBUG:boto:Data: 2023-09-29T22:00:53.554 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.555 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.555 DEBUG:boto:Params: {} 2023-09-29T22:00:53.555 DEBUG:boto:Token: None 2023-09-29T22:00:53.555 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oxq4v6HnJeHWHvcFVR5UGjzWS8Y= 2023-09-29T22:00:53.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oxq4v6HnJeHWHvcFVR5UGjzWS8Y=', 'Content-Length': '0'} 2023-09-29T22:00:53.558 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002fb784d5340618cf-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:53.559 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.559 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.559 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.560 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.560 DEBUG:boto:Method: GET 2023-09-29T22:00:53.560 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.560 DEBUG:boto:Data: 2023-09-29T22:00:53.560 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.561 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.561 DEBUG:boto:Params: {} 2023-09-29T22:00:53.561 DEBUG:boto:Token: None 2023-09-29T22:00:53.562 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0= 2023-09-29T22:00:53.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0=', 'Content-Length': '0'} 2023-09-29T22:00:53.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000305495dd41245f66-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.565 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.565 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.565 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.566 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.566 DEBUG:boto:Method: GET 2023-09-29T22:00:53.566 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.566 DEBUG:boto:Data: 2023-09-29T22:00:53.567 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.567 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.567 DEBUG:boto:Params: {} 2023-09-29T22:00:53.567 DEBUG:boto:Token: None 2023-09-29T22:00:53.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0= 2023-09-29T22:00:53.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0=', 'Content-Length': '0'} 2023-09-29T22:00:53.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009680d7636f78932e-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.571 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.571 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.572 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.572 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.572 DEBUG:boto:Method: GET 2023-09-29T22:00:53.572 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.572 DEBUG:boto:Data: 2023-09-29T22:00:53.573 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.573 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.573 DEBUG:boto:Params: {} 2023-09-29T22:00:53.573 DEBUG:boto:Token: None 2023-09-29T22:00:53.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000676db03886e38c2a-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.626 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.627 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.627 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.627 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.627 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.628 DEBUG:boto:Method: GET 2023-09-29T22:00:53.628 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.628 DEBUG:boto:Data: 2023-09-29T22:00:53.628 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.628 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.629 DEBUG:boto:Params: {} 2023-09-29T22:00:53.629 DEBUG:boto:Token: None 2023-09-29T22:00:53.629 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bbf1c9590e08ae3-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.634 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.634 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.634 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.634 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.634 DEBUG:boto:Path: /hcnphz-12/dummy 2023-09-29T22:00:53.635 DEBUG:boto:Data: 2023-09-29T22:00:53.635 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.635 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.635 DEBUG:boto:Params: {} 2023-09-29T22:00:53.635 DEBUG:boto:Token: None 2023-09-29T22:00:53.635 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy 2023-09-29T22:00:53.635 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u6X24b0X+sDsg4+ugWNZOstyw+M= 2023-09-29T22:00:53.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u6X24b0X+sDsg4+ugWNZOstyw+M=', 'Content-Length': '0'} 2023-09-29T22:00:53.639 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003c99a7e425553702-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.639 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.639 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.639 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.640 DEBUG:boto:Path: /hcnphz-12/dummy 2023-09-29T22:00:53.640 DEBUG:boto:Data: 2023-09-29T22:00:53.640 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.640 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.640 DEBUG:boto:Params: {} 2023-09-29T22:00:53.640 DEBUG:boto:Token: None 2023-09-29T22:00:53.640 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy 2023-09-29T22:00:53.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u6X24b0X+sDsg4+ugWNZOstyw+M= 2023-09-29T22:00:53.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u6X24b0X+sDsg4+ugWNZOstyw+M=', 'Content-Length': '0'} 2023-09-29T22:00:53.644 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e948518a70a532f5-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:00:53.645 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.645 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.645 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.645 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.645 DEBUG:boto:Method: GET 2023-09-29T22:00:53.646 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.646 DEBUG:boto:Data: 2023-09-29T22:00:53.646 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.646 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.646 DEBUG:boto:Params: {} 2023-09-29T22:00:53.646 DEBUG:boto:Token: None 2023-09-29T22:00:53.646 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0= 2023-09-29T22:00:53.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0=', 'Content-Length': '0'} 2023-09-29T22:00:53.649 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f22c3578a03437f-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.650 DEBUG:boto:path=/hcnphz-12/dummy 2023-09-29T22:00:53.650 DEBUG:boto:auth_path=/hcnphz-12/dummy 2023-09-29T22:00:53.650 DEBUG:boto:path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.650 DEBUG:boto:auth_path=/hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.650 DEBUG:boto:Method: GET 2023-09-29T22:00:53.650 DEBUG:boto:Path: /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.650 DEBUG:boto:Data: 2023-09-29T22:00:53.650 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.651 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.651 DEBUG:boto:Params: {} 2023-09-29T22:00:53.651 DEBUG:boto:Token: None 2023-09-29T22:00:53.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/dummy?versionId=OizGUzRwgLNMw1lqQo..6LMJ3ALbugp 2023-09-29T22:00:53.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0= 2023-09-29T22:00:53.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dZ9NsIbHwXCZ+y4Utod+nyDE4/0=', 'Content-Length': '0'} 2023-09-29T22:00:53.654 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:34 GMT'), ('x-amz-version-id', 'OizGUzRwgLNMw1lqQo..6LMJ3ALbugp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c66733b5a94e2ef6-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.654 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.655 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.655 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.655 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.655 DEBUG:boto:Method: GET 2023-09-29T22:00:53.655 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.655 DEBUG:boto:Data: 2023-09-29T22:00:53.655 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.655 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.656 DEBUG:boto:Params: {} 2023-09-29T22:00:53.656 DEBUG:boto:Token: None 2023-09-29T22:00:53.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000691faf4a99dd7f1b-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.661 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.661 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.661 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.661 DEBUG:boto:path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.661 DEBUG:boto:auth_path=/hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.662 DEBUG:boto:Method: GET 2023-09-29T22:00:53.662 DEBUG:boto:Path: /hcnphz-12/?prefix=dummy 2023-09-29T22:00:53.662 DEBUG:boto:Data: 2023-09-29T22:00:53.662 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.662 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.662 DEBUG:boto:Params: {} 2023-09-29T22:00:53.663 DEBUG:boto:Token: None 2023-09-29T22:00:53.663 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e74905e5394747b-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.711 DEBUG:boto:b'hcnphz-12dummy1000falsedummy2023-09-29T21:58:34.242Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:53.711 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.711 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.711 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.712 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.712 DEBUG:boto:Method: GET 2023-09-29T22:00:53.712 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.712 DEBUG:boto:Data: 2023-09-29T22:00:53.712 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.712 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.712 DEBUG:boto:Params: {} 2023-09-29T22:00:53.712 DEBUG:boto:Token: None 2023-09-29T22:00:53.713 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg= 2023-09-29T22:00:53.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg=', 'Content-Length': '0'} 2023-09-29T22:00:53.716 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a577b46d95f0e711-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.716 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.716 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.716 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.716 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.717 DEBUG:boto:Method: GET 2023-09-29T22:00:53.717 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.717 DEBUG:boto:Data: 2023-09-29T22:00:53.717 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.717 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.717 DEBUG:boto:Params: {} 2023-09-29T22:00:53.717 DEBUG:boto:Token: None 2023-09-29T22:00:53.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg= 2023-09-29T22:00:53.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg=', 'Content-Length': '0'} 2023-09-29T22:00:53.721 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ff1a6989c1a7f7b2-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.762 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.763 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.763 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.763 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.763 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.763 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.763 DEBUG:boto:Data: 2023-09-29T22:00:53.763 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.763 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.763 DEBUG:boto:Params: {} 2023-09-29T22:00:53.764 DEBUG:boto:Token: None 2023-09-29T22:00:53.764 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RJrPerKbhxYFZ0H+YgervTlRlBA= 2023-09-29T22:00:53.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RJrPerKbhxYFZ0H+YgervTlRlBA=', 'Content-Length': '0'} 2023-09-29T22:00:53.767 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f671a4cee4335fc2-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.767 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.767 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.767 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.768 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.768 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.768 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.768 DEBUG:boto:Data: 2023-09-29T22:00:53.768 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.768 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.768 DEBUG:boto:Params: {} 2023-09-29T22:00:53.768 DEBUG:boto:Token: None 2023-09-29T22:00:53.769 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.769 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RJrPerKbhxYFZ0H+YgervTlRlBA= 2023-09-29T22:00:53.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RJrPerKbhxYFZ0H+YgervTlRlBA=', 'Content-Length': '0'} 2023-09-29T22:00:53.771 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ad53e4dc885fbc37-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:53.772 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.772 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.772 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.772 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.772 DEBUG:boto:Method: GET 2023-09-29T22:00:53.772 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.772 DEBUG:boto:Data: 2023-09-29T22:00:53.772 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.773 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.773 DEBUG:boto:Params: {} 2023-09-29T22:00:53.773 DEBUG:boto:Token: None 2023-09-29T22:00:53.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg= 2023-09-29T22:00:53.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg=', 'Content-Length': '0'} 2023-09-29T22:00:53.776 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000563afc411c238cd5-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.822 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.823 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.823 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.823 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.823 DEBUG:boto:Method: GET 2023-09-29T22:00:53.823 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.823 DEBUG:boto:Data: 2023-09-29T22:00:53.823 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.824 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.824 DEBUG:boto:Params: {} 2023-09-29T22:00:53.824 DEBUG:boto:Token: None 2023-09-29T22:00:53.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg= 2023-09-29T22:00:53.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg=', 'Content-Length': '0'} 2023-09-29T22:00:53.828 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5eb9471e312aade-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.828 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.828 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.829 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.829 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.829 DEBUG:boto:Method: GET 2023-09-29T22:00:53.829 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.829 DEBUG:boto:Data: 2023-09-29T22:00:53.829 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.830 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.830 DEBUG:boto:Params: {} 2023-09-29T22:00:53.830 DEBUG:boto:Token: None 2023-09-29T22:00:53.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008e93ad46c8609b7-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.882 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.883 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.883 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.883 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.883 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.883 DEBUG:boto:Method: GET 2023-09-29T22:00:53.883 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.883 DEBUG:boto:Data: 2023-09-29T22:00:53.884 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.884 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.884 DEBUG:boto:Params: {} 2023-09-29T22:00:53.884 DEBUG:boto:Token: None 2023-09-29T22:00:53.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000555d63f7f7282bd8-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.889 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.890 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.890 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.890 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.890 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T22:00:53.890 DEBUG:boto:Data: 2023-09-29T22:00:53.890 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.890 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.891 DEBUG:boto:Params: {} 2023-09-29T22:00:53.891 DEBUG:boto:Token: None 2023-09-29T22:00:53.891 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1 2023-09-29T22:00:53.891 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oVm7M9WSEqltd1ityaB6v1vPKV8= 2023-09-29T22:00:53.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oVm7M9WSEqltd1ityaB6v1vPKV8=', 'Content-Length': '0'} 2023-09-29T22:00:53.895 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000257252dd5373b496-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.895 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.895 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.895 DEBUG:boto:Method: HEAD 2023-09-29T22:00:53.895 DEBUG:boto:Path: /hcnphz-12/obj-a1 2023-09-29T22:00:53.895 DEBUG:boto:Data: 2023-09-29T22:00:53.895 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.896 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.896 DEBUG:boto:Params: {} 2023-09-29T22:00:53.896 DEBUG:boto:Token: None 2023-09-29T22:00:53.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1 2023-09-29T22:00:53.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oVm7M9WSEqltd1ityaB6v1vPKV8= 2023-09-29T22:00:53.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oVm7M9WSEqltd1ityaB6v1vPKV8=', 'Content-Length': '0'} 2023-09-29T22:00:53.899 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000718518fbec43e987-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a1 2023-09-29T22:00:53.900 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.900 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.900 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.900 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.900 DEBUG:boto:Method: GET 2023-09-29T22:00:53.900 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.901 DEBUG:boto:Data: 2023-09-29T22:00:53.901 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.901 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.901 DEBUG:boto:Params: {} 2023-09-29T22:00:53.901 DEBUG:boto:Token: None 2023-09-29T22:00:53.901 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg= 2023-09-29T22:00:53.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg=', 'Content-Length': '0'} 2023-09-29T22:00:53.904 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000093e53302095d27d2-0065174915-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.946 DEBUG:boto:path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.946 DEBUG:boto:auth_path=/hcnphz-12/obj-a1 2023-09-29T22:00:53.947 DEBUG:boto:path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.947 DEBUG:boto:auth_path=/hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.947 DEBUG:boto:Method: GET 2023-09-29T22:00:53.947 DEBUG:boto:Path: /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.947 DEBUG:boto:Data: 2023-09-29T22:00:53.947 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:53.947 DEBUG:boto:Port: 8000 2023-09-29T22:00:53.947 DEBUG:boto:Params: {} 2023-09-29T22:00:53.948 DEBUG:boto:Token: None 2023-09-29T22:00:53.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/obj-a1?versionId=fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw 2023-09-29T22:00:53.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg= 2023-09-29T22:00:53.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OpfOtu4SNAShXzo10cedDj1nUEg=', 'Content-Length': '0'} 2023-09-29T22:00:53.951 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'fzFGsS8CCPNG-j6RTQmhJCbo2lIPyGw'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bc91c9351c31178-0065174915-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.951 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.951 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.951 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.951 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.952 DEBUG:boto:Method: GET 2023-09-29T22:00:53.952 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.952 DEBUG:boto:Data: 2023-09-29T22:00:53.952 DEBUG:boto:Headers: {} 2023-09-29T22:00:53.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:53.952 DEBUG:boto:Port: 8001 2023-09-29T22:00:53.952 DEBUG:boto:Params: {} 2023-09-29T22:00:53.952 DEBUG:boto:Token: None 2023-09-29T22:00:53.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:53 GMT /hcnphz-12/ 2023-09-29T22:00:53.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w= 2023-09-29T22:00:53.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R3aqyhqjQJAeq447uhpRjP1xJ4w=', 'Content-Length': '0'} 2023-09-29T22:00:53.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2ade8c9e46810fb-0065174915-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:53.998 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:53.999 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:53.999 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:53.999 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.999 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.999 DEBUG:boto:Method: GET 2023-09-29T22:00:53.999 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a1 2023-09-29T22:00:53.999 DEBUG:boto:Data: 2023-09-29T22:00:54.000 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.000 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.000 DEBUG:boto:Params: {} 2023-09-29T22:00:54.000 DEBUG:boto:Token: None 2023-09-29T22:00:54.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/ 2023-09-29T22:00:54.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs= 2023-09-29T22:00:54.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs=', 'Content-Length': '0'} 2023-09-29T22:00:54.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072d996390ac65586-0065174915-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.005 DEBUG:boto:b'hcnphz-12obj-a11000falseobj-a12023-09-29T21:58:54.735Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:54.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:54.006 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.006 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.006 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.006 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.006 DEBUG:boto:Method: GET 2023-09-29T22:00:54.006 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.007 DEBUG:boto:Data: 2023-09-29T22:00:54.007 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.007 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.007 DEBUG:boto:Params: {} 2023-09-29T22:00:54.007 DEBUG:boto:Token: None 2023-09-29T22:00:54.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w= 2023-09-29T22:00:54.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w=', 'Content-Length': '0'} 2023-09-29T22:00:54.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe63c407962a9963-0065174916-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.010 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.010 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.011 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.011 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.011 DEBUG:boto:Method: GET 2023-09-29T22:00:54.011 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.011 DEBUG:boto:Data: 2023-09-29T22:00:54.011 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.011 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.011 DEBUG:boto:Params: {} 2023-09-29T22:00:54.011 DEBUG:boto:Token: None 2023-09-29T22:00:54.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w= 2023-09-29T22:00:54.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w=', 'Content-Length': '0'} 2023-09-29T22:00:54.014 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b2beb90fc3d77329-0065174916-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.054 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.055 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.055 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.055 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.055 DEBUG:boto:Method: HEAD 2023-09-29T22:00:54.055 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.055 DEBUG:boto:Data: 2023-09-29T22:00:54.055 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.055 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.055 DEBUG:boto:Params: {} 2023-09-29T22:00:54.056 DEBUG:boto:Token: None 2023-09-29T22:00:54.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q41rMvuZykIR9Op1r6ta8HozVcU= 2023-09-29T22:00:54.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q41rMvuZykIR9Op1r6ta8HozVcU=', 'Content-Length': '0'} 2023-09-29T22:00:54.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002b97d4a34bb39020-0065174916-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.059 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.059 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.059 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.059 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.060 DEBUG:boto:Method: HEAD 2023-09-29T22:00:54.060 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.060 DEBUG:boto:Data: 2023-09-29T22:00:54.060 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.060 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.060 DEBUG:boto:Params: {} 2023-09-29T22:00:54.060 DEBUG:boto:Token: None 2023-09-29T22:00:54.060 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q41rMvuZykIR9Op1r6ta8HozVcU= 2023-09-29T22:00:54.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q41rMvuZykIR9Op1r6ta8HozVcU=', 'Content-Length': '0'} 2023-09-29T22:00:54.063 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb3ea63acf2f5d5f-0065174916-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:54.064 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.064 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.064 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.064 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.064 DEBUG:boto:Method: GET 2023-09-29T22:00:54.064 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.064 DEBUG:boto:Data: 2023-09-29T22:00:54.065 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.065 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.065 DEBUG:boto:Params: {} 2023-09-29T22:00:54.065 DEBUG:boto:Token: None 2023-09-29T22:00:54.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w= 2023-09-29T22:00:54.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w=', 'Content-Length': '0'} 2023-09-29T22:00:54.068 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000635cbc60ad35838e-0065174916-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.110 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.111 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.111 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.111 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.111 DEBUG:boto:Method: GET 2023-09-29T22:00:54.111 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.111 DEBUG:boto:Data: 2023-09-29T22:00:54.111 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.111 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.111 DEBUG:boto:Params: {} 2023-09-29T22:00:54.112 DEBUG:boto:Token: None 2023-09-29T22:00:54.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w= 2023-09-29T22:00:54.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w=', 'Content-Length': '0'} 2023-09-29T22:00:54.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aa069a0086b9813c-0065174916-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.162 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:54.163 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:54.163 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.163 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.163 DEBUG:boto:Method: GET 2023-09-29T22:00:54.163 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.163 DEBUG:boto:Data: 2023-09-29T22:00:54.163 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.163 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.164 DEBUG:boto:Params: {} 2023-09-29T22:00:54.164 DEBUG:boto:Token: None 2023-09-29T22:00:54.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/ 2023-09-29T22:00:54.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs= 2023-09-29T22:00:54.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs=', 'Content-Length': '0'} 2023-09-29T22:00:54.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8987ed364fdae2c-0065174916-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.169 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:54.169 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:54.169 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:54.169 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.169 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.170 DEBUG:boto:Method: GET 2023-09-29T22:00:54.170 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.170 DEBUG:boto:Data: 2023-09-29T22:00:54.170 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.170 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.170 DEBUG:boto:Params: {} 2023-09-29T22:00:54.170 DEBUG:boto:Token: None 2023-09-29T22:00:54.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/ 2023-09-29T22:00:54.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs= 2023-09-29T22:00:54.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs=', 'Content-Length': '0'} 2023-09-29T22:00:54.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018669ae98eee9eac-0065174916-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.222 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:54.223 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.223 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.223 DEBUG:boto:Method: HEAD 2023-09-29T22:00:54.223 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T22:00:54.223 DEBUG:boto:Data: 2023-09-29T22:00:54.223 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.225 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.225 DEBUG:boto:Params: {} 2023-09-29T22:00:54.225 DEBUG:boto:Token: None 2023-09-29T22:00:54.225 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2 2023-09-29T22:00:54.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPNgEsl/+1yvKWmm7x0+L4pRT84= 2023-09-29T22:00:54.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPNgEsl/+1yvKWmm7x0+L4pRT84=', 'Content-Length': '0'} 2023-09-29T22:00:54.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004870898206e6967a-0065174916-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.232 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.232 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.232 DEBUG:boto:Method: HEAD 2023-09-29T22:00:54.232 DEBUG:boto:Path: /hcnphz-12/obj-a2 2023-09-29T22:00:54.232 DEBUG:boto:Data: 2023-09-29T22:00:54.232 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.232 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.232 DEBUG:boto:Params: {} 2023-09-29T22:00:54.233 DEBUG:boto:Token: None 2023-09-29T22:00:54.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2 2023-09-29T22:00:54.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPNgEsl/+1yvKWmm7x0+L4pRT84= 2023-09-29T22:00:54.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPNgEsl/+1yvKWmm7x0+L4pRT84=', 'Content-Length': '0'} 2023-09-29T22:00:54.237 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000054890ac49021079b-0065174916-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-a2 2023-09-29T22:00:54.238 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.238 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.238 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.238 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.238 DEBUG:boto:Method: GET 2023-09-29T22:00:54.238 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.238 DEBUG:boto:Data: 2023-09-29T22:00:54.238 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.238 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.239 DEBUG:boto:Params: {} 2023-09-29T22:00:54.239 DEBUG:boto:Token: None 2023-09-29T22:00:54.239 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w= 2023-09-29T22:00:54.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w=', 'Content-Length': '0'} 2023-09-29T22:00:54.242 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c74510997178b5c2-0065174916-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.242 DEBUG:boto:path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.242 DEBUG:boto:auth_path=/hcnphz-12/obj-a2 2023-09-29T22:00:54.242 DEBUG:boto:path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.242 DEBUG:boto:auth_path=/hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.242 DEBUG:boto:Method: GET 2023-09-29T22:00:54.242 DEBUG:boto:Path: /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.243 DEBUG:boto:Data: 2023-09-29T22:00:54.243 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.243 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.243 DEBUG:boto:Params: {} 2023-09-29T22:00:54.243 DEBUG:boto:Token: None 2023-09-29T22:00:54.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/obj-a2?versionId=I72531cJulXMyB6yxvPBWj7fn2AYfOP 2023-09-29T22:00:54.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w= 2023-09-29T22:00:54.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZR3BDq7aNhcQ0AMiMUaeMPy4L/w=', 'Content-Length': '0'} 2023-09-29T22:00:54.246 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 21:58:54 GMT'), ('x-amz-version-id', 'I72531cJulXMyB6yxvPBWj7fn2AYfOP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f5ce03ecf99337c6-0065174916-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.294 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:54.294 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:54.295 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.295 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.295 DEBUG:boto:Method: GET 2023-09-29T22:00:54.295 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.295 DEBUG:boto:Data: 2023-09-29T22:00:54.295 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:00:54.295 DEBUG:boto:Port: 8001 2023-09-29T22:00:54.295 DEBUG:boto:Params: {} 2023-09-29T22:00:54.296 DEBUG:boto:Token: None 2023-09-29T22:00:54.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/ 2023-09-29T22:00:54.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs= 2023-09-29T22:00:54.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs=', 'Content-Length': '0'} 2023-09-29T22:00:54.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c36fc524b059320-0065174916-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.300 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:54.301 DEBUG:boto:path=/hcnphz-12/ 2023-09-29T22:00:54.301 DEBUG:boto:auth_path=/hcnphz-12/ 2023-09-29T22:00:54.301 DEBUG:boto:path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.301 DEBUG:boto:auth_path=/hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.301 DEBUG:boto:Method: GET 2023-09-29T22:00:54.301 DEBUG:boto:Path: /hcnphz-12/?prefix=obj-a2 2023-09-29T22:00:54.301 DEBUG:boto:Data: 2023-09-29T22:00:54.301 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.302 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.302 DEBUG:boto:Params: {} 2023-09-29T22:00:54.302 DEBUG:boto:Token: None 2023-09-29T22:00:54.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-12/ 2023-09-29T22:00:54.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs= 2023-09-29T22:00:54.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HyM/WjPLKoUY30vGg07B094pqGs=', 'Content-Length': '0'} 2023-09-29T22:00:54.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c63581112c865db6-0065174916-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.306 DEBUG:boto:b'hcnphz-12obj-a21000falseobj-a22023-09-29T21:58:54.881Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:00:54.307 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-12 zones={a2, a1} 2023-09-29T22:00:54.307 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2023-09-29T22:00:54.308 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:00:54.308 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:00:54.309 DEBUG:boto:Method: PUT 2023-09-29T22:00:54.309 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:00:54.309 DEBUG:boto:Data: 2023-09-29T22:00:54.309 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.309 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.309 DEBUG:boto:Params: {} 2023-09-29T22:00:54.309 DEBUG:boto:Token: None 2023-09-29T22:00:54.309 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-13/ 2023-09-29T22:00:54.310 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1WUGqqigsiEWrwbhjCoxVHExkxo= 2023-09-29T22:00:54.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1WUGqqigsiEWrwbhjCoxVHExkxo=', 'Content-Length': '0'} 2023-09-29T22:00:54.332 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002dc2036681d3215c-0065174916-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.332 DEBUG:rgw_multi.tests:created bucket=hcnphz-13 2023-09-29T22:00:54.332 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:00:54.332 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:00:54.333 DEBUG:boto:path=/hcnphz-13/?versioning 2023-09-29T22:00:54.333 DEBUG:boto:auth_path=/hcnphz-13/?versioning 2023-09-29T22:00:54.333 DEBUG:boto:Method: PUT 2023-09-29T22:00:54.333 DEBUG:boto:Path: /hcnphz-13/?versioning 2023-09-29T22:00:54.333 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:00:54.333 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.333 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.333 DEBUG:boto:Params: {} 2023-09-29T22:00:54.334 DEBUG:boto:Token: None 2023-09-29T22:00:54.334 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-13/?versioning 2023-09-29T22:00:54.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RIRBi/4598Wzs8/XNs6S1u5WHoQ= 2023-09-29T22:00:54.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RIRBi/4598Wzs8/XNs6S1u5WHoQ=', 'Content-Length': '228'} 2023-09-29T22:00:54.342 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000814ceb103c8772cb-0065174916-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:00:54.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:00:54.342 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:00:54.391 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:54.391 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:54.442 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:00:54.467 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:00:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024854.342638_73.1", 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:00:54.342638Z" 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:00:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:00:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:00:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:00:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:00:54.469 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:00:54.470 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696024854.342638_73.1', 2: '', 3: ''} 2023-09-29T22:00:54.470 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:00:54.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:00:54.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:00:54.471 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:00:54.514 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:54.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:54.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:54.561+0000 7f0edf2b8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:00:54.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:54.561+0000 7f0edf2b8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:00:54.589 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:00:54.589 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:00:54.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:00:54.591 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:00:54.592 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024854.342638_73.1", 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:00:54.342638Z", 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:00:54.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:00:54.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:00:54.595 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:00:54.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:00:54Z" 2023-09-29T22:00:54.597 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:00:54.598 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696024854.342638_73.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:00:54.342638Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:00:54Z" } 2023-09-29T22:00:54.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696024854.342638_73.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:00:54.342638Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:00:54.599 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696024854.342638_73.1', 2: '', 3: ''} 2023-09-29T22:00:54.599 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696024854.342638_73.1', 2: '', 3: ''} 2023-09-29T22:00:54.599 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:00:54.600 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:00:54.600 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:00:54.600 DEBUG:boto:Method: HEAD 2023-09-29T22:00:54.600 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:00:54.600 DEBUG:boto:Data: 2023-09-29T22:00:54.600 DEBUG:boto:Headers: {} 2023-09-29T22:00:54.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.601 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.601 DEBUG:boto:Params: {} 2023-09-29T22:00:54.601 DEBUG:boto:Token: None 2023-09-29T22:00:54.601 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-13/ 2023-09-29T22:00:54.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AdC+RVcERWCtnxk2s4rofDaGw9g= 2023-09-29T22:00:54.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AdC+RVcERWCtnxk2s4rofDaGw9g=', 'Content-Length': '0'} 2023-09-29T22:00:54.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e93f302ec201f10-0065174916-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.605 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:00:54.606 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:00:54.606 DEBUG:boto:Method: PUT 2023-09-29T22:00:54.606 DEBUG:boto:Path: /hcnphz-13/dummy 2023-09-29T22:00:54.606 DEBUG:boto:Data: 2023-09-29T22:00:54.606 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:00:54.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:00:54.606 DEBUG:boto:Port: 8000 2023-09-29T22:00:54.606 DEBUG:boto:Params: {} 2023-09-29T22:00:54.607 DEBUG:boto:Token: None 2023-09-29T22:00:54.607 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:00:54 GMT /hcnphz-13/dummy 2023-09-29T22:00:54.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XkS6E557Y5tOgxLFDT2Nesavkq8= 2023-09-29T22:00:54.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:00:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XkS6E557Y5tOgxLFDT2Nesavkq8='} 2023-09-29T22:00:54.629 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-amz-request-id', 'tx00000561189425a50139a-0065174916-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:00:54.629 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-13 2023-09-29T22:00:54.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:54.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:00:54.630 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:00:54.681 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:00:54.681 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:00:54.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:54.741+0000 7ff5c173df40 1 waiting to reach incremental sync.. 2023-09-29T22:00:59.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:00:59.741+0000 7ff5c173df40 1 waiting to reach incremental sync.. 2023-09-29T22:01:04.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:04.741+0000 7ff5c173df40 1 waiting to reach incremental sync.. 2023-09-29T22:01:09.747 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:09.745+0000 7ff5c173df40 1 waiting to reach incremental sync.. 2023-09-29T22:01:14.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:14.749+0000 7ff5c173df40 1 bucket sync caught up with source: 2023-09-29T22:01:14.751 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-09-29T22:01:14.751 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.5.11, , , , , ] 2023-09-29T22:01:14.751 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:14.749+0000 7ff5c173df40 0 bucket checkpoint complete 2023-09-29T22:01:14.778 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-13 2023-09-29T22:01:14.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:01:14.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:01:14.779 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:01:14.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:01:14.827 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:01:14.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:14.893+0000 7fc9e854bf40 1 waiting to reach incremental sync.. 2023-09-29T22:01:19.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:19.893+0000 7fc9e854bf40 1 waiting to reach incremental sync.. 2023-09-29T22:01:24.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:24.893+0000 7fc9e854bf40 1 waiting to reach incremental sync.. 2023-09-29T22:01:29.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:29.893+0000 7fc9e854bf40 1 waiting to reach incremental sync.. 2023-09-29T22:01:34.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:34.896+0000 7fc9e854bf40 1 bucket sync caught up with source: 2023-09-29T22:01:34.899 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.12.11, , , , , ] 2023-09-29T22:01:34.899 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.12.11, , , , , ] 2023-09-29T22:01:34.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:34.896+0000 7fc9e854bf40 0 bucket checkpoint complete 2023-09-29T22:01:34.926 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-13 zones={a2, a1} 2023-09-29T22:01:34.926 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:34.926 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:34.927 DEBUG:boto:Method: HEAD 2023-09-29T22:01:34.927 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:34.927 DEBUG:boto:Data: 2023-09-29T22:01:34.927 DEBUG:boto:Headers: {} 2023-09-29T22:01:34.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:34.927 DEBUG:boto:Port: 8001 2023-09-29T22:01:34.927 DEBUG:boto:Params: {} 2023-09-29T22:01:34.928 DEBUG:boto:Token: None 2023-09-29T22:01:34.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:34 GMT /hcnphz-13/ 2023-09-29T22:01:34.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:llkWYdJWhRBfG5vnvTUQWA+0NB8= 2023-09-29T22:01:34.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:llkWYdJWhRBfG5vnvTUQWA+0NB8=', 'Content-Length': '0'} 2023-09-29T22:01:34.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8061f1713b12fb8-006517493e-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:34.933 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:34.933 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:34.933 DEBUG:boto:Method: HEAD 2023-09-29T22:01:34.933 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:34.933 DEBUG:boto:Data: 2023-09-29T22:01:34.934 DEBUG:boto:Headers: {} 2023-09-29T22:01:34.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:34.934 DEBUG:boto:Port: 8000 2023-09-29T22:01:34.934 DEBUG:boto:Params: {} 2023-09-29T22:01:34.934 DEBUG:boto:Token: None 2023-09-29T22:01:34.935 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:34 GMT /hcnphz-13/ 2023-09-29T22:01:34.935 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:llkWYdJWhRBfG5vnvTUQWA+0NB8= 2023-09-29T22:01:34.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:llkWYdJWhRBfG5vnvTUQWA+0NB8=', 'Content-Length': '0'} 2023-09-29T22:01:34.938 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000441e96cf5804d14d-006517493e-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:34.939 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:01:34.939 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:34.939 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:34.939 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.939 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.940 DEBUG:boto:Method: GET 2023-09-29T22:01:34.940 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.940 DEBUG:boto:Data: 2023-09-29T22:01:34.940 DEBUG:boto:Headers: {} 2023-09-29T22:01:34.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:34.940 DEBUG:boto:Port: 8001 2023-09-29T22:01:34.940 DEBUG:boto:Params: {} 2023-09-29T22:01:34.941 DEBUG:boto:Token: None 2023-09-29T22:01:34.941 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:34 GMT /hcnphz-13/?versions 2023-09-29T22:01:34.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QZt8cIo+3j0Hh6ZXlLBTUJajIIE= 2023-09-29T22:01:34.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QZt8cIo+3j0Hh6ZXlLBTUJajIIE=', 'Content-Length': '0'} 2023-09-29T22:01:34.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008dbd4e7dd7e5875a-006517493e-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:34.986 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:34.987 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:01:34.987 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:01:34.988 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:34.988 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:34.988 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.988 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.988 DEBUG:boto:Method: GET 2023-09-29T22:01:34.988 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.988 DEBUG:boto:Data: 2023-09-29T22:01:34.988 DEBUG:boto:Headers: {} 2023-09-29T22:01:34.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:34.989 DEBUG:boto:Port: 8000 2023-09-29T22:01:34.989 DEBUG:boto:Params: {} 2023-09-29T22:01:34.989 DEBUG:boto:Token: None 2023-09-29T22:01:34.989 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:34 GMT /hcnphz-13/?versions 2023-09-29T22:01:34.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QZt8cIo+3j0Hh6ZXlLBTUJajIIE= 2023-09-29T22:01:34.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QZt8cIo+3j0Hh6ZXlLBTUJajIIE=', 'Content-Length': '0'} 2023-09-29T22:01:34.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043b4be588fa79b20-006517493e-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:34.996 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:34.996 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:01:34.996 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:34.996 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:34.996 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.997 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.997 DEBUG:boto:Method: GET 2023-09-29T22:01:34.997 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:34.997 DEBUG:boto:Data: 2023-09-29T22:01:34.997 DEBUG:boto:Headers: {} 2023-09-29T22:01:34.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:34.997 DEBUG:boto:Port: 8001 2023-09-29T22:01:34.997 DEBUG:boto:Params: {} 2023-09-29T22:01:34.998 DEBUG:boto:Token: None 2023-09-29T22:01:34.998 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:34 GMT /hcnphz-13/?versions 2023-09-29T22:01:34.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QZt8cIo+3j0Hh6ZXlLBTUJajIIE= 2023-09-29T22:01:34.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QZt8cIo+3j0Hh6ZXlLBTUJajIIE=', 'Content-Length': '0'} 2023-09-29T22:01:35.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac445ce7474fcda9-006517493e-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.002 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:35.002 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.002 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.002 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:35.002 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:35.002 DEBUG:boto:Method: GET 2023-09-29T22:01:35.003 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:01:35.003 DEBUG:boto:Data: 2023-09-29T22:01:35.003 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.003 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.003 DEBUG:boto:Params: {} 2023-09-29T22:01:35.003 DEBUG:boto:Token: None 2023-09-29T22:01:35.003 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/?versions 2023-09-29T22:01:35.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ejX3dcFHB4lID6VekF2PSZ37fbI= 2023-09-29T22:01:35.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ejX3dcFHB4lID6VekF2PSZ37fbI=', 'Content-Length': '0'} 2023-09-29T22:01:35.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dea971c5124fcba-006517493f-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.007 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:35.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:01:35.007 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.007 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.008 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.008 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.008 DEBUG:boto:Method: GET 2023-09-29T22:01:35.008 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.008 DEBUG:boto:Data: 2023-09-29T22:01:35.008 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.008 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.008 DEBUG:boto:Params: {} 2023-09-29T22:01:35.008 DEBUG:boto:Token: None 2023-09-29T22:01:35.009 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w= 2023-09-29T22:01:35.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w=', 'Content-Length': '0'} 2023-09-29T22:01:35.011 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d21107186f90fe8b-006517493f-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.011 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.011 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.012 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.012 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.012 DEBUG:boto:Method: GET 2023-09-29T22:01:35.012 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.012 DEBUG:boto:Data: 2023-09-29T22:01:35.012 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.012 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.012 DEBUG:boto:Params: {} 2023-09-29T22:01:35.013 DEBUG:boto:Token: None 2023-09-29T22:01:35.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w= 2023-09-29T22:01:35.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w=', 'Content-Length': '0'} 2023-09-29T22:01:35.015 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000851a1749d377abd0-006517493f-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.016 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.016 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.016 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.016 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.016 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.016 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.016 DEBUG:boto:Data: 2023-09-29T22:01:35.016 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.017 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.017 DEBUG:boto:Params: {} 2023-09-29T22:01:35.017 DEBUG:boto:Token: None 2023-09-29T22:01:35.017 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wd2PWyqph251HB4Io4umJZIGrrs= 2023-09-29T22:01:35.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wd2PWyqph251HB4Io4umJZIGrrs=', 'Content-Length': '0'} 2023-09-29T22:01:35.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007dcc67fbf64dcc1f-006517493f-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.019 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.020 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.020 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.020 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.020 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.020 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.020 DEBUG:boto:Data: 2023-09-29T22:01:35.020 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.020 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.021 DEBUG:boto:Params: {} 2023-09-29T22:01:35.021 DEBUG:boto:Token: None 2023-09-29T22:01:35.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wd2PWyqph251HB4Io4umJZIGrrs= 2023-09-29T22:01:35.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wd2PWyqph251HB4Io4umJZIGrrs=', 'Content-Length': '0'} 2023-09-29T22:01:35.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005edd043e05109d78-006517493f-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:01:35.024 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.024 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.024 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.024 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.024 DEBUG:boto:Method: GET 2023-09-29T22:01:35.024 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.024 DEBUG:boto:Data: 2023-09-29T22:01:35.024 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.024 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.025 DEBUG:boto:Params: {} 2023-09-29T22:01:35.025 DEBUG:boto:Token: None 2023-09-29T22:01:35.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w= 2023-09-29T22:01:35.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w=', 'Content-Length': '0'} 2023-09-29T22:01:35.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b96166386d79412c-006517493f-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.028 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.028 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.028 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.028 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.028 DEBUG:boto:Method: GET 2023-09-29T22:01:35.028 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.028 DEBUG:boto:Data: 2023-09-29T22:01:35.028 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.029 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.029 DEBUG:boto:Params: {} 2023-09-29T22:01:35.029 DEBUG:boto:Token: None 2023-09-29T22:01:35.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w= 2023-09-29T22:01:35.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w=', 'Content-Length': '0'} 2023-09-29T22:01:35.031 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ec39a8f24a227064-006517493f-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.032 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.032 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.032 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.032 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.032 DEBUG:boto:Method: GET 2023-09-29T22:01:35.032 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.032 DEBUG:boto:Data: 2023-09-29T22:01:35.032 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.033 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.033 DEBUG:boto:Params: {} 2023-09-29T22:01:35.033 DEBUG:boto:Token: None 2023-09-29T22:01:35.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.033 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4= 2023-09-29T22:01:35.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4=', 'Content-Length': '0'} 2023-09-29T22:01:35.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c03147d6c61ec39-006517493f-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.078 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:35.079 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.079 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.079 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.079 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.079 DEBUG:boto:Method: GET 2023-09-29T22:01:35.079 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.079 DEBUG:boto:Data: 2023-09-29T22:01:35.080 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.080 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.080 DEBUG:boto:Params: {} 2023-09-29T22:01:35.080 DEBUG:boto:Token: None 2023-09-29T22:01:35.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4= 2023-09-29T22:01:35.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4=', 'Content-Length': '0'} 2023-09-29T22:01:35.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064661ea8c736c56e-006517493f-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.086 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:35.086 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.086 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.086 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.086 DEBUG:boto:Path: /hcnphz-13/dummy 2023-09-29T22:01:35.086 DEBUG:boto:Data: 2023-09-29T22:01:35.087 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.087 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.087 DEBUG:boto:Params: {} 2023-09-29T22:01:35.087 DEBUG:boto:Token: None 2023-09-29T22:01:35.087 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy 2023-09-29T22:01:35.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbTn78HP342RCfD3C5phXUZZYzg= 2023-09-29T22:01:35.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbTn78HP342RCfD3C5phXUZZYzg=', 'Content-Length': '0'} 2023-09-29T22:01:35.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002cd3499593bb76fd-006517493f-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.091 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.091 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.091 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.091 DEBUG:boto:Path: /hcnphz-13/dummy 2023-09-29T22:01:35.091 DEBUG:boto:Data: 2023-09-29T22:01:35.091 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.092 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.092 DEBUG:boto:Params: {} 2023-09-29T22:01:35.092 DEBUG:boto:Token: None 2023-09-29T22:01:35.092 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy 2023-09-29T22:01:35.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbTn78HP342RCfD3C5phXUZZYzg= 2023-09-29T22:01:35.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbTn78HP342RCfD3C5phXUZZYzg=', 'Content-Length': '0'} 2023-09-29T22:01:35.095 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c0dbca8438437a3-006517493f-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:01:35.096 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.096 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.096 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.096 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.096 DEBUG:boto:Method: GET 2023-09-29T22:01:35.096 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.096 DEBUG:boto:Data: 2023-09-29T22:01:35.097 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.097 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.097 DEBUG:boto:Params: {} 2023-09-29T22:01:35.097 DEBUG:boto:Token: None 2023-09-29T22:01:35.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w= 2023-09-29T22:01:35.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w=', 'Content-Length': '0'} 2023-09-29T22:01:35.100 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000054553efeffe20fcf-006517493f-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.100 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:01:35.100 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:01:35.100 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.100 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.101 DEBUG:boto:Method: GET 2023-09-29T22:01:35.101 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.101 DEBUG:boto:Data: 2023-09-29T22:01:35.101 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.101 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.101 DEBUG:boto:Params: {} 2023-09-29T22:01:35.101 DEBUG:boto:Token: None 2023-09-29T22:01:35.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:01:35.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w= 2023-09-29T22:01:35.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66PAQpUCoS71W9KXfEYRoBJxo0w=', 'Content-Length': '0'} 2023-09-29T22:01:35.104 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001107327b9aea1d50-006517493f-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.105 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.105 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.105 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.105 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.105 DEBUG:boto:Method: GET 2023-09-29T22:01:35.105 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.105 DEBUG:boto:Data: 2023-09-29T22:01:35.105 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.106 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.106 DEBUG:boto:Params: {} 2023-09-29T22:01:35.106 DEBUG:boto:Token: None 2023-09-29T22:01:35.106 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.106 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4= 2023-09-29T22:01:35.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4=', 'Content-Length': '0'} 2023-09-29T22:01:35.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077735c5b8f9d47b7-006517493f-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.158 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:35.159 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.159 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.159 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.159 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.159 DEBUG:boto:Method: GET 2023-09-29T22:01:35.160 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:01:35.160 DEBUG:boto:Data: 2023-09-29T22:01:35.160 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.160 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.160 DEBUG:boto:Params: {} 2023-09-29T22:01:35.160 DEBUG:boto:Token: None 2023-09-29T22:01:35.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4= 2023-09-29T22:01:35.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GQqz7M3TIJLyobw4XuWXZVFDfW4=', 'Content-Length': '0'} 2023-09-29T22:01:35.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8d77dcf73aefb64-006517493f-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.166 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:01:35.166 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-13 zones={a2, a1} 2023-09-29T22:01:35.166 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.166 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.167 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.167 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.167 DEBUG:boto:Data: 2023-09-29T22:01:35.167 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.167 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.167 DEBUG:boto:Params: {} 2023-09-29T22:01:35.167 DEBUG:boto:Token: None 2023-09-29T22:01:35.167 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.172 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e98d792ee7ebd649-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.172 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.172 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.173 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.173 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.173 DEBUG:boto:Data: 2023-09-29T22:01:35.173 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.173 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.173 DEBUG:boto:Params: {} 2023-09-29T22:01:35.173 DEBUG:boto:Token: None 2023-09-29T22:01:35.173 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.219 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-amz-request-id', 'tx000007e60217938eca37e-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.219 DEBUG:rgw_multi.tests:zone=a1 version=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:01:35.219 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.219 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.219 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.219 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.219 DEBUG:boto:Data: 2023-09-29T22:01:35.220 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.220 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.220 DEBUG:boto:Params: {} 2023-09-29T22:01:35.220 DEBUG:boto:Token: None 2023-09-29T22:01:35.220 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.224 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000806486fa8e238326-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.225 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.225 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.225 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.225 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.225 DEBUG:boto:Data: 2023-09-29T22:01:35.225 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.226 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.226 DEBUG:boto:Params: {} 2023-09-29T22:01:35.226 DEBUG:boto:Token: None 2023-09-29T22:01:35.226 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.226 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-amz-request-id', 'tx000002a8e18378378157d-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.248 DEBUG:rgw_multi.tests:zone=a2 version=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:01:35.248 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.248 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.248 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.248 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.248 DEBUG:boto:Data: 2023-09-29T22:01:35.248 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.248 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.249 DEBUG:boto:Params: {} 2023-09-29T22:01:35.249 DEBUG:boto:Token: None 2023-09-29T22:01:35.249 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008536fe01a5e2851e-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.253 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.253 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.253 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.253 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.253 DEBUG:boto:Data: 2023-09-29T22:01:35.254 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.254 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.254 DEBUG:boto:Params: {} 2023-09-29T22:01:35.254 DEBUG:boto:Token: None 2023-09-29T22:01:35.254 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.254 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.271 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-amz-request-id', 'tx00000c477c154128f384b-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.272 DEBUG:rgw_multi.tests:zone=a1 version=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:01:35.272 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.272 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.272 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.272 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.272 DEBUG:boto:Data: 2023-09-29T22:01:35.272 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.273 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.273 DEBUG:boto:Params: {} 2023-09-29T22:01:35.273 DEBUG:boto:Token: None 2023-09-29T22:01:35.273 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '8'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038558c0a0f6455f8-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.277 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.277 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.277 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.277 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.277 DEBUG:boto:Data: 2023-09-29T22:01:35.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.277 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.277 DEBUG:boto:Params: {} 2023-09-29T22:01:35.278 DEBUG:boto:Token: None 2023-09-29T22:01:35.278 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.278 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-amz-request-id', 'tx00000b7a220620b724be0-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.323 DEBUG:rgw_multi.tests:zone=a2 version=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:01:35.323 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.323 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.323 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.323 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.323 DEBUG:boto:Data: 2023-09-29T22:01:35.324 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.324 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.324 DEBUG:boto:Params: {} 2023-09-29T22:01:35.324 DEBUG:boto:Token: None 2023-09-29T22:01:35.324 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.328 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f38212597300e648-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.329 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.329 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.329 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.329 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.329 DEBUG:boto:Data: 2023-09-29T22:01:35.329 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.329 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.330 DEBUG:boto:Params: {} 2023-09-29T22:01:35.330 DEBUG:boto:Token: None 2023-09-29T22:01:35.330 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.379 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-amz-request-id', 'tx0000008ddc7710d2ac468-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.379 DEBUG:rgw_multi.tests:zone=a1 version=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:01:35.379 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.379 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.379 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.379 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.379 DEBUG:boto:Data: 2023-09-29T22:01:35.380 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.380 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.380 DEBUG:boto:Params: {} 2023-09-29T22:01:35.380 DEBUG:boto:Token: None 2023-09-29T22:01:35.380 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.384 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '16'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5e4702fd9747a2c-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.385 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.385 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.385 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.385 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.385 DEBUG:boto:Data: 2023-09-29T22:01:35.385 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.386 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.386 DEBUG:boto:Params: {} 2023-09-29T22:01:35.386 DEBUG:boto:Token: None 2023-09-29T22:01:35.386 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.409 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-amz-request-id', 'tx000007285dc909a9170fb-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.409 DEBUG:rgw_multi.tests:zone=a2 version=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:01:35.409 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.409 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.409 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.410 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.410 DEBUG:boto:Data: 2023-09-29T22:01:35.410 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.410 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.410 DEBUG:boto:Params: {} 2023-09-29T22:01:35.410 DEBUG:boto:Token: None 2023-09-29T22:01:35.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.411 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.415 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000088d3c153e92677d-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.415 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.415 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.415 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.415 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.415 DEBUG:boto:Data: 2023-09-29T22:01:35.415 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.416 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.416 DEBUG:boto:Params: {} 2023-09-29T22:01:35.416 DEBUG:boto:Token: None 2023-09-29T22:01:35.416 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-amz-request-id', 'tx000001d5fb213f5ca158f-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.434 DEBUG:rgw_multi.tests:zone=a1 version=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:01:35.435 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.435 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.435 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.435 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.435 DEBUG:boto:Data: 2023-09-29T22:01:35.435 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.435 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.435 DEBUG:boto:Params: {} 2023-09-29T22:01:35.436 DEBUG:boto:Token: None 2023-09-29T22:01:35.436 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aed38765c7711600-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.439 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.439 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.439 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.439 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.440 DEBUG:boto:Data: 2023-09-29T22:01:35.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.440 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.440 DEBUG:boto:Params: {} 2023-09-29T22:01:35.440 DEBUG:boto:Token: None 2023-09-29T22:01:35.440 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.486 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-amz-request-id', 'tx00000a5b721619d0741da-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.487 DEBUG:rgw_multi.tests:zone=a2 version=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:01:35.487 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.487 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.487 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.487 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.487 DEBUG:boto:Data: 2023-09-29T22:01:35.487 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.488 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.488 DEBUG:boto:Params: {} 2023-09-29T22:01:35.488 DEBUG:boto:Token: None 2023-09-29T22:01:35.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.492 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f424d7e76d283c7-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.493 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.493 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.493 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.493 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.493 DEBUG:boto:Data: 2023-09-29T22:01:35.493 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.493 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.494 DEBUG:boto:Params: {} 2023-09-29T22:01:35.494 DEBUG:boto:Token: None 2023-09-29T22:01:35.494 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.494 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.539 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-amz-request-id', 'tx000002f36c4b52c7e2f0b-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.539 DEBUG:rgw_multi.tests:zone=a1 version=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:01:35.539 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.539 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.539 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.539 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.539 DEBUG:boto:Data: 2023-09-29T22:01:35.540 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.540 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.540 DEBUG:boto:Params: {} 2023-09-29T22:01:35.540 DEBUG:boto:Token: None 2023-09-29T22:01:35.540 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.540 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.544 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '32'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005de636f672ad6daa-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.545 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.545 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.545 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.545 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.545 DEBUG:boto:Data: 2023-09-29T22:01:35.545 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.546 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.546 DEBUG:boto:Params: {} 2023-09-29T22:01:35.546 DEBUG:boto:Token: None 2023-09-29T22:01:35.546 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.546 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.565 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-amz-request-id', 'tx000003c66a8387dc24b15-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.565 DEBUG:rgw_multi.tests:zone=a2 version=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:01:35.565 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.566 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.566 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.566 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.566 DEBUG:boto:Data: 2023-09-29T22:01:35.566 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.566 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.566 DEBUG:boto:Params: {} 2023-09-29T22:01:35.566 DEBUG:boto:Token: None 2023-09-29T22:01:35.567 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.570 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a044410bc1066167-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.570 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.570 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.571 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.571 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.571 DEBUG:boto:Data: 2023-09-29T22:01:35.571 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.571 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.571 DEBUG:boto:Params: {} 2023-09-29T22:01:35.571 DEBUG:boto:Token: None 2023-09-29T22:01:35.572 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-amz-request-id', 'tx00000ddd3a5be6d01d897-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.589 DEBUG:rgw_multi.tests:zone=a1 version=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:01:35.589 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.590 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.590 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.590 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.590 DEBUG:boto:Data: 2023-09-29T22:01:35.590 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.590 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.590 DEBUG:boto:Params: {} 2023-09-29T22:01:35.590 DEBUG:boto:Token: None 2023-09-29T22:01:35.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.594 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '40'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fcc1aecbac72c05e-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.594 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.595 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.595 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.595 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.595 DEBUG:boto:Data: 2023-09-29T22:01:35.595 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.595 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.595 DEBUG:boto:Params: {} 2023-09-29T22:01:35.595 DEBUG:boto:Token: None 2023-09-29T22:01:35.596 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-amz-request-id', 'tx0000038b38d58c53c242f-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.643 DEBUG:rgw_multi.tests:zone=a2 version=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:01:35.643 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.643 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.643 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.643 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.643 DEBUG:boto:Data: 2023-09-29T22:01:35.644 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.644 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.644 DEBUG:boto:Params: {} 2023-09-29T22:01:35.644 DEBUG:boto:Token: None 2023-09-29T22:01:35.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.648 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b1d280ac1eb91a4-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.649 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.649 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.649 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.649 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.649 DEBUG:boto:Data: 2023-09-29T22:01:35.649 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.649 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.649 DEBUG:boto:Params: {} 2023-09-29T22:01:35.650 DEBUG:boto:Token: None 2023-09-29T22:01:35.650 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.650 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.670 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-amz-request-id', 'tx0000016b0d1480a34afbc-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.670 DEBUG:rgw_multi.tests:zone=a1 version=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:01:35.670 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.670 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.670 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.670 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.670 DEBUG:boto:Data: 2023-09-29T22:01:35.670 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.671 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.671 DEBUG:boto:Params: {} 2023-09-29T22:01:35.671 DEBUG:boto:Token: None 2023-09-29T22:01:35.671 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.671 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.674 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000762a1f6cb6439df5-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.675 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.675 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.675 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.675 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.675 DEBUG:boto:Data: 2023-09-29T22:01:35.675 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.676 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.676 DEBUG:boto:Params: {} 2023-09-29T22:01:35.676 DEBUG:boto:Token: None 2023-09-29T22:01:35.676 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-amz-request-id', 'tx00000e0d3e2676a1ddf85-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.693 DEBUG:rgw_multi.tests:zone=a2 version=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:01:35.693 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.693 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.693 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.693 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.693 DEBUG:boto:Data: 2023-09-29T22:01:35.694 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.694 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.694 DEBUG:boto:Params: {} 2023-09-29T22:01:35.694 DEBUG:boto:Token: None 2023-09-29T22:01:35.694 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008fca269c02335c20-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.698 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.698 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.698 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.698 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.698 DEBUG:boto:Data: 2023-09-29T22:01:35.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.698 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.699 DEBUG:boto:Params: {} 2023-09-29T22:01:35.699 DEBUG:boto:Token: None 2023-09-29T22:01:35.699 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.743 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-amz-request-id', 'tx00000fee1a669a5663038-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.743 DEBUG:rgw_multi.tests:zone=a1 version=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:01:35.743 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.743 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.743 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.743 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.743 DEBUG:boto:Data: 2023-09-29T22:01:35.744 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.744 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.744 DEBUG:boto:Params: {} 2023-09-29T22:01:35.744 DEBUG:boto:Token: None 2023-09-29T22:01:35.744 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '56'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000038c854866d29ffeb-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.749 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.749 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.749 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.749 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.749 DEBUG:boto:Data: 2023-09-29T22:01:35.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.750 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.750 DEBUG:boto:Params: {} 2023-09-29T22:01:35.750 DEBUG:boto:Token: None 2023-09-29T22:01:35.750 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.750 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-amz-request-id', 'tx000006ab641779c30c8be-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.799 DEBUG:rgw_multi.tests:zone=a2 version=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:01:35.799 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.799 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.799 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.800 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.800 DEBUG:boto:Data: 2023-09-29T22:01:35.800 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.800 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.800 DEBUG:boto:Params: {} 2023-09-29T22:01:35.800 DEBUG:boto:Token: None 2023-09-29T22:01:35.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.805 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000741e112af37bb6c7-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.805 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.805 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.805 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.806 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.806 DEBUG:boto:Data: 2023-09-29T22:01:35.806 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.806 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.806 DEBUG:boto:Params: {} 2023-09-29T22:01:35.806 DEBUG:boto:Token: None 2023-09-29T22:01:35.806 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.826 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-amz-request-id', 'tx000001f6f82d6a630390b-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.826 DEBUG:rgw_multi.tests:zone=a1 version=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:01:35.826 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.826 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.827 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.827 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.827 DEBUG:boto:Data: 2023-09-29T22:01:35.827 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.827 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.827 DEBUG:boto:Params: {} 2023-09-29T22:01:35.827 DEBUG:boto:Token: None 2023-09-29T22:01:35.827 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.828 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.831 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '64'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b7f7635345593a87-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.831 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.831 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.831 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.831 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.831 DEBUG:boto:Data: 2023-09-29T22:01:35.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.832 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.832 DEBUG:boto:Params: {} 2023-09-29T22:01:35.832 DEBUG:boto:Token: None 2023-09-29T22:01:35.832 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.851 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-amz-request-id', 'tx00000cbf629d7ec950223-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.851 DEBUG:rgw_multi.tests:zone=a2 version=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:01:35.851 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.852 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.852 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.852 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.852 DEBUG:boto:Data: 2023-09-29T22:01:35.852 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.852 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.852 DEBUG:boto:Params: {} 2023-09-29T22:01:35.853 DEBUG:boto:Token: None 2023-09-29T22:01:35.853 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.853 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.857 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007b6b39980b7934c6-006517493f-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.857 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.857 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.857 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.858 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.858 DEBUG:boto:Data: 2023-09-29T22:01:35.858 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:01:35.858 DEBUG:boto:Port: 8000 2023-09-29T22:01:35.858 DEBUG:boto:Params: {} 2023-09-29T22:01:35.858 DEBUG:boto:Token: None 2023-09-29T22:01:35.858 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.859 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-amz-request-id', 'tx00000966169b451425178-006517493f-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.903 DEBUG:rgw_multi.tests:zone=a1 version=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:01:35.903 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:01:35.903 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:01:35.903 DEBUG:boto:Method: HEAD 2023-09-29T22:01:35.903 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:01:35.903 DEBUG:boto:Data: 2023-09-29T22:01:35.904 DEBUG:boto:Headers: {} 2023-09-29T22:01:35.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.905 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.905 DEBUG:boto:Params: {} 2023-09-29T22:01:35.905 DEBUG:boto:Token: None 2023-09-29T22:01:35.905 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/ 2023-09-29T22:01:35.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y= 2023-09-29T22:01:35.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQ8M3Hhxkx88mALkhvpZ2ydx06Y=', 'Content-Length': '0'} 2023-09-29T22:01:35.909 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006b24d56d51de2ea6-006517493f-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.910 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:01:35.910 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:01:35.910 DEBUG:boto:Method: PUT 2023-09-29T22:01:35.910 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:01:35.910 DEBUG:boto:Data: 2023-09-29T22:01:35.910 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue'} 2023-09-29T22:01:35.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:01:35.911 DEBUG:boto:Port: 8001 2023-09-29T22:01:35.911 DEBUG:boto:Params: {} 2023-09-29T22:01:35.911 DEBUG:boto:Token: None 2023-09-29T22:01:35.911 DEBUG:boto:StringToSign: PUT lmY06/L8E1cH1nU2kr9LHg== application/octet-stream Fri, 29 Sep 2023 22:01:35 GMT /hcnphz-13/obj 2023-09-29T22:01:35.911 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4= 2023-09-29T22:01:35.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'lmY06/L8E1cH1nU2kr9LHg==', 'Content-Length': '8', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:01:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pVkykO5bmBV8nY/HKR2hEmVZ8g4='} 2023-09-29T22:01:35.955 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-amz-request-id', 'tx00000d735392df1a87e88-006517493f-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:01:35.955 DEBUG:rgw_multi.tests:zone=a2 version=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:01:35.955 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-13 2023-09-29T22:01:35.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:01:35.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:01:35.956 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-13 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:01:36.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:01:36.011 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:01:36.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:36.068+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:01:36.068 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-09-29T22:01:36.068 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:01:41.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:41.068+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:01:41.070 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-09-29T22:01:41.070 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:01:46.073 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:46.072+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:01:46.073 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.5.11, , , , , ] 2023-09-29T22:01:46.073 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:01:51.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:51.072+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:01:51.075 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000001.35.6, , ] 2023-09-29T22:01:51.075 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:01:56.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:01:56.076+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:01:56.076 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000001.35.6, , ] 2023-09-29T22:01:56.076 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:01.079 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:01.076+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:01.080 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000001.35.6, , ] 2023-09-29T22:02:01.080 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:06.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:06.080+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:06.081 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000001.35.6, , ] 2023-09-29T22:02:06.081 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:11.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:11.083+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:11.084 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:11.084 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:16.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:16.083+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:16.086 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:16.086 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:21.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:21.087+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:21.089 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:21.089 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:26.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:26.087+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:26.091 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:26.091 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:31.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:31.091+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:31.093 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:31.093 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:36.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:36.091+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:36.095 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:36.095 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:41.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:41.095+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:41.097 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:41.097 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:46.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:46.095+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:46.099 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:46.100 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:51.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:51.098+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:51.102 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:51.102 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:02:56.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:02:56.102+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:02:56.105 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:02:56.105 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:01.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:01.102+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:01.107 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:03:01.107 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:06.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:06.106+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:06.109 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:03:06.109 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:11.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:11.106+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:11.111 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:03:11.112 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:16.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:16.110+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:16.114 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:03:16.114 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:21.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:21.114+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:21.116 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:03:21.116 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:26.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:26.113+0000 7f7808d08f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:26.119 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000015.63.6, , ] 2023-09-29T22:03:26.119 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:31.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:31.117+0000 7f7808d08f40 1 bucket sync caught up with source: 2023-09-29T22:03:31.121 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000003.7.10, , , 00000000031.103.9, , ] 2023-09-29T22:03:31.121 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000003.7.10, , , 00000000020.72.12, , ] 2023-09-29T22:03:31.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:31.117+0000 7f7808d08f40 0 bucket checkpoint complete 2023-09-29T22:03:31.147 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-13 2023-09-29T22:03:31.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:31.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-13', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:31.148 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-13 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:03:31.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:31.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:31.265 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:31.261+0000 7f3cfcc13f40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:31.265 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000027.87.6, , ] 2023-09-29T22:03:31.265 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.97.12, , ] 2023-09-29T22:03:36.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:36.265+0000 7f3cfcc13f40 1 bucket sync caught up with source: 2023-09-29T22:03:36.268 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000002.12.11, , , 00000000031.97.12, , ] 2023-09-29T22:03:36.268 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000002.12.11, , , 00000000031.97.12, , ] 2023-09-29T22:03:36.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:36.265+0000 7f3cfcc13f40 0 bucket checkpoint complete 2023-09-29T22:03:36.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-13 zones={a2, a1} 2023-09-29T22:03:36.295 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.295 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.295 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.295 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:03:36.295 DEBUG:boto:Data: 2023-09-29T22:03:36.295 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.296 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.296 DEBUG:boto:Params: {} 2023-09-29T22:03:36.296 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:03:36.296 DEBUG:boto:Token: None 2023-09-29T22:03:36.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nvVl9op/GeeTWmugErbCceVYENY= 2023-09-29T22:03:36.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nvVl9op/GeeTWmugErbCceVYENY=', 'Content-Length': '0'} 2023-09-29T22:03:36.302 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ab58c285f350351-00651749b8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.303 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.303 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.303 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.303 DEBUG:boto:Path: /hcnphz-13/ 2023-09-29T22:03:36.303 DEBUG:boto:Data: 2023-09-29T22:03:36.303 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.303 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.304 DEBUG:boto:Params: {} 2023-09-29T22:03:36.304 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:03:36.304 DEBUG:boto:Token: None 2023-09-29T22:03:36.304 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.304 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nvVl9op/GeeTWmugErbCceVYENY= 2023-09-29T22:03:36.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nvVl9op/GeeTWmugErbCceVYENY=', 'Content-Length': '0'} 2023-09-29T22:03:36.311 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '160'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e9a2400b931682-00651749b8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.311 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:03:36.311 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.312 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.312 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.312 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.312 DEBUG:boto:Method: GET 2023-09-29T22:03:36.312 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.312 DEBUG:boto:Data: 2023-09-29T22:03:36.312 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.312 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.312 DEBUG:boto:Params: {} 2023-09-29T22:03:36.313 DEBUG:boto:Token: None 2023-09-29T22:03:36.313 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/?versions 2023-09-29T22:03:36.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI= 2023-09-29T22:03:36.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI=', 'Content-Length': '0'} 2023-09-29T22:03:36.319 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5fbaf5e25813726-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.319 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDysj0dBRPeSLOTMc-kPu0YmgNYl1uCvtrue2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNgfalse2023-09-29T22:01:35.915Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7false2023-09-29T22:01:35.811Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYffalse2023-09-29T22:01:35.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6Ifalse2023-09-29T22:01:35.703Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrpSMuc1GJlB4zfJhxcLig4aLeB9wbxbfalse2023-09-29T22:01:35.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4Tfalse2023-09-29T22:01:35.680Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPSroL9.BBptEm21WZRCI8FrA0kZDA96false2023-09-29T22:01:35.654Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7e8K1DAqREpBLGcWpWU7padLejgV62qfalse2023-09-29T22:01:35.600Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxYKB5fMv0wyZy-frsPf-.TEL0ca4Pgcfalse2023-09-29T22:01:35.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7YyFpAWutMJGjWIJTn9o65BehGeETgWfalse2023-09-29T22:01:35.498Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-false2023-09-29T22:01:35.550Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19w3gH6L6XVWwe2N7qkyQteADnJRlTSfalse2023-09-29T22:01:35.445Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU8xudiwYAFkZ2HZnqEX0HOcHshGnyncfalse2023-09-29T22:01:35.420Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAPoMvqvvam5IdSmRUTel8cjnttV00WZfalse2023-09-29T22:01:35.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFlz4F8gyQg2Sv8j31HOgKaQJV5oYrkpfalse2023-09-29T22:01:35.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU0kFiWYrndp08LH2Ryrtmjw75rahc7sfalse2023-09-29T22:01:35.258Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-false2023-09-29T22:01:35.282Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiri9FYdBGldQRD.PWnNbydapWa3BHP8false2023-09-29T22:01:35.231Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnt.n3hC6tYeenMgygbhwVyjvoejWUSOfalse2023-09-29T22:01:35.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.320 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.322 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:03:36.323 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.323 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.323 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.323 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.323 DEBUG:boto:Method: GET 2023-09-29T22:03:36.323 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.323 DEBUG:boto:Data: 2023-09-29T22:03:36.323 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.324 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.324 DEBUG:boto:Params: {} 2023-09-29T22:03:36.324 DEBUG:boto:Token: None 2023-09-29T22:03:36.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/?versions 2023-09-29T22:03:36.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI= 2023-09-29T22:03:36.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI=', 'Content-Length': '0'} 2023-09-29T22:03:36.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503019074ab76b84-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.329 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDysj0dBRPeSLOTMc-kPu0YmgNYl1uCvtrue2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNgfalse2023-09-29T22:01:35.915Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7false2023-09-29T22:01:35.811Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYffalse2023-09-29T22:01:35.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6Ifalse2023-09-29T22:01:35.703Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrpSMuc1GJlB4zfJhxcLig4aLeB9wbxbfalse2023-09-29T22:01:35.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4Tfalse2023-09-29T22:01:35.680Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPSroL9.BBptEm21WZRCI8FrA0kZDA96false2023-09-29T22:01:35.654Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7e8K1DAqREpBLGcWpWU7padLejgV62qfalse2023-09-29T22:01:35.600Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxYKB5fMv0wyZy-frsPf-.TEL0ca4Pgcfalse2023-09-29T22:01:35.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7YyFpAWutMJGjWIJTn9o65BehGeETgWfalse2023-09-29T22:01:35.498Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-false2023-09-29T22:01:35.550Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19w3gH6L6XVWwe2N7qkyQteADnJRlTSfalse2023-09-29T22:01:35.445Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU8xudiwYAFkZ2HZnqEX0HOcHshGnyncfalse2023-09-29T22:01:35.420Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAPoMvqvvam5IdSmRUTel8cjnttV00WZfalse2023-09-29T22:01:35.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFlz4F8gyQg2Sv8j31HOgKaQJV5oYrkpfalse2023-09-29T22:01:35.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU0kFiWYrndp08LH2Ryrtmjw75rahc7sfalse2023-09-29T22:01:35.258Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-false2023-09-29T22:01:35.282Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiri9FYdBGldQRD.PWnNbydapWa3BHP8false2023-09-29T22:01:35.231Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnt.n3hC6tYeenMgygbhwVyjvoejWUSOfalse2023-09-29T22:01:35.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.330 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:36.333 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.333 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.333 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.333 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.333 DEBUG:boto:Method: GET 2023-09-29T22:03:36.333 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.333 DEBUG:boto:Data: 2023-09-29T22:03:36.333 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.333 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.334 DEBUG:boto:Params: {} 2023-09-29T22:03:36.334 DEBUG:boto:Token: None 2023-09-29T22:03:36.334 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/?versions 2023-09-29T22:03:36.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI= 2023-09-29T22:03:36.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI=', 'Content-Length': '0'} 2023-09-29T22:03:36.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000460005c83ef85235-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.340 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDysj0dBRPeSLOTMc-kPu0YmgNYl1uCvtrue2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNgfalse2023-09-29T22:01:35.915Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7false2023-09-29T22:01:35.811Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYffalse2023-09-29T22:01:35.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6Ifalse2023-09-29T22:01:35.703Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrpSMuc1GJlB4zfJhxcLig4aLeB9wbxbfalse2023-09-29T22:01:35.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4Tfalse2023-09-29T22:01:35.680Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPSroL9.BBptEm21WZRCI8FrA0kZDA96false2023-09-29T22:01:35.654Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7e8K1DAqREpBLGcWpWU7padLejgV62qfalse2023-09-29T22:01:35.600Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxYKB5fMv0wyZy-frsPf-.TEL0ca4Pgcfalse2023-09-29T22:01:35.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7YyFpAWutMJGjWIJTn9o65BehGeETgWfalse2023-09-29T22:01:35.498Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-false2023-09-29T22:01:35.550Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19w3gH6L6XVWwe2N7qkyQteADnJRlTSfalse2023-09-29T22:01:35.445Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU8xudiwYAFkZ2HZnqEX0HOcHshGnyncfalse2023-09-29T22:01:35.420Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAPoMvqvvam5IdSmRUTel8cjnttV00WZfalse2023-09-29T22:01:35.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFlz4F8gyQg2Sv8j31HOgKaQJV5oYrkpfalse2023-09-29T22:01:35.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU0kFiWYrndp08LH2Ryrtmjw75rahc7sfalse2023-09-29T22:01:35.258Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-false2023-09-29T22:01:35.282Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiri9FYdBGldQRD.PWnNbydapWa3BHP8false2023-09-29T22:01:35.231Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnt.n3hC6tYeenMgygbhwVyjvoejWUSOfalse2023-09-29T22:01:35.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.341 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.341 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.341 DEBUG:boto:path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.341 DEBUG:boto:auth_path=/hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.341 DEBUG:boto:Method: GET 2023-09-29T22:03:36.342 DEBUG:boto:Path: /hcnphz-13/?versions&max-keys=999 2023-09-29T22:03:36.342 DEBUG:boto:Data: 2023-09-29T22:03:36.342 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.342 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.342 DEBUG:boto:Params: {} 2023-09-29T22:03:36.342 DEBUG:boto:Token: None 2023-09-29T22:03:36.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/?versions 2023-09-29T22:03:36.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI= 2023-09-29T22:03:36.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sKZ9K87jBp0LnfVczx+49tHipRI=', 'Content-Length': '0'} 2023-09-29T22:03:36.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178917e774dcb935-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.348 DEBUG:boto:b'hcnphz-13999falsedummyUqEpVKuFnoPJjnOtByVjVM6HyHrP6VPtrue2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjDysj0dBRPeSLOTMc-kPu0YmgNYl1uCvtrue2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjtpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNgfalse2023-09-29T22:01:35.915Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7false2023-09-29T22:01:35.811Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjzRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYffalse2023-09-29T22:01:35.836Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6Ifalse2023-09-29T22:01:35.703Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjrpSMuc1GJlB4zfJhxcLig4aLeB9wbxbfalse2023-09-29T22:01:35.754Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4Tfalse2023-09-29T22:01:35.680Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjPSroL9.BBptEm21WZRCI8FrA0kZDA96false2023-09-29T22:01:35.654Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7e8K1DAqREpBLGcWpWU7padLejgV62qfalse2023-09-29T22:01:35.600Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjxYKB5fMv0wyZy-frsPf-.TEL0ca4Pgcfalse2023-09-29T22:01:35.575Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7YyFpAWutMJGjWIJTn9o65BehGeETgWfalse2023-09-29T22:01:35.498Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-false2023-09-29T22:01:35.550Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19w3gH6L6XVWwe2N7qkyQteADnJRlTSfalse2023-09-29T22:01:35.445Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU8xudiwYAFkZ2HZnqEX0HOcHshGnyncfalse2023-09-29T22:01:35.420Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjAPoMvqvvam5IdSmRUTel8cjnttV00WZfalse2023-09-29T22:01:35.334Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjFlz4F8gyQg2Sv8j31HOgKaQJV5oYrkpfalse2023-09-29T22:01:35.390Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjU0kFiWYrndp08LH2Ryrtmjw75rahc7sfalse2023-09-29T22:01:35.258Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjqUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-false2023-09-29T22:01:35.282Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjiri9FYdBGldQRD.PWnNbydapWa3BHP8false2023-09-29T22:01:35.231Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnt.n3hC6tYeenMgygbhwVyjvoejWUSOfalse2023-09-29T22:01:35.178Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:03:36.349 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.350 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.350 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.350 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.350 DEBUG:boto:Method: GET 2023-09-29T22:03:36.350 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.350 DEBUG:boto:Data: 2023-09-29T22:03:36.350 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.350 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.350 DEBUG:boto:Params: {} 2023-09-29T22:03:36.351 DEBUG:boto:Token: None 2023-09-29T22:03:36.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A= 2023-09-29T22:03:36.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A=', 'Content-Length': '0'} 2023-09-29T22:03:36.353 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c3b46e71e6c6b08-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.354 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.354 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.354 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.354 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.354 DEBUG:boto:Method: GET 2023-09-29T22:03:36.354 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.354 DEBUG:boto:Data: 2023-09-29T22:03:36.354 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.355 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.355 DEBUG:boto:Params: {} 2023-09-29T22:03:36.355 DEBUG:boto:Token: None 2023-09-29T22:03:36.355 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A= 2023-09-29T22:03:36.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A=', 'Content-Length': '0'} 2023-09-29T22:03:36.357 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e69b119c82d9eb6a-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.358 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.358 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.358 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.358 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.358 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.358 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.358 DEBUG:boto:Data: 2023-09-29T22:03:36.358 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.359 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.359 DEBUG:boto:Params: {} 2023-09-29T22:03:36.359 DEBUG:boto:Token: None 2023-09-29T22:03:36.359 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.359 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ytvysBDyc3xCg3MAZbuCEvJCyJE= 2023-09-29T22:03:36.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ytvysBDyc3xCg3MAZbuCEvJCyJE=', 'Content-Length': '0'} 2023-09-29T22:03:36.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b0cf8aaccff75f8f-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.362 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.362 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.362 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.362 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.362 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.362 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.362 DEBUG:boto:Data: 2023-09-29T22:03:36.363 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.363 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.363 DEBUG:boto:Params: {} 2023-09-29T22:03:36.363 DEBUG:boto:Token: None 2023-09-29T22:03:36.363 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.363 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ytvysBDyc3xCg3MAZbuCEvJCyJE= 2023-09-29T22:03:36.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ytvysBDyc3xCg3MAZbuCEvJCyJE=', 'Content-Length': '0'} 2023-09-29T22:03:36.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c445d5657865f9a7-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:03:36.366 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.366 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.366 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.366 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.366 DEBUG:boto:Method: GET 2023-09-29T22:03:36.367 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.367 DEBUG:boto:Data: 2023-09-29T22:03:36.367 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.367 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.367 DEBUG:boto:Params: {} 2023-09-29T22:03:36.367 DEBUG:boto:Token: None 2023-09-29T22:03:36.367 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.367 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A= 2023-09-29T22:03:36.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A=', 'Content-Length': '0'} 2023-09-29T22:03:36.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009c278c5b6043d0aa-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.370 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.370 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.371 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.371 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.371 DEBUG:boto:Method: GET 2023-09-29T22:03:36.371 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.371 DEBUG:boto:Data: 2023-09-29T22:03:36.371 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.371 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.371 DEBUG:boto:Params: {} 2023-09-29T22:03:36.371 DEBUG:boto:Token: None 2023-09-29T22:03:36.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A= 2023-09-29T22:03:36.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A=', 'Content-Length': '0'} 2023-09-29T22:03:36.374 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005ff2089b165f7876-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.374 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.375 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.375 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.375 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.375 DEBUG:boto:Method: GET 2023-09-29T22:03:36.375 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.375 DEBUG:boto:Data: 2023-09-29T22:03:36.375 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.375 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.375 DEBUG:boto:Params: {} 2023-09-29T22:03:36.376 DEBUG:boto:Token: None 2023-09-29T22:03:36.376 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a3e21f04154f21f-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.422 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.423 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.423 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.423 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.423 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.423 DEBUG:boto:Method: GET 2023-09-29T22:03:36.423 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.424 DEBUG:boto:Data: 2023-09-29T22:03:36.424 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.424 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.424 DEBUG:boto:Params: {} 2023-09-29T22:03:36.424 DEBUG:boto:Token: None 2023-09-29T22:03:36.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbeaf9d22fc60e5b-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.429 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.430 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.430 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.430 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.430 DEBUG:boto:Path: /hcnphz-13/dummy 2023-09-29T22:03:36.430 DEBUG:boto:Data: 2023-09-29T22:03:36.431 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.431 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.431 DEBUG:boto:Params: {} 2023-09-29T22:03:36.431 DEBUG:boto:Token: None 2023-09-29T22:03:36.431 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy 2023-09-29T22:03:36.431 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v98g95Asq5L+wZrgyFDieaU+fG0= 2023-09-29T22:03:36.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v98g95Asq5L+wZrgyFDieaU+fG0=', 'Content-Length': '0'} 2023-09-29T22:03:36.435 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b73f3cbf54605702-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.435 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.435 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.436 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.436 DEBUG:boto:Path: /hcnphz-13/dummy 2023-09-29T22:03:36.436 DEBUG:boto:Data: 2023-09-29T22:03:36.436 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.436 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.436 DEBUG:boto:Params: {} 2023-09-29T22:03:36.436 DEBUG:boto:Token: None 2023-09-29T22:03:36.436 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy 2023-09-29T22:03:36.437 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v98g95Asq5L+wZrgyFDieaU+fG0= 2023-09-29T22:03:36.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v98g95Asq5L+wZrgyFDieaU+fG0=', 'Content-Length': '0'} 2023-09-29T22:03:36.440 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a2a10a1deabcb907-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T22:03:36.441 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.441 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.441 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.441 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.441 DEBUG:boto:Method: GET 2023-09-29T22:03:36.441 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.441 DEBUG:boto:Data: 2023-09-29T22:03:36.441 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.442 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.442 DEBUG:boto:Params: {} 2023-09-29T22:03:36.442 DEBUG:boto:Token: None 2023-09-29T22:03:36.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A= 2023-09-29T22:03:36.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A=', 'Content-Length': '0'} 2023-09-29T22:03:36.445 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008d96ef0c29f69a3e-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.445 DEBUG:boto:path=/hcnphz-13/dummy 2023-09-29T22:03:36.445 DEBUG:boto:auth_path=/hcnphz-13/dummy 2023-09-29T22:03:36.445 DEBUG:boto:path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.446 DEBUG:boto:auth_path=/hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.446 DEBUG:boto:Method: GET 2023-09-29T22:03:36.446 DEBUG:boto:Path: /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.446 DEBUG:boto:Data: 2023-09-29T22:03:36.446 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.446 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.446 DEBUG:boto:Params: {} 2023-09-29T22:03:36.446 DEBUG:boto:Token: None 2023-09-29T22:03:36.447 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/dummy?versionId=UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP 2023-09-29T22:03:36.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A= 2023-09-29T22:03:36.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7NtAs+MOBq6Ore4q5PMre2V6h4A=', 'Content-Length': '0'} 2023-09-29T22:03:36.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:00:54 GMT'), ('x-amz-version-id', 'UqEpVKuFnoPJjnOtByVjVM6HyHrP6VP'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000023442e6815754988-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.450 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.450 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.450 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.450 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.450 DEBUG:boto:Method: GET 2023-09-29T22:03:36.451 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.451 DEBUG:boto:Data: 2023-09-29T22:03:36.451 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.451 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.451 DEBUG:boto:Params: {} 2023-09-29T22:03:36.451 DEBUG:boto:Token: None 2023-09-29T22:03:36.451 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ac6b57e6c508f2e-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.456 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.457 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.457 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.457 DEBUG:boto:path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.457 DEBUG:boto:auth_path=/hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.457 DEBUG:boto:Method: GET 2023-09-29T22:03:36.457 DEBUG:boto:Path: /hcnphz-13/?prefix=dummy 2023-09-29T22:03:36.457 DEBUG:boto:Data: 2023-09-29T22:03:36.457 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.458 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.458 DEBUG:boto:Params: {} 2023-09-29T22:03:36.458 DEBUG:boto:Token: None 2023-09-29T22:03:36.458 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.458 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be76d7fa5ee26e77-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.463 DEBUG:boto:b'hcnphz-13dummy1000falsedummy2023-09-29T22:00:54.612Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.464 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.464 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.464 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.464 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.464 DEBUG:boto:Method: GET 2023-09-29T22:03:36.464 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.464 DEBUG:boto:Data: 2023-09-29T22:03:36.464 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.465 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.465 DEBUG:boto:Params: {} 2023-09-29T22:03:36.465 DEBUG:boto:Token: None 2023-09-29T22:03:36.465 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000977a08fe3be33bfc-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.510 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.510 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.511 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.511 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.511 DEBUG:boto:Method: GET 2023-09-29T22:03:36.511 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.511 DEBUG:boto:Data: 2023-09-29T22:03:36.511 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.511 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.511 DEBUG:boto:Params: {} 2023-09-29T22:03:36.512 DEBUG:boto:Token: None 2023-09-29T22:03:36.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.515 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003fff157507b81968-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.515 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.515 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.515 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.515 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.516 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.516 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.516 DEBUG:boto:Data: 2023-09-29T22:03:36.516 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.516 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.516 DEBUG:boto:Params: {} 2023-09-29T22:03:36.516 DEBUG:boto:Token: None 2023-09-29T22:03:36.516 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL://pOrnI5A+Z+Zr8iBf8CGC5P+54= 2023-09-29T22:03:36.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL://pOrnI5A+Z+Zr8iBf8CGC5P+54=', 'Content-Length': '0'} 2023-09-29T22:03:36.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008b5260ba6d51bfdc-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.519 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.520 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.520 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.520 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.520 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.520 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.520 DEBUG:boto:Data: 2023-09-29T22:03:36.520 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.520 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.520 DEBUG:boto:Params: {} 2023-09-29T22:03:36.521 DEBUG:boto:Token: None 2023-09-29T22:03:36.521 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL://pOrnI5A+Z+Zr8iBf8CGC5P+54= 2023-09-29T22:03:36.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL://pOrnI5A+Z+Zr8iBf8CGC5P+54=', 'Content-Length': '0'} 2023-09-29T22:03:36.523 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072fbb4468a03ee10-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.524 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.524 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.524 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.524 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.524 DEBUG:boto:Method: GET 2023-09-29T22:03:36.524 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.525 DEBUG:boto:Data: 2023-09-29T22:03:36.525 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.525 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.525 DEBUG:boto:Params: {} 2023-09-29T22:03:36.525 DEBUG:boto:Token: None 2023-09-29T22:03:36.525 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.528 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000003517f28a9a0c042-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.529 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.529 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.529 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.529 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.529 DEBUG:boto:Method: GET 2023-09-29T22:03:36.529 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.530 DEBUG:boto:Data: 2023-09-29T22:03:36.530 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.530 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.530 DEBUG:boto:Params: {} 2023-09-29T22:03:36.530 DEBUG:boto:Token: None 2023-09-29T22:03:36.530 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.533 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000076dbd660da415d33-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.574 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.575 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.575 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.575 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.575 DEBUG:boto:Method: GET 2023-09-29T22:03:36.575 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.575 DEBUG:boto:Data: 2023-09-29T22:03:36.575 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.576 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.576 DEBUG:boto:Params: {} 2023-09-29T22:03:36.576 DEBUG:boto:Token: None 2023-09-29T22:03:36.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bfa398ca1c92eb3-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.581 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.582 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.582 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.582 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.582 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.582 DEBUG:boto:Method: GET 2023-09-29T22:03:36.582 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.582 DEBUG:boto:Data: 2023-09-29T22:03:36.583 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.583 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.583 DEBUG:boto:Params: {} 2023-09-29T22:03:36.583 DEBUG:boto:Token: None 2023-09-29T22:03:36.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.583 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335d8444b3b842ee-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.589 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.589 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.589 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.589 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.589 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:36.589 DEBUG:boto:Data: 2023-09-29T22:03:36.590 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.590 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.590 DEBUG:boto:Params: {} 2023-09-29T22:03:36.590 DEBUG:boto:Token: None 2023-09-29T22:03:36.590 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj 2023-09-29T22:03:36.590 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y= 2023-09-29T22:03:36.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y=', 'Content-Length': '0'} 2023-09-29T22:03:36.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000963c9a207065948a-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.595 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.595 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.595 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.596 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:36.596 DEBUG:boto:Data: 2023-09-29T22:03:36.596 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.596 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.596 DEBUG:boto:Params: {} 2023-09-29T22:03:36.596 DEBUG:boto:Token: None 2023-09-29T22:03:36.596 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj 2023-09-29T22:03:36.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y= 2023-09-29T22:03:36.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y=', 'Content-Length': '0'} 2023-09-29T22:03:36.602 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006749a6a20b90c304-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.602 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.603 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.603 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.603 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.603 DEBUG:boto:Method: GET 2023-09-29T22:03:36.603 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.603 DEBUG:boto:Data: 2023-09-29T22:03:36.603 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.603 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.604 DEBUG:boto:Params: {} 2023-09-29T22:03:36.604 DEBUG:boto:Token: None 2023-09-29T22:03:36.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.606 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000eeca2528f7767fec-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.650 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.651 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.651 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.651 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.651 DEBUG:boto:Method: GET 2023-09-29T22:03:36.651 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.651 DEBUG:boto:Data: 2023-09-29T22:03:36.651 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.651 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.652 DEBUG:boto:Params: {} 2023-09-29T22:03:36.652 DEBUG:boto:Token: None 2023-09-29T22:03:36.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.655 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000024563f60a35d5bf6-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.655 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.655 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.656 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.656 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.656 DEBUG:boto:Method: GET 2023-09-29T22:03:36.656 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.656 DEBUG:boto:Data: 2023-09-29T22:03:36.656 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.656 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.656 DEBUG:boto:Params: {} 2023-09-29T22:03:36.657 DEBUG:boto:Token: None 2023-09-29T22:03:36.657 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a7d55c12b516b2-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.662 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.662 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.662 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.662 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.663 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.663 DEBUG:boto:Method: GET 2023-09-29T22:03:36.663 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.663 DEBUG:boto:Data: 2023-09-29T22:03:36.663 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.663 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.663 DEBUG:boto:Params: {} 2023-09-29T22:03:36.663 DEBUG:boto:Token: None 2023-09-29T22:03:36.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000078f96d219af37483-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.710 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.711 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.711 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.711 DEBUG:boto:path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.711 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.711 DEBUG:boto:Method: GET 2023-09-29T22:03:36.711 DEBUG:boto:Path: /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.712 DEBUG:boto:Data: 2023-09-29T22:03:36.712 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.712 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.712 DEBUG:boto:Params: {} 2023-09-29T22:03:36.712 DEBUG:boto:Token: None 2023-09-29T22:03:36.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI= 2023-09-29T22:03:36.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI=', 'Content-Length': '0'} 2023-09-29T22:03:36.715 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000004ec9a5bff61f94b-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.716 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.716 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.716 DEBUG:boto:path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.716 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.716 DEBUG:boto:Method: GET 2023-09-29T22:03:36.716 DEBUG:boto:Path: /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.716 DEBUG:boto:Data: 2023-09-29T22:03:36.716 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.717 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.717 DEBUG:boto:Params: {} 2023-09-29T22:03:36.717 DEBUG:boto:Token: None 2023-09-29T22:03:36.717 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.717 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI= 2023-09-29T22:03:36.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI=', 'Content-Length': '0'} 2023-09-29T22:03:36.720 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002657c7ee80af8111-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.720 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.721 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.721 DEBUG:boto:path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.721 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.721 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.721 DEBUG:boto:Path: /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.721 DEBUG:boto:Data: 2023-09-29T22:03:36.721 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.721 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.722 DEBUG:boto:Params: {} 2023-09-29T22:03:36.722 DEBUG:boto:Token: None 2023-09-29T22:03:36.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtZ4iTcWBX9f1eNZLaqeaeTTFWI= 2023-09-29T22:03:36.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtZ4iTcWBX9f1eNZLaqeaeTTFWI=', 'Content-Length': '0'} 2023-09-29T22:03:36.724 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cb6c8c395941cd3-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.725 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.725 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.725 DEBUG:boto:path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.725 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.725 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.725 DEBUG:boto:Path: /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.725 DEBUG:boto:Data: 2023-09-29T22:03:36.725 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.726 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.726 DEBUG:boto:Params: {} 2023-09-29T22:03:36.726 DEBUG:boto:Token: None 2023-09-29T22:03:36.726 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtZ4iTcWBX9f1eNZLaqeaeTTFWI= 2023-09-29T22:03:36.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtZ4iTcWBX9f1eNZLaqeaeTTFWI=', 'Content-Length': '0'} 2023-09-29T22:03:36.729 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001c04ded4176f8551-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.729 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.729 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.729 DEBUG:boto:path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.729 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.730 DEBUG:boto:Method: GET 2023-09-29T22:03:36.730 DEBUG:boto:Path: /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.730 DEBUG:boto:Data: 2023-09-29T22:03:36.730 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.730 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.730 DEBUG:boto:Params: {} 2023-09-29T22:03:36.730 DEBUG:boto:Token: None 2023-09-29T22:03:36.731 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.731 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI= 2023-09-29T22:03:36.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI=', 'Content-Length': '0'} 2023-09-29T22:03:36.733 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c073edb2f428a4a-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.734 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.734 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.734 DEBUG:boto:path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.734 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.734 DEBUG:boto:Method: GET 2023-09-29T22:03:36.734 DEBUG:boto:Path: /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.734 DEBUG:boto:Data: 2023-09-29T22:03:36.734 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.735 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.735 DEBUG:boto:Params: {} 2023-09-29T22:03:36.735 DEBUG:boto:Token: None 2023-09-29T22:03:36.735 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg 2023-09-29T22:03:36.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI= 2023-09-29T22:03:36.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qtm4A6VGKJLpMPa6Ww8aOOaDUlI=', 'Content-Length': '0'} 2023-09-29T22:03:36.738 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'tpfoDFrpYtqK8U.YmC7Q7ZzY57dAZNg'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cf296b4a13e3465f-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.782 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.783 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.783 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.783 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.783 DEBUG:boto:Method: GET 2023-09-29T22:03:36.783 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.783 DEBUG:boto:Data: 2023-09-29T22:03:36.783 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.783 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.784 DEBUG:boto:Params: {} 2023-09-29T22:03:36.784 DEBUG:boto:Token: None 2023-09-29T22:03:36.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000349ef97c8f9f8cc2-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.789 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.790 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.790 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.790 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.790 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.790 DEBUG:boto:Method: GET 2023-09-29T22:03:36.790 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.790 DEBUG:boto:Data: 2023-09-29T22:03:36.791 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.791 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.791 DEBUG:boto:Params: {} 2023-09-29T22:03:36.791 DEBUG:boto:Token: None 2023-09-29T22:03:36.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.791 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fca56743abd0a9f-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.838 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.839 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.839 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.839 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.839 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:36.839 DEBUG:boto:Data: 2023-09-29T22:03:36.839 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.840 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.840 DEBUG:boto:Params: {} 2023-09-29T22:03:36.840 DEBUG:boto:Token: None 2023-09-29T22:03:36.840 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj 2023-09-29T22:03:36.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y= 2023-09-29T22:03:36.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y=', 'Content-Length': '0'} 2023-09-29T22:03:36.846 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cbdb62b9b0f2d7b6-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.846 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.847 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.847 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.847 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:36.847 DEBUG:boto:Data: 2023-09-29T22:03:36.847 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.847 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.847 DEBUG:boto:Params: {} 2023-09-29T22:03:36.847 DEBUG:boto:Token: None 2023-09-29T22:03:36.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj 2023-09-29T22:03:36.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y= 2023-09-29T22:03:36.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a7lQb00zIMuSaj3J1Pu368X8K5Y=', 'Content-Length': '0'} 2023-09-29T22:03:36.853 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029c793e3b45ed869-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.854 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.854 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.854 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.854 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.854 DEBUG:boto:Method: GET 2023-09-29T22:03:36.854 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.855 DEBUG:boto:Data: 2023-09-29T22:03:36.855 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.855 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.855 DEBUG:boto:Params: {} 2023-09-29T22:03:36.855 DEBUG:boto:Token: None 2023-09-29T22:03:36.855 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.858 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008fba526b640b348b-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.898 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.899 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.899 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.899 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.899 DEBUG:boto:Method: GET 2023-09-29T22:03:36.899 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.899 DEBUG:boto:Data: 2023-09-29T22:03:36.899 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.899 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.900 DEBUG:boto:Params: {} 2023-09-29T22:03:36.900 DEBUG:boto:Token: None 2023-09-29T22:03:36.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:36.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8= 2023-09-29T22:03:36.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4NY5YsuIM8Y5Vkb9QWKtzcznd8=', 'Content-Length': '0'} 2023-09-29T22:03:36.903 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046505fd5744742ef-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.903 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.903 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.903 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.903 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.904 DEBUG:boto:Method: GET 2023-09-29T22:03:36.904 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.904 DEBUG:boto:Data: 2023-09-29T22:03:36.904 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.904 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.904 DEBUG:boto:Params: {} 2023-09-29T22:03:36.904 DEBUG:boto:Token: None 2023-09-29T22:03:36.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005080e298b936e2c0-00651749b8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.910 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.910 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:36.910 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:36.910 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.910 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:36.911 DEBUG:boto:Method: GET 2023-09-29T22:03:36.911 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:36.911 DEBUG:boto:Data: 2023-09-29T22:03:36.911 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.911 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.911 DEBUG:boto:Params: {} 2023-09-29T22:03:36.911 DEBUG:boto:Token: None 2023-09-29T22:03:36.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/ 2023-09-29T22:03:36.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA= 2023-09-29T22:03:36.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/JvwkgMCuHKBocgWd3b1OW4QhXA=', 'Content-Length': '0'} 2023-09-29T22:03:36.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000492c79d6e93925e-00651749b8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.916 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:36.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.917 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.917 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.917 DEBUG:boto:path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.917 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.917 DEBUG:boto:Method: GET 2023-09-29T22:03:36.918 DEBUG:boto:Path: /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.918 DEBUG:boto:Data: 2023-09-29T22:03:36.918 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.918 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.918 DEBUG:boto:Params: {} 2023-09-29T22:03:36.918 DEBUG:boto:Token: None 2023-09-29T22:03:36.918 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.919 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2DBz+2zmU4nq99/zrR7PcnV7Isc= 2023-09-29T22:03:36.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2DBz+2zmU4nq99/zrR7PcnV7Isc=', 'Content-Length': '0'} 2023-09-29T22:03:36.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d0edb9d00ad08ceb-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.922 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.922 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.922 DEBUG:boto:path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.922 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.922 DEBUG:boto:Method: GET 2023-09-29T22:03:36.922 DEBUG:boto:Path: /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.922 DEBUG:boto:Data: 2023-09-29T22:03:36.922 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.923 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.923 DEBUG:boto:Params: {} 2023-09-29T22:03:36.923 DEBUG:boto:Token: None 2023-09-29T22:03:36.923 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2DBz+2zmU4nq99/zrR7PcnV7Isc= 2023-09-29T22:03:36.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2DBz+2zmU4nq99/zrR7PcnV7Isc=', 'Content-Length': '0'} 2023-09-29T22:03:36.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a886f03cc3f3210a-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.966 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.967 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.967 DEBUG:boto:path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.967 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.967 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.967 DEBUG:boto:Path: /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.967 DEBUG:boto:Data: 2023-09-29T22:03:36.967 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.967 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.968 DEBUG:boto:Params: {} 2023-09-29T22:03:36.968 DEBUG:boto:Token: None 2023-09-29T22:03:36.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsJK3C2sRwFLGkskfq0iEPnwRzU= 2023-09-29T22:03:36.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsJK3C2sRwFLGkskfq0iEPnwRzU=', 'Content-Length': '0'} 2023-09-29T22:03:36.970 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005ea7653f0533e8e5-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.971 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.971 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.971 DEBUG:boto:path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.971 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.971 DEBUG:boto:Method: HEAD 2023-09-29T22:03:36.971 DEBUG:boto:Path: /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.971 DEBUG:boto:Data: 2023-09-29T22:03:36.972 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:36.972 DEBUG:boto:Port: 8000 2023-09-29T22:03:36.972 DEBUG:boto:Params: {} 2023-09-29T22:03:36.972 DEBUG:boto:Token: None 2023-09-29T22:03:36.972 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsJK3C2sRwFLGkskfq0iEPnwRzU= 2023-09-29T22:03:36.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsJK3C2sRwFLGkskfq0iEPnwRzU=', 'Content-Length': '0'} 2023-09-29T22:03:36.975 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c2a32cbd4312fdc4-00651749b8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:36.975 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:36.975 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:36.975 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:36.976 DEBUG:boto:path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.976 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.976 DEBUG:boto:Method: GET 2023-09-29T22:03:36.976 DEBUG:boto:Path: /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.976 DEBUG:boto:Data: 2023-09-29T22:03:36.976 DEBUG:boto:Headers: {} 2023-09-29T22:03:36.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:36.976 DEBUG:boto:Port: 8001 2023-09-29T22:03:36.976 DEBUG:boto:Params: {} 2023-09-29T22:03:36.977 DEBUG:boto:Token: None 2023-09-29T22:03:36.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:36 GMT /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:36.977 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2DBz+2zmU4nq99/zrR7PcnV7Isc= 2023-09-29T22:03:36.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2DBz+2zmU4nq99/zrR7PcnV7Isc=', 'Content-Length': '0'} 2023-09-29T22:03:36.980 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000006078f5b9c8b8ac8-00651749b8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.022 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.023 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.023 DEBUG:boto:path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:37.023 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:37.023 DEBUG:boto:Method: GET 2023-09-29T22:03:37.023 DEBUG:boto:Path: /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:37.023 DEBUG:boto:Data: 2023-09-29T22:03:37.023 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.023 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.023 DEBUG:boto:Params: {} 2023-09-29T22:03:37.024 DEBUG:boto:Token: None 2023-09-29T22:03:37.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7 2023-09-29T22:03:37.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RnHdL9wkpV/JYFdRO6g82EOSA4A= 2023-09-29T22:03:37.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RnHdL9wkpV/JYFdRO6g82EOSA4A=', 'Content-Length': '0'} 2023-09-29T22:03:37.027 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'FPa-hmJ8d9cWUO3cSsdsSHRwBEv3uy7'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003d510822b097054f-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.027 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.027 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.027 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.027 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.028 DEBUG:boto:Method: GET 2023-09-29T22:03:37.028 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.028 DEBUG:boto:Data: 2023-09-29T22:03:37.028 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.028 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.028 DEBUG:boto:Params: {} 2023-09-29T22:03:37.028 DEBUG:boto:Token: None 2023-09-29T22:03:37.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e1e76d1cb9e3c6f-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.034 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.035 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.035 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.035 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.035 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.035 DEBUG:boto:Method: GET 2023-09-29T22:03:37.035 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.036 DEBUG:boto:Data: 2023-09-29T22:03:37.036 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.036 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.036 DEBUG:boto:Params: {} 2023-09-29T22:03:37.036 DEBUG:boto:Token: None 2023-09-29T22:03:37.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc84e6c700865fad-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.041 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.041 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.042 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.042 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.042 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.042 DEBUG:boto:Data: 2023-09-29T22:03:37.042 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.042 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.042 DEBUG:boto:Params: {} 2023-09-29T22:03:37.042 DEBUG:boto:Token: None 2023-09-29T22:03:37.043 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.043 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.047 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000f81bca7df0ad21e-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.047 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.048 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.048 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.048 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.048 DEBUG:boto:Data: 2023-09-29T22:03:37.048 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.048 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.048 DEBUG:boto:Params: {} 2023-09-29T22:03:37.048 DEBUG:boto:Token: None 2023-09-29T22:03:37.049 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.053 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b11bffc105fa73a5-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.053 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.053 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.053 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.054 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.054 DEBUG:boto:Method: GET 2023-09-29T22:03:37.054 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.054 DEBUG:boto:Data: 2023-09-29T22:03:37.054 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.054 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.054 DEBUG:boto:Params: {} 2023-09-29T22:03:37.054 DEBUG:boto:Token: None 2023-09-29T22:03:37.055 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.055 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.057 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005139e0b598b92ee0-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.098 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.099 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.099 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.099 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.099 DEBUG:boto:Method: GET 2023-09-29T22:03:37.099 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.099 DEBUG:boto:Data: 2023-09-29T22:03:37.099 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.100 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.100 DEBUG:boto:Params: {} 2023-09-29T22:03:37.100 DEBUG:boto:Token: None 2023-09-29T22:03:37.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d79bd9151325ecb7-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.103 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.103 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.103 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.103 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.104 DEBUG:boto:Method: GET 2023-09-29T22:03:37.104 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.104 DEBUG:boto:Data: 2023-09-29T22:03:37.104 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.104 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.104 DEBUG:boto:Params: {} 2023-09-29T22:03:37.104 DEBUG:boto:Token: None 2023-09-29T22:03:37.105 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000521dfc0814a44d23-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.109 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.109 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.110 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.110 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.110 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.110 DEBUG:boto:Method: GET 2023-09-29T22:03:37.110 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.110 DEBUG:boto:Data: 2023-09-29T22:03:37.110 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.110 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.111 DEBUG:boto:Params: {} 2023-09-29T22:03:37.111 DEBUG:boto:Token: None 2023-09-29T22:03:37.111 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.111 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c62ff499d427139-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.115 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.116 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.116 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.116 DEBUG:boto:path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.116 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.117 DEBUG:boto:Method: GET 2023-09-29T22:03:37.117 DEBUG:boto:Path: /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.117 DEBUG:boto:Data: 2023-09-29T22:03:37.117 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.117 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.117 DEBUG:boto:Params: {} 2023-09-29T22:03:37.118 DEBUG:boto:Token: None 2023-09-29T22:03:37.118 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.118 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE= 2023-09-29T22:03:37.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE=', 'Content-Length': '0'} 2023-09-29T22:03:37.121 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000008cbf47ae912361-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.121 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.121 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.121 DEBUG:boto:path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.121 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.121 DEBUG:boto:Method: GET 2023-09-29T22:03:37.122 DEBUG:boto:Path: /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.122 DEBUG:boto:Data: 2023-09-29T22:03:37.122 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.122 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.122 DEBUG:boto:Params: {} 2023-09-29T22:03:37.122 DEBUG:boto:Token: None 2023-09-29T22:03:37.122 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE= 2023-09-29T22:03:37.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE=', 'Content-Length': '0'} 2023-09-29T22:03:37.125 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026ef8517d2582a86-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.125 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.126 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.126 DEBUG:boto:path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.126 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.126 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.126 DEBUG:boto:Path: /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.126 DEBUG:boto:Data: 2023-09-29T22:03:37.126 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.126 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.127 DEBUG:boto:Params: {} 2023-09-29T22:03:37.127 DEBUG:boto:Token: None 2023-09-29T22:03:37.127 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.127 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eIuX9LaiMjRukuk9n6shQdUDZwQ= 2023-09-29T22:03:37.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eIuX9LaiMjRukuk9n6shQdUDZwQ=', 'Content-Length': '0'} 2023-09-29T22:03:37.130 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000079c04af697f393e1-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.130 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.130 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.130 DEBUG:boto:path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.130 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.131 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.131 DEBUG:boto:Path: /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.131 DEBUG:boto:Data: 2023-09-29T22:03:37.131 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.131 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.131 DEBUG:boto:Params: {} 2023-09-29T22:03:37.131 DEBUG:boto:Token: None 2023-09-29T22:03:37.131 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eIuX9LaiMjRukuk9n6shQdUDZwQ= 2023-09-29T22:03:37.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eIuX9LaiMjRukuk9n6shQdUDZwQ=', 'Content-Length': '0'} 2023-09-29T22:03:37.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a1a501fa29bf616-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.135 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.135 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.135 DEBUG:boto:path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.135 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.135 DEBUG:boto:Method: GET 2023-09-29T22:03:37.135 DEBUG:boto:Path: /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.135 DEBUG:boto:Data: 2023-09-29T22:03:37.135 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.135 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.136 DEBUG:boto:Params: {} 2023-09-29T22:03:37.136 DEBUG:boto:Token: None 2023-09-29T22:03:37.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE= 2023-09-29T22:03:37.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE=', 'Content-Length': '0'} 2023-09-29T22:03:37.139 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008303bf674a095ab3-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.139 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.139 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.139 DEBUG:boto:path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.139 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.139 DEBUG:boto:Method: GET 2023-09-29T22:03:37.140 DEBUG:boto:Path: /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.140 DEBUG:boto:Data: 2023-09-29T22:03:37.140 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.140 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.140 DEBUG:boto:Params: {} 2023-09-29T22:03:37.140 DEBUG:boto:Token: None 2023-09-29T22:03:37.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf 2023-09-29T22:03:37.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE= 2023-09-29T22:03:37.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mHocPqg6Rtb56eZiopittJ9tuYE=', 'Content-Length': '0'} 2023-09-29T22:03:37.143 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'zRMn3Mtw8IGfoPG1RXFWkzIAQ27OnYf'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000399ad5ce622b3793-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.144 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.144 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.144 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.144 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.144 DEBUG:boto:Method: GET 2023-09-29T22:03:37.144 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.144 DEBUG:boto:Data: 2023-09-29T22:03:37.144 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.145 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.145 DEBUG:boto:Params: {} 2023-09-29T22:03:37.145 DEBUG:boto:Token: None 2023-09-29T22:03:37.145 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da6a54f30c3857c4-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.194 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.195 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.195 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.195 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.195 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.195 DEBUG:boto:Method: GET 2023-09-29T22:03:37.195 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.195 DEBUG:boto:Data: 2023-09-29T22:03:37.196 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.196 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.196 DEBUG:boto:Params: {} 2023-09-29T22:03:37.196 DEBUG:boto:Token: None 2023-09-29T22:03:37.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f6ddfdc4f50ed20-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.202 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.202 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.202 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.202 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.203 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.203 DEBUG:boto:Data: 2023-09-29T22:03:37.203 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.203 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.203 DEBUG:boto:Params: {} 2023-09-29T22:03:37.203 DEBUG:boto:Token: None 2023-09-29T22:03:37.203 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.209 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000726f3005f9b8b15b-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.210 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.210 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.210 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.210 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.210 DEBUG:boto:Data: 2023-09-29T22:03:37.211 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.211 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.211 DEBUG:boto:Params: {} 2023-09-29T22:03:37.211 DEBUG:boto:Token: None 2023-09-29T22:03:37.211 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c358955ecc0de1ad-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.216 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.216 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.217 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.217 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.217 DEBUG:boto:Method: GET 2023-09-29T22:03:37.217 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.217 DEBUG:boto:Data: 2023-09-29T22:03:37.217 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.217 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.217 DEBUG:boto:Params: {} 2023-09-29T22:03:37.218 DEBUG:boto:Token: None 2023-09-29T22:03:37.218 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.218 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029520c67ee117bbe-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.262 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.262 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.263 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.263 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.263 DEBUG:boto:Method: GET 2023-09-29T22:03:37.263 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.263 DEBUG:boto:Data: 2023-09-29T22:03:37.263 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.263 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.263 DEBUG:boto:Params: {} 2023-09-29T22:03:37.264 DEBUG:boto:Token: None 2023-09-29T22:03:37.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.267 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000084738ab4e962c913-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.267 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.267 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.267 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.267 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.267 DEBUG:boto:Method: GET 2023-09-29T22:03:37.268 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.268 DEBUG:boto:Data: 2023-09-29T22:03:37.268 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.268 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.268 DEBUG:boto:Params: {} 2023-09-29T22:03:37.268 DEBUG:boto:Token: None 2023-09-29T22:03:37.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d95c36a104b10d1-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.314 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.315 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.315 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.315 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.315 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.315 DEBUG:boto:Method: GET 2023-09-29T22:03:37.315 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.316 DEBUG:boto:Data: 2023-09-29T22:03:37.316 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.316 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.316 DEBUG:boto:Params: {} 2023-09-29T22:03:37.316 DEBUG:boto:Token: None 2023-09-29T22:03:37.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce45c9a3569473a-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.322 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.322 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.323 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.323 DEBUG:boto:path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.323 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.323 DEBUG:boto:Method: GET 2023-09-29T22:03:37.323 DEBUG:boto:Path: /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.323 DEBUG:boto:Data: 2023-09-29T22:03:37.324 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.324 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.324 DEBUG:boto:Params: {} 2023-09-29T22:03:37.324 DEBUG:boto:Token: None 2023-09-29T22:03:37.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c= 2023-09-29T22:03:37.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c=', 'Content-Length': '0'} 2023-09-29T22:03:37.328 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ab83a68e724bcdcf-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.328 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.328 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.328 DEBUG:boto:path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.328 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.329 DEBUG:boto:Method: GET 2023-09-29T22:03:37.329 DEBUG:boto:Path: /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.329 DEBUG:boto:Data: 2023-09-29T22:03:37.329 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.329 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.329 DEBUG:boto:Params: {} 2023-09-29T22:03:37.329 DEBUG:boto:Token: None 2023-09-29T22:03:37.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c= 2023-09-29T22:03:37.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c=', 'Content-Length': '0'} 2023-09-29T22:03:37.332 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000376150dcce8332fc-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.332 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.332 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.333 DEBUG:boto:path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.333 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.333 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.333 DEBUG:boto:Path: /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.333 DEBUG:boto:Data: 2023-09-29T22:03:37.333 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.333 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.333 DEBUG:boto:Params: {} 2023-09-29T22:03:37.334 DEBUG:boto:Token: None 2023-09-29T22:03:37.334 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T3Nm2O8H4EKJVw8t3F0ePptEl+4= 2023-09-29T22:03:37.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T3Nm2O8H4EKJVw8t3F0ePptEl+4=', 'Content-Length': '0'} 2023-09-29T22:03:37.336 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001df5980f55ce435c-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.337 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.337 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.337 DEBUG:boto:path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.337 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.337 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.337 DEBUG:boto:Path: /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.337 DEBUG:boto:Data: 2023-09-29T22:03:37.337 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.338 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.338 DEBUG:boto:Params: {} 2023-09-29T22:03:37.338 DEBUG:boto:Token: None 2023-09-29T22:03:37.338 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T3Nm2O8H4EKJVw8t3F0ePptEl+4= 2023-09-29T22:03:37.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T3Nm2O8H4EKJVw8t3F0ePptEl+4=', 'Content-Length': '0'} 2023-09-29T22:03:37.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e86d47691b251fe5-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.341 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.341 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.342 DEBUG:boto:path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.342 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.342 DEBUG:boto:Method: GET 2023-09-29T22:03:37.342 DEBUG:boto:Path: /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.342 DEBUG:boto:Data: 2023-09-29T22:03:37.342 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.342 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.342 DEBUG:boto:Params: {} 2023-09-29T22:03:37.343 DEBUG:boto:Token: None 2023-09-29T22:03:37.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c= 2023-09-29T22:03:37.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c=', 'Content-Length': '0'} 2023-09-29T22:03:37.346 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000089faaa242787f37a-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.386 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.386 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.387 DEBUG:boto:path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.387 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.387 DEBUG:boto:Method: GET 2023-09-29T22:03:37.387 DEBUG:boto:Path: /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.387 DEBUG:boto:Data: 2023-09-29T22:03:37.387 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.387 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.387 DEBUG:boto:Params: {} 2023-09-29T22:03:37.388 DEBUG:boto:Token: None 2023-09-29T22:03:37.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I 2023-09-29T22:03:37.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c= 2023-09-29T22:03:37.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lGmvxXqgMRQjw/7xkzORfHSER1c=', 'Content-Length': '0'} 2023-09-29T22:03:37.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '4BQXR5JQPMJiT.C-F6Q.QL.pc3Jel6I'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004bf7a2a7add20f6f-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.391 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.391 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.391 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.391 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.391 DEBUG:boto:Method: GET 2023-09-29T22:03:37.391 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.392 DEBUG:boto:Data: 2023-09-29T22:03:37.392 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.392 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.392 DEBUG:boto:Params: {} 2023-09-29T22:03:37.392 DEBUG:boto:Token: None 2023-09-29T22:03:37.392 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ddee194b01b24e-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.438 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.439 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.439 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.439 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.439 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.439 DEBUG:boto:Method: GET 2023-09-29T22:03:37.439 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.439 DEBUG:boto:Data: 2023-09-29T22:03:37.440 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.440 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.440 DEBUG:boto:Params: {} 2023-09-29T22:03:37.440 DEBUG:boto:Token: None 2023-09-29T22:03:37.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000904ace397d2968ab-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.445 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.446 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.446 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.446 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.446 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.446 DEBUG:boto:Data: 2023-09-29T22:03:37.446 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.447 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.447 DEBUG:boto:Params: {} 2023-09-29T22:03:37.447 DEBUG:boto:Token: None 2023-09-29T22:03:37.447 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.453 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7a65367d2c56fb1-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.453 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.453 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.453 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.454 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.454 DEBUG:boto:Data: 2023-09-29T22:03:37.454 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.454 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.454 DEBUG:boto:Params: {} 2023-09-29T22:03:37.454 DEBUG:boto:Token: None 2023-09-29T22:03:37.454 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d9383ffe1304e8b8-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.460 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.460 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.460 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.460 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.460 DEBUG:boto:Method: GET 2023-09-29T22:03:37.460 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.461 DEBUG:boto:Data: 2023-09-29T22:03:37.461 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.461 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.461 DEBUG:boto:Params: {} 2023-09-29T22:03:37.461 DEBUG:boto:Token: None 2023-09-29T22:03:37.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000067395420aca0a504-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.507 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.507 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.507 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.507 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.507 DEBUG:boto:Method: GET 2023-09-29T22:03:37.507 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.507 DEBUG:boto:Data: 2023-09-29T22:03:37.508 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.508 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.508 DEBUG:boto:Params: {} 2023-09-29T22:03:37.508 DEBUG:boto:Token: None 2023-09-29T22:03:37.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000007129dc3121cac45-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.512 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.513 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.513 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.513 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.513 DEBUG:boto:Method: GET 2023-09-29T22:03:37.513 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.514 DEBUG:boto:Data: 2023-09-29T22:03:37.514 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.514 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.514 DEBUG:boto:Params: {} 2023-09-29T22:03:37.515 DEBUG:boto:Token: None 2023-09-29T22:03:37.515 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.515 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.520 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000707d09fc8042d41d-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.562 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.563 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.563 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.563 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.563 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.564 DEBUG:boto:Method: GET 2023-09-29T22:03:37.564 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.564 DEBUG:boto:Data: 2023-09-29T22:03:37.564 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.564 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.564 DEBUG:boto:Params: {} 2023-09-29T22:03:37.565 DEBUG:boto:Token: None 2023-09-29T22:03:37.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b51a658ddb9bcc00-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.571 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.571 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.572 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.572 DEBUG:boto:path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.572 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.572 DEBUG:boto:Method: GET 2023-09-29T22:03:37.572 DEBUG:boto:Path: /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.572 DEBUG:boto:Data: 2023-09-29T22:03:37.573 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.573 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.573 DEBUG:boto:Params: {} 2023-09-29T22:03:37.573 DEBUG:boto:Token: None 2023-09-29T22:03:37.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw= 2023-09-29T22:03:37.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw=', 'Content-Length': '0'} 2023-09-29T22:03:37.576 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009f35684a59d0b3e2-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.577 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.577 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.577 DEBUG:boto:path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.577 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.577 DEBUG:boto:Method: GET 2023-09-29T22:03:37.578 DEBUG:boto:Path: /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.578 DEBUG:boto:Data: 2023-09-29T22:03:37.578 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.578 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.578 DEBUG:boto:Params: {} 2023-09-29T22:03:37.578 DEBUG:boto:Token: None 2023-09-29T22:03:37.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw= 2023-09-29T22:03:37.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw=', 'Content-Length': '0'} 2023-09-29T22:03:37.582 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e6d6ade3150d2b88-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.622 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.623 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.623 DEBUG:boto:path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.623 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.623 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.623 DEBUG:boto:Path: /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.623 DEBUG:boto:Data: 2023-09-29T22:03:37.624 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.624 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.624 DEBUG:boto:Params: {} 2023-09-29T22:03:37.624 DEBUG:boto:Token: None 2023-09-29T22:03:37.624 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.625 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JN6rZysBsHuR1tpwei6JHBbz45s= 2023-09-29T22:03:37.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JN6rZysBsHuR1tpwei6JHBbz45s=', 'Content-Length': '0'} 2023-09-29T22:03:37.628 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c549b4e9bee522d9-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.628 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.628 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.628 DEBUG:boto:path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.628 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.628 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.629 DEBUG:boto:Path: /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.629 DEBUG:boto:Data: 2023-09-29T22:03:37.629 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.629 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.629 DEBUG:boto:Params: {} 2023-09-29T22:03:37.629 DEBUG:boto:Token: None 2023-09-29T22:03:37.629 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.630 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JN6rZysBsHuR1tpwei6JHBbz45s= 2023-09-29T22:03:37.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JN6rZysBsHuR1tpwei6JHBbz45s=', 'Content-Length': '0'} 2023-09-29T22:03:37.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cc0e99b474fe1a8-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.633 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.633 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.633 DEBUG:boto:path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.633 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.633 DEBUG:boto:Method: GET 2023-09-29T22:03:37.634 DEBUG:boto:Path: /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.634 DEBUG:boto:Data: 2023-09-29T22:03:37.634 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.634 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.634 DEBUG:boto:Params: {} 2023-09-29T22:03:37.634 DEBUG:boto:Token: None 2023-09-29T22:03:37.634 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.635 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw= 2023-09-29T22:03:37.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw=', 'Content-Length': '0'} 2023-09-29T22:03:37.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b4eaf43a86fd9e9-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.638 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.638 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.638 DEBUG:boto:path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.638 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.639 DEBUG:boto:Method: GET 2023-09-29T22:03:37.639 DEBUG:boto:Path: /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.639 DEBUG:boto:Data: 2023-09-29T22:03:37.639 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.639 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.639 DEBUG:boto:Params: {} 2023-09-29T22:03:37.639 DEBUG:boto:Token: None 2023-09-29T22:03:37.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb 2023-09-29T22:03:37.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw= 2023-09-29T22:03:37.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VB2Gv9lgSiR1f1CcAagIch0aSYw=', 'Content-Length': '0'} 2023-09-29T22:03:37.642 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'rpSMuc1GJlB4zfJhxcLig4aLeB9wbxb'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000366fb19f642eb1bd-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.643 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.643 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.643 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.643 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.643 DEBUG:boto:Method: GET 2023-09-29T22:03:37.644 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.644 DEBUG:boto:Data: 2023-09-29T22:03:37.644 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.644 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.644 DEBUG:boto:Params: {} 2023-09-29T22:03:37.644 DEBUG:boto:Token: None 2023-09-29T22:03:37.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0289e9d94ec2a17-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.650 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.650 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.650 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.651 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.651 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.651 DEBUG:boto:Method: GET 2023-09-29T22:03:37.651 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.651 DEBUG:boto:Data: 2023-09-29T22:03:37.651 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.652 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.652 DEBUG:boto:Params: {} 2023-09-29T22:03:37.652 DEBUG:boto:Token: None 2023-09-29T22:03:37.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83b35c20da29a3d-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.658 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.658 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.658 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.659 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.659 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.659 DEBUG:boto:Data: 2023-09-29T22:03:37.659 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.660 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.660 DEBUG:boto:Params: {} 2023-09-29T22:03:37.660 DEBUG:boto:Token: None 2023-09-29T22:03:37.661 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.667 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000022a8f354f95b7c6b-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.667 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.667 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.667 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.667 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.667 DEBUG:boto:Data: 2023-09-29T22:03:37.668 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.668 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.668 DEBUG:boto:Params: {} 2023-09-29T22:03:37.668 DEBUG:boto:Token: None 2023-09-29T22:03:37.668 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.675 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005cd85a920ce48bae-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.675 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.675 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.675 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.676 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.676 DEBUG:boto:Method: GET 2023-09-29T22:03:37.676 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.676 DEBUG:boto:Data: 2023-09-29T22:03:37.676 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.676 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.677 DEBUG:boto:Params: {} 2023-09-29T22:03:37.677 DEBUG:boto:Token: None 2023-09-29T22:03:37.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.680 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005dbc8d3494c45711-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.722 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.723 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.723 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.723 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.723 DEBUG:boto:Method: GET 2023-09-29T22:03:37.723 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.723 DEBUG:boto:Data: 2023-09-29T22:03:37.723 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.723 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.724 DEBUG:boto:Params: {} 2023-09-29T22:03:37.724 DEBUG:boto:Token: None 2023-09-29T22:03:37.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.727 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f4d418454dbe61db-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.727 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.727 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.727 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.728 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.728 DEBUG:boto:Method: GET 2023-09-29T22:03:37.728 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.728 DEBUG:boto:Data: 2023-09-29T22:03:37.728 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.728 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.728 DEBUG:boto:Params: {} 2023-09-29T22:03:37.729 DEBUG:boto:Token: None 2023-09-29T22:03:37.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cda2baa3b6aa718-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.774 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.775 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.775 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.776 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.776 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.776 DEBUG:boto:Method: GET 2023-09-29T22:03:37.776 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.776 DEBUG:boto:Data: 2023-09-29T22:03:37.776 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.776 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.776 DEBUG:boto:Params: {} 2023-09-29T22:03:37.777 DEBUG:boto:Token: None 2023-09-29T22:03:37.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e3cba0510c05fd2-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.782 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.783 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.783 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.783 DEBUG:boto:path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.783 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.783 DEBUG:boto:Method: GET 2023-09-29T22:03:37.783 DEBUG:boto:Path: /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.783 DEBUG:boto:Data: 2023-09-29T22:03:37.783 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.783 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.784 DEBUG:boto:Params: {} 2023-09-29T22:03:37.784 DEBUG:boto:Token: None 2023-09-29T22:03:37.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o= 2023-09-29T22:03:37.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o=', 'Content-Length': '0'} 2023-09-29T22:03:37.787 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000014c3a8be1fe91cd-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.787 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.787 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.787 DEBUG:boto:path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.787 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.788 DEBUG:boto:Method: GET 2023-09-29T22:03:37.788 DEBUG:boto:Path: /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.788 DEBUG:boto:Data: 2023-09-29T22:03:37.788 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.788 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.788 DEBUG:boto:Params: {} 2023-09-29T22:03:37.788 DEBUG:boto:Token: None 2023-09-29T22:03:37.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.789 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o= 2023-09-29T22:03:37.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o=', 'Content-Length': '0'} 2023-09-29T22:03:37.791 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bde57bb3d2a5ae01-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.792 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.792 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.792 DEBUG:boto:path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.792 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.792 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.792 DEBUG:boto:Path: /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.792 DEBUG:boto:Data: 2023-09-29T22:03:37.793 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.793 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.793 DEBUG:boto:Params: {} 2023-09-29T22:03:37.793 DEBUG:boto:Token: None 2023-09-29T22:03:37.793 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BAgND+KBBykMtJ91wjbeJgCX3fg= 2023-09-29T22:03:37.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BAgND+KBBykMtJ91wjbeJgCX3fg=', 'Content-Length': '0'} 2023-09-29T22:03:37.796 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008c9fee67fd3c7dd9-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.796 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.796 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.796 DEBUG:boto:path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.796 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.797 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.797 DEBUG:boto:Path: /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.797 DEBUG:boto:Data: 2023-09-29T22:03:37.797 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.797 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.797 DEBUG:boto:Params: {} 2023-09-29T22:03:37.797 DEBUG:boto:Token: None 2023-09-29T22:03:37.797 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BAgND+KBBykMtJ91wjbeJgCX3fg= 2023-09-29T22:03:37.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BAgND+KBBykMtJ91wjbeJgCX3fg=', 'Content-Length': '0'} 2023-09-29T22:03:37.800 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000448a50fe976ef1f7-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.800 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.801 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.801 DEBUG:boto:path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.801 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.801 DEBUG:boto:Method: GET 2023-09-29T22:03:37.801 DEBUG:boto:Path: /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.801 DEBUG:boto:Data: 2023-09-29T22:03:37.801 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.801 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.801 DEBUG:boto:Params: {} 2023-09-29T22:03:37.802 DEBUG:boto:Token: None 2023-09-29T22:03:37.802 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o= 2023-09-29T22:03:37.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o=', 'Content-Length': '0'} 2023-09-29T22:03:37.805 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000046dc3b4a2e0150ab-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.850 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.851 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.851 DEBUG:boto:path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.851 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.851 DEBUG:boto:Method: GET 2023-09-29T22:03:37.851 DEBUG:boto:Path: /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.851 DEBUG:boto:Data: 2023-09-29T22:03:37.851 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.852 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.852 DEBUG:boto:Params: {} 2023-09-29T22:03:37.852 DEBUG:boto:Token: None 2023-09-29T22:03:37.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T 2023-09-29T22:03:37.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o= 2023-09-29T22:03:37.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pt2PnElIBNBfzjRIUASdXqHfD8o=', 'Content-Length': '0'} 2023-09-29T22:03:37.855 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '8Z.rdkd7FqrAI1m5Vv35gdyEZZJNW4T'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000485afb8ef9083526-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.856 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.856 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.856 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.856 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.856 DEBUG:boto:Method: GET 2023-09-29T22:03:37.856 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.856 DEBUG:boto:Data: 2023-09-29T22:03:37.857 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.857 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.857 DEBUG:boto:Params: {} 2023-09-29T22:03:37.857 DEBUG:boto:Token: None 2023-09-29T22:03:37.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ca24daff4ad3c8c-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.862 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.862 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.862 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.863 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.863 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.863 DEBUG:boto:Method: GET 2023-09-29T22:03:37.863 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.863 DEBUG:boto:Data: 2023-09-29T22:03:37.863 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.863 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.863 DEBUG:boto:Params: {} 2023-09-29T22:03:37.863 DEBUG:boto:Token: None 2023-09-29T22:03:37.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067d13c019431cfaa-00651749b9-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.910 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:37.911 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.911 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.911 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.911 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.911 DEBUG:boto:Data: 2023-09-29T22:03:37.911 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.912 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.912 DEBUG:boto:Params: {} 2023-09-29T22:03:37.912 DEBUG:boto:Token: None 2023-09-29T22:03:37.912 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.918 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006a8b5e5d9d94d50e-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.918 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.919 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.919 DEBUG:boto:Method: HEAD 2023-09-29T22:03:37.919 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:37.919 DEBUG:boto:Data: 2023-09-29T22:03:37.919 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.919 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.919 DEBUG:boto:Params: {} 2023-09-29T22:03:37.919 DEBUG:boto:Token: None 2023-09-29T22:03:37.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj 2023-09-29T22:03:37.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE= 2023-09-29T22:03:37.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/G00eFicCsNg4FUliSU19E53kmE=', 'Content-Length': '0'} 2023-09-29T22:03:37.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000072cd5c54459dabed-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:37.925 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.926 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.926 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.926 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.926 DEBUG:boto:Method: GET 2023-09-29T22:03:37.926 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.926 DEBUG:boto:Data: 2023-09-29T22:03:37.926 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.926 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.926 DEBUG:boto:Params: {} 2023-09-29T22:03:37.927 DEBUG:boto:Token: None 2023-09-29T22:03:37.927 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000779ec202512d1f2a-00651749b9-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.974 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:37.974 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:37.975 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.975 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.975 DEBUG:boto:Method: GET 2023-09-29T22:03:37.975 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.975 DEBUG:boto:Data: 2023-09-29T22:03:37.975 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:37.975 DEBUG:boto:Port: 8000 2023-09-29T22:03:37.975 DEBUG:boto:Params: {} 2023-09-29T22:03:37.975 DEBUG:boto:Token: None 2023-09-29T22:03:37.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:37.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk= 2023-09-29T22:03:37.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znncS/slE72ELaJJORTBvSsWBtk=', 'Content-Length': '0'} 2023-09-29T22:03:37.978 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a0072b00ed1660fa-00651749b9-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:37.979 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:37.979 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:37.979 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.979 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:37.979 DEBUG:boto:Method: GET 2023-09-29T22:03:37.979 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:37.980 DEBUG:boto:Data: 2023-09-29T22:03:37.980 DEBUG:boto:Headers: {} 2023-09-29T22:03:37.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:37.980 DEBUG:boto:Port: 8001 2023-09-29T22:03:37.980 DEBUG:boto:Params: {} 2023-09-29T22:03:37.980 DEBUG:boto:Token: None 2023-09-29T22:03:37.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:37 GMT /hcnphz-13/ 2023-09-29T22:03:37.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc= 2023-09-29T22:03:37.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+agXE1VsQLGwi6YynXmouOs5aoc=', 'Content-Length': '0'} 2023-09-29T22:03:37.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dee90f03b225df7-00651749b9-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.026 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.027 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.027 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.027 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.027 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.027 DEBUG:boto:Method: GET 2023-09-29T22:03:38.027 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.027 DEBUG:boto:Data: 2023-09-29T22:03:38.028 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.028 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.028 DEBUG:boto:Params: {} 2023-09-29T22:03:38.028 DEBUG:boto:Token: None 2023-09-29T22:03:38.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.028 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.031 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d607583da04e3f0-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.031 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.032 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.032 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.032 DEBUG:boto:path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.032 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.032 DEBUG:boto:Method: GET 2023-09-29T22:03:38.033 DEBUG:boto:Path: /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.033 DEBUG:boto:Data: 2023-09-29T22:03:38.033 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.033 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.033 DEBUG:boto:Params: {} 2023-09-29T22:03:38.033 DEBUG:boto:Token: None 2023-09-29T22:03:38.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig= 2023-09-29T22:03:38.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig=', 'Content-Length': '0'} 2023-09-29T22:03:38.036 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000b3fd465ea23e4ac-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.036 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.037 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.037 DEBUG:boto:path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.037 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.037 DEBUG:boto:Method: GET 2023-09-29T22:03:38.037 DEBUG:boto:Path: /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.037 DEBUG:boto:Data: 2023-09-29T22:03:38.037 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.038 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.038 DEBUG:boto:Params: {} 2023-09-29T22:03:38.038 DEBUG:boto:Token: None 2023-09-29T22:03:38.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig= 2023-09-29T22:03:38.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig=', 'Content-Length': '0'} 2023-09-29T22:03:38.041 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2427a108bb184e6-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.041 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.041 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.041 DEBUG:boto:path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.041 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.042 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.042 DEBUG:boto:Path: /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.042 DEBUG:boto:Data: 2023-09-29T22:03:38.042 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.042 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.042 DEBUG:boto:Params: {} 2023-09-29T22:03:38.042 DEBUG:boto:Token: None 2023-09-29T22:03:38.043 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.043 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6BcGYVUfESOsBOgY9lkPGXDYATM= 2023-09-29T22:03:38.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6BcGYVUfESOsBOgY9lkPGXDYATM=', 'Content-Length': '0'} 2023-09-29T22:03:38.045 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d649d878bb21ea45-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.045 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.046 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.046 DEBUG:boto:path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.046 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.046 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.046 DEBUG:boto:Path: /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.046 DEBUG:boto:Data: 2023-09-29T22:03:38.046 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.047 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.047 DEBUG:boto:Params: {} 2023-09-29T22:03:38.047 DEBUG:boto:Token: None 2023-09-29T22:03:38.047 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6BcGYVUfESOsBOgY9lkPGXDYATM= 2023-09-29T22:03:38.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6BcGYVUfESOsBOgY9lkPGXDYATM=', 'Content-Length': '0'} 2023-09-29T22:03:38.050 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000008d385fd4ab57cd5-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.050 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.050 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.050 DEBUG:boto:path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.050 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.051 DEBUG:boto:Method: GET 2023-09-29T22:03:38.051 DEBUG:boto:Path: /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.051 DEBUG:boto:Data: 2023-09-29T22:03:38.051 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.051 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.051 DEBUG:boto:Params: {} 2023-09-29T22:03:38.052 DEBUG:boto:Token: None 2023-09-29T22:03:38.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig= 2023-09-29T22:03:38.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig=', 'Content-Length': '0'} 2023-09-29T22:03:38.054 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a45b5f03440ab5a4-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.098 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.099 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.099 DEBUG:boto:path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.099 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.099 DEBUG:boto:Method: GET 2023-09-29T22:03:38.099 DEBUG:boto:Path: /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.099 DEBUG:boto:Data: 2023-09-29T22:03:38.099 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.100 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.100 DEBUG:boto:Params: {} 2023-09-29T22:03:38.100 DEBUG:boto:Token: None 2023-09-29T22:03:38.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=PSroL9.BBptEm21WZRCI8FrA0kZDA96 2023-09-29T22:03:38.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig= 2023-09-29T22:03:38.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g5TKpb+LvlHhoW1yd7MJu8BHQig=', 'Content-Length': '0'} 2023-09-29T22:03:38.103 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'PSroL9.BBptEm21WZRCI8FrA0kZDA96'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a9aa7c59c8ec7997-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.104 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.104 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.104 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.104 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.104 DEBUG:boto:Method: GET 2023-09-29T22:03:38.104 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.104 DEBUG:boto:Data: 2023-09-29T22:03:38.105 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.105 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.105 DEBUG:boto:Params: {} 2023-09-29T22:03:38.105 DEBUG:boto:Token: None 2023-09-29T22:03:38.105 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.106 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.111 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8a5f57a4ccb70f-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.158 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.159 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.159 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.159 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.159 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.159 DEBUG:boto:Method: GET 2023-09-29T22:03:38.159 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.160 DEBUG:boto:Data: 2023-09-29T22:03:38.160 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.160 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.160 DEBUG:boto:Params: {} 2023-09-29T22:03:38.160 DEBUG:boto:Token: None 2023-09-29T22:03:38.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.166 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4298ea43804e393-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.166 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.166 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.167 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.167 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.167 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.167 DEBUG:boto:Data: 2023-09-29T22:03:38.167 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.167 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.168 DEBUG:boto:Params: {} 2023-09-29T22:03:38.168 DEBUG:boto:Token: None 2023-09-29T22:03:38.168 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.173 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007401e481007bcf86-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.173 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.173 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.173 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.173 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.174 DEBUG:boto:Data: 2023-09-29T22:03:38.174 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.174 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.174 DEBUG:boto:Params: {} 2023-09-29T22:03:38.174 DEBUG:boto:Token: None 2023-09-29T22:03:38.174 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004be95fdf5d866991-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.179 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.179 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.179 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.180 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.180 DEBUG:boto:Method: GET 2023-09-29T22:03:38.180 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.180 DEBUG:boto:Data: 2023-09-29T22:03:38.180 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.180 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.180 DEBUG:boto:Params: {} 2023-09-29T22:03:38.180 DEBUG:boto:Token: None 2023-09-29T22:03:38.181 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe74f07f64e29a7f-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.226 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.227 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.227 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.227 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.227 DEBUG:boto:Method: GET 2023-09-29T22:03:38.227 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.227 DEBUG:boto:Data: 2023-09-29T22:03:38.227 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.227 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.227 DEBUG:boto:Params: {} 2023-09-29T22:03:38.228 DEBUG:boto:Token: None 2023-09-29T22:03:38.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.228 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.231 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b0a14ab517dcf9af-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.231 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.231 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.231 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.232 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.232 DEBUG:boto:Method: GET 2023-09-29T22:03:38.232 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.232 DEBUG:boto:Data: 2023-09-29T22:03:38.232 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.232 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.232 DEBUG:boto:Params: {} 2023-09-29T22:03:38.232 DEBUG:boto:Token: None 2023-09-29T22:03:38.233 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7feaac801047fbb-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.282 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.283 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.283 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.283 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.283 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.283 DEBUG:boto:Method: GET 2023-09-29T22:03:38.283 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.283 DEBUG:boto:Data: 2023-09-29T22:03:38.284 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.284 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.284 DEBUG:boto:Params: {} 2023-09-29T22:03:38.284 DEBUG:boto:Token: None 2023-09-29T22:03:38.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044ee671a6bad68d0-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.290 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.290 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.291 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.291 DEBUG:boto:path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.291 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.291 DEBUG:boto:Method: GET 2023-09-29T22:03:38.291 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.291 DEBUG:boto:Data: 2023-09-29T22:03:38.291 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.291 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.292 DEBUG:boto:Params: {} 2023-09-29T22:03:38.292 DEBUG:boto:Token: None 2023-09-29T22:03:38.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g= 2023-09-29T22:03:38.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g=', 'Content-Length': '0'} 2023-09-29T22:03:38.295 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b8a4cbfd34087a5d-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.295 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.296 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.296 DEBUG:boto:path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.296 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.296 DEBUG:boto:Method: GET 2023-09-29T22:03:38.296 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.296 DEBUG:boto:Data: 2023-09-29T22:03:38.296 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.297 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.297 DEBUG:boto:Params: {} 2023-09-29T22:03:38.297 DEBUG:boto:Token: None 2023-09-29T22:03:38.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g= 2023-09-29T22:03:38.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g=', 'Content-Length': '0'} 2023-09-29T22:03:38.300 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000025c2071033b3e4a6-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.301 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.301 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.301 DEBUG:boto:path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.301 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.301 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.301 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.301 DEBUG:boto:Data: 2023-09-29T22:03:38.301 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.302 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.302 DEBUG:boto:Params: {} 2023-09-29T22:03:38.302 DEBUG:boto:Token: None 2023-09-29T22:03:38.302 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gNWGCT7mOb+hV0jrnJuIDBCX1sE= 2023-09-29T22:03:38.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gNWGCT7mOb+hV0jrnJuIDBCX1sE=', 'Content-Length': '0'} 2023-09-29T22:03:38.304 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d08d230081c035d-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.305 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.305 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.305 DEBUG:boto:path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.305 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.305 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.305 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.305 DEBUG:boto:Data: 2023-09-29T22:03:38.306 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.306 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.306 DEBUG:boto:Params: {} 2023-09-29T22:03:38.306 DEBUG:boto:Token: None 2023-09-29T22:03:38.306 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gNWGCT7mOb+hV0jrnJuIDBCX1sE= 2023-09-29T22:03:38.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gNWGCT7mOb+hV0jrnJuIDBCX1sE=', 'Content-Length': '0'} 2023-09-29T22:03:38.309 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5f5c94d1f120336-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.309 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.309 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.309 DEBUG:boto:path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.309 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.310 DEBUG:boto:Method: GET 2023-09-29T22:03:38.310 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.310 DEBUG:boto:Data: 2023-09-29T22:03:38.310 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.310 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.310 DEBUG:boto:Params: {} 2023-09-29T22:03:38.310 DEBUG:boto:Token: None 2023-09-29T22:03:38.310 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g= 2023-09-29T22:03:38.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g=', 'Content-Length': '0'} 2023-09-29T22:03:38.313 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000756f19d5cc57ba8f-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.314 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.314 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.314 DEBUG:boto:path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.314 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.314 DEBUG:boto:Method: GET 2023-09-29T22:03:38.314 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.315 DEBUG:boto:Data: 2023-09-29T22:03:38.315 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.315 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.315 DEBUG:boto:Params: {} 2023-09-29T22:03:38.315 DEBUG:boto:Token: None 2023-09-29T22:03:38.315 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7e8K1DAqREpBLGcWpWU7padLejgV62q 2023-09-29T22:03:38.315 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g= 2023-09-29T22:03:38.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8MEjZn24Yj4Se+wP/uPD95KG51g=', 'Content-Length': '0'} 2023-09-29T22:03:38.318 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7e8K1DAqREpBLGcWpWU7padLejgV62q'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007821f0a297f479f0-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.318 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.318 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.318 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.319 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.319 DEBUG:boto:Method: GET 2023-09-29T22:03:38.319 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.319 DEBUG:boto:Data: 2023-09-29T22:03:38.319 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.319 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.319 DEBUG:boto:Params: {} 2023-09-29T22:03:38.319 DEBUG:boto:Token: None 2023-09-29T22:03:38.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbdb9eadefb25a4d-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.325 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.325 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.325 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.325 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.326 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.326 DEBUG:boto:Method: GET 2023-09-29T22:03:38.326 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.326 DEBUG:boto:Data: 2023-09-29T22:03:38.326 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.326 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.326 DEBUG:boto:Params: {} 2023-09-29T22:03:38.326 DEBUG:boto:Token: None 2023-09-29T22:03:38.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df4b667621410cb-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.331 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.332 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.332 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.332 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.332 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.332 DEBUG:boto:Data: 2023-09-29T22:03:38.332 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.332 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.332 DEBUG:boto:Params: {} 2023-09-29T22:03:38.333 DEBUG:boto:Token: None 2023-09-29T22:03:38.333 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.337 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000007ee3e3134b4ac6-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.338 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.338 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.338 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.338 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.338 DEBUG:boto:Data: 2023-09-29T22:03:38.338 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.338 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.339 DEBUG:boto:Params: {} 2023-09-29T22:03:38.339 DEBUG:boto:Token: None 2023-09-29T22:03:38.339 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.339 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.344 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ab7362859aebb09-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.344 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.344 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.344 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.345 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.345 DEBUG:boto:Method: GET 2023-09-29T22:03:38.345 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.345 DEBUG:boto:Data: 2023-09-29T22:03:38.345 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.345 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.345 DEBUG:boto:Params: {} 2023-09-29T22:03:38.345 DEBUG:boto:Token: None 2023-09-29T22:03:38.346 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.346 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.348 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000029647730e00934eb-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.390 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.390 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.391 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.391 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.391 DEBUG:boto:Method: GET 2023-09-29T22:03:38.391 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.391 DEBUG:boto:Data: 2023-09-29T22:03:38.391 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.391 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.391 DEBUG:boto:Params: {} 2023-09-29T22:03:38.392 DEBUG:boto:Token: None 2023-09-29T22:03:38.392 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.395 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078841540ed102ba6-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.395 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.395 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.395 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.395 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.395 DEBUG:boto:Method: GET 2023-09-29T22:03:38.395 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.396 DEBUG:boto:Data: 2023-09-29T22:03:38.396 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.396 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.396 DEBUG:boto:Params: {} 2023-09-29T22:03:38.396 DEBUG:boto:Token: None 2023-09-29T22:03:38.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8a473eda6df6972-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.446 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.447 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.447 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.447 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.447 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.447 DEBUG:boto:Method: GET 2023-09-29T22:03:38.447 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.447 DEBUG:boto:Data: 2023-09-29T22:03:38.448 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.448 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.448 DEBUG:boto:Params: {} 2023-09-29T22:03:38.448 DEBUG:boto:Token: None 2023-09-29T22:03:38.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2b6916db5346a5d-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.454 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.455 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.455 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.455 DEBUG:boto:path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.455 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.455 DEBUG:boto:Method: GET 2023-09-29T22:03:38.455 DEBUG:boto:Path: /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.455 DEBUG:boto:Data: 2023-09-29T22:03:38.455 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.456 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.456 DEBUG:boto:Params: {} 2023-09-29T22:03:38.456 DEBUG:boto:Token: None 2023-09-29T22:03:38.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA= 2023-09-29T22:03:38.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA=', 'Content-Length': '0'} 2023-09-29T22:03:38.459 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a61da72d31211196-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.459 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.460 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.460 DEBUG:boto:path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.460 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.460 DEBUG:boto:Method: GET 2023-09-29T22:03:38.460 DEBUG:boto:Path: /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.460 DEBUG:boto:Data: 2023-09-29T22:03:38.460 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.460 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.460 DEBUG:boto:Params: {} 2023-09-29T22:03:38.461 DEBUG:boto:Token: None 2023-09-29T22:03:38.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA= 2023-09-29T22:03:38.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA=', 'Content-Length': '0'} 2023-09-29T22:03:38.464 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007eb90e79d3e8677e-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.464 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.464 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.464 DEBUG:boto:path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.464 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.464 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.465 DEBUG:boto:Path: /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.465 DEBUG:boto:Data: 2023-09-29T22:03:38.465 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.465 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.465 DEBUG:boto:Params: {} 2023-09-29T22:03:38.465 DEBUG:boto:Token: None 2023-09-29T22:03:38.465 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v7eiIfamvKRGK647n6AlZILzQ4E= 2023-09-29T22:03:38.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v7eiIfamvKRGK647n6AlZILzQ4E=', 'Content-Length': '0'} 2023-09-29T22:03:38.468 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000020fec85dea2e68aa-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.469 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.469 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.469 DEBUG:boto:path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.469 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.469 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.469 DEBUG:boto:Path: /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.469 DEBUG:boto:Data: 2023-09-29T22:03:38.469 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.470 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.470 DEBUG:boto:Params: {} 2023-09-29T22:03:38.470 DEBUG:boto:Token: None 2023-09-29T22:03:38.470 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.470 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v7eiIfamvKRGK647n6AlZILzQ4E= 2023-09-29T22:03:38.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v7eiIfamvKRGK647n6AlZILzQ4E=', 'Content-Length': '0'} 2023-09-29T22:03:38.473 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000b7cf32a0a8cfcf45-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.473 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.473 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.473 DEBUG:boto:path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.473 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.473 DEBUG:boto:Method: GET 2023-09-29T22:03:38.474 DEBUG:boto:Path: /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.474 DEBUG:boto:Data: 2023-09-29T22:03:38.474 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.474 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.474 DEBUG:boto:Params: {} 2023-09-29T22:03:38.474 DEBUG:boto:Token: None 2023-09-29T22:03:38.474 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA= 2023-09-29T22:03:38.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA=', 'Content-Length': '0'} 2023-09-29T22:03:38.477 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e81f67d0ab377400-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.522 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.523 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.523 DEBUG:boto:path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.523 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.523 DEBUG:boto:Method: GET 2023-09-29T22:03:38.523 DEBUG:boto:Path: /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.523 DEBUG:boto:Data: 2023-09-29T22:03:38.523 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.523 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.523 DEBUG:boto:Params: {} 2023-09-29T22:03:38.524 DEBUG:boto:Token: None 2023-09-29T22:03:38.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc 2023-09-29T22:03:38.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA= 2023-09-29T22:03:38.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OY3+kAlzIIztM+8KILbI9rQinQA=', 'Content-Length': '0'} 2023-09-29T22:03:38.527 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'xYKB5fMv0wyZy-frsPf-.TEL0ca4Pgc'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000011e0ad332395cc9-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.527 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.527 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.527 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.528 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.528 DEBUG:boto:Method: GET 2023-09-29T22:03:38.528 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.528 DEBUG:boto:Data: 2023-09-29T22:03:38.528 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.528 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.528 DEBUG:boto:Params: {} 2023-09-29T22:03:38.528 DEBUG:boto:Token: None 2023-09-29T22:03:38.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e812291060471073-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.574 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.575 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.575 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.575 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.575 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.576 DEBUG:boto:Method: GET 2023-09-29T22:03:38.576 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.576 DEBUG:boto:Data: 2023-09-29T22:03:38.576 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.576 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.576 DEBUG:boto:Params: {} 2023-09-29T22:03:38.576 DEBUG:boto:Token: None 2023-09-29T22:03:38.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f9ca7fb8d404dfc-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.582 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.583 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.583 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.583 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.583 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.583 DEBUG:boto:Data: 2023-09-29T22:03:38.583 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.583 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.583 DEBUG:boto:Params: {} 2023-09-29T22:03:38.584 DEBUG:boto:Token: None 2023-09-29T22:03:38.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.589 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006fce67fe70e305fe-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.590 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.590 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.590 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.590 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.590 DEBUG:boto:Data: 2023-09-29T22:03:38.590 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.590 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.590 DEBUG:boto:Params: {} 2023-09-29T22:03:38.591 DEBUG:boto:Token: None 2023-09-29T22:03:38.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.596 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000979cc7040d2afa36-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.596 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.597 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.597 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.597 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.597 DEBUG:boto:Method: GET 2023-09-29T22:03:38.597 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.597 DEBUG:boto:Data: 2023-09-29T22:03:38.597 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.597 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.597 DEBUG:boto:Params: {} 2023-09-29T22:03:38.598 DEBUG:boto:Token: None 2023-09-29T22:03:38.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.598 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.601 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000c7240078fcaa715-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.642 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.643 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.643 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.643 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.643 DEBUG:boto:Method: GET 2023-09-29T22:03:38.643 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.643 DEBUG:boto:Data: 2023-09-29T22:03:38.643 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.644 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.644 DEBUG:boto:Params: {} 2023-09-29T22:03:38.644 DEBUG:boto:Token: None 2023-09-29T22:03:38.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.647 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000062257c4e0d738576-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.648 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.648 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.648 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.648 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.648 DEBUG:boto:Method: GET 2023-09-29T22:03:38.648 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.648 DEBUG:boto:Data: 2023-09-29T22:03:38.648 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.649 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.649 DEBUG:boto:Params: {} 2023-09-29T22:03:38.649 DEBUG:boto:Token: None 2023-09-29T22:03:38.649 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e10918b534497c48-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.654 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.654 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.654 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.654 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.654 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.654 DEBUG:boto:Method: GET 2023-09-29T22:03:38.655 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.655 DEBUG:boto:Data: 2023-09-29T22:03:38.655 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.655 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.655 DEBUG:boto:Params: {} 2023-09-29T22:03:38.655 DEBUG:boto:Token: None 2023-09-29T22:03:38.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ee81d0e5629034-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.661 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.662 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.662 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.662 DEBUG:boto:path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.662 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.662 DEBUG:boto:Method: GET 2023-09-29T22:03:38.662 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.662 DEBUG:boto:Data: 2023-09-29T22:03:38.662 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.663 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.663 DEBUG:boto:Params: {} 2023-09-29T22:03:38.663 DEBUG:boto:Token: None 2023-09-29T22:03:38.663 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk= 2023-09-29T22:03:38.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk=', 'Content-Length': '0'} 2023-09-29T22:03:38.665 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007789d0e693be18fc-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.666 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.666 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.666 DEBUG:boto:path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.666 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.666 DEBUG:boto:Method: GET 2023-09-29T22:03:38.666 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.666 DEBUG:boto:Data: 2023-09-29T22:03:38.667 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.667 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.667 DEBUG:boto:Params: {} 2023-09-29T22:03:38.667 DEBUG:boto:Token: None 2023-09-29T22:03:38.667 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.667 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk= 2023-09-29T22:03:38.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk=', 'Content-Length': '0'} 2023-09-29T22:03:38.669 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e263bce07a7f34bf-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.670 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.670 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.670 DEBUG:boto:path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.670 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.670 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.670 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.670 DEBUG:boto:Data: 2023-09-29T22:03:38.670 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.671 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.671 DEBUG:boto:Params: {} 2023-09-29T22:03:38.671 DEBUG:boto:Token: None 2023-09-29T22:03:38.671 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.671 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rg1zYLNZLvP3DUUp5LmMkkhVwEc= 2023-09-29T22:03:38.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rg1zYLNZLvP3DUUp5LmMkkhVwEc=', 'Content-Length': '0'} 2023-09-29T22:03:38.673 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ef284ca5c0e29fc9-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.673 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.674 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.674 DEBUG:boto:path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.674 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.674 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.674 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.674 DEBUG:boto:Data: 2023-09-29T22:03:38.674 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.674 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.675 DEBUG:boto:Params: {} 2023-09-29T22:03:38.675 DEBUG:boto:Token: None 2023-09-29T22:03:38.675 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rg1zYLNZLvP3DUUp5LmMkkhVwEc= 2023-09-29T22:03:38.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rg1zYLNZLvP3DUUp5LmMkkhVwEc=', 'Content-Length': '0'} 2023-09-29T22:03:38.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000078157155af338847-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.678 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.678 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.678 DEBUG:boto:path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.679 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.679 DEBUG:boto:Method: GET 2023-09-29T22:03:38.679 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.679 DEBUG:boto:Data: 2023-09-29T22:03:38.679 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.679 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.679 DEBUG:boto:Params: {} 2023-09-29T22:03:38.679 DEBUG:boto:Token: None 2023-09-29T22:03:38.679 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk= 2023-09-29T22:03:38.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk=', 'Content-Length': '0'} 2023-09-29T22:03:38.682 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000002e142d19a4ec9a9-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.726 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.727 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.727 DEBUG:boto:path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.727 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.727 DEBUG:boto:Method: GET 2023-09-29T22:03:38.727 DEBUG:boto:Path: /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.727 DEBUG:boto:Data: 2023-09-29T22:03:38.727 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.727 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.728 DEBUG:boto:Params: {} 2023-09-29T22:03:38.728 DEBUG:boto:Token: None 2023-09-29T22:03:38.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=7YyFpAWutMJGjWIJTn9o65BehGeETgW 2023-09-29T22:03:38.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk= 2023-09-29T22:03:38.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kImV7kMBzZNhPilS1WEamTkCTRk=', 'Content-Length': '0'} 2023-09-29T22:03:38.731 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '7YyFpAWutMJGjWIJTn9o65BehGeETgW'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca92f32e40ee3592-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.731 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.732 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.732 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.732 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.732 DEBUG:boto:Method: GET 2023-09-29T22:03:38.732 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.732 DEBUG:boto:Data: 2023-09-29T22:03:38.732 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.732 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.732 DEBUG:boto:Params: {} 2023-09-29T22:03:38.733 DEBUG:boto:Token: None 2023-09-29T22:03:38.733 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037c32b4bfd932476-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.778 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.779 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.779 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.779 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.779 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.780 DEBUG:boto:Method: GET 2023-09-29T22:03:38.780 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.780 DEBUG:boto:Data: 2023-09-29T22:03:38.780 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.780 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.780 DEBUG:boto:Params: {} 2023-09-29T22:03:38.780 DEBUG:boto:Token: None 2023-09-29T22:03:38.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b79fc6bbd6e46fe2-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.785 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.786 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.786 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.786 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.786 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.786 DEBUG:boto:Data: 2023-09-29T22:03:38.786 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.786 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.787 DEBUG:boto:Params: {} 2023-09-29T22:03:38.787 DEBUG:boto:Token: None 2023-09-29T22:03:38.787 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.787 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.792 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f6d1c2432f0e8152-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.792 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.792 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.793 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.793 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.793 DEBUG:boto:Data: 2023-09-29T22:03:38.793 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.793 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.793 DEBUG:boto:Params: {} 2023-09-29T22:03:38.793 DEBUG:boto:Token: None 2023-09-29T22:03:38.793 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.794 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:38.798 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000047078b78648e49c9-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.799 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.799 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.799 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.799 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.799 DEBUG:boto:Method: GET 2023-09-29T22:03:38.799 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.799 DEBUG:boto:Data: 2023-09-29T22:03:38.799 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.799 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.800 DEBUG:boto:Params: {} 2023-09-29T22:03:38.800 DEBUG:boto:Token: None 2023-09-29T22:03:38.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.803 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008ce4c4cccad7e4ac-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.846 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.847 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.847 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.847 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.847 DEBUG:boto:Method: GET 2023-09-29T22:03:38.847 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.847 DEBUG:boto:Data: 2023-09-29T22:03:38.847 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.847 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.848 DEBUG:boto:Params: {} 2023-09-29T22:03:38.848 DEBUG:boto:Token: None 2023-09-29T22:03:38.848 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:38.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA= 2023-09-29T22:03:38.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9rLjl1nr1u3hl43dNX0a3g2yPVA=', 'Content-Length': '0'} 2023-09-29T22:03:38.851 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000012599c6a28f1c91c-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.851 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.851 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.852 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.852 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.852 DEBUG:boto:Method: GET 2023-09-29T22:03:38.852 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.852 DEBUG:boto:Data: 2023-09-29T22:03:38.852 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.852 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.852 DEBUG:boto:Params: {} 2023-09-29T22:03:38.853 DEBUG:boto:Token: None 2023-09-29T22:03:38.853 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.853 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013437e691b329ac2-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.902 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.903 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.903 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.903 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.903 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.903 DEBUG:boto:Method: GET 2023-09-29T22:03:38.903 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.903 DEBUG:boto:Data: 2023-09-29T22:03:38.904 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.904 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.904 DEBUG:boto:Params: {} 2023-09-29T22:03:38.904 DEBUG:boto:Token: None 2023-09-29T22:03:38.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044f1fa57410e2f76-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.910 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.911 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.911 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.911 DEBUG:boto:path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.911 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.911 DEBUG:boto:Method: GET 2023-09-29T22:03:38.911 DEBUG:boto:Path: /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.912 DEBUG:boto:Data: 2023-09-29T22:03:38.912 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.912 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.912 DEBUG:boto:Params: {} 2023-09-29T22:03:38.912 DEBUG:boto:Token: None 2023-09-29T22:03:38.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE= 2023-09-29T22:03:38.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE=', 'Content-Length': '0'} 2023-09-29T22:03:38.915 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000565ea0c763a1ef5b-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.916 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.916 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.916 DEBUG:boto:path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.916 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.916 DEBUG:boto:Method: GET 2023-09-29T22:03:38.917 DEBUG:boto:Path: /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.917 DEBUG:boto:Data: 2023-09-29T22:03:38.917 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.917 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.917 DEBUG:boto:Params: {} 2023-09-29T22:03:38.918 DEBUG:boto:Token: None 2023-09-29T22:03:38.918 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.918 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE= 2023-09-29T22:03:38.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE=', 'Content-Length': '0'} 2023-09-29T22:03:38.921 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ffd1227bf742736-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.921 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.921 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.921 DEBUG:boto:path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.922 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.922 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.922 DEBUG:boto:Path: /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.922 DEBUG:boto:Data: 2023-09-29T22:03:38.922 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.922 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.922 DEBUG:boto:Params: {} 2023-09-29T22:03:38.922 DEBUG:boto:Token: None 2023-09-29T22:03:38.923 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BQsiAHJS98JN8tZMJRMq4jx3E+4= 2023-09-29T22:03:38.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BQsiAHJS98JN8tZMJRMq4jx3E+4=', 'Content-Length': '0'} 2023-09-29T22:03:38.925 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000563e3302c216a6ed-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.926 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.926 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.926 DEBUG:boto:path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.926 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.926 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.926 DEBUG:boto:Path: /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.926 DEBUG:boto:Data: 2023-09-29T22:03:38.926 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.927 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.927 DEBUG:boto:Params: {} 2023-09-29T22:03:38.927 DEBUG:boto:Token: None 2023-09-29T22:03:38.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BQsiAHJS98JN8tZMJRMq4jx3E+4= 2023-09-29T22:03:38.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BQsiAHJS98JN8tZMJRMq4jx3E+4=', 'Content-Length': '0'} 2023-09-29T22:03:38.930 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f1213db536333eca-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:38.930 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.931 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.931 DEBUG:boto:path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.931 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.931 DEBUG:boto:Method: GET 2023-09-29T22:03:38.931 DEBUG:boto:Path: /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.931 DEBUG:boto:Data: 2023-09-29T22:03:38.931 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.931 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.931 DEBUG:boto:Params: {} 2023-09-29T22:03:38.932 DEBUG:boto:Token: None 2023-09-29T22:03:38.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE= 2023-09-29T22:03:38.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE=', 'Content-Length': '0'} 2023-09-29T22:03:38.935 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fcac91ff589bc7ab-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.935 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.935 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.935 DEBUG:boto:path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.935 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.935 DEBUG:boto:Method: GET 2023-09-29T22:03:38.935 DEBUG:boto:Path: /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.936 DEBUG:boto:Data: 2023-09-29T22:03:38.936 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.936 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.936 DEBUG:boto:Params: {} 2023-09-29T22:03:38.936 DEBUG:boto:Token: None 2023-09-29T22:03:38.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj?versionId=ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.- 2023-09-29T22:03:38.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE= 2023-09-29T22:03:38.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAaXc3WQpJ6vyU5XmUOW00HtguE=', 'Content-Length': '0'} 2023-09-29T22:03:38.939 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'ZXhw9NcJ43Sw2vgJjs.y.bBIqR-P4.-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f266ed503d34958c-00651749ba-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.939 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.939 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.940 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.940 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.940 DEBUG:boto:Method: GET 2023-09-29T22:03:38.940 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.940 DEBUG:boto:Data: 2023-09-29T22:03:38.940 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.940 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.940 DEBUG:boto:Params: {} 2023-09-29T22:03:38.941 DEBUG:boto:Token: None 2023-09-29T22:03:38.941 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.945 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000274d1a7a50d6c0df-00651749ba-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.986 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.987 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:38.987 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:38.987 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.987 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:38.988 DEBUG:boto:Method: GET 2023-09-29T22:03:38.988 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:38.988 DEBUG:boto:Data: 2023-09-29T22:03:38.988 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:38.988 DEBUG:boto:Port: 8000 2023-09-29T22:03:38.988 DEBUG:boto:Params: {} 2023-09-29T22:03:38.988 DEBUG:boto:Token: None 2023-09-29T22:03:38.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/ 2023-09-29T22:03:38.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c= 2023-09-29T22:03:38.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sLuXII1ZuOk8WbcaZmM09IQES2c=', 'Content-Length': '0'} 2023-09-29T22:03:38.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a4b8d94c388979-00651749ba-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:38.994 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:38.994 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:38.995 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:38.995 DEBUG:boto:Method: HEAD 2023-09-29T22:03:38.995 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:38.995 DEBUG:boto:Data: 2023-09-29T22:03:38.996 DEBUG:boto:Headers: {} 2023-09-29T22:03:38.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:38.996 DEBUG:boto:Port: 8001 2023-09-29T22:03:38.996 DEBUG:boto:Params: {} 2023-09-29T22:03:38.996 DEBUG:boto:Token: None 2023-09-29T22:03:38.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:38 GMT /hcnphz-13/obj 2023-09-29T22:03:38.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ= 2023-09-29T22:03:38.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFdJ4ZhySvpWeHNF0e1suAFVVtQ=', 'Content-Length': '0'} 2023-09-29T22:03:39.002 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000682ca15573771650-00651749ba-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.003 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.003 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.003 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.003 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.003 DEBUG:boto:Data: 2023-09-29T22:03:39.003 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.003 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.003 DEBUG:boto:Params: {} 2023-09-29T22:03:39.004 DEBUG:boto:Token: None 2023-09-29T22:03:39.004 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.010 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c6348661abf6187a-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.011 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.011 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.011 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.011 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.011 DEBUG:boto:Method: GET 2023-09-29T22:03:39.011 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.011 DEBUG:boto:Data: 2023-09-29T22:03:39.011 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.011 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.012 DEBUG:boto:Params: {} 2023-09-29T22:03:39.012 DEBUG:boto:Token: None 2023-09-29T22:03:39.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.015 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d29bafc10bf3f94e-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.062 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.063 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.063 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.063 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.063 DEBUG:boto:Method: GET 2023-09-29T22:03:39.063 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.063 DEBUG:boto:Data: 2023-09-29T22:03:39.063 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.064 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.064 DEBUG:boto:Params: {} 2023-09-29T22:03:39.064 DEBUG:boto:Token: None 2023-09-29T22:03:39.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.067 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a3f6806708f88825-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.068 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.068 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.068 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.068 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.068 DEBUG:boto:Method: GET 2023-09-29T22:03:39.068 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.068 DEBUG:boto:Data: 2023-09-29T22:03:39.069 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.069 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.069 DEBUG:boto:Params: {} 2023-09-29T22:03:39.069 DEBUG:boto:Token: None 2023-09-29T22:03:39.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4a0a31ee9eaca6d-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.114 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.115 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.115 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.115 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.115 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.115 DEBUG:boto:Method: GET 2023-09-29T22:03:39.116 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.116 DEBUG:boto:Data: 2023-09-29T22:03:39.116 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.116 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.116 DEBUG:boto:Params: {} 2023-09-29T22:03:39.116 DEBUG:boto:Token: None 2023-09-29T22:03:39.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.122 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000634da822717daded-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.122 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.122 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.123 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.123 DEBUG:boto:path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.123 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.123 DEBUG:boto:Method: GET 2023-09-29T22:03:39.123 DEBUG:boto:Path: /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.123 DEBUG:boto:Data: 2023-09-29T22:03:39.123 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.123 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.123 DEBUG:boto:Params: {} 2023-09-29T22:03:39.124 DEBUG:boto:Token: None 2023-09-29T22:03:39.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA= 2023-09-29T22:03:39.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA=', 'Content-Length': '0'} 2023-09-29T22:03:39.127 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002825f49a07242efe-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.127 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.127 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.127 DEBUG:boto:path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.127 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.127 DEBUG:boto:Method: GET 2023-09-29T22:03:39.127 DEBUG:boto:Path: /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.128 DEBUG:boto:Data: 2023-09-29T22:03:39.128 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.128 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.128 DEBUG:boto:Params: {} 2023-09-29T22:03:39.128 DEBUG:boto:Token: None 2023-09-29T22:03:39.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA= 2023-09-29T22:03:39.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA=', 'Content-Length': '0'} 2023-09-29T22:03:39.131 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ea85d12544b82663-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.174 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.175 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.175 DEBUG:boto:path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.175 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.175 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.175 DEBUG:boto:Path: /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.175 DEBUG:boto:Data: 2023-09-29T22:03:39.175 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.175 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.176 DEBUG:boto:Params: {} 2023-09-29T22:03:39.176 DEBUG:boto:Token: None 2023-09-29T22:03:39.176 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oNqbvVmVGkTJ/GV5U8lLrO1DfK8= 2023-09-29T22:03:39.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oNqbvVmVGkTJ/GV5U8lLrO1DfK8=', 'Content-Length': '0'} 2023-09-29T22:03:39.179 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005b3bdb8700a4ee33-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.179 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.179 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.179 DEBUG:boto:path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.179 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.180 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.180 DEBUG:boto:Path: /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.180 DEBUG:boto:Data: 2023-09-29T22:03:39.180 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.180 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.180 DEBUG:boto:Params: {} 2023-09-29T22:03:39.180 DEBUG:boto:Token: None 2023-09-29T22:03:39.180 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oNqbvVmVGkTJ/GV5U8lLrO1DfK8= 2023-09-29T22:03:39.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oNqbvVmVGkTJ/GV5U8lLrO1DfK8=', 'Content-Length': '0'} 2023-09-29T22:03:39.183 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e84f44dcb72a7da9-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.184 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.184 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.184 DEBUG:boto:path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.184 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.184 DEBUG:boto:Method: GET 2023-09-29T22:03:39.184 DEBUG:boto:Path: /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.184 DEBUG:boto:Data: 2023-09-29T22:03:39.184 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.185 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.185 DEBUG:boto:Params: {} 2023-09-29T22:03:39.185 DEBUG:boto:Token: None 2023-09-29T22:03:39.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.185 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA= 2023-09-29T22:03:39.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA=', 'Content-Length': '0'} 2023-09-29T22:03:39.188 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f8299a7d06186fd1-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.188 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.188 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.188 DEBUG:boto:path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.189 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.189 DEBUG:boto:Method: GET 2023-09-29T22:03:39.189 DEBUG:boto:Path: /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.189 DEBUG:boto:Data: 2023-09-29T22:03:39.189 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.189 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.189 DEBUG:boto:Params: {} 2023-09-29T22:03:39.189 DEBUG:boto:Token: None 2023-09-29T22:03:39.189 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=19w3gH6L6XVWwe2N7qkyQteADnJRlTS 2023-09-29T22:03:39.190 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA= 2023-09-29T22:03:39.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OZN2ZVez6DGU/QhAglu9VE+fnOA=', 'Content-Length': '0'} 2023-09-29T22:03:39.192 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', '19w3gH6L6XVWwe2N7qkyQteADnJRlTS'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001671daa7bfcd853c-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.193 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.193 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.193 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.193 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.193 DEBUG:boto:Method: GET 2023-09-29T22:03:39.193 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.193 DEBUG:boto:Data: 2023-09-29T22:03:39.194 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.194 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.194 DEBUG:boto:Params: {} 2023-09-29T22:03:39.194 DEBUG:boto:Token: None 2023-09-29T22:03:39.194 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000600e550219a64359-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.246 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.247 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.247 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.247 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.247 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.247 DEBUG:boto:Method: GET 2023-09-29T22:03:39.247 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.249 DEBUG:boto:Data: 2023-09-29T22:03:39.249 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.249 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.249 DEBUG:boto:Params: {} 2023-09-29T22:03:39.250 DEBUG:boto:Token: None 2023-09-29T22:03:39.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092d09676ad4c83e3-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.255 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.255 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.256 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.256 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.256 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.256 DEBUG:boto:Data: 2023-09-29T22:03:39.256 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.256 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.256 DEBUG:boto:Params: {} 2023-09-29T22:03:39.256 DEBUG:boto:Token: None 2023-09-29T22:03:39.257 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.262 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003abe35b3d0776195-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.263 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.263 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.263 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.263 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.263 DEBUG:boto:Data: 2023-09-29T22:03:39.263 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.263 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.264 DEBUG:boto:Params: {} 2023-09-29T22:03:39.264 DEBUG:boto:Token: None 2023-09-29T22:03:39.264 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.270 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000abceff9c96a5b628-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.270 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.270 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.270 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.271 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.271 DEBUG:boto:Method: GET 2023-09-29T22:03:39.271 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.271 DEBUG:boto:Data: 2023-09-29T22:03:39.271 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.271 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.271 DEBUG:boto:Params: {} 2023-09-29T22:03:39.271 DEBUG:boto:Token: None 2023-09-29T22:03:39.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.275 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a791c9ea0a79d4b6-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.318 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.319 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.319 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.319 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.319 DEBUG:boto:Method: GET 2023-09-29T22:03:39.319 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.319 DEBUG:boto:Data: 2023-09-29T22:03:39.319 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.319 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.319 DEBUG:boto:Params: {} 2023-09-29T22:03:39.320 DEBUG:boto:Token: None 2023-09-29T22:03:39.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.323 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a7473ea7cfaa8bb0-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.323 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.323 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.323 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.323 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.324 DEBUG:boto:Method: GET 2023-09-29T22:03:39.324 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.324 DEBUG:boto:Data: 2023-09-29T22:03:39.324 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.324 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.324 DEBUG:boto:Params: {} 2023-09-29T22:03:39.324 DEBUG:boto:Token: None 2023-09-29T22:03:39.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eda13262ad64bdda-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.370 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.371 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.371 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.371 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.371 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.372 DEBUG:boto:Method: GET 2023-09-29T22:03:39.372 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.372 DEBUG:boto:Data: 2023-09-29T22:03:39.372 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.372 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.372 DEBUG:boto:Params: {} 2023-09-29T22:03:39.372 DEBUG:boto:Token: None 2023-09-29T22:03:39.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e0b7bd2cfb08d86-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.378 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.378 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.378 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.378 DEBUG:boto:path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.379 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.379 DEBUG:boto:Method: GET 2023-09-29T22:03:39.379 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.379 DEBUG:boto:Data: 2023-09-29T22:03:39.379 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.379 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.379 DEBUG:boto:Params: {} 2023-09-29T22:03:39.379 DEBUG:boto:Token: None 2023-09-29T22:03:39.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs= 2023-09-29T22:03:39.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs=', 'Content-Length': '0'} 2023-09-29T22:03:39.383 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f2afaea70d2bbc57-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.383 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.383 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.383 DEBUG:boto:path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.383 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.384 DEBUG:boto:Method: GET 2023-09-29T22:03:39.384 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.384 DEBUG:boto:Data: 2023-09-29T22:03:39.384 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.384 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.384 DEBUG:boto:Params: {} 2023-09-29T22:03:39.384 DEBUG:boto:Token: None 2023-09-29T22:03:39.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs= 2023-09-29T22:03:39.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs=', 'Content-Length': '0'} 2023-09-29T22:03:39.387 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000ca46b647a7baa723-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.388 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.388 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.388 DEBUG:boto:path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.388 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.388 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.388 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.388 DEBUG:boto:Data: 2023-09-29T22:03:39.389 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.389 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.389 DEBUG:boto:Params: {} 2023-09-29T22:03:39.389 DEBUG:boto:Token: None 2023-09-29T22:03:39.389 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.389 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Hv2owagHwQazS2bs2UP84AXbWGI= 2023-09-29T22:03:39.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Hv2owagHwQazS2bs2UP84AXbWGI=', 'Content-Length': '0'} 2023-09-29T22:03:39.392 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082eb5d8248e4df68-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.392 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.392 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.392 DEBUG:boto:path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.392 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.393 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.393 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.393 DEBUG:boto:Data: 2023-09-29T22:03:39.393 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.393 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.393 DEBUG:boto:Params: {} 2023-09-29T22:03:39.393 DEBUG:boto:Token: None 2023-09-29T22:03:39.393 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Hv2owagHwQazS2bs2UP84AXbWGI= 2023-09-29T22:03:39.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Hv2owagHwQazS2bs2UP84AXbWGI=', 'Content-Length': '0'} 2023-09-29T22:03:39.396 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002d99f797f8350c02-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.397 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.397 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.397 DEBUG:boto:path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.397 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.397 DEBUG:boto:Method: GET 2023-09-29T22:03:39.397 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.398 DEBUG:boto:Data: 2023-09-29T22:03:39.398 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.398 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.398 DEBUG:boto:Params: {} 2023-09-29T22:03:39.398 DEBUG:boto:Token: None 2023-09-29T22:03:39.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.399 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs= 2023-09-29T22:03:39.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs=', 'Content-Length': '0'} 2023-09-29T22:03:39.401 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d5578e7d793cd419-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.442 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.443 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.443 DEBUG:boto:path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.443 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.443 DEBUG:boto:Method: GET 2023-09-29T22:03:39.444 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.444 DEBUG:boto:Data: 2023-09-29T22:03:39.444 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.445 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.445 DEBUG:boto:Params: {} 2023-09-29T22:03:39.445 DEBUG:boto:Token: None 2023-09-29T22:03:39.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=U8xudiwYAFkZ2HZnqEX0HOcHshGnync 2023-09-29T22:03:39.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs= 2023-09-29T22:03:39.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IB8bEniw/Eq8Zb60eM5pUpLssQs=', 'Content-Length': '0'} 2023-09-29T22:03:39.448 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U8xudiwYAFkZ2HZnqEX0HOcHshGnync'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000077d094c05e6b4449-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.449 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.449 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.449 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.449 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.449 DEBUG:boto:Method: GET 2023-09-29T22:03:39.449 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.449 DEBUG:boto:Data: 2023-09-29T22:03:39.449 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.450 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.450 DEBUG:boto:Params: {} 2023-09-29T22:03:39.450 DEBUG:boto:Token: None 2023-09-29T22:03:39.450 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.450 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.455 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d323afe34b55ab53-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.498 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.499 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.499 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.499 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.499 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.499 DEBUG:boto:Method: GET 2023-09-29T22:03:39.499 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.499 DEBUG:boto:Data: 2023-09-29T22:03:39.499 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.500 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.500 DEBUG:boto:Params: {} 2023-09-29T22:03:39.500 DEBUG:boto:Token: None 2023-09-29T22:03:39.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.505 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a5d160b6131f72b-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.506 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.506 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.506 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.506 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.507 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.507 DEBUG:boto:Data: 2023-09-29T22:03:39.507 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.507 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.507 DEBUG:boto:Params: {} 2023-09-29T22:03:39.507 DEBUG:boto:Token: None 2023-09-29T22:03:39.507 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.507 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.513 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000061e40b217d9f08d6-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.513 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.513 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.513 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.514 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.514 DEBUG:boto:Data: 2023-09-29T22:03:39.514 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.514 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.514 DEBUG:boto:Params: {} 2023-09-29T22:03:39.514 DEBUG:boto:Token: None 2023-09-29T22:03:39.514 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.519 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d044956fcc4ee87e-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.520 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.520 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.520 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.520 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.520 DEBUG:boto:Method: GET 2023-09-29T22:03:39.520 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.520 DEBUG:boto:Data: 2023-09-29T22:03:39.521 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.521 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.521 DEBUG:boto:Params: {} 2023-09-29T22:03:39.521 DEBUG:boto:Token: None 2023-09-29T22:03:39.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.524 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aae79be06bb9ace3-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.566 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.567 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.567 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.567 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.567 DEBUG:boto:Method: GET 2023-09-29T22:03:39.567 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.567 DEBUG:boto:Data: 2023-09-29T22:03:39.567 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.567 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.568 DEBUG:boto:Params: {} 2023-09-29T22:03:39.568 DEBUG:boto:Token: None 2023-09-29T22:03:39.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.571 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e0365f62a28ab217-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.571 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.571 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.572 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.572 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.572 DEBUG:boto:Method: GET 2023-09-29T22:03:39.572 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.572 DEBUG:boto:Data: 2023-09-29T22:03:39.572 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.572 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.573 DEBUG:boto:Params: {} 2023-09-29T22:03:39.573 DEBUG:boto:Token: None 2023-09-29T22:03:39.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070739b6729f25c5c-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.578 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.579 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.579 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.579 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.579 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.579 DEBUG:boto:Method: GET 2023-09-29T22:03:39.579 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.579 DEBUG:boto:Data: 2023-09-29T22:03:39.580 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.580 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.580 DEBUG:boto:Params: {} 2023-09-29T22:03:39.580 DEBUG:boto:Token: None 2023-09-29T22:03:39.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005d13bf3c0293680-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.585 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.586 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.586 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.586 DEBUG:boto:path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.586 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.586 DEBUG:boto:Method: GET 2023-09-29T22:03:39.587 DEBUG:boto:Path: /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.587 DEBUG:boto:Data: 2023-09-29T22:03:39.587 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.587 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.587 DEBUG:boto:Params: {} 2023-09-29T22:03:39.587 DEBUG:boto:Token: None 2023-09-29T22:03:39.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q= 2023-09-29T22:03:39.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q=', 'Content-Length': '0'} 2023-09-29T22:03:39.591 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000e2b20ffaa5a9889-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.591 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.591 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.591 DEBUG:boto:path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.591 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.592 DEBUG:boto:Method: GET 2023-09-29T22:03:39.592 DEBUG:boto:Path: /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.592 DEBUG:boto:Data: 2023-09-29T22:03:39.592 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.592 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.592 DEBUG:boto:Params: {} 2023-09-29T22:03:39.592 DEBUG:boto:Token: None 2023-09-29T22:03:39.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q= 2023-09-29T22:03:39.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q=', 'Content-Length': '0'} 2023-09-29T22:03:39.595 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001cab3f8a6b049c97-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.596 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.596 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.596 DEBUG:boto:path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.596 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.596 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.596 DEBUG:boto:Path: /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.596 DEBUG:boto:Data: 2023-09-29T22:03:39.596 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.597 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.597 DEBUG:boto:Params: {} 2023-09-29T22:03:39.597 DEBUG:boto:Token: None 2023-09-29T22:03:39.597 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NRbxwyaAjZ8Y+cinNlDVJZhrgNU= 2023-09-29T22:03:39.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NRbxwyaAjZ8Y+cinNlDVJZhrgNU=', 'Content-Length': '0'} 2023-09-29T22:03:39.600 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007cd36a524486bba1-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.600 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.601 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.601 DEBUG:boto:path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.601 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.601 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.601 DEBUG:boto:Path: /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.601 DEBUG:boto:Data: 2023-09-29T22:03:39.601 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.601 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.602 DEBUG:boto:Params: {} 2023-09-29T22:03:39.602 DEBUG:boto:Token: None 2023-09-29T22:03:39.602 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.602 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NRbxwyaAjZ8Y+cinNlDVJZhrgNU= 2023-09-29T22:03:39.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NRbxwyaAjZ8Y+cinNlDVJZhrgNU=', 'Content-Length': '0'} 2023-09-29T22:03:39.605 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000cb72fc8d267c88ee-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.605 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.605 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.605 DEBUG:boto:path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.605 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.605 DEBUG:boto:Method: GET 2023-09-29T22:03:39.606 DEBUG:boto:Path: /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.606 DEBUG:boto:Data: 2023-09-29T22:03:39.606 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.606 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.606 DEBUG:boto:Params: {} 2023-09-29T22:03:39.606 DEBUG:boto:Token: None 2023-09-29T22:03:39.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q= 2023-09-29T22:03:39.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q=', 'Content-Length': '0'} 2023-09-29T22:03:39.610 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000003cb6f670acffd13d-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.650 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.650 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.651 DEBUG:boto:path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.651 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.651 DEBUG:boto:Method: GET 2023-09-29T22:03:39.651 DEBUG:boto:Path: /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.651 DEBUG:boto:Data: 2023-09-29T22:03:39.651 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.651 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.652 DEBUG:boto:Params: {} 2023-09-29T22:03:39.652 DEBUG:boto:Token: None 2023-09-29T22:03:39.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=APoMvqvvam5IdSmRUTel8cjnttV00WZ 2023-09-29T22:03:39.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q= 2023-09-29T22:03:39.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VsuClPve6UkrTepIMcebLYqot/Q=', 'Content-Length': '0'} 2023-09-29T22:03:39.657 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'APoMvqvvam5IdSmRUTel8cjnttV00WZ'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e8c1dbfa7b9e86cb-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.658 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.658 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.658 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.658 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.658 DEBUG:boto:Method: GET 2023-09-29T22:03:39.658 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.658 DEBUG:boto:Data: 2023-09-29T22:03:39.658 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.659 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.659 DEBUG:boto:Params: {} 2023-09-29T22:03:39.659 DEBUG:boto:Token: None 2023-09-29T22:03:39.659 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058acc3aa8a123056-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.664 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.665 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.665 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.665 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.665 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.665 DEBUG:boto:Method: GET 2023-09-29T22:03:39.665 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.666 DEBUG:boto:Data: 2023-09-29T22:03:39.666 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.666 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.666 DEBUG:boto:Params: {} 2023-09-29T22:03:39.666 DEBUG:boto:Token: None 2023-09-29T22:03:39.666 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108be43d5e3bc4e6-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.671 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.672 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.672 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.672 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.672 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.672 DEBUG:boto:Data: 2023-09-29T22:03:39.672 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.673 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.673 DEBUG:boto:Params: {} 2023-09-29T22:03:39.673 DEBUG:boto:Token: None 2023-09-29T22:03:39.673 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.678 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c66090f4c22a04b-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.678 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.679 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.679 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.679 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.679 DEBUG:boto:Data: 2023-09-29T22:03:39.679 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.679 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.679 DEBUG:boto:Params: {} 2023-09-29T22:03:39.679 DEBUG:boto:Token: None 2023-09-29T22:03:39.680 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.685 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1193287f627aab1-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.685 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.685 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.685 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.685 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.685 DEBUG:boto:Method: GET 2023-09-29T22:03:39.686 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.686 DEBUG:boto:Data: 2023-09-29T22:03:39.686 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.686 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.686 DEBUG:boto:Params: {} 2023-09-29T22:03:39.686 DEBUG:boto:Token: None 2023-09-29T22:03:39.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.687 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.689 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ad03c78cf25d051-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.730 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.730 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.731 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.731 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.731 DEBUG:boto:Method: GET 2023-09-29T22:03:39.731 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.731 DEBUG:boto:Data: 2023-09-29T22:03:39.731 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.731 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.731 DEBUG:boto:Params: {} 2023-09-29T22:03:39.731 DEBUG:boto:Token: None 2023-09-29T22:03:39.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.734 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000bd983b5e3465547b-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.735 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.735 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.735 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.735 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.735 DEBUG:boto:Method: GET 2023-09-29T22:03:39.735 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.735 DEBUG:boto:Data: 2023-09-29T22:03:39.736 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.736 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.736 DEBUG:boto:Params: {} 2023-09-29T22:03:39.736 DEBUG:boto:Token: None 2023-09-29T22:03:39.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e69022a1e766a776-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.742 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.742 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.742 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.742 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.742 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.742 DEBUG:boto:Method: GET 2023-09-29T22:03:39.743 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.743 DEBUG:boto:Data: 2023-09-29T22:03:39.743 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.743 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.743 DEBUG:boto:Params: {} 2023-09-29T22:03:39.743 DEBUG:boto:Token: None 2023-09-29T22:03:39.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c079c8fa750f5c2e-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.794 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.795 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.795 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.795 DEBUG:boto:path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.795 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.795 DEBUG:boto:Method: GET 2023-09-29T22:03:39.795 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.795 DEBUG:boto:Data: 2023-09-29T22:03:39.796 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.796 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.796 DEBUG:boto:Params: {} 2023-09-29T22:03:39.796 DEBUG:boto:Token: None 2023-09-29T22:03:39.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo= 2023-09-29T22:03:39.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo=', 'Content-Length': '0'} 2023-09-29T22:03:39.799 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f03d925ef3c9f0a-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.799 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.800 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.800 DEBUG:boto:path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.800 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.800 DEBUG:boto:Method: GET 2023-09-29T22:03:39.800 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.800 DEBUG:boto:Data: 2023-09-29T22:03:39.800 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.800 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.800 DEBUG:boto:Params: {} 2023-09-29T22:03:39.801 DEBUG:boto:Token: None 2023-09-29T22:03:39.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo= 2023-09-29T22:03:39.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo=', 'Content-Length': '0'} 2023-09-29T22:03:39.804 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009479da9a4618a400-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.804 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.804 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.805 DEBUG:boto:path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.805 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.805 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.805 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.805 DEBUG:boto:Data: 2023-09-29T22:03:39.805 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.805 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.805 DEBUG:boto:Params: {} 2023-09-29T22:03:39.806 DEBUG:boto:Token: None 2023-09-29T22:03:39.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vbiy9zAh0ZZnvmlU8TyK70njzUA= 2023-09-29T22:03:39.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vbiy9zAh0ZZnvmlU8TyK70njzUA=', 'Content-Length': '0'} 2023-09-29T22:03:39.809 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f87b20f8d591136-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.809 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.809 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.809 DEBUG:boto:path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.809 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.810 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.810 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.810 DEBUG:boto:Data: 2023-09-29T22:03:39.810 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.810 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.810 DEBUG:boto:Params: {} 2023-09-29T22:03:39.810 DEBUG:boto:Token: None 2023-09-29T22:03:39.810 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vbiy9zAh0ZZnvmlU8TyK70njzUA= 2023-09-29T22:03:39.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vbiy9zAh0ZZnvmlU8TyK70njzUA=', 'Content-Length': '0'} 2023-09-29T22:03:39.813 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000878161156016259a-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.814 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.814 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.814 DEBUG:boto:path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.814 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.814 DEBUG:boto:Method: GET 2023-09-29T22:03:39.814 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.814 DEBUG:boto:Data: 2023-09-29T22:03:39.815 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.815 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.815 DEBUG:boto:Params: {} 2023-09-29T22:03:39.815 DEBUG:boto:Token: None 2023-09-29T22:03:39.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.815 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo= 2023-09-29T22:03:39.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo=', 'Content-Length': '0'} 2023-09-29T22:03:39.818 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000038a792d62095349-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.858 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.858 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.859 DEBUG:boto:path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.859 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.859 DEBUG:boto:Method: GET 2023-09-29T22:03:39.859 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.859 DEBUG:boto:Data: 2023-09-29T22:03:39.859 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.859 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.859 DEBUG:boto:Params: {} 2023-09-29T22:03:39.860 DEBUG:boto:Token: None 2023-09-29T22:03:39.860 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp 2023-09-29T22:03:39.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo= 2023-09-29T22:03:39.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LMdQL0GDX49A/+70OstKnEHqOwo=', 'Content-Length': '0'} 2023-09-29T22:03:39.863 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Flz4F8gyQg2Sv8j31HOgKaQJV5oYrkp'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fe0fbd5b7e7665af-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.863 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.863 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.863 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.863 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.863 DEBUG:boto:Method: GET 2023-09-29T22:03:39.864 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.864 DEBUG:boto:Data: 2023-09-29T22:03:39.864 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.864 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.864 DEBUG:boto:Params: {} 2023-09-29T22:03:39.864 DEBUG:boto:Token: None 2023-09-29T22:03:39.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039c26b2baf0b3d43-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.869 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.869 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.870 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.870 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.870 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.870 DEBUG:boto:Method: GET 2023-09-29T22:03:39.870 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.870 DEBUG:boto:Data: 2023-09-29T22:03:39.870 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.870 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.871 DEBUG:boto:Params: {} 2023-09-29T22:03:39.871 DEBUG:boto:Token: None 2023-09-29T22:03:39.871 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.871 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddc660b89eda1473-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.918 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.919 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.919 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.919 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.919 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.919 DEBUG:boto:Data: 2023-09-29T22:03:39.919 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.920 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.920 DEBUG:boto:Params: {} 2023-09-29T22:03:39.920 DEBUG:boto:Token: None 2023-09-29T22:03:39.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.926 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000427d52ee59ab0d97-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.926 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.926 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.926 DEBUG:boto:Method: HEAD 2023-09-29T22:03:39.927 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:39.927 DEBUG:boto:Data: 2023-09-29T22:03:39.927 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.927 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.927 DEBUG:boto:Params: {} 2023-09-29T22:03:39.927 DEBUG:boto:Token: None 2023-09-29T22:03:39.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj 2023-09-29T22:03:39.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM= 2023-09-29T22:03:39.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e0lY5WIRtK/LPHW4G3+07Y+QybM=', 'Content-Length': '0'} 2023-09-29T22:03:39.933 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008a51660aa2c137f4-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:39.933 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.934 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.934 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.934 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.934 DEBUG:boto:Method: GET 2023-09-29T22:03:39.934 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.934 DEBUG:boto:Data: 2023-09-29T22:03:39.934 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.934 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.934 DEBUG:boto:Params: {} 2023-09-29T22:03:39.935 DEBUG:boto:Token: None 2023-09-29T22:03:39.935 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.935 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.938 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000004c835ae3d57c46fc-00651749bb-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.982 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:39.982 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:39.983 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.983 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.983 DEBUG:boto:Method: GET 2023-09-29T22:03:39.983 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.983 DEBUG:boto:Data: 2023-09-29T22:03:39.983 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.983 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.983 DEBUG:boto:Params: {} 2023-09-29T22:03:39.984 DEBUG:boto:Token: None 2023-09-29T22:03:39.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:39.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs= 2023-09-29T22:03:39.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:19YZkQWf/JsdqDrP5PBKv4pqdWs=', 'Content-Length': '0'} 2023-09-29T22:03:39.987 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d64bf8ad523a16f4-00651749bb-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.987 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.987 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.987 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.987 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.988 DEBUG:boto:Method: GET 2023-09-29T22:03:39.988 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.988 DEBUG:boto:Data: 2023-09-29T22:03:39.988 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:39.988 DEBUG:boto:Port: 8001 2023-09-29T22:03:39.988 DEBUG:boto:Params: {} 2023-09-29T22:03:39.988 DEBUG:boto:Token: None 2023-09-29T22:03:39.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:39.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000708e9ed70483aad8-00651749bb-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:39.993 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:39.993 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:39.994 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:39.994 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.994 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:39.994 DEBUG:boto:Method: GET 2023-09-29T22:03:39.994 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:39.994 DEBUG:boto:Data: 2023-09-29T22:03:39.994 DEBUG:boto:Headers: {} 2023-09-29T22:03:39.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:39.994 DEBUG:boto:Port: 8000 2023-09-29T22:03:39.995 DEBUG:boto:Params: {} 2023-09-29T22:03:39.995 DEBUG:boto:Token: None 2023-09-29T22:03:39.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:39 GMT /hcnphz-13/ 2023-09-29T22:03:39.995 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4= 2023-09-29T22:03:39.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QJ8ozTHraydxm5DfiVQ4NiaVvI4=', 'Content-Length': '0'} 2023-09-29T22:03:40.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cf241a3d59f4b57-00651749bb-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.000 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.001 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.001 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.001 DEBUG:boto:path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.001 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.001 DEBUG:boto:Method: GET 2023-09-29T22:03:40.001 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.001 DEBUG:boto:Data: 2023-09-29T22:03:40.001 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.002 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.002 DEBUG:boto:Params: {} 2023-09-29T22:03:40.002 DEBUG:boto:Token: None 2023-09-29T22:03:40.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8= 2023-09-29T22:03:40.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8=', 'Content-Length': '0'} 2023-09-29T22:03:40.004 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d180b8a9edb7c686-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.005 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.005 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.005 DEBUG:boto:path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.005 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.005 DEBUG:boto:Method: GET 2023-09-29T22:03:40.005 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.005 DEBUG:boto:Data: 2023-09-29T22:03:40.006 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.006 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.006 DEBUG:boto:Params: {} 2023-09-29T22:03:40.006 DEBUG:boto:Token: None 2023-09-29T22:03:40.006 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.006 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8= 2023-09-29T22:03:40.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8=', 'Content-Length': '0'} 2023-09-29T22:03:40.008 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000674d7de183d279e3-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.054 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.054 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.055 DEBUG:boto:path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.055 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.055 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.055 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.055 DEBUG:boto:Data: 2023-09-29T22:03:40.055 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.055 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.055 DEBUG:boto:Params: {} 2023-09-29T22:03:40.056 DEBUG:boto:Token: None 2023-09-29T22:03:40.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BayEFok2jmFSXQu2En4K5w6Kpls= 2023-09-29T22:03:40.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BayEFok2jmFSXQu2En4K5w6Kpls=', 'Content-Length': '0'} 2023-09-29T22:03:40.059 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006d657cd48e388b61-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.060 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.060 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.060 DEBUG:boto:path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.060 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.060 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.060 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.060 DEBUG:boto:Data: 2023-09-29T22:03:40.060 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.061 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.061 DEBUG:boto:Params: {} 2023-09-29T22:03:40.061 DEBUG:boto:Token: None 2023-09-29T22:03:40.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BayEFok2jmFSXQu2En4K5w6Kpls= 2023-09-29T22:03:40.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BayEFok2jmFSXQu2En4K5w6Kpls=', 'Content-Length': '0'} 2023-09-29T22:03:40.065 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005753f211b3248f60-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.065 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.065 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.066 DEBUG:boto:path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.066 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.066 DEBUG:boto:Method: GET 2023-09-29T22:03:40.066 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.066 DEBUG:boto:Data: 2023-09-29T22:03:40.066 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.066 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.066 DEBUG:boto:Params: {} 2023-09-29T22:03:40.066 DEBUG:boto:Token: None 2023-09-29T22:03:40.067 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.067 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8= 2023-09-29T22:03:40.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8=', 'Content-Length': '0'} 2023-09-29T22:03:40.069 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d72d81797c6c06d3-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.110 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.110 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.111 DEBUG:boto:path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.111 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.111 DEBUG:boto:Method: GET 2023-09-29T22:03:40.111 DEBUG:boto:Path: /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.111 DEBUG:boto:Data: 2023-09-29T22:03:40.111 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.111 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.111 DEBUG:boto:Params: {} 2023-09-29T22:03:40.111 DEBUG:boto:Token: None 2023-09-29T22:03:40.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=U0kFiWYrndp08LH2Ryrtmjw75rahc7s 2023-09-29T22:03:40.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8= 2023-09-29T22:03:40.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m0w2o2M6ilDvDWo6tSfu4r66Cx8=', 'Content-Length': '0'} 2023-09-29T22:03:40.115 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'U0kFiWYrndp08LH2Ryrtmjw75rahc7s'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009b1f3bd952aa0407-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.115 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.115 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.115 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.115 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.115 DEBUG:boto:Method: GET 2023-09-29T22:03:40.115 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.116 DEBUG:boto:Data: 2023-09-29T22:03:40.116 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.116 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.116 DEBUG:boto:Params: {} 2023-09-29T22:03:40.116 DEBUG:boto:Token: None 2023-09-29T22:03:40.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000343ba5761ee9eefb-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.121 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.122 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.122 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.122 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.122 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.122 DEBUG:boto:Method: GET 2023-09-29T22:03:40.122 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.123 DEBUG:boto:Data: 2023-09-29T22:03:40.123 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.123 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.123 DEBUG:boto:Params: {} 2023-09-29T22:03:40.123 DEBUG:boto:Token: None 2023-09-29T22:03:40.123 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000141c192930ffd8c6-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.128 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.128 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.128 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.128 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.128 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.129 DEBUG:boto:Data: 2023-09-29T22:03:40.129 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.129 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.129 DEBUG:boto:Params: {} 2023-09-29T22:03:40.129 DEBUG:boto:Token: None 2023-09-29T22:03:40.129 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.134 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000464177ed4f4c402a-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.134 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.134 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.135 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.135 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.135 DEBUG:boto:Data: 2023-09-29T22:03:40.135 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.135 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.135 DEBUG:boto:Params: {} 2023-09-29T22:03:40.135 DEBUG:boto:Token: None 2023-09-29T22:03:40.135 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.140 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000aec480291264c74e-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.141 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.141 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.141 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.141 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.141 DEBUG:boto:Method: GET 2023-09-29T22:03:40.141 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.141 DEBUG:boto:Data: 2023-09-29T22:03:40.141 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.142 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.142 DEBUG:boto:Params: {} 2023-09-29T22:03:40.142 DEBUG:boto:Token: None 2023-09-29T22:03:40.142 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.142 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.145 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000782d69a47a317a90-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.186 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.186 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.187 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.187 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.187 DEBUG:boto:Method: GET 2023-09-29T22:03:40.187 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.187 DEBUG:boto:Data: 2023-09-29T22:03:40.187 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.187 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.187 DEBUG:boto:Params: {} 2023-09-29T22:03:40.188 DEBUG:boto:Token: None 2023-09-29T22:03:40.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.191 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000005f2d1669958220dd-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.191 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.191 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.191 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.191 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.191 DEBUG:boto:Method: GET 2023-09-29T22:03:40.192 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.192 DEBUG:boto:Data: 2023-09-29T22:03:40.192 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.192 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.192 DEBUG:boto:Params: {} 2023-09-29T22:03:40.192 DEBUG:boto:Token: None 2023-09-29T22:03:40.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000045c4560fa2cf729-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.197 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.197 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.197 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.197 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.197 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.197 DEBUG:boto:Method: GET 2023-09-29T22:03:40.197 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.198 DEBUG:boto:Data: 2023-09-29T22:03:40.198 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.198 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.198 DEBUG:boto:Params: {} 2023-09-29T22:03:40.198 DEBUG:boto:Token: None 2023-09-29T22:03:40.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007255573f38527460-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.203 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.203 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.203 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.203 DEBUG:boto:path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.203 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.204 DEBUG:boto:Method: GET 2023-09-29T22:03:40.204 DEBUG:boto:Path: /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.204 DEBUG:boto:Data: 2023-09-29T22:03:40.204 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.204 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.204 DEBUG:boto:Params: {} 2023-09-29T22:03:40.204 DEBUG:boto:Token: None 2023-09-29T22:03:40.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY= 2023-09-29T22:03:40.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY=', 'Content-Length': '0'} 2023-09-29T22:03:40.207 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001f9602286477bb41-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.208 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.208 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.208 DEBUG:boto:path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.208 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.208 DEBUG:boto:Method: GET 2023-09-29T22:03:40.208 DEBUG:boto:Path: /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.208 DEBUG:boto:Data: 2023-09-29T22:03:40.208 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.208 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.209 DEBUG:boto:Params: {} 2023-09-29T22:03:40.209 DEBUG:boto:Token: None 2023-09-29T22:03:40.209 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.209 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY= 2023-09-29T22:03:40.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY=', 'Content-Length': '0'} 2023-09-29T22:03:40.212 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e1900156e4536001-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.212 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.212 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.212 DEBUG:boto:path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.212 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.213 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.213 DEBUG:boto:Path: /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.213 DEBUG:boto:Data: 2023-09-29T22:03:40.213 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.213 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.213 DEBUG:boto:Params: {} 2023-09-29T22:03:40.213 DEBUG:boto:Token: None 2023-09-29T22:03:40.213 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yDOVCOLmQY9sjSnJnOagrzzSf1I= 2023-09-29T22:03:40.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yDOVCOLmQY9sjSnJnOagrzzSf1I=', 'Content-Length': '0'} 2023-09-29T22:03:40.216 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000dfdf2569e6a984c3-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.216 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.216 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.216 DEBUG:boto:path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.217 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.217 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.217 DEBUG:boto:Path: /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.217 DEBUG:boto:Data: 2023-09-29T22:03:40.217 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.217 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.217 DEBUG:boto:Params: {} 2023-09-29T22:03:40.217 DEBUG:boto:Token: None 2023-09-29T22:03:40.218 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.218 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yDOVCOLmQY9sjSnJnOagrzzSf1I= 2023-09-29T22:03:40.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yDOVCOLmQY9sjSnJnOagrzzSf1I=', 'Content-Length': '0'} 2023-09-29T22:03:40.220 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000010528fa2f75e4082-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.220 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.221 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.221 DEBUG:boto:path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.221 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.221 DEBUG:boto:Method: GET 2023-09-29T22:03:40.221 DEBUG:boto:Path: /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.221 DEBUG:boto:Data: 2023-09-29T22:03:40.221 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.221 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.221 DEBUG:boto:Params: {} 2023-09-29T22:03:40.222 DEBUG:boto:Token: None 2023-09-29T22:03:40.222 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.222 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY= 2023-09-29T22:03:40.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY=', 'Content-Length': '0'} 2023-09-29T22:03:40.225 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000000fe2429f5f7f328c-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.266 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.267 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.267 DEBUG:boto:path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.267 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.267 DEBUG:boto:Method: GET 2023-09-29T22:03:40.267 DEBUG:boto:Path: /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.267 DEBUG:boto:Data: 2023-09-29T22:03:40.267 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.267 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.268 DEBUG:boto:Params: {} 2023-09-29T22:03:40.268 DEBUG:boto:Token: None 2023-09-29T22:03:40.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc- 2023-09-29T22:03:40.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY= 2023-09-29T22:03:40.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teD8XoVTzpi2jAleJk84sgWHGyY=', 'Content-Length': '0'} 2023-09-29T22:03:40.271 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'qUlx5C5B1aGaO5W8U0fuWEA3r.YkWc-'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000001584e055b7353295-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.314 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.315 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.315 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.315 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.315 DEBUG:boto:Method: GET 2023-09-29T22:03:40.315 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.315 DEBUG:boto:Data: 2023-09-29T22:03:40.315 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.315 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.315 DEBUG:boto:Params: {} 2023-09-29T22:03:40.316 DEBUG:boto:Token: None 2023-09-29T22:03:40.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058d136debb8f5b29-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.322 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.322 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.322 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.322 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.322 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.322 DEBUG:boto:Method: GET 2023-09-29T22:03:40.323 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.323 DEBUG:boto:Data: 2023-09-29T22:03:40.323 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.323 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.323 DEBUG:boto:Params: {} 2023-09-29T22:03:40.323 DEBUG:boto:Token: None 2023-09-29T22:03:40.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4c8186346550ff3-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.329 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.329 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.329 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.329 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.329 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.330 DEBUG:boto:Data: 2023-09-29T22:03:40.330 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.330 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.330 DEBUG:boto:Params: {} 2023-09-29T22:03:40.330 DEBUG:boto:Token: None 2023-09-29T22:03:40.330 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.335 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ca41d924294f010-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.335 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.335 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.336 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.336 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.336 DEBUG:boto:Data: 2023-09-29T22:03:40.336 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.336 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.336 DEBUG:boto:Params: {} 2023-09-29T22:03:40.336 DEBUG:boto:Token: None 2023-09-29T22:03:40.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.341 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000002a49bfa276623b25-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.342 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.342 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.342 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.342 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.342 DEBUG:boto:Method: GET 2023-09-29T22:03:40.342 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.342 DEBUG:boto:Data: 2023-09-29T22:03:40.342 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.343 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.343 DEBUG:boto:Params: {} 2023-09-29T22:03:40.343 DEBUG:boto:Token: None 2023-09-29T22:03:40.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.345 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000006692e0554f420906-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.386 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.387 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.387 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.387 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.387 DEBUG:boto:Method: GET 2023-09-29T22:03:40.387 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.387 DEBUG:boto:Data: 2023-09-29T22:03:40.387 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.387 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.388 DEBUG:boto:Params: {} 2023-09-29T22:03:40.388 DEBUG:boto:Token: None 2023-09-29T22:03:40.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.391 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009ea95afd06867411-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.391 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.391 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.391 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.392 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.392 DEBUG:boto:Method: GET 2023-09-29T22:03:40.392 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.392 DEBUG:boto:Data: 2023-09-29T22:03:40.392 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.392 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.392 DEBUG:boto:Params: {} 2023-09-29T22:03:40.392 DEBUG:boto:Token: None 2023-09-29T22:03:40.393 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e483c3d5fb50da64-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.438 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.439 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.439 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.439 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.439 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.439 DEBUG:boto:Method: GET 2023-09-29T22:03:40.440 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.440 DEBUG:boto:Data: 2023-09-29T22:03:40.440 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.440 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.440 DEBUG:boto:Params: {} 2023-09-29T22:03:40.440 DEBUG:boto:Token: None 2023-09-29T22:03:40.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.446 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033a6f4badd7f2755-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.446 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.446 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.447 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.447 DEBUG:boto:path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.447 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.447 DEBUG:boto:Method: GET 2023-09-29T22:03:40.447 DEBUG:boto:Path: /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.447 DEBUG:boto:Data: 2023-09-29T22:03:40.447 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.447 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.447 DEBUG:boto:Params: {} 2023-09-29T22:03:40.448 DEBUG:boto:Token: None 2023-09-29T22:03:40.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA= 2023-09-29T22:03:40.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA=', 'Content-Length': '0'} 2023-09-29T22:03:40.451 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000940fe9c5f13c48a6-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.451 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.451 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.451 DEBUG:boto:path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.451 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.451 DEBUG:boto:Method: GET 2023-09-29T22:03:40.452 DEBUG:boto:Path: /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.452 DEBUG:boto:Data: 2023-09-29T22:03:40.452 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.452 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.452 DEBUG:boto:Params: {} 2023-09-29T22:03:40.452 DEBUG:boto:Token: None 2023-09-29T22:03:40.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA= 2023-09-29T22:03:40.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA=', 'Content-Length': '0'} 2023-09-29T22:03:40.455 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000a5dc1a6a132697f5-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.498 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.498 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.499 DEBUG:boto:path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.499 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.499 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.499 DEBUG:boto:Path: /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.499 DEBUG:boto:Data: 2023-09-29T22:03:40.499 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.499 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.499 DEBUG:boto:Params: {} 2023-09-29T22:03:40.500 DEBUG:boto:Token: None 2023-09-29T22:03:40.500 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UHaR9Rk91fmYrhxovT6Ag5IGwwk= 2023-09-29T22:03:40.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UHaR9Rk91fmYrhxovT6Ag5IGwwk=', 'Content-Length': '0'} 2023-09-29T22:03:40.503 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000060bc4a17d0591cc2-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.503 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.503 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.503 DEBUG:boto:path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.503 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.503 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.504 DEBUG:boto:Path: /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.504 DEBUG:boto:Data: 2023-09-29T22:03:40.504 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.504 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.504 DEBUG:boto:Params: {} 2023-09-29T22:03:40.504 DEBUG:boto:Token: None 2023-09-29T22:03:40.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UHaR9Rk91fmYrhxovT6Ag5IGwwk= 2023-09-29T22:03:40.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UHaR9Rk91fmYrhxovT6Ag5IGwwk=', 'Content-Length': '0'} 2023-09-29T22:03:40.507 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000009be3e5b02e6c21f7-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.508 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.508 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.508 DEBUG:boto:path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.508 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.508 DEBUG:boto:Method: GET 2023-09-29T22:03:40.508 DEBUG:boto:Path: /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.509 DEBUG:boto:Data: 2023-09-29T22:03:40.509 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.509 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.509 DEBUG:boto:Params: {} 2023-09-29T22:03:40.509 DEBUG:boto:Token: None 2023-09-29T22:03:40.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA= 2023-09-29T22:03:40.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA=', 'Content-Length': '0'} 2023-09-29T22:03:40.512 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000026a9e0cb9b23c847-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.512 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.513 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.513 DEBUG:boto:path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.513 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.513 DEBUG:boto:Method: GET 2023-09-29T22:03:40.513 DEBUG:boto:Path: /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.513 DEBUG:boto:Data: 2023-09-29T22:03:40.513 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.513 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.514 DEBUG:boto:Params: {} 2023-09-29T22:03:40.514 DEBUG:boto:Token: None 2023-09-29T22:03:40.514 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=iri9FYdBGldQRD.PWnNbydapWa3BHP8 2023-09-29T22:03:40.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA= 2023-09-29T22:03:40.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gYQhzwErYVHAmbv27repqDTVXlA=', 'Content-Length': '0'} 2023-09-29T22:03:40.517 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'iri9FYdBGldQRD.PWnNbydapWa3BHP8'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000082ce74d5757957ee-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.558 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.559 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.559 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.559 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.559 DEBUG:boto:Method: GET 2023-09-29T22:03:40.559 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.559 DEBUG:boto:Data: 2023-09-29T22:03:40.559 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.559 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.559 DEBUG:boto:Params: {} 2023-09-29T22:03:40.560 DEBUG:boto:Token: None 2023-09-29T22:03:40.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df9c163702804866-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.565 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.565 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.565 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.566 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.566 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.566 DEBUG:boto:Method: GET 2023-09-29T22:03:40.566 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.566 DEBUG:boto:Data: 2023-09-29T22:03:40.566 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.566 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.566 DEBUG:boto:Params: {} 2023-09-29T22:03:40.567 DEBUG:boto:Token: None 2023-09-29T22:03:40.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dac8fb9c96dd834a-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.618 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.619 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.619 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.619 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.619 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.619 DEBUG:boto:Data: 2023-09-29T22:03:40.619 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.619 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.620 DEBUG:boto:Params: {} 2023-09-29T22:03:40.620 DEBUG:boto:Token: None 2023-09-29T22:03:40.620 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.625 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000058447e93109330a0-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.626 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.626 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.626 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.626 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.626 DEBUG:boto:Data: 2023-09-29T22:03:40.626 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.626 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.626 DEBUG:boto:Params: {} 2023-09-29T22:03:40.627 DEBUG:boto:Token: None 2023-09-29T22:03:40.627 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.632 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000d584e0bbb76c95d9-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.633 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.633 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.633 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.633 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.633 DEBUG:boto:Method: GET 2023-09-29T22:03:40.633 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.633 DEBUG:boto:Data: 2023-09-29T22:03:40.634 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.634 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.634 DEBUG:boto:Params: {} 2023-09-29T22:03:40.634 DEBUG:boto:Token: None 2023-09-29T22:03:40.634 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.634 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.637 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c7e19ab7b965fd0e-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.678 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.678 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.678 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.679 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.679 DEBUG:boto:Method: GET 2023-09-29T22:03:40.679 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.679 DEBUG:boto:Data: 2023-09-29T22:03:40.679 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.679 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.679 DEBUG:boto:Params: {} 2023-09-29T22:03:40.679 DEBUG:boto:Token: None 2023-09-29T22:03:40.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.683 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008438ebbb59172838-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.683 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.683 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.683 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.683 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.683 DEBUG:boto:Method: GET 2023-09-29T22:03:40.684 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.684 DEBUG:boto:Data: 2023-09-29T22:03:40.684 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.684 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.684 DEBUG:boto:Params: {} 2023-09-29T22:03:40.684 DEBUG:boto:Token: None 2023-09-29T22:03:40.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.684 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c7d87a178b23ea6-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.730 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.731 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.731 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.731 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.731 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.731 DEBUG:boto:Method: GET 2023-09-29T22:03:40.731 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.731 DEBUG:boto:Data: 2023-09-29T22:03:40.731 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.732 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.732 DEBUG:boto:Params: {} 2023-09-29T22:03:40.732 DEBUG:boto:Token: None 2023-09-29T22:03:40.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab8da43411fad030-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.737 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.738 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.738 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.738 DEBUG:boto:path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.738 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.738 DEBUG:boto:Method: GET 2023-09-29T22:03:40.738 DEBUG:boto:Path: /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.738 DEBUG:boto:Data: 2023-09-29T22:03:40.738 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.739 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.739 DEBUG:boto:Params: {} 2023-09-29T22:03:40.739 DEBUG:boto:Token: None 2023-09-29T22:03:40.739 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk= 2023-09-29T22:03:40.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk=', 'Content-Length': '0'} 2023-09-29T22:03:40.742 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000008f6facdc1bbeea43-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.742 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.743 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.743 DEBUG:boto:path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.743 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.743 DEBUG:boto:Method: GET 2023-09-29T22:03:40.743 DEBUG:boto:Path: /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.743 DEBUG:boto:Data: 2023-09-29T22:03:40.743 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.743 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.744 DEBUG:boto:Params: {} 2023-09-29T22:03:40.744 DEBUG:boto:Token: None 2023-09-29T22:03:40.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk= 2023-09-29T22:03:40.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk=', 'Content-Length': '0'} 2023-09-29T22:03:40.747 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000068107fcb062b84aa-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.747 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.747 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.747 DEBUG:boto:path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.747 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.748 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.748 DEBUG:boto:Path: /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.748 DEBUG:boto:Data: 2023-09-29T22:03:40.748 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.748 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.748 DEBUG:boto:Params: {} 2023-09-29T22:03:40.748 DEBUG:boto:Token: None 2023-09-29T22:03:40.749 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.749 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIkK9XruMlQnrpHRsJNU+rgEjUA= 2023-09-29T22:03:40.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIkK9XruMlQnrpHRsJNU+rgEjUA=', 'Content-Length': '0'} 2023-09-29T22:03:40.751 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000f3a913af73f64b4c-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.752 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.752 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.752 DEBUG:boto:path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.752 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.752 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.752 DEBUG:boto:Path: /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.752 DEBUG:boto:Data: 2023-09-29T22:03:40.752 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.753 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.753 DEBUG:boto:Params: {} 2023-09-29T22:03:40.753 DEBUG:boto:Token: None 2023-09-29T22:03:40.753 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIkK9XruMlQnrpHRsJNU+rgEjUA= 2023-09-29T22:03:40.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIkK9XruMlQnrpHRsJNU+rgEjUA=', 'Content-Length': '0'} 2023-09-29T22:03:40.756 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000031459b905cba9f1b-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.756 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.756 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.757 DEBUG:boto:path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.757 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.757 DEBUG:boto:Method: GET 2023-09-29T22:03:40.757 DEBUG:boto:Path: /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.758 DEBUG:boto:Data: 2023-09-29T22:03:40.758 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.758 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.758 DEBUG:boto:Params: {} 2023-09-29T22:03:40.758 DEBUG:boto:Token: None 2023-09-29T22:03:40.758 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.759 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk= 2023-09-29T22:03:40.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk=', 'Content-Length': '0'} 2023-09-29T22:03:40.762 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000e167c37bc10a6d8a-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.802 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.802 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.802 DEBUG:boto:path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.803 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.803 DEBUG:boto:Method: GET 2023-09-29T22:03:40.803 DEBUG:boto:Path: /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.803 DEBUG:boto:Data: 2023-09-29T22:03:40.803 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.803 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.803 DEBUG:boto:Params: {} 2023-09-29T22:03:40.803 DEBUG:boto:Token: None 2023-09-29T22:03:40.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=nt.n3hC6tYeenMgygbhwVyjvoejWUSO 2023-09-29T22:03:40.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk= 2023-09-29T22:03:40.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XY9d5iTAV7loDijUn9eI9Frnsdk=', 'Content-Length': '0'} 2023-09-29T22:03:40.806 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'nt.n3hC6tYeenMgygbhwVyjvoejWUSO'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000074d142195951e7f0-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.807 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.807 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.807 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.807 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.807 DEBUG:boto:Method: GET 2023-09-29T22:03:40.807 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.807 DEBUG:boto:Data: 2023-09-29T22:03:40.808 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.808 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.808 DEBUG:boto:Params: {} 2023-09-29T22:03:40.808 DEBUG:boto:Token: None 2023-09-29T22:03:40.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003300f94abdda284b-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.813 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.814 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.814 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.814 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.814 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.814 DEBUG:boto:Method: GET 2023-09-29T22:03:40.814 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.814 DEBUG:boto:Data: 2023-09-29T22:03:40.815 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.815 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.815 DEBUG:boto:Params: {} 2023-09-29T22:03:40.815 DEBUG:boto:Token: None 2023-09-29T22:03:40.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.815 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c81280f9cd5c5254-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.820 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.820 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.820 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.820 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.820 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.820 DEBUG:boto:Data: 2023-09-29T22:03:40.821 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.821 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.821 DEBUG:boto:Params: {} 2023-09-29T22:03:40.821 DEBUG:boto:Token: None 2023-09-29T22:03:40.821 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.826 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000c4f678180eb65975-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.826 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.826 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.826 DEBUG:boto:Method: HEAD 2023-09-29T22:03:40.826 DEBUG:boto:Path: /hcnphz-13/obj 2023-09-29T22:03:40.826 DEBUG:boto:Data: 2023-09-29T22:03:40.827 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.827 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.827 DEBUG:boto:Params: {} 2023-09-29T22:03:40.827 DEBUG:boto:Token: None 2023-09-29T22:03:40.827 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj 2023-09-29T22:03:40.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ= 2023-09-29T22:03:40.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:omi/oJpcQx9A7SigqmNAu9VKlGQ=', 'Content-Length': '0'} 2023-09-29T22:03:40.832 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx000007834b4b0b7b491b5-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.832 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:40.832 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.833 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.833 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.833 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.833 DEBUG:boto:Method: GET 2023-09-29T22:03:40.833 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.833 DEBUG:boto:Data: 2023-09-29T22:03:40.833 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.833 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.833 DEBUG:boto:Params: {} 2023-09-29T22:03:40.834 DEBUG:boto:Token: None 2023-09-29T22:03:40.834 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.834 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.836 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx00000fa52af71abb1dfdb-00651749bc-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.882 DEBUG:boto:path=/hcnphz-13/obj 2023-09-29T22:03:40.883 DEBUG:boto:auth_path=/hcnphz-13/obj 2023-09-29T22:03:40.883 DEBUG:boto:path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.883 DEBUG:boto:auth_path=/hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.883 DEBUG:boto:Method: GET 2023-09-29T22:03:40.883 DEBUG:boto:Path: /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.883 DEBUG:boto:Data: 2023-09-29T22:03:40.883 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.883 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.883 DEBUG:boto:Params: {} 2023-09-29T22:03:40.884 DEBUG:boto:Token: None 2023-09-29T22:03:40.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/obj?versionId=Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv 2023-09-29T22:03:40.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI= 2023-09-29T22:03:40.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J83XI7oJ1D0Nsd5n4QExrvgYUuI=', 'Content-Length': '0'} 2023-09-29T22:03:40.887 DEBUG:boto:Response headers: [('Content-Length', '8'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:01:35 GMT'), ('x-amz-version-id', 'Dysj0dBRPeSLOTMc-kPu0YmgNYl1uCv'), ('x-rgw-object-type', 'Normal'), ('ETag', '"966634ebf2fc135707d6753692bf4b1e"'), ('x-amz-request-id', 'tx0000048198b1d69939f94-00651749bc-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.887 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.887 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.887 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.887 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.887 DEBUG:boto:Method: GET 2023-09-29T22:03:40.888 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.888 DEBUG:boto:Data: 2023-09-29T22:03:40.888 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:40.888 DEBUG:boto:Port: 8001 2023-09-29T22:03:40.888 DEBUG:boto:Params: {} 2023-09-29T22:03:40.888 DEBUG:boto:Token: None 2023-09-29T22:03:40.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c035c7eb14fd339-00651749bc-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.893 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.894 DEBUG:boto:path=/hcnphz-13/ 2023-09-29T22:03:40.894 DEBUG:boto:auth_path=/hcnphz-13/ 2023-09-29T22:03:40.894 DEBUG:boto:path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.894 DEBUG:boto:auth_path=/hcnphz-13/?prefix=obj 2023-09-29T22:03:40.894 DEBUG:boto:Method: GET 2023-09-29T22:03:40.894 DEBUG:boto:Path: /hcnphz-13/?prefix=obj 2023-09-29T22:03:40.895 DEBUG:boto:Data: 2023-09-29T22:03:40.895 DEBUG:boto:Headers: {} 2023-09-29T22:03:40.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:40.895 DEBUG:boto:Port: 8000 2023-09-29T22:03:40.895 DEBUG:boto:Params: {} 2023-09-29T22:03:40.895 DEBUG:boto:Token: None 2023-09-29T22:03:40.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:40 GMT /hcnphz-13/ 2023-09-29T22:03:40.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8= 2023-09-29T22:03:40.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tFhBmN0OiqAIDIluMPDBnbFQ6t8=', 'Content-Length': '0'} 2023-09-29T22:03:40.900 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000779f061df41ae19d-00651749bc-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:40.946 DEBUG:boto:b'hcnphz-13obj1000falseobj2023-09-29T22:01:35.862Z"966634ebf2fc135707d6753692bf4b1e"8STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:40.947 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-13 zones={a2, a1} 2023-09-29T22:03:40.947 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:03:40.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:40.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:40.948 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:40.992 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:40.992 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:02:01.619643Z" 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:03:41.082 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:37.608065Z" 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.083 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:03:41.084 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000001632', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:03:41.085 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:03:41.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.085 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:41.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:41.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.214 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.215 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001632", 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:02:01.619643Z" 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:03:41.217 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.218 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:37.608065Z" 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.219 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:03:41.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.221 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:03:41Z" 2023-09-29T22:03:41.221 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:03:41.221 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001632", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:02:01.619643Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:30.832179Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:57:37.608065Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:03:41Z" } 2023-09-29T22:03:41.222 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001632', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:02:01.619643Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:30.832179Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:57:37.608065Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:03:41.222 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000001632', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:03:41.222 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000001632', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:03:41.222 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:03:41.222 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:03:41.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.223 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:41.263 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.362 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:03:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.362 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002619", 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:03:30.812688Z" 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-09-29T22:03:41.363 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:59.650497Z" 2023-09-29T22:03:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:03:41.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.364 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:03:41.365 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000002619', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:03:41.365 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:03:41.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.366 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:41.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:41.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.491 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:03:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:03:41.491 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.492 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002619", 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:03:30.812688Z" 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.493 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.494 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:03:41.496 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:59.650497Z" 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:03:41Z" 2023-09-29T22:03:41.499 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:03:41.500 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002619", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:03:30.812688Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:10.866065Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:57:59.650497Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:03:41Z" } 2023-09-29T22:03:41.500 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002619', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:03:30.812688Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:10.866065Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:57:59.650497Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:03:41.501 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000002619', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:03:41.501 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000002619', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:03:41.501 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:03:41.501 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2023-09-29T22:03:41.502 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:41.502 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:41.502 DEBUG:boto:Method: PUT 2023-09-29T22:03:41.503 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:03:41.503 DEBUG:boto:Data: 2023-09-29T22:03:41.503 DEBUG:boto:Headers: {} 2023-09-29T22:03:41.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:41.503 DEBUG:boto:Port: 8000 2023-09-29T22:03:41.503 DEBUG:boto:Params: {} 2023-09-29T22:03:41.503 DEBUG:boto:Token: None 2023-09-29T22:03:41.503 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:03:41 GMT /hcnphz-14/ 2023-09-29T22:03:41.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gZtvwr475egHrh5PbqDEkJW2/7I= 2023-09-29T22:03:41.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gZtvwr475egHrh5PbqDEkJW2/7I=', 'Content-Length': '0'} 2023-09-29T22:03:41.530 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000227507c8d8d08018-00651749bd-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:41.530 DEBUG:rgw_multi.tests:created bucket=hcnphz-14 2023-09-29T22:03:41.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.530 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:41.585 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:41.586 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.644 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:03:41.679 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:03:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:03:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:03:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025021.528935_85.1", 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:03:41.528935Z" 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.680 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:03:41.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.681 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:03:41.681 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025021.528935_85.1', 2: '', 3: ''} 2023-09-29T22:03:41.682 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:03:41.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:41.682 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:41.725 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:41.725 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:41.781+0000 7ffae44e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:03:41.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:41.785+0000 7ffae44e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:03:41.807 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.809 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025021.528935_85.1", 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:03:41.528935Z", 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.810 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:03:41.812 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:03:41Z" 2023-09-29T22:03:41.813 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:03:41.814 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025021.528935_85.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:03:41.528935Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:03:41Z" } 2023-09-29T22:03:41.814 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025021.528935_85.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:03:41.528935Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:03:41.815 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025021.528935_85.1', 2: '', 3: ''} 2023-09-29T22:03:41.815 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025021.528935_85.1', 2: '', 3: ''} 2023-09-29T22:03:41.815 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:03:41.815 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:41.815 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:41.815 DEBUG:boto:Method: HEAD 2023-09-29T22:03:41.815 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:03:41.816 DEBUG:boto:Data: 2023-09-29T22:03:41.816 DEBUG:boto:Headers: {} 2023-09-29T22:03:41.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:41.816 DEBUG:boto:Port: 8000 2023-09-29T22:03:41.816 DEBUG:boto:Params: {} 2023-09-29T22:03:41.816 DEBUG:boto:Token: None 2023-09-29T22:03:41.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:41 GMT /hcnphz-14/ 2023-09-29T22:03:41.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ORIAdQJl+DvSrwfnVD3dYWZnRCM= 2023-09-29T22:03:41.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ORIAdQJl+DvSrwfnVD3dYWZnRCM=', 'Content-Length': '0'} 2023-09-29T22:03:41.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000073f002a754be3a15-00651749bd-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:41.821 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:41.821 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:41.821 DEBUG:boto:Method: PUT 2023-09-29T22:03:41.822 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:03:41.822 DEBUG:boto:Data: 2023-09-29T22:03:41.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:03:41.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:41.822 DEBUG:boto:Port: 8000 2023-09-29T22:03:41.822 DEBUG:boto:Params: {} 2023-09-29T22:03:41.822 DEBUG:boto:Token: None 2023-09-29T22:03:41.822 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:03:41 GMT /hcnphz-14/obj 2023-09-29T22:03:41.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0R/zLD1Ad+amgL/ozkqV4y55cdE= 2023-09-29T22:03:41.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:03:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0R/zLD1Ad+amgL/ozkqV4y55cdE='} 2023-09-29T22:03:41.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f13f8554bd9ad32a-00651749bd-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:41.835 DEBUG:rgw_multi.tests:created initial version id=None 2023-09-29T22:03:41.835 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-14 2023-09-29T22:03:41.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:41.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:41.836 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:03:41.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:41.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:41.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:41.945+0000 7ffb0590bf40 1 waiting to reach incremental sync.. 2023-09-29T22:03:46.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:46.949+0000 7ffb0590bf40 1 waiting to reach incremental sync.. 2023-09-29T22:03:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:51.953+0000 7ffb0590bf40 1 bucket sync caught up with source: 2023-09-29T22:03:51.952 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:03:51.952 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:03:51.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:51.953+0000 7ffb0590bf40 0 bucket checkpoint complete 2023-09-29T22:03:51.979 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-14 2023-09-29T22:03:51.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:51.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:51.980 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:03:52.036 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:52.036 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:52.097+0000 7f25293a3f40 1 bucket sync caught up with source: 2023-09-29T22:03:52.096 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.18.6, , ] 2023-09-29T22:03:52.097 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.18.6, , ] 2023-09-29T22:03:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:52.097+0000 7f25293a3f40 0 bucket checkpoint complete 2023-09-29T22:03:52.122 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-14 zones={a2, a1} 2023-09-29T22:03:52.122 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.122 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.122 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.122 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:03:52.122 DEBUG:boto:Data: 2023-09-29T22:03:52.122 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.123 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.123 DEBUG:boto:Params: {} 2023-09-29T22:03:52.123 DEBUG:boto:Token: None 2023-09-29T22:03:52.123 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pg9t4QRL87NtQW3JJcxEFsEDXQk= 2023-09-29T22:03:52.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pg9t4QRL87NtQW3JJcxEFsEDXQk=', 'Content-Length': '0'} 2023-09-29T22:03:52.126 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c02a563736617a4-00651749c8-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.126 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.127 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.127 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.127 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:03:52.127 DEBUG:boto:Data: 2023-09-29T22:03:52.127 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.127 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.127 DEBUG:boto:Params: {} 2023-09-29T22:03:52.127 DEBUG:boto:Token: None 2023-09-29T22:03:52.128 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pg9t4QRL87NtQW3JJcxEFsEDXQk= 2023-09-29T22:03:52.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pg9t4QRL87NtQW3JJcxEFsEDXQk=', 'Content-Length': '0'} 2023-09-29T22:03:52.131 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000429be6d68831c86c-00651749c8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.131 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:03:52.131 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.131 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.131 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.131 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.132 DEBUG:boto:Method: GET 2023-09-29T22:03:52.132 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.132 DEBUG:boto:Data: 2023-09-29T22:03:52.132 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.132 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.132 DEBUG:boto:Params: {} 2023-09-29T22:03:52.132 DEBUG:boto:Token: None 2023-09-29T22:03:52.132 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/?versions 2023-09-29T22:03:52.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI= 2023-09-29T22:03:52.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI=', 'Content-Length': '0'} 2023-09-29T22:03:52.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e28927a6aaf36421-00651749c8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.136 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.137 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:52.137 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:03:52.137 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.137 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.137 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.137 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.137 DEBUG:boto:Method: GET 2023-09-29T22:03:52.137 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.137 DEBUG:boto:Data: 2023-09-29T22:03:52.138 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.138 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.138 DEBUG:boto:Params: {} 2023-09-29T22:03:52.138 DEBUG:boto:Token: None 2023-09-29T22:03:52.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/?versions 2023-09-29T22:03:52.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI= 2023-09-29T22:03:52.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI=', 'Content-Length': '0'} 2023-09-29T22:03:52.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee7e772ea78f05b5-00651749c8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.142 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:03:52.142 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.142 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.142 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.143 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.143 DEBUG:boto:Method: GET 2023-09-29T22:03:52.143 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.143 DEBUG:boto:Data: 2023-09-29T22:03:52.143 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.143 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.143 DEBUG:boto:Params: {} 2023-09-29T22:03:52.143 DEBUG:boto:Token: None 2023-09-29T22:03:52.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/?versions 2023-09-29T22:03:52.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI= 2023-09-29T22:03:52.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI=', 'Content-Length': '0'} 2023-09-29T22:03:52.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3072bffeb9a836b-00651749c8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.147 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.148 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.148 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.148 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.148 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.148 DEBUG:boto:Method: GET 2023-09-29T22:03:52.148 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:03:52.148 DEBUG:boto:Data: 2023-09-29T22:03:52.148 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.149 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.149 DEBUG:boto:Params: {} 2023-09-29T22:03:52.149 DEBUG:boto:Token: None 2023-09-29T22:03:52.149 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/?versions 2023-09-29T22:03:52.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI= 2023-09-29T22:03:52.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fNU/tF7+tw/8Dps3QrTcxiZ2yEI=', 'Content-Length': '0'} 2023-09-29T22:03:52.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a04b50625fea13ef-00651749c8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.153 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:52.154 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.154 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.154 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.154 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.154 DEBUG:boto:Method: GET 2023-09-29T22:03:52.154 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.154 DEBUG:boto:Data: 2023-09-29T22:03:52.154 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.155 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.155 DEBUG:boto:Params: {} 2023-09-29T22:03:52.155 DEBUG:boto:Token: None 2023-09-29T22:03:52.155 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.155 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE= 2023-09-29T22:03:52.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE=', 'Content-Length': '0'} 2023-09-29T22:03:52.157 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003f52fa25c3fe853c-00651749c8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.158 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.158 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.158 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.158 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.158 DEBUG:boto:Method: GET 2023-09-29T22:03:52.158 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.158 DEBUG:boto:Data: 2023-09-29T22:03:52.159 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.159 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.159 DEBUG:boto:Params: {} 2023-09-29T22:03:52.159 DEBUG:boto:Token: None 2023-09-29T22:03:52.159 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE= 2023-09-29T22:03:52.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE=', 'Content-Length': '0'} 2023-09-29T22:03:52.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ceddd5a650775a4f-00651749c8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.162 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.162 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.162 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.162 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.162 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.162 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.163 DEBUG:boto:Data: 2023-09-29T22:03:52.163 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.163 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.163 DEBUG:boto:Params: {} 2023-09-29T22:03:52.163 DEBUG:boto:Token: None 2023-09-29T22:03:52.163 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.163 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CB3IhGkX5gfgyLJn/mGKgcJzxvQ= 2023-09-29T22:03:52.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CB3IhGkX5gfgyLJn/mGKgcJzxvQ=', 'Content-Length': '0'} 2023-09-29T22:03:52.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba28483aec9dde31-00651749c8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.166 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.166 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.166 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.166 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.167 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.167 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.167 DEBUG:boto:Data: 2023-09-29T22:03:52.167 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.167 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.167 DEBUG:boto:Params: {} 2023-09-29T22:03:52.167 DEBUG:boto:Token: None 2023-09-29T22:03:52.167 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CB3IhGkX5gfgyLJn/mGKgcJzxvQ= 2023-09-29T22:03:52.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CB3IhGkX5gfgyLJn/mGKgcJzxvQ=', 'Content-Length': '0'} 2023-09-29T22:03:52.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000695fbe3b74072407-00651749c8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:52.170 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.170 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.170 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.171 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.171 DEBUG:boto:Method: GET 2023-09-29T22:03:52.171 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.171 DEBUG:boto:Data: 2023-09-29T22:03:52.171 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.171 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.171 DEBUG:boto:Params: {} 2023-09-29T22:03:52.171 DEBUG:boto:Token: None 2023-09-29T22:03:52.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE= 2023-09-29T22:03:52.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE=', 'Content-Length': '0'} 2023-09-29T22:03:52.174 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017292d0f43f64e63-00651749c8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.174 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.174 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.174 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.175 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.175 DEBUG:boto:Method: GET 2023-09-29T22:03:52.175 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.175 DEBUG:boto:Data: 2023-09-29T22:03:52.175 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.175 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.175 DEBUG:boto:Params: {} 2023-09-29T22:03:52.175 DEBUG:boto:Token: None 2023-09-29T22:03:52.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:03:52.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE= 2023-09-29T22:03:52.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G3uUPgLIGZdW1pgMllG4TrqZ9rE=', 'Content-Length': '0'} 2023-09-29T22:03:52.178 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009e8a462f95e5480a-00651749c8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.178 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.178 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.179 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.179 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.179 DEBUG:boto:Method: GET 2023-09-29T22:03:52.179 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:03:52.179 DEBUG:boto:Data: 2023-09-29T22:03:52.179 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.179 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.179 DEBUG:boto:Params: {} 2023-09-29T22:03:52.179 DEBUG:boto:Token: None 2023-09-29T22:03:52.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs= 2023-09-29T22:03:52.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs=', 'Content-Length': '0'} 2023-09-29T22:03:52.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039578d99e0c57aa1-00651749c8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.226 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.227 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.227 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.227 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.227 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.227 DEBUG:boto:Method: GET 2023-09-29T22:03:52.227 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:03:52.227 DEBUG:boto:Data: 2023-09-29T22:03:52.227 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.228 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.228 DEBUG:boto:Params: {} 2023-09-29T22:03:52.228 DEBUG:boto:Token: None 2023-09-29T22:03:52.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.228 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs= 2023-09-29T22:03:52.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs=', 'Content-Length': '0'} 2023-09-29T22:03:52.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9b574a0c9258957-00651749c8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.233 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.233 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.233 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.233 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.234 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:03:52.234 DEBUG:boto:Data: 2023-09-29T22:03:52.234 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.234 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.234 DEBUG:boto:Params: {} 2023-09-29T22:03:52.234 DEBUG:boto:Token: None 2023-09-29T22:03:52.234 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj 2023-09-29T22:03:52.235 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAYSbkS8Axxrvq3ghvf1mRzimFU= 2023-09-29T22:03:52.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAYSbkS8Axxrvq3ghvf1mRzimFU=', 'Content-Length': '0'} 2023-09-29T22:03:52.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000096dac8bc8ff8e244-00651749c8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.237 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.237 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.238 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.238 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:03:52.238 DEBUG:boto:Data: 2023-09-29T22:03:52.238 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.238 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.238 DEBUG:boto:Params: {} 2023-09-29T22:03:52.238 DEBUG:boto:Token: None 2023-09-29T22:03:52.239 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj 2023-09-29T22:03:52.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAYSbkS8Axxrvq3ghvf1mRzimFU= 2023-09-29T22:03:52.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAYSbkS8Axxrvq3ghvf1mRzimFU=', 'Content-Length': '0'} 2023-09-29T22:03:52.241 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000009babd7726eee49d1-00651749c8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:03:52.242 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.242 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.242 DEBUG:boto:Method: GET 2023-09-29T22:03:52.242 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:03:52.242 DEBUG:boto:Data: 2023-09-29T22:03:52.242 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.243 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.243 DEBUG:boto:Params: {} 2023-09-29T22:03:52.243 DEBUG:boto:Token: None 2023-09-29T22:03:52.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj 2023-09-29T22:03:52.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:23HegF9cpSw1etv1U6m5vqU3sWY= 2023-09-29T22:03:52.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:23HegF9cpSw1etv1U6m5vqU3sWY=', 'Content-Length': '0'} 2023-09-29T22:03:52.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ae60df62fa52fbac-00651749c8-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.246 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.246 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.246 DEBUG:boto:Method: GET 2023-09-29T22:03:52.246 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:03:52.246 DEBUG:boto:Data: 2023-09-29T22:03:52.246 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.246 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.247 DEBUG:boto:Params: {} 2023-09-29T22:03:52.247 DEBUG:boto:Token: None 2023-09-29T22:03:52.247 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj 2023-09-29T22:03:52.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:23HegF9cpSw1etv1U6m5vqU3sWY= 2023-09-29T22:03:52.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:23HegF9cpSw1etv1U6m5vqU3sWY=', 'Content-Length': '0'} 2023-09-29T22:03:52.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000005d31c0c4e607ee22-00651749c8-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.250 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.250 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.250 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.250 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.250 DEBUG:boto:Method: GET 2023-09-29T22:03:52.250 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:03:52.251 DEBUG:boto:Data: 2023-09-29T22:03:52.251 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:03:52.251 DEBUG:boto:Port: 8001 2023-09-29T22:03:52.251 DEBUG:boto:Params: {} 2023-09-29T22:03:52.251 DEBUG:boto:Token: None 2023-09-29T22:03:52.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs= 2023-09-29T22:03:52.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs=', 'Content-Length': '0'} 2023-09-29T22:03:52.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000952cc59e88139fdf-00651749c8-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.255 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.256 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.256 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.256 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.256 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:03:52.256 DEBUG:boto:Method: GET 2023-09-29T22:03:52.256 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:03:52.257 DEBUG:boto:Data: 2023-09-29T22:03:52.257 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.257 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.257 DEBUG:boto:Params: {} 2023-09-29T22:03:52.257 DEBUG:boto:Token: None 2023-09-29T22:03:52.257 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs= 2023-09-29T22:03:52.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KzymWjcXxM1sQXD6nZsuk5hePZs=', 'Content-Length': '0'} 2023-09-29T22:03:52.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007543b41f0abfae7c-00651749c8-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.262 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:03:52.262 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-14 zones={a2, a1} 2023-09-29T22:03:52.262 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.262 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.263 DEBUG:boto:path=/hcnphz-14/?versioning 2023-09-29T22:03:52.263 DEBUG:boto:auth_path=/hcnphz-14/?versioning 2023-09-29T22:03:52.263 DEBUG:boto:Method: PUT 2023-09-29T22:03:52.263 DEBUG:boto:Path: /hcnphz-14/?versioning 2023-09-29T22:03:52.263 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:03:52.263 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.263 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.263 DEBUG:boto:Params: {} 2023-09-29T22:03:52.263 DEBUG:boto:Token: None 2023-09-29T22:03:52.264 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/?versioning 2023-09-29T22:03:52.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PR9wJrIYuUX/eoqT5S8gAZLpXRI= 2023-09-29T22:03:52.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PR9wJrIYuUX/eoqT5S8gAZLpXRI=', 'Content-Length': '228'} 2023-09-29T22:03:52.273 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006eaed18bf3c4f03e-00651749c8-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.273 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:52.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:52.273 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:52.317 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:52.317 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:52.372 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:03:52.398 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:03:52.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:03:52.398 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025032.273580_94.1", 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:03:52.273580Z" 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:52.399 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:03:52.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.400 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:52.400 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:03:52.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:52.400 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:03:52.401 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025032.273580_94.1', 2: '', 3: ''} 2023-09-29T22:03:52.401 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:03:52.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:52.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:03:52.402 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:03:52.444 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:52.444 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:52.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:52.497+0000 7feadaea2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:03:52.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:52.497+0000 7feadaea2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:03:52.520 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:03:52.520 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.521 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:52.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025032.273580_94.1", 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:52.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:03:52.273580Z", 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:52.524 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:03:52.525 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:52.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:03:52Z" 2023-09-29T22:03:52.527 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:03:52.528 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025032.273580_94.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:03:52.273580Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:03:52Z" } 2023-09-29T22:03:52.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025032.273580_94.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:03:52.273580Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:03:52.529 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025032.273580_94.1', 2: '', 3: ''} 2023-09-29T22:03:52.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025032.273580_94.1', 2: '', 3: ''} 2023-09-29T22:03:52.529 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:03:52.529 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:03:52.529 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:03:52.529 DEBUG:boto:Method: HEAD 2023-09-29T22:03:52.529 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:03:52.529 DEBUG:boto:Data: 2023-09-29T22:03:52.529 DEBUG:boto:Headers: {} 2023-09-29T22:03:52.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.530 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.530 DEBUG:boto:Params: {} 2023-09-29T22:03:52.530 DEBUG:boto:Token: None 2023-09-29T22:03:52.530 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/ 2023-09-29T22:03:52.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pg9t4QRL87NtQW3JJcxEFsEDXQk= 2023-09-29T22:03:52.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pg9t4QRL87NtQW3JJcxEFsEDXQk=', 'Content-Length': '0'} 2023-09-29T22:03:52.534 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf47869f649359ff-00651749c8-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.534 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:03:52.534 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:03:52.534 DEBUG:boto:Method: PUT 2023-09-29T22:03:52.535 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:03:52.535 DEBUG:boto:Data: 2023-09-29T22:03:52.535 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:03:52.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:03:52.535 DEBUG:boto:Port: 8000 2023-09-29T22:03:52.535 DEBUG:boto:Params: {} 2023-09-29T22:03:52.535 DEBUG:boto:Token: None 2023-09-29T22:03:52.535 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:03:52 GMT /hcnphz-14/obj 2023-09-29T22:03:52.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H+JseGq2LXDaZqwO2MD4Lp/ttRk= 2023-09-29T22:03:52.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:03:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H+JseGq2LXDaZqwO2MD4Lp/ttRk='} 2023-09-29T22:03:52.555 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-amz-request-id', 'tx00000e53a4b549769c130-00651749c8-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:03:52.555 DEBUG:rgw_multi.tests:created new version id=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:03:52.555 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-14 2023-09-29T22:03:52.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:52.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:03:52.556 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:03:52.625 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:03:52.625 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:03:52.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:52.689+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:52.688 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:03:52.689 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:03:57.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:03:57.689+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:03:57.691 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:03:57.691 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:02.693+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:02.693 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:04:02.693 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:07.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:07.696+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:07.696 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:04:07.696 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:12.698 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:12.696+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:12.699 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:04:12.699 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:17.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:17.700+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:17.701 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:04:17.701 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:22.703 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:22.700+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:22.703 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:04:22.703 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:27.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:27.704+0000 7f4b77e8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:27.706 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.9.6, , ] 2023-09-29T22:04:27.706 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:32.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:32.708+0000 7f4b77e8bf40 1 bucket sync caught up with source: 2023-09-29T22:04:32.708 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:32.708 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:32.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:32.708+0000 7f4b77e8bf40 0 bucket checkpoint complete 2023-09-29T22:04:32.735 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-14 2023-09-29T22:04:32.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:04:32.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:04:32.735 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:04:32.784 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:04:32.785 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:04:32.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:32.852+0000 7f398c98ff40 1 bucket sync caught up with source: 2023-09-29T22:04:32.852 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.22.13, , ] 2023-09-29T22:04:32.852 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.22.13, , ] 2023-09-29T22:04:32.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:32.852+0000 7f398c98ff40 0 bucket checkpoint complete 2023-09-29T22:04:32.877 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-14 zones={a2, a1} 2023-09-29T22:04:32.878 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.878 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.878 DEBUG:boto:Method: HEAD 2023-09-29T22:04:32.878 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:04:32.878 DEBUG:boto:Data: 2023-09-29T22:04:32.878 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.879 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.879 DEBUG:boto:Params: {} 2023-09-29T22:04:32.879 DEBUG:boto:Token: None 2023-09-29T22:04:32.879 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/ 2023-09-29T22:04:32.879 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oKY3v9IHZllaVd6yxqMrmsODKrk= 2023-09-29T22:04:32.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oKY3v9IHZllaVd6yxqMrmsODKrk=', 'Content-Length': '0'} 2023-09-29T22:04:32.883 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003db6a4194cc53e78-00651749f0-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.883 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.883 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.883 DEBUG:boto:Method: HEAD 2023-09-29T22:04:32.883 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:04:32.884 DEBUG:boto:Data: 2023-09-29T22:04:32.884 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:32.884 DEBUG:boto:Port: 8000 2023-09-29T22:04:32.884 DEBUG:boto:Params: {} 2023-09-29T22:04:32.884 DEBUG:boto:Token: None 2023-09-29T22:04:32.884 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/ 2023-09-29T22:04:32.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oKY3v9IHZllaVd6yxqMrmsODKrk= 2023-09-29T22:04:32.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oKY3v9IHZllaVd6yxqMrmsODKrk=', 'Content-Length': '0'} 2023-09-29T22:04:32.888 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc7c49ed15c4b5ae-00651749f0-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.888 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:04:32.888 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.888 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.888 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.888 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.888 DEBUG:boto:Method: GET 2023-09-29T22:04:32.888 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.889 DEBUG:boto:Data: 2023-09-29T22:04:32.889 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.889 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.889 DEBUG:boto:Params: {} 2023-09-29T22:04:32.889 DEBUG:boto:Token: None 2023-09-29T22:04:32.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/?versions 2023-09-29T22:04:32.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo= 2023-09-29T22:04:32.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo=', 'Content-Length': '0'} 2023-09-29T22:04:32.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc9a7da1d2f5a999-00651749f0-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.893 DEBUG:boto:b'hcnphz-14999falseobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibtrue2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:32.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:04:32.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:04:32.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:04:32.894 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.894 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.895 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.895 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.895 DEBUG:boto:Method: GET 2023-09-29T22:04:32.895 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.895 DEBUG:boto:Data: 2023-09-29T22:04:32.895 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:32.895 DEBUG:boto:Port: 8000 2023-09-29T22:04:32.895 DEBUG:boto:Params: {} 2023-09-29T22:04:32.895 DEBUG:boto:Token: None 2023-09-29T22:04:32.897 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/?versions 2023-09-29T22:04:32.897 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo= 2023-09-29T22:04:32.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo=', 'Content-Length': '0'} 2023-09-29T22:04:32.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000335400e852bff4a7-00651749f0-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.946 DEBUG:boto:b'hcnphz-14999falseobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibtrue2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:32.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:04:32.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:04:32.947 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.948 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.948 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.948 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.948 DEBUG:boto:Method: GET 2023-09-29T22:04:32.948 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.949 DEBUG:boto:Data: 2023-09-29T22:04:32.949 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.950 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.950 DEBUG:boto:Params: {} 2023-09-29T22:04:32.950 DEBUG:boto:Token: None 2023-09-29T22:04:32.951 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/?versions 2023-09-29T22:04:32.951 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo= 2023-09-29T22:04:32.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo=', 'Content-Length': '0'} 2023-09-29T22:04:32.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aad04f7278e14aff-00651749f0-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.956 DEBUG:boto:b'hcnphz-14999falseobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibtrue2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:32.957 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.957 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.958 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.958 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.958 DEBUG:boto:Method: GET 2023-09-29T22:04:32.958 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:04:32.958 DEBUG:boto:Data: 2023-09-29T22:04:32.959 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:32.959 DEBUG:boto:Port: 8000 2023-09-29T22:04:32.959 DEBUG:boto:Params: {} 2023-09-29T22:04:32.959 DEBUG:boto:Token: None 2023-09-29T22:04:32.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/?versions 2023-09-29T22:04:32.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo= 2023-09-29T22:04:32.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kAyQx6BStjwAhVRjAUlVDJsRZXo=', 'Content-Length': '0'} 2023-09-29T22:04:32.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a153ad47f3de5fd5-00651749f0-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.964 DEBUG:boto:b'hcnphz-14999falseobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibtrue2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobjnullfalse2023-09-29T22:03:41.827Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:32.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:04:32.965 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:32.965 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:32.965 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.965 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.966 DEBUG:boto:Method: GET 2023-09-29T22:04:32.966 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.966 DEBUG:boto:Data: 2023-09-29T22:04:32.966 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.967 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.967 DEBUG:boto:Params: {} 2023-09-29T22:04:32.967 DEBUG:boto:Token: None 2023-09-29T22:04:32.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4= 2023-09-29T22:04:32.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4=', 'Content-Length': '0'} 2023-09-29T22:04:32.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a874ef0e88243fac-00651749f0-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.971 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:32.971 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:32.971 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.971 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.971 DEBUG:boto:Method: GET 2023-09-29T22:04:32.972 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.972 DEBUG:boto:Data: 2023-09-29T22:04:32.972 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:32.972 DEBUG:boto:Port: 8000 2023-09-29T22:04:32.972 DEBUG:boto:Params: {} 2023-09-29T22:04:32.972 DEBUG:boto:Token: None 2023-09-29T22:04:32.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.973 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4= 2023-09-29T22:04:32.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4=', 'Content-Length': '0'} 2023-09-29T22:04:32.975 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000052503576d68cb46d-00651749f0-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.976 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:32.976 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:32.976 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.976 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.976 DEBUG:boto:Method: HEAD 2023-09-29T22:04:32.977 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.977 DEBUG:boto:Data: 2023-09-29T22:04:32.977 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.977 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.977 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.977 DEBUG:boto:Params: {} 2023-09-29T22:04:32.978 DEBUG:boto:Token: None 2023-09-29T22:04:32.978 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.978 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xx+df7yXTmEf7JAVMo+HUxRVRkM= 2023-09-29T22:04:32.978 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xx+df7yXTmEf7JAVMo+HUxRVRkM=', 'Content-Length': '0'} 2023-09-29T22:04:32.980 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f528d8f544d82512-00651749f0-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.981 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:32.981 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:32.981 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.981 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.982 DEBUG:boto:Method: HEAD 2023-09-29T22:04:32.982 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.982 DEBUG:boto:Data: 2023-09-29T22:04:32.982 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:32.982 DEBUG:boto:Port: 8000 2023-09-29T22:04:32.982 DEBUG:boto:Params: {} 2023-09-29T22:04:32.982 DEBUG:boto:Token: None 2023-09-29T22:04:32.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xx+df7yXTmEf7JAVMo+HUxRVRkM= 2023-09-29T22:04:32.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xx+df7yXTmEf7JAVMo+HUxRVRkM=', 'Content-Length': '0'} 2023-09-29T22:04:32.985 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003268a8762075cb1c-00651749f0-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:04:32.986 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:32.986 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:32.986 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.986 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.986 DEBUG:boto:Method: GET 2023-09-29T22:04:32.986 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.986 DEBUG:boto:Data: 2023-09-29T22:04:32.986 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.987 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.987 DEBUG:boto:Params: {} 2023-09-29T22:04:32.987 DEBUG:boto:Token: None 2023-09-29T22:04:32.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.987 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4= 2023-09-29T22:04:32.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4=', 'Content-Length': '0'} 2023-09-29T22:04:32.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d5c7de1a1270ed49-00651749f0-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.990 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:32.990 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:32.990 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.990 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.990 DEBUG:boto:Method: GET 2023-09-29T22:04:32.991 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.991 DEBUG:boto:Data: 2023-09-29T22:04:32.991 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:32.991 DEBUG:boto:Port: 8000 2023-09-29T22:04:32.991 DEBUG:boto:Params: {} 2023-09-29T22:04:32.991 DEBUG:boto:Token: None 2023-09-29T22:04:32.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:32.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4= 2023-09-29T22:04:32.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0WZHAVLCwQjevuhtge6ET3drBE4=', 'Content-Length': '0'} 2023-09-29T22:04:32.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d845a60f281f5d4f-00651749f0-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:32.994 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:32.994 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:32.995 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:32.995 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:32.995 DEBUG:boto:Method: GET 2023-09-29T22:04:32.995 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:32.995 DEBUG:boto:Data: 2023-09-29T22:04:32.995 DEBUG:boto:Headers: {} 2023-09-29T22:04:32.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:32.995 DEBUG:boto:Port: 8001 2023-09-29T22:04:32.996 DEBUG:boto:Params: {} 2023-09-29T22:04:32.996 DEBUG:boto:Token: None 2023-09-29T22:04:32.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:32 GMT /hcnphz-14/ 2023-09-29T22:04:32.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YhKS5gztH1/cVCH1K3pwmwwjAFQ= 2023-09-29T22:04:32.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YhKS5gztH1/cVCH1K3pwmwwjAFQ=', 'Content-Length': '0'} 2023-09-29T22:04:33.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed0d6411e017e87e-00651749f0-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.046 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.047 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.047 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.047 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.047 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.048 DEBUG:boto:Method: GET 2023-09-29T22:04:33.048 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.048 DEBUG:boto:Data: 2023-09-29T22:04:33.048 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.048 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.048 DEBUG:boto:Params: {} 2023-09-29T22:04:33.048 DEBUG:boto:Token: None 2023-09-29T22:04:33.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fc98f12147a9793-00651749f1-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.053 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.054 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.054 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.054 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.054 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:04:33.054 DEBUG:boto:Data: 2023-09-29T22:04:33.054 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.054 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.054 DEBUG:boto:Params: {} 2023-09-29T22:04:33.055 DEBUG:boto:Token: None 2023-09-29T22:04:33.055 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj 2023-09-29T22:04:33.055 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0= 2023-09-29T22:04:33.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0=', 'Content-Length': '0'} 2023-09-29T22:04:33.059 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b894d2a182d051b4-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.059 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.059 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.059 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.059 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:04:33.059 DEBUG:boto:Data: 2023-09-29T22:04:33.060 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.060 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.060 DEBUG:boto:Params: {} 2023-09-29T22:04:33.060 DEBUG:boto:Token: None 2023-09-29T22:04:33.060 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj 2023-09-29T22:04:33.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0= 2023-09-29T22:04:33.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0=', 'Content-Length': '0'} 2023-09-29T22:04:33.064 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000088cebe87d744c189-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:04:33.064 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.064 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.064 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.064 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.065 DEBUG:boto:Method: GET 2023-09-29T22:04:33.065 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.065 DEBUG:boto:Data: 2023-09-29T22:04:33.065 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.065 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.065 DEBUG:boto:Params: {} 2023-09-29T22:04:33.065 DEBUG:boto:Token: None 2023-09-29T22:04:33.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs= 2023-09-29T22:04:33.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs=', 'Content-Length': '0'} 2023-09-29T22:04:33.068 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a665a1011ac6c528-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.069 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.069 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.069 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.069 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.069 DEBUG:boto:Method: GET 2023-09-29T22:04:33.069 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.069 DEBUG:boto:Data: 2023-09-29T22:04:33.069 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.070 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.070 DEBUG:boto:Params: {} 2023-09-29T22:04:33.070 DEBUG:boto:Token: None 2023-09-29T22:04:33.070 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.070 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs= 2023-09-29T22:04:33.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs=', 'Content-Length': '0'} 2023-09-29T22:04:33.073 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000007faf6db7addcaa67-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.073 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.073 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.073 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.073 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.073 DEBUG:boto:Method: GET 2023-09-29T22:04:33.074 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.074 DEBUG:boto:Data: 2023-09-29T22:04:33.074 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.074 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.074 DEBUG:boto:Params: {} 2023-09-29T22:04:33.074 DEBUG:boto:Token: None 2023-09-29T22:04:33.074 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.075 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008355ad394e6a8fbb-00651749f1-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.122 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.123 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.123 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.123 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.123 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.124 DEBUG:boto:Method: GET 2023-09-29T22:04:33.124 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.124 DEBUG:boto:Data: 2023-09-29T22:04:33.125 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.125 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.125 DEBUG:boto:Params: {} 2023-09-29T22:04:33.125 DEBUG:boto:Token: None 2023-09-29T22:04:33.125 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.125 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7132fde9d0d6a61-00651749f1-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.130 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:04:33.131 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.131 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.131 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.131 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.131 DEBUG:boto:Method: GET 2023-09-29T22:04:33.131 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.131 DEBUG:boto:Data: 2023-09-29T22:04:33.132 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.133 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.133 DEBUG:boto:Params: {} 2023-09-29T22:04:33.133 DEBUG:boto:Token: None 2023-09-29T22:04:33.134 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.134 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY= 2023-09-29T22:04:33.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY=', 'Content-Length': '0'} 2023-09-29T22:04:33.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000505c6fd687f47087-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.137 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.137 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.137 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.138 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.138 DEBUG:boto:Method: GET 2023-09-29T22:04:33.138 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.138 DEBUG:boto:Data: 2023-09-29T22:04:33.138 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.138 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.138 DEBUG:boto:Params: {} 2023-09-29T22:04:33.138 DEBUG:boto:Token: None 2023-09-29T22:04:33.139 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.139 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY= 2023-09-29T22:04:33.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY=', 'Content-Length': '0'} 2023-09-29T22:04:33.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001b8456baaca320ee-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.142 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.142 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.142 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.142 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.142 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.142 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.142 DEBUG:boto:Data: 2023-09-29T22:04:33.142 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.143 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.143 DEBUG:boto:Params: {} 2023-09-29T22:04:33.143 DEBUG:boto:Token: None 2023-09-29T22:04:33.143 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.143 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XyHt8hVxZAeCCHkDz8FUQb5JxfY= 2023-09-29T22:04:33.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XyHt8hVxZAeCCHkDz8FUQb5JxfY=', 'Content-Length': '0'} 2023-09-29T22:04:33.146 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006c00cf54a255e2f3-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.146 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.146 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.146 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.147 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.147 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.147 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.147 DEBUG:boto:Data: 2023-09-29T22:04:33.147 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.147 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.147 DEBUG:boto:Params: {} 2023-09-29T22:04:33.147 DEBUG:boto:Token: None 2023-09-29T22:04:33.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XyHt8hVxZAeCCHkDz8FUQb5JxfY= 2023-09-29T22:04:33.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XyHt8hVxZAeCCHkDz8FUQb5JxfY=', 'Content-Length': '0'} 2023-09-29T22:04:33.150 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e69da25cd3aa02ec-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:04:33.151 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.151 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.151 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.151 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.151 DEBUG:boto:Method: GET 2023-09-29T22:04:33.151 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.151 DEBUG:boto:Data: 2023-09-29T22:04:33.152 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.152 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.152 DEBUG:boto:Params: {} 2023-09-29T22:04:33.152 DEBUG:boto:Token: None 2023-09-29T22:04:33.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY= 2023-09-29T22:04:33.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY=', 'Content-Length': '0'} 2023-09-29T22:04:33.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ba7e1f6b728be40e-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.156 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.156 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.156 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.156 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.157 DEBUG:boto:Method: GET 2023-09-29T22:04:33.157 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.157 DEBUG:boto:Data: 2023-09-29T22:04:33.158 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.158 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.159 DEBUG:boto:Params: {} 2023-09-29T22:04:33.159 DEBUG:boto:Token: None 2023-09-29T22:04:33.159 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:04:33.160 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY= 2023-09-29T22:04:33.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WFiivkf0PowTR2ngparuaH55uRY=', 'Content-Length': '0'} 2023-09-29T22:04:33.163 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:41 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000099456d5cd50a9ca4-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.164 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.164 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.164 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.164 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.164 DEBUG:boto:Method: GET 2023-09-29T22:04:33.165 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.165 DEBUG:boto:Data: 2023-09-29T22:04:33.165 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.165 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.166 DEBUG:boto:Params: {} 2023-09-29T22:04:33.166 DEBUG:boto:Token: None 2023-09-29T22:04:33.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2da365f9ce9fedf-00651749f1-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.214 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.215 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.215 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.216 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.216 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.216 DEBUG:boto:Method: GET 2023-09-29T22:04:33.216 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.216 DEBUG:boto:Data: 2023-09-29T22:04:33.216 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.217 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.217 DEBUG:boto:Params: {} 2023-09-29T22:04:33.217 DEBUG:boto:Token: None 2023-09-29T22:04:33.218 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.218 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004454d4062905a86b-00651749f1-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.222 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.223 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.223 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.223 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.223 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:04:33.224 DEBUG:boto:Data: 2023-09-29T22:04:33.224 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.224 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.224 DEBUG:boto:Params: {} 2023-09-29T22:04:33.225 DEBUG:boto:Token: None 2023-09-29T22:04:33.225 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj 2023-09-29T22:04:33.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0= 2023-09-29T22:04:33.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0=', 'Content-Length': '0'} 2023-09-29T22:04:33.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d62db60a4d616ab7-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.229 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.229 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.229 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.229 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:04:33.229 DEBUG:boto:Data: 2023-09-29T22:04:33.230 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.230 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.230 DEBUG:boto:Params: {} 2023-09-29T22:04:33.230 DEBUG:boto:Token: None 2023-09-29T22:04:33.230 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj 2023-09-29T22:04:33.231 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0= 2023-09-29T22:04:33.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mkKRyBp+G00nCuFBShDaGoCChJ0=', 'Content-Length': '0'} 2023-09-29T22:04:33.234 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c15a3ec9923a9990-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:04:33.234 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.235 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.235 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.235 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.235 DEBUG:boto:Method: GET 2023-09-29T22:04:33.235 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.235 DEBUG:boto:Data: 2023-09-29T22:04:33.235 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.235 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.235 DEBUG:boto:Params: {} 2023-09-29T22:04:33.236 DEBUG:boto:Token: None 2023-09-29T22:04:33.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs= 2023-09-29T22:04:33.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs=', 'Content-Length': '0'} 2023-09-29T22:04:33.239 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000072b4f48b4173d0ba-00651749f1-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.239 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.239 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.239 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.240 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.240 DEBUG:boto:Method: GET 2023-09-29T22:04:33.240 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.240 DEBUG:boto:Data: 2023-09-29T22:04:33.241 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.241 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.241 DEBUG:boto:Params: {} 2023-09-29T22:04:33.241 DEBUG:boto:Token: None 2023-09-29T22:04:33.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:04:33.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs= 2023-09-29T22:04:33.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TQoJ9EOeAaWWfsH99HYwKWFNzxs=', 'Content-Length': '0'} 2023-09-29T22:04:33.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000006ef3434e8dc45a3d-00651749f1-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.245 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.245 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.245 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.246 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.246 DEBUG:boto:Method: GET 2023-09-29T22:04:33.246 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.246 DEBUG:boto:Data: 2023-09-29T22:04:33.246 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:04:33.247 DEBUG:boto:Port: 8001 2023-09-29T22:04:33.247 DEBUG:boto:Params: {} 2023-09-29T22:04:33.247 DEBUG:boto:Token: None 2023-09-29T22:04:33.247 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.252 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009b77bfbb37bd4d0-00651749f1-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.294 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.295 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.295 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.295 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.296 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:04:33.296 DEBUG:boto:Method: GET 2023-09-29T22:04:33.296 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:04:33.296 DEBUG:boto:Data: 2023-09-29T22:04:33.296 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.297 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.297 DEBUG:boto:Params: {} 2023-09-29T22:04:33.297 DEBUG:boto:Token: None 2023-09-29T22:04:33.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE= 2023-09-29T22:04:33.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OkC+fVYxwY06rnOTtkwn9D6XAdE=', 'Content-Length': '0'} 2023-09-29T22:04:33.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8ba6b6b8ade7e5e-00651749f1-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.303 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:04:33.303 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-14 zones={a2, a1} 2023-09-29T22:04:33.303 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.303 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.304 DEBUG:boto:path=/hcnphz-14/?versioning 2023-09-29T22:04:33.304 DEBUG:boto:auth_path=/hcnphz-14/?versioning 2023-09-29T22:04:33.304 DEBUG:boto:Method: PUT 2023-09-29T22:04:33.304 DEBUG:boto:Path: /hcnphz-14/?versioning 2023-09-29T22:04:33.304 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:04:33.304 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.304 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.304 DEBUG:boto:Params: {} 2023-09-29T22:04:33.305 DEBUG:boto:Token: None 2023-09-29T22:04:33.305 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/?versioning 2023-09-29T22:04:33.305 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DgLqwb3h04na2v+jGqOXvOgI7PA= 2023-09-29T22:04:33.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DgLqwb3h04na2v+jGqOXvOgI7PA=', 'Content-Length': '230'} 2023-09-29T22:04:33.316 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000686ef710f882d147-00651749f1-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:04:33.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:04:33.317 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:04:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:04:33.361 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:04:33.419 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:04:33.444 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:04:33.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.444 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:04:33.444 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025073.316218_96.1", 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:04:33.316218Z" 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:04:33.446 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:04:33.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:04:33.446 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:04:33.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:04:33.447 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:04:33.448 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025073.316218_96.1', 2: '', 3: ''} 2023-09-29T22:04:33.448 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:04:33.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:04:33.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:04:33.449 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:04:33.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:04:33.541 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:04:33.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:33.588+0000 7f3e5ee37f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:04:33.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:33.588+0000 7f3e5ee37f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:04:33.613 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:04:33.613 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:04:33.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:04:33.615 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025073.316218_96.1", 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:04:33.616 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:04:33.316218Z", 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:04:33.617 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:04:33.618 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:04:33.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:04:33Z" 2023-09-29T22:04:33.620 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:04:33.621 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025073.316218_96.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:04:33.316218Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:04:33Z" } 2023-09-29T22:04:33.621 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025073.316218_96.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:04:33.316218Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:04:33.622 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025073.316218_96.1', 2: '', 3: ''} 2023-09-29T22:04:33.622 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025073.316218_96.1', 2: '', 3: ''} 2023-09-29T22:04:33.622 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:04:33.622 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:04:33.622 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:04:33.622 DEBUG:boto:Method: HEAD 2023-09-29T22:04:33.622 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:04:33.623 DEBUG:boto:Data: 2023-09-29T22:04:33.623 DEBUG:boto:Headers: {} 2023-09-29T22:04:33.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.623 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.623 DEBUG:boto:Params: {} 2023-09-29T22:04:33.623 DEBUG:boto:Token: None 2023-09-29T22:04:33.623 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/ 2023-09-29T22:04:33.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e4VcFU7z1c1BS0p4QXCpU57WHnA= 2023-09-29T22:04:33.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e4VcFU7z1c1BS0p4QXCpU57WHnA=', 'Content-Length': '0'} 2023-09-29T22:04:33.627 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c001b4d7e00be23a-00651749f1-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.628 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:04:33.628 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:04:33.628 DEBUG:boto:Method: PUT 2023-09-29T22:04:33.628 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:04:33.628 DEBUG:boto:Data: 2023-09-29T22:04:33.628 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:04:33.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:04:33.629 DEBUG:boto:Port: 8000 2023-09-29T22:04:33.629 DEBUG:boto:Params: {} 2023-09-29T22:04:33.629 DEBUG:boto:Token: None 2023-09-29T22:04:33.629 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:04:33 GMT /hcnphz-14/obj 2023-09-29T22:04:33.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Hfx8j6dE67pLXSyNRPbNmciMVmQ= 2023-09-29T22:04:33.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:04:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Hfx8j6dE67pLXSyNRPbNmciMVmQ='} 2023-09-29T22:04:33.678 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008dba0f582da4d45a-00651749f1-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:04:33.679 DEBUG:rgw_multi.tests:created null version id=None 2023-09-29T22:04:33.679 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-14 2023-09-29T22:04:33.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:04:33.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:04:33.679 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-14 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:04:33.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:04:33.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:04:33.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:33.792+0000 7ff27571df40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:33.795 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:33.796 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:04:38.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:38.792+0000 7ff27571df40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:38.794 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:38.794 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:04:43.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:43.796+0000 7ff27571df40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:43.796 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:43.797 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:04:48.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:48.795+0000 7ff27571df40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:48.799 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.12.13, , ] 2023-09-29T22:04:48.799 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:04:53.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:53.799+0000 7ff27571df40 1 bucket sync caught up with source: 2023-09-29T22:04:53.801 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:04:53.801 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.16.18, , ] 2023-09-29T22:04:53.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:53.799+0000 7ff27571df40 0 bucket checkpoint complete 2023-09-29T22:04:53.829 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-14 2023-09-29T22:04:53.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:04:53.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-14', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:04:53.830 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-14 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:04:53.880 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:04:53.880 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:04:53.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:53.939+0000 7efe23f3ef40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:53.941 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.22.13, , ] 2023-09-29T22:04:53.941 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.27.18, , ] 2023-09-29T22:04:58.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:04:58.943+0000 7efe23f3ef40 1 waiting for incremental sync to catch up: 2023-09-29T22:04:58.944 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.22.13, , ] 2023-09-29T22:04:58.944 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.27.18, , ] 2023-09-29T22:05:03.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:03.943+0000 7efe23f3ef40 1 waiting for incremental sync to catch up: 2023-09-29T22:05:03.946 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.22.13, , ] 2023-09-29T22:05:03.946 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.27.18, , ] 2023-09-29T22:05:08.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:08.947+0000 7efe23f3ef40 1 waiting for incremental sync to catch up: 2023-09-29T22:05:08.949 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.22.13, , ] 2023-09-29T22:05:08.949 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.27.18, , ] 2023-09-29T22:05:13.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:13.947+0000 7efe23f3ef40 1 bucket sync caught up with source: 2023-09-29T22:05:13.951 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000005.27.18, , ] 2023-09-29T22:05:13.951 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000005.27.18, , ] 2023-09-29T22:05:13.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:13.947+0000 7efe23f3ef40 0 bucket checkpoint complete 2023-09-29T22:05:13.981 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-14 zones={a2, a1} 2023-09-29T22:05:13.981 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:13.981 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:13.982 DEBUG:boto:Method: HEAD 2023-09-29T22:05:13.982 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:05:13.982 DEBUG:boto:Data: 2023-09-29T22:05:13.982 DEBUG:boto:Headers: {} 2023-09-29T22:05:13.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:13.982 DEBUG:boto:Port: 8001 2023-09-29T22:05:13.982 DEBUG:boto:Params: {} 2023-09-29T22:05:13.982 DEBUG:boto:Token: None 2023-09-29T22:05:13.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:13 GMT /hcnphz-14/ 2023-09-29T22:05:13.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2nCUQCxUzGh2pdizdtbUUizpF8= 2023-09-29T22:05:13.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2nCUQCxUzGh2pdizdtbUUizpF8=', 'Content-Length': '0'} 2023-09-29T22:05:13.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e4097a52c47a226-0065174a19-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:13.987 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:13.987 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:13.987 DEBUG:boto:Method: HEAD 2023-09-29T22:05:13.987 DEBUG:boto:Path: /hcnphz-14/ 2023-09-29T22:05:13.987 DEBUG:boto:Data: 2023-09-29T22:05:13.987 DEBUG:boto:Headers: {} 2023-09-29T22:05:13.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:13.987 DEBUG:boto:Port: 8000 2023-09-29T22:05:13.987 DEBUG:boto:Params: {} 2023-09-29T22:05:13.988 DEBUG:boto:Token: None 2023-09-29T22:05:13.988 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:13 GMT /hcnphz-14/ 2023-09-29T22:05:13.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2nCUQCxUzGh2pdizdtbUUizpF8= 2023-09-29T22:05:13.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2nCUQCxUzGh2pdizdtbUUizpF8=', 'Content-Length': '0'} 2023-09-29T22:05:13.991 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d399a2234864724c-0065174a19-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:13.991 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:05:13.992 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:13.992 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:13.992 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:13.992 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:13.992 DEBUG:boto:Method: GET 2023-09-29T22:05:13.992 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:13.992 DEBUG:boto:Data: 2023-09-29T22:05:13.992 DEBUG:boto:Headers: {} 2023-09-29T22:05:13.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:13.993 DEBUG:boto:Port: 8001 2023-09-29T22:05:13.993 DEBUG:boto:Params: {} 2023-09-29T22:05:13.993 DEBUG:boto:Token: None 2023-09-29T22:05:13.993 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:13 GMT /hcnphz-14/?versions 2023-09-29T22:05:13.993 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Finzk+Kw2sKT5NRQRmjv+k6Or8M= 2023-09-29T22:05:13.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Finzk+Kw2sKT5NRQRmjv+k6Or8M=', 'Content-Length': '0'} 2023-09-29T22:05:13.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075877d43ce45c274-0065174a19-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:13.997 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibfalse2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:13.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:13.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:13.997 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:05:13.998 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:13.998 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:13.998 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:13.998 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:13.998 DEBUG:boto:Method: GET 2023-09-29T22:05:13.998 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:13.998 DEBUG:boto:Data: 2023-09-29T22:05:13.998 DEBUG:boto:Headers: {} 2023-09-29T22:05:13.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:13.998 DEBUG:boto:Port: 8000 2023-09-29T22:05:13.999 DEBUG:boto:Params: {} 2023-09-29T22:05:13.999 DEBUG:boto:Token: None 2023-09-29T22:05:13.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:13 GMT /hcnphz-14/?versions 2023-09-29T22:05:13.999 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Finzk+Kw2sKT5NRQRmjv+k6Or8M= 2023-09-29T22:05:13.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Finzk+Kw2sKT5NRQRmjv+k6Or8M=', 'Content-Length': '0'} 2023-09-29T22:05:14.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d737b594b8092951-0065174a19-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.002 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibfalse2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:14.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:14.003 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.003 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.003 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:14.003 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:14.003 DEBUG:boto:Method: GET 2023-09-29T22:05:14.003 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:14.003 DEBUG:boto:Data: 2023-09-29T22:05:14.003 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.004 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.004 DEBUG:boto:Params: {} 2023-09-29T22:05:14.004 DEBUG:boto:Token: None 2023-09-29T22:05:14.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/?versions 2023-09-29T22:05:14.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P2LMaq9x9uBVi1dBYoxepPY43DU= 2023-09-29T22:05:14.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P2LMaq9x9uBVi1dBYoxepPY43DU=', 'Content-Length': '0'} 2023-09-29T22:05:14.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000372c08437af7757-0065174a1a-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.008 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibfalse2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.009 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.009 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.009 DEBUG:boto:path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:14.009 DEBUG:boto:auth_path=/hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:14.009 DEBUG:boto:Method: GET 2023-09-29T22:05:14.009 DEBUG:boto:Path: /hcnphz-14/?versions&max-keys=999 2023-09-29T22:05:14.009 DEBUG:boto:Data: 2023-09-29T22:05:14.009 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.010 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.010 DEBUG:boto:Params: {} 2023-09-29T22:05:14.010 DEBUG:boto:Token: None 2023-09-29T22:05:14.010 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/?versions 2023-09-29T22:05:14.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P2LMaq9x9uBVi1dBYoxepPY43DU= 2023-09-29T22:05:14.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P2LMaq9x9uBVi1dBYoxepPY43DU=', 'Content-Length': '0'} 2023-09-29T22:05:14.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024444c30eebb8003-0065174a1a-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.014 DEBUG:boto:b'hcnphz-14999falseobjnulltrue2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormalobj0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tibfalse2023-09-29T22:03:52.539Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:14.015 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.015 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.015 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.015 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.015 DEBUG:boto:Method: GET 2023-09-29T22:05:14.015 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.015 DEBUG:boto:Data: 2023-09-29T22:05:14.015 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.016 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.016 DEBUG:boto:Params: {} 2023-09-29T22:05:14.016 DEBUG:boto:Token: None 2023-09-29T22:05:14.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM= 2023-09-29T22:05:14.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM=', 'Content-Length': '0'} 2023-09-29T22:05:14.019 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000f7ee5707a17210c3-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.019 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.019 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.019 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.019 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.020 DEBUG:boto:Method: GET 2023-09-29T22:05:14.020 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.020 DEBUG:boto:Data: 2023-09-29T22:05:14.020 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.020 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.020 DEBUG:boto:Params: {} 2023-09-29T22:05:14.020 DEBUG:boto:Token: None 2023-09-29T22:05:14.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM= 2023-09-29T22:05:14.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM=', 'Content-Length': '0'} 2023-09-29T22:05:14.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c3b77785f53577fa-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.023 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.023 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.023 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.024 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.024 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.024 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.024 DEBUG:boto:Data: 2023-09-29T22:05:14.024 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.024 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.024 DEBUG:boto:Params: {} 2023-09-29T22:05:14.024 DEBUG:boto:Token: None 2023-09-29T22:05:14.025 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aS1ptlFZNb9xLlSTj7uHPmnkwJA= 2023-09-29T22:05:14.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aS1ptlFZNb9xLlSTj7uHPmnkwJA=', 'Content-Length': '0'} 2023-09-29T22:05:14.027 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000979021ee1108ffcd-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.027 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.027 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.027 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.027 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.027 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.027 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.027 DEBUG:boto:Data: 2023-09-29T22:05:14.028 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.028 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.028 DEBUG:boto:Params: {} 2023-09-29T22:05:14.028 DEBUG:boto:Token: None 2023-09-29T22:05:14.028 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.028 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aS1ptlFZNb9xLlSTj7uHPmnkwJA= 2023-09-29T22:05:14.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aS1ptlFZNb9xLlSTj7uHPmnkwJA=', 'Content-Length': '0'} 2023-09-29T22:05:14.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000003df268a1f39de7aa-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:14.030 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.031 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.031 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.031 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.031 DEBUG:boto:Method: GET 2023-09-29T22:05:14.031 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.031 DEBUG:boto:Data: 2023-09-29T22:05:14.031 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.031 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.031 DEBUG:boto:Params: {} 2023-09-29T22:05:14.032 DEBUG:boto:Token: None 2023-09-29T22:05:14.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM= 2023-09-29T22:05:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM=', 'Content-Length': '0'} 2023-09-29T22:05:14.033 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000e98dbe670b6d4c0-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.034 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.034 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.034 DEBUG:boto:path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.034 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.034 DEBUG:boto:Method: GET 2023-09-29T22:05:14.034 DEBUG:boto:Path: /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.034 DEBUG:boto:Data: 2023-09-29T22:05:14.035 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.035 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.035 DEBUG:boto:Params: {} 2023-09-29T22:05:14.035 DEBUG:boto:Token: None 2023-09-29T22:05:14.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=null 2023-09-29T22:05:14.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM= 2023-09-29T22:05:14.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bx+G1nfu0h7UwIokct3zIoZ9dsM=', 'Content-Length': '0'} 2023-09-29T22:05:14.037 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000ed8cb65e2c0296f8-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.037 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.037 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.038 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.038 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.038 DEBUG:boto:Method: GET 2023-09-29T22:05:14.038 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.038 DEBUG:boto:Data: 2023-09-29T22:05:14.038 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.038 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.038 DEBUG:boto:Params: {} 2023-09-29T22:05:14.038 DEBUG:boto:Token: None 2023-09-29T22:05:14.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.039 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a55fc9d951fd1a8b-0065174a1a-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.090 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.091 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.091 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.091 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.091 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.091 DEBUG:boto:Method: GET 2023-09-29T22:05:14.091 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.092 DEBUG:boto:Data: 2023-09-29T22:05:14.092 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.092 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.092 DEBUG:boto:Params: {} 2023-09-29T22:05:14.092 DEBUG:boto:Token: None 2023-09-29T22:05:14.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa31fca3c6bd6d86-0065174a1a-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.097 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.098 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.098 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.098 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.098 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.098 DEBUG:boto:Data: 2023-09-29T22:05:14.098 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.099 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.099 DEBUG:boto:Params: {} 2023-09-29T22:05:14.099 DEBUG:boto:Token: None 2023-09-29T22:05:14.099 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.099 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo= 2023-09-29T22:05:14.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo=', 'Content-Length': '0'} 2023-09-29T22:05:14.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000c174cc14303490b4-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.102 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.102 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.103 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.103 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.103 DEBUG:boto:Data: 2023-09-29T22:05:14.103 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.103 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.103 DEBUG:boto:Params: {} 2023-09-29T22:05:14.103 DEBUG:boto:Token: None 2023-09-29T22:05:14.103 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.103 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo= 2023-09-29T22:05:14.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo=', 'Content-Length': '0'} 2023-09-29T22:05:14.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000540b6c4723b6fe53-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:14.107 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.107 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.107 DEBUG:boto:Method: GET 2023-09-29T22:05:14.107 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.107 DEBUG:boto:Data: 2023-09-29T22:05:14.107 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.108 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.108 DEBUG:boto:Params: {} 2023-09-29T22:05:14.108 DEBUG:boto:Token: None 2023-09-29T22:05:14.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU= 2023-09-29T22:05:14.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU=', 'Content-Length': '0'} 2023-09-29T22:05:14.111 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000611bb177bdd31ee4-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.111 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.111 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.111 DEBUG:boto:Method: GET 2023-09-29T22:05:14.111 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.111 DEBUG:boto:Data: 2023-09-29T22:05:14.111 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.112 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.112 DEBUG:boto:Params: {} 2023-09-29T22:05:14.112 DEBUG:boto:Token: None 2023-09-29T22:05:14.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU= 2023-09-29T22:05:14.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU=', 'Content-Length': '0'} 2023-09-29T22:05:14.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000fcdf7b2af4a470f2-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.115 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.115 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.115 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.115 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.115 DEBUG:boto:Method: GET 2023-09-29T22:05:14.115 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.115 DEBUG:boto:Data: 2023-09-29T22:05:14.115 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.116 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.116 DEBUG:boto:Params: {} 2023-09-29T22:05:14.116 DEBUG:boto:Token: None 2023-09-29T22:05:14.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.121 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ed55469f3ddfc19-0065174a1a-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.121 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.122 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.122 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.122 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.122 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.122 DEBUG:boto:Method: GET 2023-09-29T22:05:14.123 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.123 DEBUG:boto:Data: 2023-09-29T22:05:14.123 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.123 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.123 DEBUG:boto:Params: {} 2023-09-29T22:05:14.123 DEBUG:boto:Token: None 2023-09-29T22:05:14.123 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000660e3496c5097f0-0065174a1a-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.174 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:14.175 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.175 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.175 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.175 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.176 DEBUG:boto:Method: GET 2023-09-29T22:05:14.176 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.176 DEBUG:boto:Data: 2023-09-29T22:05:14.176 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.176 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.176 DEBUG:boto:Params: {} 2023-09-29T22:05:14.176 DEBUG:boto:Token: None 2023-09-29T22:05:14.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.177 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo= 2023-09-29T22:05:14.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo=', 'Content-Length': '0'} 2023-09-29T22:05:14.180 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000092f7cfc74af78876-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.180 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.180 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.180 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.180 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.180 DEBUG:boto:Method: GET 2023-09-29T22:05:14.180 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.181 DEBUG:boto:Data: 2023-09-29T22:05:14.181 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.181 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.181 DEBUG:boto:Params: {} 2023-09-29T22:05:14.181 DEBUG:boto:Token: None 2023-09-29T22:05:14.181 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo= 2023-09-29T22:05:14.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo=', 'Content-Length': '0'} 2023-09-29T22:05:14.184 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000f72ef4047a77ba4-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.184 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.184 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.184 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.185 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.185 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.185 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.185 DEBUG:boto:Data: 2023-09-29T22:05:14.185 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.185 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.185 DEBUG:boto:Params: {} 2023-09-29T22:05:14.185 DEBUG:boto:Token: None 2023-09-29T22:05:14.186 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.186 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z+5ef7FvXtD1WXVSfX4YcER1zkE= 2023-09-29T22:05:14.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z+5ef7FvXtD1WXVSfX4YcER1zkE=', 'Content-Length': '0'} 2023-09-29T22:05:14.188 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000008abe246e679d6d4c-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.189 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.189 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.189 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.189 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.189 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.189 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.189 DEBUG:boto:Data: 2023-09-29T22:05:14.189 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.190 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.190 DEBUG:boto:Params: {} 2023-09-29T22:05:14.190 DEBUG:boto:Token: None 2023-09-29T22:05:14.190 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.190 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z+5ef7FvXtD1WXVSfX4YcER1zkE= 2023-09-29T22:05:14.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z+5ef7FvXtD1WXVSfX4YcER1zkE=', 'Content-Length': '0'} 2023-09-29T22:05:14.193 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000077659f7f03ffda6c-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:14.194 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.194 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.194 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.194 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.194 DEBUG:boto:Method: GET 2023-09-29T22:05:14.194 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.195 DEBUG:boto:Data: 2023-09-29T22:05:14.195 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.195 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.195 DEBUG:boto:Params: {} 2023-09-29T22:05:14.195 DEBUG:boto:Token: None 2023-09-29T22:05:14.195 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo= 2023-09-29T22:05:14.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo=', 'Content-Length': '0'} 2023-09-29T22:05:14.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000b170c3dafed47180-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.199 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.200 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.200 DEBUG:boto:path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.200 DEBUG:boto:auth_path=/hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.200 DEBUG:boto:Method: GET 2023-09-29T22:05:14.201 DEBUG:boto:Path: /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.201 DEBUG:boto:Data: 2023-09-29T22:05:14.201 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.201 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.201 DEBUG:boto:Params: {} 2023-09-29T22:05:14.202 DEBUG:boto:Token: None 2023-09-29T22:05:14.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj?versionId=0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib 2023-09-29T22:05:14.202 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo= 2023-09-29T22:05:14.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nwtbwy4W0kCZX+sjxV+Lyh6CMbo=', 'Content-Length': '0'} 2023-09-29T22:05:14.205 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:03:52 GMT'), ('x-amz-version-id', '0PE5fTtGf8ewxzbYZgehlJ3e0Dv1tib'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000aed4eda27f2998fe-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.205 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.205 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.205 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.206 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.206 DEBUG:boto:Method: GET 2023-09-29T22:05:14.206 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.206 DEBUG:boto:Data: 2023-09-29T22:05:14.206 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.206 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.206 DEBUG:boto:Params: {} 2023-09-29T22:05:14.207 DEBUG:boto:Token: None 2023-09-29T22:05:14.207 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b23a7e1b7379b3a-0065174a1a-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.254 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.255 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.255 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.255 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.255 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.255 DEBUG:boto:Method: GET 2023-09-29T22:05:14.256 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.256 DEBUG:boto:Data: 2023-09-29T22:05:14.256 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.256 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.256 DEBUG:boto:Params: {} 2023-09-29T22:05:14.256 DEBUG:boto:Token: None 2023-09-29T22:05:14.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b1d92431866467b-0065174a1a-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.262 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.262 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.262 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.263 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.263 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.263 DEBUG:boto:Data: 2023-09-29T22:05:14.263 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.264 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.264 DEBUG:boto:Params: {} 2023-09-29T22:05:14.264 DEBUG:boto:Token: None 2023-09-29T22:05:14.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo= 2023-09-29T22:05:14.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo=', 'Content-Length': '0'} 2023-09-29T22:05:14.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000001d0bd3c6f2ca6bf7-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.269 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.269 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.269 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.269 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.269 DEBUG:boto:Data: 2023-09-29T22:05:14.269 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.270 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.270 DEBUG:boto:Params: {} 2023-09-29T22:05:14.270 DEBUG:boto:Token: None 2023-09-29T22:05:14.270 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo= 2023-09-29T22:05:14.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OHW32V5LUVBYVxageiBOjQWIjNo=', 'Content-Length': '0'} 2023-09-29T22:05:14.273 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000771707f2b5bb67a4-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:14.273 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.273 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.273 DEBUG:boto:Method: GET 2023-09-29T22:05:14.274 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.274 DEBUG:boto:Data: 2023-09-29T22:05:14.274 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.274 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.274 DEBUG:boto:Params: {} 2023-09-29T22:05:14.274 DEBUG:boto:Token: None 2023-09-29T22:05:14.274 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU= 2023-09-29T22:05:14.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU=', 'Content-Length': '0'} 2023-09-29T22:05:14.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000477d93f36d03aed0-0065174a1a-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.278 DEBUG:boto:path=/hcnphz-14/obj 2023-09-29T22:05:14.278 DEBUG:boto:auth_path=/hcnphz-14/obj 2023-09-29T22:05:14.278 DEBUG:boto:Method: GET 2023-09-29T22:05:14.278 DEBUG:boto:Path: /hcnphz-14/obj 2023-09-29T22:05:14.278 DEBUG:boto:Data: 2023-09-29T22:05:14.279 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.279 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.279 DEBUG:boto:Params: {} 2023-09-29T22:05:14.279 DEBUG:boto:Token: None 2023-09-29T22:05:14.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/obj 2023-09-29T22:05:14.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU= 2023-09-29T22:05:14.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Ac37UkXpJRdo/6yHpTgI25p1NU=', 'Content-Length': '0'} 2023-09-29T22:05:14.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:04:33 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000039f8d31415f0fe8a-0065174a1a-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.282 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.282 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.283 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.283 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.283 DEBUG:boto:Method: GET 2023-09-29T22:05:14.283 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.283 DEBUG:boto:Data: 2023-09-29T22:05:14.283 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.284 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.284 DEBUG:boto:Params: {} 2023-09-29T22:05:14.284 DEBUG:boto:Token: None 2023-09-29T22:05:14.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000370c3c3e2c5e2154-0065174a1a-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.289 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.289 DEBUG:boto:path=/hcnphz-14/ 2023-09-29T22:05:14.289 DEBUG:boto:auth_path=/hcnphz-14/ 2023-09-29T22:05:14.290 DEBUG:boto:path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.290 DEBUG:boto:auth_path=/hcnphz-14/?prefix=obj 2023-09-29T22:05:14.290 DEBUG:boto:Method: GET 2023-09-29T22:05:14.290 DEBUG:boto:Path: /hcnphz-14/?prefix=obj 2023-09-29T22:05:14.290 DEBUG:boto:Data: 2023-09-29T22:05:14.290 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.290 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.291 DEBUG:boto:Params: {} 2023-09-29T22:05:14.291 DEBUG:boto:Token: None 2023-09-29T22:05:14.291 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-14/ 2023-09-29T22:05:14.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw= 2023-09-29T22:05:14.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIwNaHJhJoIAwSPO10/3ia/zchw=', 'Content-Length': '0'} 2023-09-29T22:05:14.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a5b5bf1be7781c4-0065174a1a-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.338 DEBUG:boto:b'hcnphz-14obj1000falseobj2023-09-29T22:04:33.633Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:14.339 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-14 zones={a2, a1} 2023-09-29T22:05:14.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2023-09-29T22:05:14.340 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-15 2023-09-29T22:05:14.340 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:14.341 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:14.341 DEBUG:boto:Method: PUT 2023-09-29T22:05:14.341 DEBUG:boto:Path: /hcnphz-15/ 2023-09-29T22:05:14.341 DEBUG:boto:Data: 2023-09-29T22:05:14.341 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.341 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.341 DEBUG:boto:Params: {} 2023-09-29T22:05:14.341 DEBUG:boto:Token: None 2023-09-29T22:05:14.342 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-15/ 2023-09-29T22:05:14.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sAFXg6Ku20npDpiOJDODmf7X/Mo= 2023-09-29T22:05:14.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sAFXg6Ku20npDpiOJDODmf7X/Mo=', 'Content-Length': '0'} 2023-09-29T22:05:14.370 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ad162ccee1599f91-0065174a1a-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.370 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-16 2023-09-29T22:05:14.371 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:14.371 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:14.371 DEBUG:boto:Method: PUT 2023-09-29T22:05:14.371 DEBUG:boto:Path: /hcnphz-16/ 2023-09-29T22:05:14.371 DEBUG:boto:Data: 2023-09-29T22:05:14.371 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.371 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.371 DEBUG:boto:Params: {} 2023-09-29T22:05:14.372 DEBUG:boto:Token: None 2023-09-29T22:05:14.372 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-16/ 2023-09-29T22:05:14.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5cVLeNSh7Dn4hrdfNFnb6IplFZo= 2023-09-29T22:05:14.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5cVLeNSh7Dn4hrdfNFnb6IplFZo=', 'Content-Length': '0'} 2023-09-29T22:05:14.411 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006793b87c881cbbf9-0065174a1a-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.412 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:14.412 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:14.412 DEBUG:boto:path=/hcnphz-15/?versioning 2023-09-29T22:05:14.412 DEBUG:boto:auth_path=/hcnphz-15/?versioning 2023-09-29T22:05:14.412 DEBUG:boto:Method: PUT 2023-09-29T22:05:14.412 DEBUG:boto:Path: /hcnphz-15/?versioning 2023-09-29T22:05:14.412 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:05:14.413 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.413 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.413 DEBUG:boto:Params: {} 2023-09-29T22:05:14.413 DEBUG:boto:Token: None 2023-09-29T22:05:14.413 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-15/?versioning 2023-09-29T22:05:14.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mkoeYd0UfAoU/pI83LiVFO+Ovjk= 2023-09-29T22:05:14.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mkoeYd0UfAoU/pI83LiVFO+Ovjk=', 'Content-Length': '228'} 2023-09-29T22:05:14.424 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b2e0bc20b53e0936-0065174a1a-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.424 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:14.425 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:14.425 DEBUG:boto:path=/hcnphz-16/?versioning 2023-09-29T22:05:14.425 DEBUG:boto:auth_path=/hcnphz-16/?versioning 2023-09-29T22:05:14.425 DEBUG:boto:Method: PUT 2023-09-29T22:05:14.425 DEBUG:boto:Path: /hcnphz-16/?versioning 2023-09-29T22:05:14.425 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:05:14.425 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.425 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.426 DEBUG:boto:Params: {} 2023-09-29T22:05:14.426 DEBUG:boto:Token: None 2023-09-29T22:05:14.426 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-16/?versioning 2023-09-29T22:05:14.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCIBmMRkXgIlC+YCi/wfEVbsCNY= 2023-09-29T22:05:14.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCIBmMRkXgIlC+YCi/wfEVbsCNY=', 'Content-Length': '228'} 2023-09-29T22:05:14.445 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000138f38a57163660d-0065174a1a-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.445 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:14.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:14.446 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:05:14.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:14.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:14.545 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:05:14.566 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:05:14.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:05:14.567 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:05:14.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025114.442683_108.1", 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:05:14.442683Z" 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.569 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:14.569 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:05:14.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:14.569 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:05:14.570 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025114.442683_108.1', 2: '', 3: ''} 2023-09-29T22:05:14.570 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:05:14.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:14.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:14.571 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:05:14.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:14.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:14.668 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:14.667+0000 7fe518d16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:05:14.669 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:14.667+0000 7fe518d16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:05:14.694 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:05:14.694 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:05:14.694 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:05:14.694 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:14.695 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:05:14.696 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025114.442683_108.1", 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:05:14.442683Z", 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:14.698 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:05:14.699 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:05:14.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:14.701 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:05:14Z" 2023-09-29T22:05:14.701 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:05:14.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025114.442683_108.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:05:14.442683Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:05:14Z" } 2023-09-29T22:05:14.702 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025114.442683_108.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:05:14.442683Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:05:14.702 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025114.442683_108.1', 2: '', 3: ''} 2023-09-29T22:05:14.702 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025114.442683_108.1', 2: '', 3: ''} 2023-09-29T22:05:14.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:05:14.702 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:14.702 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:14.703 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.703 DEBUG:boto:Path: /hcnphz-15/ 2023-09-29T22:05:14.703 DEBUG:boto:Data: 2023-09-29T22:05:14.703 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.703 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.703 DEBUG:boto:Params: {} 2023-09-29T22:05:14.703 DEBUG:boto:Token: None 2023-09-29T22:05:14.703 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-15/ 2023-09-29T22:05:14.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2gAYO3TLqNqyVvy24PaSKGHpfcY= 2023-09-29T22:05:14.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2gAYO3TLqNqyVvy24PaSKGHpfcY=', 'Content-Length': '0'} 2023-09-29T22:05:14.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a62f3fa03753a0b-0065174a1a-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.707 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:14.708 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:14.708 DEBUG:boto:Method: PUT 2023-09-29T22:05:14.708 DEBUG:boto:Path: /hcnphz-15/obj 2023-09-29T22:05:14.708 DEBUG:boto:Data: 2023-09-29T22:05:14.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:05:14.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.708 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.708 DEBUG:boto:Params: {} 2023-09-29T22:05:14.708 DEBUG:boto:Token: None 2023-09-29T22:05:14.709 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-15/obj 2023-09-29T22:05:14.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2/v5JVMb33V+PmahKf+flHSh8rE= 2023-09-29T22:05:14.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2/v5JVMb33V+PmahKf+flHSh8rE='} 2023-09-29T22:05:14.754 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-amz-request-id', 'tx00000c5bb8af62055686b-0065174a1a-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.755 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:14.755 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:14.755 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.755 DEBUG:boto:Path: /hcnphz-15/ 2023-09-29T22:05:14.755 DEBUG:boto:Data: 2023-09-29T22:05:14.755 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.755 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.756 DEBUG:boto:Params: {} 2023-09-29T22:05:14.756 DEBUG:boto:Token: None 2023-09-29T22:05:14.756 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-15/ 2023-09-29T22:05:14.756 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2gAYO3TLqNqyVvy24PaSKGHpfcY= 2023-09-29T22:05:14.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2gAYO3TLqNqyVvy24PaSKGHpfcY=', 'Content-Length': '0'} 2023-09-29T22:05:14.760 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec1f065c2142118-0065174a1a-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.760 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:14.760 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:14.760 DEBUG:boto:Method: DELETE 2023-09-29T22:05:14.760 DEBUG:boto:Path: /hcnphz-15/obj 2023-09-29T22:05:14.760 DEBUG:boto:Data: 2023-09-29T22:05:14.760 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:14.761 DEBUG:boto:Port: 8000 2023-09-29T22:05:14.761 DEBUG:boto:Params: {} 2023-09-29T22:05:14.761 DEBUG:boto:Token: None 2023-09-29T22:05:14.761 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-15/obj 2023-09-29T22:05:14.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5MxfFjL4WR5eJX9IgSpbhiwESLA= 2023-09-29T22:05:14.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5MxfFjL4WR5eJX9IgSpbhiwESLA=', 'Content-Length': '0'} 2023-09-29T22:05:14.776 DEBUG:boto:Response headers: [('x-amz-version-id', 'KG7.Z7NioPKYrr8.glgE9cf2UQGvjL.'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000008611d129df32917-0065174a1a-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.776 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:14.776 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:14.777 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.777 DEBUG:boto:Path: /hcnphz-16/ 2023-09-29T22:05:14.777 DEBUG:boto:Data: 2023-09-29T22:05:14.777 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.777 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.777 DEBUG:boto:Params: {} 2023-09-29T22:05:14.777 DEBUG:boto:Token: None 2023-09-29T22:05:14.777 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-16/ 2023-09-29T22:05:14.778 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WkuQA8sqgY4jR3q25kq+78BbmKo= 2023-09-29T22:05:14.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WkuQA8sqgY4jR3q25kq+78BbmKo=', 'Content-Length': '0'} 2023-09-29T22:05:14.781 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000becd60e03ff3b34c-0065174a1a-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.781 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:14.782 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:14.782 DEBUG:boto:Method: PUT 2023-09-29T22:05:14.782 DEBUG:boto:Path: /hcnphz-16/obj 2023-09-29T22:05:14.782 DEBUG:boto:Data: 2023-09-29T22:05:14.782 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:05:14.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.782 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.782 DEBUG:boto:Params: {} 2023-09-29T22:05:14.782 DEBUG:boto:Token: None 2023-09-29T22:05:14.783 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-16/obj 2023-09-29T22:05:14.783 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xDGxgPw1cXDOAFUV2Lb63UxocT8= 2023-09-29T22:05:14.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xDGxgPw1cXDOAFUV2Lb63UxocT8='} 2023-09-29T22:05:14.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-amz-request-id', 'tx0000018118640ecc46c20-0065174a1a-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.805 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:14.805 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:14.805 DEBUG:boto:Method: HEAD 2023-09-29T22:05:14.805 DEBUG:boto:Path: /hcnphz-16/ 2023-09-29T22:05:14.805 DEBUG:boto:Data: 2023-09-29T22:05:14.805 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.805 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.806 DEBUG:boto:Params: {} 2023-09-29T22:05:14.806 DEBUG:boto:Token: None 2023-09-29T22:05:14.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-16/ 2023-09-29T22:05:14.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WkuQA8sqgY4jR3q25kq+78BbmKo= 2023-09-29T22:05:14.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WkuQA8sqgY4jR3q25kq+78BbmKo=', 'Content-Length': '0'} 2023-09-29T22:05:14.809 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f1098a2c0e7a3e18-0065174a1a-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.810 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:14.810 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:14.810 DEBUG:boto:Method: DELETE 2023-09-29T22:05:14.810 DEBUG:boto:Path: /hcnphz-16/obj 2023-09-29T22:05:14.810 DEBUG:boto:Data: 2023-09-29T22:05:14.810 DEBUG:boto:Headers: {} 2023-09-29T22:05:14.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:14.810 DEBUG:boto:Port: 8001 2023-09-29T22:05:14.810 DEBUG:boto:Params: {} 2023-09-29T22:05:14.811 DEBUG:boto:Token: None 2023-09-29T22:05:14.811 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:05:14 GMT /hcnphz-16/obj 2023-09-29T22:05:14.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+hBcibs6eUFA3xK0K665eluktUg= 2023-09-29T22:05:14.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+hBcibs6eUFA3xK0K665eluktUg=', 'Content-Length': '0'} 2023-09-29T22:05:14.824 DEBUG:boto:Response headers: [('x-amz-version-id', 'NlhuNJLAtnjraMd157-8AHhKWtMZSLD'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx0000050467896c4157996-0065174a1a-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:14.825 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-15 2023-09-29T22:05:14.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:14.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-15', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:14.825 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-15 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:05:14.875 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:14.875 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:14.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:14.939+0000 7fc5a8b36f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:19.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:19.939+0000 7fc5a8b36f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:24.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:24.939+0000 7fc5a8b36f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:29.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:29.942+0000 7fc5a8b36f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:34.942+0000 7fc5a8b36f40 1 bucket sync caught up with source: 2023-09-29T22:05:34.947 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.8.12, , ] 2023-09-29T22:05:34.947 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.8.12, , ] 2023-09-29T22:05:34.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:34.942+0000 7fc5a8b36f40 0 bucket checkpoint complete 2023-09-29T22:05:34.975 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-15 2023-09-29T22:05:34.976 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:34.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-15', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:34.976 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-15 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:05:35.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:35.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:35.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:35.090+0000 7fb79a049f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:40.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:40.090+0000 7fb79a049f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:45.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:45.090+0000 7fb79a049f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:50.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:50.090+0000 7fb79a049f40 1 waiting to reach incremental sync.. 2023-09-29T22:05:55.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:55.094+0000 7fb79a049f40 1 bucket sync caught up with source: 2023-09-29T22:05:55.099 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.15.6, , ] 2023-09-29T22:05:55.099 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000002.15.6, , ] 2023-09-29T22:05:55.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:55.094+0000 7fb79a049f40 0 bucket checkpoint complete 2023-09-29T22:05:55.127 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-15 zones={a2, a1} 2023-09-29T22:05:55.127 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.127 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.127 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.128 DEBUG:boto:Path: /hcnphz-15/ 2023-09-29T22:05:55.128 DEBUG:boto:Data: 2023-09-29T22:05:55.128 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.128 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.128 DEBUG:boto:Params: {} 2023-09-29T22:05:55.128 DEBUG:boto:Token: None 2023-09-29T22:05:55.128 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/ 2023-09-29T22:05:55.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wizj50UH86+tRTmZTkuLiXse04I= 2023-09-29T22:05:55.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wizj50UH86+tRTmZTkuLiXse04I=', 'Content-Length': '0'} 2023-09-29T22:05:55.132 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a55c2847f5671a6-0065174a43-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.133 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.133 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.133 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.133 DEBUG:boto:Path: /hcnphz-15/ 2023-09-29T22:05:55.133 DEBUG:boto:Data: 2023-09-29T22:05:55.133 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.133 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.133 DEBUG:boto:Params: {} 2023-09-29T22:05:55.134 DEBUG:boto:Token: None 2023-09-29T22:05:55.134 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/ 2023-09-29T22:05:55.134 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wizj50UH86+tRTmZTkuLiXse04I= 2023-09-29T22:05:55.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wizj50UH86+tRTmZTkuLiXse04I=', 'Content-Length': '0'} 2023-09-29T22:05:55.138 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a177ba59f0fa5d2e-0065174a43-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.138 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:05:55.138 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.138 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.138 DEBUG:boto:path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.138 DEBUG:boto:auth_path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.138 DEBUG:boto:Method: GET 2023-09-29T22:05:55.139 DEBUG:boto:Path: /hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.139 DEBUG:boto:Data: 2023-09-29T22:05:55.139 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.139 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.139 DEBUG:boto:Params: {} 2023-09-29T22:05:55.139 DEBUG:boto:Token: None 2023-09-29T22:05:55.139 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/?versions 2023-09-29T22:05:55.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk= 2023-09-29T22:05:55.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk=', 'Content-Length': '0'} 2023-09-29T22:05:55.143 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff8145b0820b9062-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.143 DEBUG:boto:b'hcnphz-15999falseobjKG7.Z7NioPKYrr8.glgE9cf2UQGvjL.true2023-09-29T22:05:14.767Zrgw-multisite-test-userMultisite Test UserNormalobj5SiejQShV6y3I80ULA-TYeOTIA0oMoCfalse2023-09-29T22:05:14.713Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.144 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:05:55.144 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.144 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.144 DEBUG:boto:path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.145 DEBUG:boto:auth_path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.145 DEBUG:boto:Method: GET 2023-09-29T22:05:55.145 DEBUG:boto:Path: /hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.145 DEBUG:boto:Data: 2023-09-29T22:05:55.145 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.145 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.145 DEBUG:boto:Params: {} 2023-09-29T22:05:55.145 DEBUG:boto:Token: None 2023-09-29T22:05:55.145 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/?versions 2023-09-29T22:05:55.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk= 2023-09-29T22:05:55.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk=', 'Content-Length': '0'} 2023-09-29T22:05:55.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ef1fb7985a3e9b9-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.150 DEBUG:boto:b'hcnphz-15999falseobjKG7.Z7NioPKYrr8.glgE9cf2UQGvjL.true2023-09-29T22:05:14.767Zrgw-multisite-test-userMultisite Test UserNormalobj5SiejQShV6y3I80ULA-TYeOTIA0oMoCfalse2023-09-29T22:05:14.713Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.150 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.151 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.151 DEBUG:boto:path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.151 DEBUG:boto:auth_path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.151 DEBUG:boto:Method: GET 2023-09-29T22:05:55.151 DEBUG:boto:Path: /hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.151 DEBUG:boto:Data: 2023-09-29T22:05:55.151 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.151 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.151 DEBUG:boto:Params: {} 2023-09-29T22:05:55.152 DEBUG:boto:Token: None 2023-09-29T22:05:55.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/?versions 2023-09-29T22:05:55.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk= 2023-09-29T22:05:55.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk=', 'Content-Length': '0'} 2023-09-29T22:05:55.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ea587c53054c7ae-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.202 DEBUG:boto:b'hcnphz-15999falseobjKG7.Z7NioPKYrr8.glgE9cf2UQGvjL.true2023-09-29T22:05:14.767Zrgw-multisite-test-userMultisite Test UserNormalobj5SiejQShV6y3I80ULA-TYeOTIA0oMoCfalse2023-09-29T22:05:14.713Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.203 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.203 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.203 DEBUG:boto:path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.203 DEBUG:boto:auth_path=/hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.203 DEBUG:boto:Method: GET 2023-09-29T22:05:55.204 DEBUG:boto:Path: /hcnphz-15/?versions&max-keys=999 2023-09-29T22:05:55.204 DEBUG:boto:Data: 2023-09-29T22:05:55.204 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.204 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.204 DEBUG:boto:Params: {} 2023-09-29T22:05:55.204 DEBUG:boto:Token: None 2023-09-29T22:05:55.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/?versions 2023-09-29T22:05:55.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk= 2023-09-29T22:05:55.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIxdFK6+wcEoOEZ9gt+pY1VFeWk=', 'Content-Length': '0'} 2023-09-29T22:05:55.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a15e283a53d00179-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.209 DEBUG:boto:b'hcnphz-15999falseobjKG7.Z7NioPKYrr8.glgE9cf2UQGvjL.true2023-09-29T22:05:14.767Zrgw-multisite-test-userMultisite Test UserNormalobj5SiejQShV6y3I80ULA-TYeOTIA0oMoCfalse2023-09-29T22:05:14.713Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:55.210 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.210 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.210 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.210 DEBUG:boto:Path: /hcnphz-15/obj 2023-09-29T22:05:55.210 DEBUG:boto:Data: 2023-09-29T22:05:55.213 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.213 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.213 DEBUG:boto:Params: {} 2023-09-29T22:05:55.213 DEBUG:boto:Token: None 2023-09-29T22:05:55.214 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj 2023-09-29T22:05:55.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU= 2023-09-29T22:05:55.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU=', 'Content-Length': '0'} 2023-09-29T22:05:55.217 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000004a6aba26201c93e9-0065174a43-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.218 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.218 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.218 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.218 DEBUG:boto:Path: /hcnphz-15/obj 2023-09-29T22:05:55.218 DEBUG:boto:Data: 2023-09-29T22:05:55.218 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.219 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.219 DEBUG:boto:Params: {} 2023-09-29T22:05:55.219 DEBUG:boto:Token: None 2023-09-29T22:05:55.219 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj 2023-09-29T22:05:55.219 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU= 2023-09-29T22:05:55.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU=', 'Content-Length': '0'} 2023-09-29T22:05:55.223 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000f9ec719cbc8000e0-0065174a43-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:55.223 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.223 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.223 DEBUG:boto:path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.223 DEBUG:boto:auth_path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.224 DEBUG:boto:Method: GET 2023-09-29T22:05:55.224 DEBUG:boto:Path: /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.224 DEBUG:boto:Data: 2023-09-29T22:05:55.224 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.224 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.224 DEBUG:boto:Params: {} 2023-09-29T22:05:55.224 DEBUG:boto:Token: None 2023-09-29T22:05:55.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs= 2023-09-29T22:05:55.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs=', 'Content-Length': '0'} 2023-09-29T22:05:55.227 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000030d254c43e2dc3ad-0065174a43-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.228 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.228 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.228 DEBUG:boto:path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.228 DEBUG:boto:auth_path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.228 DEBUG:boto:Method: GET 2023-09-29T22:05:55.228 DEBUG:boto:Path: /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.228 DEBUG:boto:Data: 2023-09-29T22:05:55.228 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.229 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.229 DEBUG:boto:Params: {} 2023-09-29T22:05:55.229 DEBUG:boto:Token: None 2023-09-29T22:05:55.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs= 2023-09-29T22:05:55.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs=', 'Content-Length': '0'} 2023-09-29T22:05:55.232 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000000c89b3ea16df0e26-0065174a43-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.232 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.232 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.232 DEBUG:boto:path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.232 DEBUG:boto:auth_path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.232 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.232 DEBUG:boto:Path: /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.233 DEBUG:boto:Data: 2023-09-29T22:05:55.233 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.233 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.233 DEBUG:boto:Params: {} 2023-09-29T22:05:55.233 DEBUG:boto:Token: None 2023-09-29T22:05:55.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SNb+LK+PGfXW3uR5eErKBLJO890= 2023-09-29T22:05:55.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SNb+LK+PGfXW3uR5eErKBLJO890=', 'Content-Length': '0'} 2023-09-29T22:05:55.236 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d20143c5da0f5122-0065174a43-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.236 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.236 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.236 DEBUG:boto:path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.237 DEBUG:boto:auth_path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.237 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.237 DEBUG:boto:Path: /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.237 DEBUG:boto:Data: 2023-09-29T22:05:55.237 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.237 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.237 DEBUG:boto:Params: {} 2023-09-29T22:05:55.237 DEBUG:boto:Token: None 2023-09-29T22:05:55.237 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SNb+LK+PGfXW3uR5eErKBLJO890= 2023-09-29T22:05:55.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SNb+LK+PGfXW3uR5eErKBLJO890=', 'Content-Length': '0'} 2023-09-29T22:05:55.240 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000e0dc17e99fdf1ec7-0065174a43-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:55.241 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.241 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.241 DEBUG:boto:path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.241 DEBUG:boto:auth_path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.241 DEBUG:boto:Method: GET 2023-09-29T22:05:55.241 DEBUG:boto:Path: /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.241 DEBUG:boto:Data: 2023-09-29T22:05:55.241 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.242 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.242 DEBUG:boto:Params: {} 2023-09-29T22:05:55.242 DEBUG:boto:Token: None 2023-09-29T22:05:55.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs= 2023-09-29T22:05:55.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs=', 'Content-Length': '0'} 2023-09-29T22:05:55.245 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000d38a7c9e83d08aca-0065174a43-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.246 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.246 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.246 DEBUG:boto:path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.246 DEBUG:boto:auth_path=/hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.246 DEBUG:boto:Method: GET 2023-09-29T22:05:55.246 DEBUG:boto:Path: /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.246 DEBUG:boto:Data: 2023-09-29T22:05:55.246 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.247 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.247 DEBUG:boto:Params: {} 2023-09-29T22:05:55.247 DEBUG:boto:Token: None 2023-09-29T22:05:55.247 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj?versionId=5SiejQShV6y3I80ULA-TYeOTIA0oMoC 2023-09-29T22:05:55.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs= 2023-09-29T22:05:55.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xBp+xQHUw0nFKHqWZbz33CYKOGs=', 'Content-Length': '0'} 2023-09-29T22:05:55.250 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', '5SiejQShV6y3I80ULA-TYeOTIA0oMoC'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a10c0febb9a4cef4-0065174a43-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.250 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.250 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.250 DEBUG:boto:path=/hcnphz-15/?prefix=obj 2023-09-29T22:05:55.250 DEBUG:boto:auth_path=/hcnphz-15/?prefix=obj 2023-09-29T22:05:55.251 DEBUG:boto:Method: GET 2023-09-29T22:05:55.251 DEBUG:boto:Path: /hcnphz-15/?prefix=obj 2023-09-29T22:05:55.251 DEBUG:boto:Data: 2023-09-29T22:05:55.251 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.251 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.251 DEBUG:boto:Params: {} 2023-09-29T22:05:55.251 DEBUG:boto:Token: None 2023-09-29T22:05:55.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/ 2023-09-29T22:05:55.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OQpgzEiN3vQ+kFdOKADwmwE45fk= 2023-09-29T22:05:55.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OQpgzEiN3vQ+kFdOKADwmwE45fk=', 'Content-Length': '0'} 2023-09-29T22:05:55.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fee4a958b0e6ee4-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.298 DEBUG:boto:b'hcnphz-15obj1000false' 2023-09-29T22:05:55.299 DEBUG:boto:path=/hcnphz-15/ 2023-09-29T22:05:55.299 DEBUG:boto:auth_path=/hcnphz-15/ 2023-09-29T22:05:55.299 DEBUG:boto:path=/hcnphz-15/?prefix=obj 2023-09-29T22:05:55.299 DEBUG:boto:auth_path=/hcnphz-15/?prefix=obj 2023-09-29T22:05:55.299 DEBUG:boto:Method: GET 2023-09-29T22:05:55.299 DEBUG:boto:Path: /hcnphz-15/?prefix=obj 2023-09-29T22:05:55.300 DEBUG:boto:Data: 2023-09-29T22:05:55.300 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.300 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.300 DEBUG:boto:Params: {} 2023-09-29T22:05:55.300 DEBUG:boto:Token: None 2023-09-29T22:05:55.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/ 2023-09-29T22:05:55.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OQpgzEiN3vQ+kFdOKADwmwE45fk= 2023-09-29T22:05:55.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OQpgzEiN3vQ+kFdOKADwmwE45fk=', 'Content-Length': '0'} 2023-09-29T22:05:55.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c4dd3d869e72f6cc-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.305 DEBUG:boto:b'hcnphz-15obj1000false' 2023-09-29T22:05:55.305 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.306 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.306 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.306 DEBUG:boto:Path: /hcnphz-15/obj 2023-09-29T22:05:55.306 DEBUG:boto:Data: 2023-09-29T22:05:55.306 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.306 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.306 DEBUG:boto:Params: {} 2023-09-29T22:05:55.307 DEBUG:boto:Token: None 2023-09-29T22:05:55.307 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj 2023-09-29T22:05:55.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU= 2023-09-29T22:05:55.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU=', 'Content-Length': '0'} 2023-09-29T22:05:55.310 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000024a4ea72fba7545-0065174a43-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.310 DEBUG:boto:path=/hcnphz-15/obj 2023-09-29T22:05:55.311 DEBUG:boto:auth_path=/hcnphz-15/obj 2023-09-29T22:05:55.311 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.311 DEBUG:boto:Path: /hcnphz-15/obj 2023-09-29T22:05:55.311 DEBUG:boto:Data: 2023-09-29T22:05:55.311 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.311 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.311 DEBUG:boto:Params: {} 2023-09-29T22:05:55.311 DEBUG:boto:Token: None 2023-09-29T22:05:55.312 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-15/obj 2023-09-29T22:05:55.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU= 2023-09-29T22:05:55.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a2TZiBWr8cwP4GJbowWGvuGq9xU=', 'Content-Length': '0'} 2023-09-29T22:05:55.315 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e804dca08e51a805-0065174a43-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.316 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-15 zones={a2, a1} 2023-09-29T22:05:55.316 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-16 2023-09-29T22:05:55.316 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:55.316 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-16', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:55.316 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-16 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:05:55.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:55.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:55.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:55.434+0000 7f84fa9c9f40 1 bucket sync caught up with source: 2023-09-29T22:05:55.439 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000002.9.6, , ] 2023-09-29T22:05:55.439 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000002.9.6, , ] 2023-09-29T22:05:55.439 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:55.434+0000 7f84fa9c9f40 0 bucket checkpoint complete 2023-09-29T22:05:55.464 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-16 2023-09-29T22:05:55.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:55.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-16', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:55.465 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-16 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:05:55.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:55.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:55.583 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:55.582+0000 7f6e215b1f40 1 bucket sync caught up with source: 2023-09-29T22:05:55.584 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.21.12, , ] 2023-09-29T22:05:55.584 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.21.12, , ] 2023-09-29T22:05:55.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:55.582+0000 7f6e215b1f40 0 bucket checkpoint complete 2023-09-29T22:05:55.608 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-16 zones={a2, a1} 2023-09-29T22:05:55.608 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.608 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.608 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.608 DEBUG:boto:Path: /hcnphz-16/ 2023-09-29T22:05:55.608 DEBUG:boto:Data: 2023-09-29T22:05:55.609 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.609 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.609 DEBUG:boto:Params: {} 2023-09-29T22:05:55.609 DEBUG:boto:Token: None 2023-09-29T22:05:55.609 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/ 2023-09-29T22:05:55.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pvuZlfrNiFZMectAzrKh4WfL0tY= 2023-09-29T22:05:55.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pvuZlfrNiFZMectAzrKh4WfL0tY=', 'Content-Length': '0'} 2023-09-29T22:05:55.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044b4dd73699b294f-0065174a43-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.613 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.614 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.614 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.614 DEBUG:boto:Path: /hcnphz-16/ 2023-09-29T22:05:55.614 DEBUG:boto:Data: 2023-09-29T22:05:55.614 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.614 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.614 DEBUG:boto:Params: {} 2023-09-29T22:05:55.615 DEBUG:boto:Token: None 2023-09-29T22:05:55.615 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/ 2023-09-29T22:05:55.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pvuZlfrNiFZMectAzrKh4WfL0tY= 2023-09-29T22:05:55.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pvuZlfrNiFZMectAzrKh4WfL0tY=', 'Content-Length': '0'} 2023-09-29T22:05:55.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ac73eacaaaaa3c1d-0065174a43-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.618 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:05:55.618 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.618 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.618 DEBUG:boto:path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.618 DEBUG:boto:auth_path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.619 DEBUG:boto:Method: GET 2023-09-29T22:05:55.619 DEBUG:boto:Path: /hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.619 DEBUG:boto:Data: 2023-09-29T22:05:55.619 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.619 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.619 DEBUG:boto:Params: {} 2023-09-29T22:05:55.619 DEBUG:boto:Token: None 2023-09-29T22:05:55.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/?versions 2023-09-29T22:05:55.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0= 2023-09-29T22:05:55.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0=', 'Content-Length': '0'} 2023-09-29T22:05:55.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d03f8c9cb75620d5-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.626 DEBUG:boto:b'hcnphz-16999falseobjNlhuNJLAtnjraMd157-8AHhKWtMZSLDtrue2023-09-29T22:05:14.816Zrgw-multisite-test-userMultisite Test UserNormalobjYKjMwWUe3bI-cLceZQH8q4CQB3pgaa2false2023-09-29T22:05:14.788Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.626 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:05:55.626 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.627 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.627 DEBUG:boto:path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.627 DEBUG:boto:auth_path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.627 DEBUG:boto:Method: GET 2023-09-29T22:05:55.627 DEBUG:boto:Path: /hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.627 DEBUG:boto:Data: 2023-09-29T22:05:55.627 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.627 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.627 DEBUG:boto:Params: {} 2023-09-29T22:05:55.628 DEBUG:boto:Token: None 2023-09-29T22:05:55.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/?versions 2023-09-29T22:05:55.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0= 2023-09-29T22:05:55.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0=', 'Content-Length': '0'} 2023-09-29T22:05:55.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3b804dfddb37268-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.631 DEBUG:boto:b'hcnphz-16999falseobjNlhuNJLAtnjraMd157-8AHhKWtMZSLDtrue2023-09-29T22:05:14.816Zrgw-multisite-test-userMultisite Test UserNormalobjYKjMwWUe3bI-cLceZQH8q4CQB3pgaa2false2023-09-29T22:05:14.788Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:05:55.632 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.632 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.632 DEBUG:boto:path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.632 DEBUG:boto:auth_path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.632 DEBUG:boto:Method: GET 2023-09-29T22:05:55.633 DEBUG:boto:Path: /hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.633 DEBUG:boto:Data: 2023-09-29T22:05:55.633 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.633 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.633 DEBUG:boto:Params: {} 2023-09-29T22:05:55.633 DEBUG:boto:Token: None 2023-09-29T22:05:55.633 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/?versions 2023-09-29T22:05:55.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0= 2023-09-29T22:05:55.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0=', 'Content-Length': '0'} 2023-09-29T22:05:55.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026b645bf7661d2dc-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.678 DEBUG:boto:b'hcnphz-16999falseobjNlhuNJLAtnjraMd157-8AHhKWtMZSLDtrue2023-09-29T22:05:14.816Zrgw-multisite-test-userMultisite Test UserNormalobjYKjMwWUe3bI-cLceZQH8q4CQB3pgaa2false2023-09-29T22:05:14.788Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.679 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.679 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.679 DEBUG:boto:path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.679 DEBUG:boto:auth_path=/hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.679 DEBUG:boto:Method: GET 2023-09-29T22:05:55.680 DEBUG:boto:Path: /hcnphz-16/?versions&max-keys=999 2023-09-29T22:05:55.680 DEBUG:boto:Data: 2023-09-29T22:05:55.680 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.680 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.680 DEBUG:boto:Params: {} 2023-09-29T22:05:55.680 DEBUG:boto:Token: None 2023-09-29T22:05:55.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/?versions 2023-09-29T22:05:55.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0= 2023-09-29T22:05:55.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WgThoAAf6ygcqbR8bIS8Je7Y8f0=', 'Content-Length': '0'} 2023-09-29T22:05:55.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d253085949508b1-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.685 DEBUG:boto:b'hcnphz-16999falseobjNlhuNJLAtnjraMd157-8AHhKWtMZSLDtrue2023-09-29T22:05:14.816Zrgw-multisite-test-userMultisite Test UserNormalobjYKjMwWUe3bI-cLceZQH8q4CQB3pgaa2false2023-09-29T22:05:14.788Z"d41d8cd98f00b204e9800998ecf8427e"0STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:05:55.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:55.685 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.685 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.686 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.686 DEBUG:boto:Path: /hcnphz-16/obj 2023-09-29T22:05:55.686 DEBUG:boto:Data: 2023-09-29T22:05:55.686 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.686 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.686 DEBUG:boto:Params: {} 2023-09-29T22:05:55.686 DEBUG:boto:Token: None 2023-09-29T22:05:55.686 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj 2023-09-29T22:05:55.687 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q= 2023-09-29T22:05:55.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q=', 'Content-Length': '0'} 2023-09-29T22:05:55.690 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000001883c377031cc031-0065174a43-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.690 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.690 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.690 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.690 DEBUG:boto:Path: /hcnphz-16/obj 2023-09-29T22:05:55.691 DEBUG:boto:Data: 2023-09-29T22:05:55.691 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.691 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.691 DEBUG:boto:Params: {} 2023-09-29T22:05:55.691 DEBUG:boto:Token: None 2023-09-29T22:05:55.691 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj 2023-09-29T22:05:55.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q= 2023-09-29T22:05:55.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q=', 'Content-Length': '0'} 2023-09-29T22:05:55.694 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000007e977bb41c196d1-0065174a43-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:55.695 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.695 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.695 DEBUG:boto:path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.695 DEBUG:boto:auth_path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.695 DEBUG:boto:Method: GET 2023-09-29T22:05:55.695 DEBUG:boto:Path: /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.695 DEBUG:boto:Data: 2023-09-29T22:05:55.696 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.696 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.696 DEBUG:boto:Params: {} 2023-09-29T22:05:55.696 DEBUG:boto:Token: None 2023-09-29T22:05:55.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go= 2023-09-29T22:05:55.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go=', 'Content-Length': '0'} 2023-09-29T22:05:55.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000694c261d6785d147-0065174a43-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.699 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.699 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.700 DEBUG:boto:path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.700 DEBUG:boto:auth_path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.700 DEBUG:boto:Method: GET 2023-09-29T22:05:55.700 DEBUG:boto:Path: /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.700 DEBUG:boto:Data: 2023-09-29T22:05:55.700 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.700 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.700 DEBUG:boto:Params: {} 2023-09-29T22:05:55.701 DEBUG:boto:Token: None 2023-09-29T22:05:55.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go= 2023-09-29T22:05:55.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go=', 'Content-Length': '0'} 2023-09-29T22:05:55.703 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx000002b099188d63ddc0b-0065174a43-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.704 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.704 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.704 DEBUG:boto:path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.704 DEBUG:boto:auth_path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.704 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.704 DEBUG:boto:Path: /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.704 DEBUG:boto:Data: 2023-09-29T22:05:55.705 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.705 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.705 DEBUG:boto:Params: {} 2023-09-29T22:05:55.705 DEBUG:boto:Token: None 2023-09-29T22:05:55.705 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f0AlQ2OP5sn2qwD6sllbNxQYVBY= 2023-09-29T22:05:55.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f0AlQ2OP5sn2qwD6sllbNxQYVBY=', 'Content-Length': '0'} 2023-09-29T22:05:55.708 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000137ca639eb181067-0065174a43-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.708 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.708 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.709 DEBUG:boto:path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.709 DEBUG:boto:auth_path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.709 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.709 DEBUG:boto:Path: /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.709 DEBUG:boto:Data: 2023-09-29T22:05:55.709 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.709 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.709 DEBUG:boto:Params: {} 2023-09-29T22:05:55.709 DEBUG:boto:Token: None 2023-09-29T22:05:55.710 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f0AlQ2OP5sn2qwD6sllbNxQYVBY= 2023-09-29T22:05:55.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f0AlQ2OP5sn2qwD6sllbNxQYVBY=', 'Content-Length': '0'} 2023-09-29T22:05:55.712 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000081dee03d172e5963-0065174a43-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:05:55.713 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.713 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.713 DEBUG:boto:path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.713 DEBUG:boto:auth_path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.713 DEBUG:boto:Method: GET 2023-09-29T22:05:55.713 DEBUG:boto:Path: /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.713 DEBUG:boto:Data: 2023-09-29T22:05:55.714 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.714 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.714 DEBUG:boto:Params: {} 2023-09-29T22:05:55.714 DEBUG:boto:Token: None 2023-09-29T22:05:55.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go= 2023-09-29T22:05:55.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go=', 'Content-Length': '0'} 2023-09-29T22:05:55.717 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx00000a1918bd8aea4e8ff-0065174a43-1126-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.717 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.717 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.718 DEBUG:boto:path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.718 DEBUG:boto:auth_path=/hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.718 DEBUG:boto:Method: GET 2023-09-29T22:05:55.718 DEBUG:boto:Path: /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.718 DEBUG:boto:Data: 2023-09-29T22:05:55.718 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.718 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.718 DEBUG:boto:Params: {} 2023-09-29T22:05:55.718 DEBUG:boto:Token: None 2023-09-29T22:05:55.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj?versionId=YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2 2023-09-29T22:05:55.719 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go= 2023-09-29T22:05:55.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4LCbDSAZxc4c9ACzjMgE8fMH+Go=', 'Content-Length': '0'} 2023-09-29T22:05:55.722 DEBUG:boto:Response headers: [('Content-Length', '0'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:05:14 GMT'), ('x-amz-version-id', 'YKjMwWUe3bI-cLceZQH8q4CQB3pgaa2'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('x-amz-request-id', 'tx0000017f174b8fa86c2af-0065174a43-1122-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.722 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.722 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.722 DEBUG:boto:path=/hcnphz-16/?prefix=obj 2023-09-29T22:05:55.723 DEBUG:boto:auth_path=/hcnphz-16/?prefix=obj 2023-09-29T22:05:55.723 DEBUG:boto:Method: GET 2023-09-29T22:05:55.723 DEBUG:boto:Path: /hcnphz-16/?prefix=obj 2023-09-29T22:05:55.723 DEBUG:boto:Data: 2023-09-29T22:05:55.723 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.723 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.723 DEBUG:boto:Params: {} 2023-09-29T22:05:55.724 DEBUG:boto:Token: None 2023-09-29T22:05:55.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/ 2023-09-29T22:05:55.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TbhHkzdMhkINBlMDXoK4ErVU5qQ= 2023-09-29T22:05:55.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TbhHkzdMhkINBlMDXoK4ErVU5qQ=', 'Content-Length': '0'} 2023-09-29T22:05:55.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000678de5640b682d5e-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.770 DEBUG:boto:b'hcnphz-16obj1000false' 2023-09-29T22:05:55.771 DEBUG:boto:path=/hcnphz-16/ 2023-09-29T22:05:55.771 DEBUG:boto:auth_path=/hcnphz-16/ 2023-09-29T22:05:55.771 DEBUG:boto:path=/hcnphz-16/?prefix=obj 2023-09-29T22:05:55.771 DEBUG:boto:auth_path=/hcnphz-16/?prefix=obj 2023-09-29T22:05:55.772 DEBUG:boto:Method: GET 2023-09-29T22:05:55.772 DEBUG:boto:Path: /hcnphz-16/?prefix=obj 2023-09-29T22:05:55.772 DEBUG:boto:Data: 2023-09-29T22:05:55.772 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.772 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.772 DEBUG:boto:Params: {} 2023-09-29T22:05:55.773 DEBUG:boto:Token: None 2023-09-29T22:05:55.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/ 2023-09-29T22:05:55.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TbhHkzdMhkINBlMDXoK4ErVU5qQ= 2023-09-29T22:05:55.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TbhHkzdMhkINBlMDXoK4ErVU5qQ=', 'Content-Length': '0'} 2023-09-29T22:05:55.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd39154b886f133-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.778 DEBUG:boto:b'hcnphz-16obj1000false' 2023-09-29T22:05:55.779 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.779 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.779 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.779 DEBUG:boto:Path: /hcnphz-16/obj 2023-09-29T22:05:55.780 DEBUG:boto:Data: 2023-09-29T22:05:55.780 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.780 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.780 DEBUG:boto:Params: {} 2023-09-29T22:05:55.780 DEBUG:boto:Token: None 2023-09-29T22:05:55.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj 2023-09-29T22:05:55.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q= 2023-09-29T22:05:55.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q=', 'Content-Length': '0'} 2023-09-29T22:05:55.784 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009c62c878c12649ee-0065174a43-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.785 DEBUG:boto:path=/hcnphz-16/obj 2023-09-29T22:05:55.785 DEBUG:boto:auth_path=/hcnphz-16/obj 2023-09-29T22:05:55.785 DEBUG:boto:Method: HEAD 2023-09-29T22:05:55.785 DEBUG:boto:Path: /hcnphz-16/obj 2023-09-29T22:05:55.785 DEBUG:boto:Data: 2023-09-29T22:05:55.785 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.785 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.786 DEBUG:boto:Params: {} 2023-09-29T22:05:55.786 DEBUG:boto:Token: None 2023-09-29T22:05:55.786 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-16/obj 2023-09-29T22:05:55.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q= 2023-09-29T22:05:55.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uhiZzSrsr7fb4uxzcjIzoRZ28/Q=', 'Content-Length': '0'} 2023-09-29T22:05:55.789 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000009a735bdfad5e98b8-0065174a43-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-16 zones={a2, a1} 2023-09-29T22:05:55.790 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2023-09-29T22:05:55.791 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-17 2023-09-29T22:05:55.791 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:05:55.792 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:05:55.792 DEBUG:boto:Method: PUT 2023-09-29T22:05:55.792 DEBUG:boto:Path: /hcnphz-17/ 2023-09-29T22:05:55.792 DEBUG:boto:Data: 2023-09-29T22:05:55.792 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.792 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.792 DEBUG:boto:Params: {} 2023-09-29T22:05:55.793 DEBUG:boto:Token: None 2023-09-29T22:05:55.793 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-17/ 2023-09-29T22:05:55.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HjiAGOQ2S/cz2u6TUb8txajxpQk= 2023-09-29T22:05:55.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HjiAGOQ2S/cz2u6TUb8txajxpQk=', 'Content-Length': '0'} 2023-09-29T22:05:55.818 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000078fb95f8367947ac-0065174a43-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.818 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-18 2023-09-29T22:05:55.818 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:05:55.819 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:05:55.819 DEBUG:boto:Method: PUT 2023-09-29T22:05:55.819 DEBUG:boto:Path: /hcnphz-18/ 2023-09-29T22:05:55.819 DEBUG:boto:Data: 2023-09-29T22:05:55.819 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.819 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.819 DEBUG:boto:Params: {} 2023-09-29T22:05:55.820 DEBUG:boto:Token: None 2023-09-29T22:05:55.820 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-18/ 2023-09-29T22:05:55.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F5mmL05/c2WfATQqpHqJ8tlNNf0= 2023-09-29T22:05:55.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F5mmL05/c2WfATQqpHqJ8tlNNf0=', 'Content-Length': '0'} 2023-09-29T22:05:55.861 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000047aa75cbf18ec113-0065174a43-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.862 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:05:55.862 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:05:55.862 DEBUG:boto:path=/hcnphz-17/?versioning 2023-09-29T22:05:55.862 DEBUG:boto:auth_path=/hcnphz-17/?versioning 2023-09-29T22:05:55.862 DEBUG:boto:Method: PUT 2023-09-29T22:05:55.862 DEBUG:boto:Path: /hcnphz-17/?versioning 2023-09-29T22:05:55.862 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:05:55.863 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.863 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.863 DEBUG:boto:Params: {} 2023-09-29T22:05:55.863 DEBUG:boto:Token: None 2023-09-29T22:05:55.863 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-17/?versioning 2023-09-29T22:05:55.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5j16FDT6Vwlzz1wPsHSd5RsM4Cs= 2023-09-29T22:05:55.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5j16FDT6Vwlzz1wPsHSd5RsM4Cs=', 'Content-Length': '228'} 2023-09-29T22:05:55.873 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d588d6333f2db0e8-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.873 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:05:55.873 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:05:55.873 DEBUG:boto:path=/hcnphz-17/?versioning 2023-09-29T22:05:55.874 DEBUG:boto:auth_path=/hcnphz-17/?versioning 2023-09-29T22:05:55.874 DEBUG:boto:Method: PUT 2023-09-29T22:05:55.874 DEBUG:boto:Path: /hcnphz-17/?versioning 2023-09-29T22:05:55.874 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:05:55.874 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:55.874 DEBUG:boto:Port: 8000 2023-09-29T22:05:55.874 DEBUG:boto:Params: {} 2023-09-29T22:05:55.875 DEBUG:boto:Token: None 2023-09-29T22:05:55.876 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-17/?versioning 2023-09-29T22:05:55.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5j16FDT6Vwlzz1wPsHSd5RsM4Cs= 2023-09-29T22:05:55.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5j16FDT6Vwlzz1wPsHSd5RsM4Cs=', 'Content-Length': '230'} 2023-09-29T22:05:55.882 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c224556e363b536c-0065174a43-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.882 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:05:55.882 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:05:55.883 DEBUG:boto:path=/hcnphz-18/?versioning 2023-09-29T22:05:55.883 DEBUG:boto:auth_path=/hcnphz-18/?versioning 2023-09-29T22:05:55.883 DEBUG:boto:Method: PUT 2023-09-29T22:05:55.883 DEBUG:boto:Path: /hcnphz-18/?versioning 2023-09-29T22:05:55.883 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:05:55.883 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.883 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.883 DEBUG:boto:Params: {} 2023-09-29T22:05:55.883 DEBUG:boto:Token: None 2023-09-29T22:05:55.884 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-18/?versioning 2023-09-29T22:05:55.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mLYRKv7HJSFNyWf5q2tYA8bUK88= 2023-09-29T22:05:55.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mLYRKv7HJSFNyWf5q2tYA8bUK88=', 'Content-Length': '228'} 2023-09-29T22:05:55.896 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000951922b9760b5a99-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.896 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:05:55.896 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:05:55.896 DEBUG:boto:path=/hcnphz-18/?versioning 2023-09-29T22:05:55.896 DEBUG:boto:auth_path=/hcnphz-18/?versioning 2023-09-29T22:05:55.896 DEBUG:boto:Method: PUT 2023-09-29T22:05:55.896 DEBUG:boto:Path: /hcnphz-18/?versioning 2023-09-29T22:05:55.897 DEBUG:boto:Data: Suspended Disabled 2023-09-29T22:05:55.897 DEBUG:boto:Headers: {} 2023-09-29T22:05:55.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:55.897 DEBUG:boto:Port: 8001 2023-09-29T22:05:55.897 DEBUG:boto:Params: {} 2023-09-29T22:05:55.897 DEBUG:boto:Token: None 2023-09-29T22:05:55.897 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:05:55 GMT /hcnphz-18/?versioning 2023-09-29T22:05:55.897 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mLYRKv7HJSFNyWf5q2tYA8bUK88= 2023-09-29T22:05:55.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mLYRKv7HJSFNyWf5q2tYA8bUK88=', 'Content-Length': '230'} 2023-09-29T22:05:55.916 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096ac0dbf18c71c4d-0065174a43-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:55.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:55.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:55.917 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:05:55.960 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:55.960 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:56.007 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:05:56.036 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:05:56.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.036 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025155.908656_125.1", 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:05:55.908656Z" 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.037 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:56.038 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:05:56.039 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025155.908656_125.1', 2: '', 3: ''} 2023-09-29T22:05:56.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:05:56.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:56.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:05:56.040 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:05:56.082 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:56.082 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:56.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:56.130+0000 7fe153273f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:05:56.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:56.130+0000 7fe153273f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:05:56.159 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:05:56.159 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:05:56.159 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:05:56.159 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:56.160 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.161 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025155.908656_125.1", 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:05:55.908656Z", 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:56.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:56.163 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:05:56.164 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:05:56.165 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:05:56.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:05:56.166 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:05:56Z" 2023-09-29T22:05:56.166 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:05:56.167 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025155.908656_125.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:05:55.908656Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:05:56Z" } 2023-09-29T22:05:56.167 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025155.908656_125.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:05:55.908656Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:05:56.167 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025155.908656_125.1', 2: '', 3: ''} 2023-09-29T22:05:56.167 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025155.908656_125.1', 2: '', 3: ''} 2023-09-29T22:05:56.167 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:05:56.168 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:05:56.168 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:05:56.168 DEBUG:boto:Method: HEAD 2023-09-29T22:05:56.168 DEBUG:boto:Path: /hcnphz-17/ 2023-09-29T22:05:56.168 DEBUG:boto:Data: 2023-09-29T22:05:56.168 DEBUG:boto:Headers: {} 2023-09-29T22:05:56.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:56.168 DEBUG:boto:Port: 8000 2023-09-29T22:05:56.168 DEBUG:boto:Params: {} 2023-09-29T22:05:56.169 DEBUG:boto:Token: None 2023-09-29T22:05:56.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-17/ 2023-09-29T22:05:56.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EwrhtEMDvsoy1MkxeRQ5/0wk7+8= 2023-09-29T22:05:56.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EwrhtEMDvsoy1MkxeRQ5/0wk7+8=', 'Content-Length': '0'} 2023-09-29T22:05:56.173 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063d3dd394573cc1d-0065174a44-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.173 DEBUG:boto:path=/hcnphz-17/obj 2023-09-29T22:05:56.173 DEBUG:boto:auth_path=/hcnphz-17/obj 2023-09-29T22:05:56.173 DEBUG:boto:Method: PUT 2023-09-29T22:05:56.173 DEBUG:boto:Path: /hcnphz-17/obj 2023-09-29T22:05:56.173 DEBUG:boto:Data: 2023-09-29T22:05:56.174 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:05:56.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:56.174 DEBUG:boto:Port: 8000 2023-09-29T22:05:56.174 DEBUG:boto:Params: {} 2023-09-29T22:05:56.174 DEBUG:boto:Token: None 2023-09-29T22:05:56.174 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-17/obj 2023-09-29T22:05:56.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+7nZ6JH1yHjOx4AKS70gYisJ2Xo= 2023-09-29T22:05:56.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+7nZ6JH1yHjOx4AKS70gYisJ2Xo='} 2023-09-29T22:05:56.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000008af183a1c02f952-0065174a44-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.223 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:05:56.223 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:05:56.223 DEBUG:boto:Method: HEAD 2023-09-29T22:05:56.223 DEBUG:boto:Path: /hcnphz-17/ 2023-09-29T22:05:56.223 DEBUG:boto:Data: 2023-09-29T22:05:56.223 DEBUG:boto:Headers: {} 2023-09-29T22:05:56.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:56.224 DEBUG:boto:Port: 8000 2023-09-29T22:05:56.224 DEBUG:boto:Params: {} 2023-09-29T22:05:56.224 DEBUG:boto:Token: None 2023-09-29T22:05:56.224 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-17/ 2023-09-29T22:05:56.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EwrhtEMDvsoy1MkxeRQ5/0wk7+8= 2023-09-29T22:05:56.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EwrhtEMDvsoy1MkxeRQ5/0wk7+8=', 'Content-Length': '0'} 2023-09-29T22:05:56.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000778960c7da3e14ab-0065174a44-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.228 DEBUG:boto:path=/hcnphz-17/obj 2023-09-29T22:05:56.228 DEBUG:boto:auth_path=/hcnphz-17/obj 2023-09-29T22:05:56.228 DEBUG:boto:Method: DELETE 2023-09-29T22:05:56.229 DEBUG:boto:Path: /hcnphz-17/obj 2023-09-29T22:05:56.229 DEBUG:boto:Data: 2023-09-29T22:05:56.229 DEBUG:boto:Headers: {} 2023-09-29T22:05:56.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:05:56.229 DEBUG:boto:Port: 8000 2023-09-29T22:05:56.229 DEBUG:boto:Params: {} 2023-09-29T22:05:56.229 DEBUG:boto:Token: None 2023-09-29T22:05:56.229 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-17/obj 2023-09-29T22:05:56.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGoboVQnOrwIzTeWMa5CAGdy9is= 2023-09-29T22:05:56.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGoboVQnOrwIzTeWMa5CAGdy9is=', 'Content-Length': '0'} 2023-09-29T22:05:56.249 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000007de2663bbdc40ab6-0065174a44-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.250 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:05:56.250 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:05:56.250 DEBUG:boto:Method: HEAD 2023-09-29T22:05:56.250 DEBUG:boto:Path: /hcnphz-18/ 2023-09-29T22:05:56.250 DEBUG:boto:Data: 2023-09-29T22:05:56.251 DEBUG:boto:Headers: {} 2023-09-29T22:05:56.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:56.251 DEBUG:boto:Port: 8001 2023-09-29T22:05:56.251 DEBUG:boto:Params: {} 2023-09-29T22:05:56.251 DEBUG:boto:Token: None 2023-09-29T22:05:56.251 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-18/ 2023-09-29T22:05:56.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:shZFXM/rdZHLcJ0NMuUafQALgdQ= 2023-09-29T22:05:56.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:shZFXM/rdZHLcJ0NMuUafQALgdQ=', 'Content-Length': '0'} 2023-09-29T22:05:56.255 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000722a71313832ca28-0065174a44-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.255 DEBUG:boto:path=/hcnphz-18/obj 2023-09-29T22:05:56.255 DEBUG:boto:auth_path=/hcnphz-18/obj 2023-09-29T22:05:56.255 DEBUG:boto:Method: PUT 2023-09-29T22:05:56.255 DEBUG:boto:Path: /hcnphz-18/obj 2023-09-29T22:05:56.256 DEBUG:boto:Data: 2023-09-29T22:05:56.256 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue'} 2023-09-29T22:05:56.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:56.256 DEBUG:boto:Port: 8001 2023-09-29T22:05:56.256 DEBUG:boto:Params: {} 2023-09-29T22:05:56.256 DEBUG:boto:Token: None 2023-09-29T22:05:56.256 DEBUG:boto:StringToSign: PUT 1B2M2Y8AsgTpgAmY7PhCfg== application/octet-stream Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-18/obj 2023-09-29T22:05:56.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OZqtal7JVbx2s5bi6rnvyCfJGXY= 2023-09-29T22:05:56.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '1B2M2Y8AsgTpgAmY7PhCfg==', 'Content-Length': '0', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OZqtal7JVbx2s5bi6rnvyCfJGXY='} 2023-09-29T22:05:56.268 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"d41d8cd98f00b204e9800998ecf8427e"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001adf9b1230fda810-0065174a44-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.268 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:05:56.268 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:05:56.269 DEBUG:boto:Method: HEAD 2023-09-29T22:05:56.269 DEBUG:boto:Path: /hcnphz-18/ 2023-09-29T22:05:56.269 DEBUG:boto:Data: 2023-09-29T22:05:56.269 DEBUG:boto:Headers: {} 2023-09-29T22:05:56.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:56.269 DEBUG:boto:Port: 8001 2023-09-29T22:05:56.269 DEBUG:boto:Params: {} 2023-09-29T22:05:56.269 DEBUG:boto:Token: None 2023-09-29T22:05:56.270 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-18/ 2023-09-29T22:05:56.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:shZFXM/rdZHLcJ0NMuUafQALgdQ= 2023-09-29T22:05:56.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:shZFXM/rdZHLcJ0NMuUafQALgdQ=', 'Content-Length': '0'} 2023-09-29T22:05:56.273 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea332ef8e3d8c157-0065174a44-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.274 DEBUG:boto:path=/hcnphz-18/obj 2023-09-29T22:05:56.274 DEBUG:boto:auth_path=/hcnphz-18/obj 2023-09-29T22:05:56.274 DEBUG:boto:Method: DELETE 2023-09-29T22:05:56.274 DEBUG:boto:Path: /hcnphz-18/obj 2023-09-29T22:05:56.274 DEBUG:boto:Data: 2023-09-29T22:05:56.274 DEBUG:boto:Headers: {} 2023-09-29T22:05:56.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:05:56.274 DEBUG:boto:Port: 8001 2023-09-29T22:05:56.275 DEBUG:boto:Params: {} 2023-09-29T22:05:56.275 DEBUG:boto:Token: None 2023-09-29T22:05:56.275 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:05:56 GMT /hcnphz-18/obj 2023-09-29T22:05:56.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5qB4d0K74SnnZ2515b3WDXWAZ/4= 2023-09-29T22:05:56.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:05:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5qB4d0K74SnnZ2515b3WDXWAZ/4=', 'Content-Length': '0'} 2023-09-29T22:05:56.294 DEBUG:boto:Response headers: [('x-amz-version-id', 'null'), ('x-amz-delete-marker', 'true'), ('x-amz-request-id', 'tx000003ebe02e25e6780a0-0065174a44-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:05:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:05:56.295 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-17 2023-09-29T22:05:56.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:56.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-17', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:05:56.295 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-17 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:05:56.346 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:05:56.346 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:05:56.410 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:05:56.406+0000 7f0768630f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:01.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:01.410+0000 7f0768630f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:06.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:06.409+0000 7f0768630f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:11.413 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:11.409+0000 7f0768630f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:16.417 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:16.413+0000 7f0768630f40 1 bucket sync caught up with source: 2023-09-29T22:06:16.418 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.12.7, , ] 2023-09-29T22:06:16.418 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.12.7, , ] 2023-09-29T22:06:16.418 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:16.413+0000 7f0768630f40 0 bucket checkpoint complete 2023-09-29T22:06:16.445 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-17 2023-09-29T22:06:16.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:06:16.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-17', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:06:16.446 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-17 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:06:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:16.495 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:16.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:16.553+0000 7fdbb1b63f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:21.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:21.561+0000 7fdbb1b63f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:26.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:26.561+0000 7fdbb1b63f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:31.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:31.561+0000 7fdbb1b63f40 1 waiting to reach incremental sync.. 2023-09-29T22:06:36.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:36.565+0000 7fdbb1b63f40 1 bucket sync caught up with source: 2023-09-29T22:06:36.565 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.23.14, , ] 2023-09-29T22:06:36.566 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.23.14, , ] 2023-09-29T22:06:36.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:36.565+0000 7fdbb1b63f40 0 bucket checkpoint complete 2023-09-29T22:06:36.594 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-17 zones={a2, a1} 2023-09-29T22:06:36.594 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:06:36.595 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:06:36.595 DEBUG:boto:Method: HEAD 2023-09-29T22:06:36.595 DEBUG:boto:Path: /hcnphz-17/ 2023-09-29T22:06:36.595 DEBUG:boto:Data: 2023-09-29T22:06:36.595 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:36.595 DEBUG:boto:Port: 8001 2023-09-29T22:06:36.595 DEBUG:boto:Params: {} 2023-09-29T22:06:36.596 DEBUG:boto:Token: None 2023-09-29T22:06:36.596 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/ 2023-09-29T22:06:36.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FbWswjXsx8fYS43WzHAxhsD9B2U= 2023-09-29T22:06:36.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FbWswjXsx8fYS43WzHAxhsD9B2U=', 'Content-Length': '0'} 2023-09-29T22:06:36.600 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a4c6c4ed58428df-0065174a6c-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.600 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:06:36.600 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:06:36.601 DEBUG:boto:Method: HEAD 2023-09-29T22:06:36.601 DEBUG:boto:Path: /hcnphz-17/ 2023-09-29T22:06:36.601 DEBUG:boto:Data: 2023-09-29T22:06:36.601 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:36.601 DEBUG:boto:Port: 8000 2023-09-29T22:06:36.601 DEBUG:boto:Params: {} 2023-09-29T22:06:36.601 DEBUG:boto:Token: None 2023-09-29T22:06:36.601 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/ 2023-09-29T22:06:36.602 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FbWswjXsx8fYS43WzHAxhsD9B2U= 2023-09-29T22:06:36.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FbWswjXsx8fYS43WzHAxhsD9B2U=', 'Content-Length': '0'} 2023-09-29T22:06:36.605 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005b4d02b44ca689ec-0065174a6c-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.605 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:06:36.606 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:06:36.606 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:06:36.606 DEBUG:boto:path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.606 DEBUG:boto:auth_path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.606 DEBUG:boto:Method: GET 2023-09-29T22:06:36.606 DEBUG:boto:Path: /hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.606 DEBUG:boto:Data: 2023-09-29T22:06:36.606 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:36.607 DEBUG:boto:Port: 8001 2023-09-29T22:06:36.607 DEBUG:boto:Params: {} 2023-09-29T22:06:36.607 DEBUG:boto:Token: None 2023-09-29T22:06:36.607 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/?versions 2023-09-29T22:06:36.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg= 2023-09-29T22:06:36.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg=', 'Content-Length': '0'} 2023-09-29T22:06:36.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9ddd28e5ce1283e-0065174a6c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.658 DEBUG:boto:b'hcnphz-17999falseobjnulltrue2023-09-29T22:05:56.235Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:36.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:06:36.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:06:36.659 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:06:36.659 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:06:36.660 DEBUG:boto:path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.660 DEBUG:boto:auth_path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.660 DEBUG:boto:Method: GET 2023-09-29T22:06:36.660 DEBUG:boto:Path: /hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.660 DEBUG:boto:Data: 2023-09-29T22:06:36.660 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:36.660 DEBUG:boto:Port: 8000 2023-09-29T22:06:36.660 DEBUG:boto:Params: {} 2023-09-29T22:06:36.661 DEBUG:boto:Token: None 2023-09-29T22:06:36.661 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/?versions 2023-09-29T22:06:36.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg= 2023-09-29T22:06:36.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg=', 'Content-Length': '0'} 2023-09-29T22:06:36.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d31fef3e5eab8c9d-0065174a6c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.666 DEBUG:boto:b'hcnphz-17999falseobjnulltrue2023-09-29T22:05:56.235Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:36.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:06:36.666 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:06:36.666 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:06:36.667 DEBUG:boto:path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.667 DEBUG:boto:auth_path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.667 DEBUG:boto:Method: GET 2023-09-29T22:06:36.667 DEBUG:boto:Path: /hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.667 DEBUG:boto:Data: 2023-09-29T22:06:36.667 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:36.667 DEBUG:boto:Port: 8001 2023-09-29T22:06:36.667 DEBUG:boto:Params: {} 2023-09-29T22:06:36.667 DEBUG:boto:Token: None 2023-09-29T22:06:36.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/?versions 2023-09-29T22:06:36.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg= 2023-09-29T22:06:36.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg=', 'Content-Length': '0'} 2023-09-29T22:06:36.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077e10560cccdf43f-0065174a6c-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.672 DEBUG:boto:b'hcnphz-17999falseobjnulltrue2023-09-29T22:05:56.235Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:36.673 DEBUG:boto:path=/hcnphz-17/ 2023-09-29T22:06:36.673 DEBUG:boto:auth_path=/hcnphz-17/ 2023-09-29T22:06:36.673 DEBUG:boto:path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.673 DEBUG:boto:auth_path=/hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.673 DEBUG:boto:Method: GET 2023-09-29T22:06:36.673 DEBUG:boto:Path: /hcnphz-17/?versions&max-keys=999 2023-09-29T22:06:36.673 DEBUG:boto:Data: 2023-09-29T22:06:36.674 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:36.674 DEBUG:boto:Port: 8000 2023-09-29T22:06:36.674 DEBUG:boto:Params: {} 2023-09-29T22:06:36.674 DEBUG:boto:Token: None 2023-09-29T22:06:36.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/?versions 2023-09-29T22:06:36.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg= 2023-09-29T22:06:36.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3OsM+ZQ1qye9KRs5Ci1HudkV8Pg=', 'Content-Length': '0'} 2023-09-29T22:06:36.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b260409950e3fd07-0065174a6c-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.722 DEBUG:boto:b'hcnphz-17999falseobjnulltrue2023-09-29T22:05:56.235Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:36.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:06:36.723 DEBUG:boto:path=/hcnphz-17/obj 2023-09-29T22:06:36.723 DEBUG:boto:auth_path=/hcnphz-17/obj 2023-09-29T22:06:36.723 DEBUG:boto:Method: HEAD 2023-09-29T22:06:36.723 DEBUG:boto:Path: /hcnphz-17/obj 2023-09-29T22:06:36.724 DEBUG:boto:Data: 2023-09-29T22:06:36.724 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:36.724 DEBUG:boto:Port: 8001 2023-09-29T22:06:36.724 DEBUG:boto:Params: {} 2023-09-29T22:06:36.724 DEBUG:boto:Token: None 2023-09-29T22:06:36.724 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/obj 2023-09-29T22:06:36.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JKjVpWv51Ap1kZ+y5HkFjAJ+kfE= 2023-09-29T22:06:36.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JKjVpWv51Ap1kZ+y5HkFjAJ+kfE=', 'Content-Length': '0'} 2023-09-29T22:06:36.727 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a9229978188b7f5d-0065174a6c-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.727 DEBUG:boto:path=/hcnphz-17/obj 2023-09-29T22:06:36.727 DEBUG:boto:auth_path=/hcnphz-17/obj 2023-09-29T22:06:36.727 DEBUG:boto:Method: HEAD 2023-09-29T22:06:36.727 DEBUG:boto:Path: /hcnphz-17/obj 2023-09-29T22:06:36.728 DEBUG:boto:Data: 2023-09-29T22:06:36.728 DEBUG:boto:Headers: {} 2023-09-29T22:06:36.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:36.728 DEBUG:boto:Port: 8000 2023-09-29T22:06:36.728 DEBUG:boto:Params: {} 2023-09-29T22:06:36.728 DEBUG:boto:Token: None 2023-09-29T22:06:36.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:36 GMT /hcnphz-17/obj 2023-09-29T22:06:36.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JKjVpWv51Ap1kZ+y5HkFjAJ+kfE= 2023-09-29T22:06:36.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JKjVpWv51Ap1kZ+y5HkFjAJ+kfE=', 'Content-Length': '0'} 2023-09-29T22:06:36.731 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000a8f0edc29f51fc33-0065174a6c-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:36.731 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-17 zones={a2, a1} 2023-09-29T22:06:36.731 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-18 2023-09-29T22:06:36.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:06:36.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-18', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:06:36.732 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-18 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:06:36.784 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:36.785 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:36.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:36.853+0000 7ff30a0e5f40 1 bucket sync caught up with source: 2023-09-29T22:06:36.853 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.5.14, , ] 2023-09-29T22:06:36.853 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.5.14, , ] 2023-09-29T22:06:36.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:36.853+0000 7ff30a0e5f40 0 bucket checkpoint complete 2023-09-29T22:06:36.879 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-18 2023-09-29T22:06:36.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:06:36.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-18', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:06:36.879 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-18 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:06:36.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:36.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:36.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:36.993+0000 7f2028d60f40 1 bucket sync caught up with source: 2023-09-29T22:06:36.994 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000003.16.7, , ] 2023-09-29T22:06:36.994 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000003.16.7, , ] 2023-09-29T22:06:36.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:36.993+0000 7f2028d60f40 0 bucket checkpoint complete 2023-09-29T22:06:37.021 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-18 zones={a2, a1} 2023-09-29T22:06:37.021 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:06:37.021 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:06:37.021 DEBUG:boto:Method: HEAD 2023-09-29T22:06:37.022 DEBUG:boto:Path: /hcnphz-18/ 2023-09-29T22:06:37.022 DEBUG:boto:Data: 2023-09-29T22:06:37.022 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.022 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.022 DEBUG:boto:Params: {} 2023-09-29T22:06:37.022 DEBUG:boto:Token: None 2023-09-29T22:06:37.023 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/ 2023-09-29T22:06:37.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YbufFrqgm5rc4grN0Y9gBVlX8ck= 2023-09-29T22:06:37.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YbufFrqgm5rc4grN0Y9gBVlX8ck=', 'Content-Length': '0'} 2023-09-29T22:06:37.026 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000121113edf622e0fa-0065174a6d-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.026 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:06:37.026 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:06:37.026 DEBUG:boto:Method: HEAD 2023-09-29T22:06:37.026 DEBUG:boto:Path: /hcnphz-18/ 2023-09-29T22:06:37.026 DEBUG:boto:Data: 2023-09-29T22:06:37.027 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.027 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.027 DEBUG:boto:Params: {} 2023-09-29T22:06:37.027 DEBUG:boto:Token: None 2023-09-29T22:06:37.027 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/ 2023-09-29T22:06:37.027 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YbufFrqgm5rc4grN0Y9gBVlX8ck= 2023-09-29T22:06:37.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YbufFrqgm5rc4grN0Y9gBVlX8ck=', 'Content-Length': '0'} 2023-09-29T22:06:37.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4e82afa591e06c9-0065174a6d-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.030 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:06:37.031 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:06:37.031 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:06:37.031 DEBUG:boto:path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.031 DEBUG:boto:auth_path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.031 DEBUG:boto:Method: GET 2023-09-29T22:06:37.031 DEBUG:boto:Path: /hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.031 DEBUG:boto:Data: 2023-09-29T22:06:37.031 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.032 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.032 DEBUG:boto:Params: {} 2023-09-29T22:06:37.032 DEBUG:boto:Token: None 2023-09-29T22:06:37.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/?versions 2023-09-29T22:06:37.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc= 2023-09-29T22:06:37.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc=', 'Content-Length': '0'} 2023-09-29T22:06:37.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e296897c07e1ec66-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.035 DEBUG:boto:b'hcnphz-18999falseobjnulltrue2023-09-29T22:05:56.280Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:37.036 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:06:37.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:06:37.036 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:06:37.036 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:06:37.036 DEBUG:boto:path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.037 DEBUG:boto:auth_path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.037 DEBUG:boto:Method: GET 2023-09-29T22:06:37.037 DEBUG:boto:Path: /hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.037 DEBUG:boto:Data: 2023-09-29T22:06:37.037 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.037 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.037 DEBUG:boto:Params: {} 2023-09-29T22:06:37.038 DEBUG:boto:Token: None 2023-09-29T22:06:37.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/?versions 2023-09-29T22:06:37.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc= 2023-09-29T22:06:37.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc=', 'Content-Length': '0'} 2023-09-29T22:06:37.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084134fd08f6f16c9-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.042 DEBUG:boto:b'hcnphz-18999falseobjnulltrue2023-09-29T22:05:56.280Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:37.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2023-09-29T22:06:37.042 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:06:37.043 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:06:37.043 DEBUG:boto:path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.043 DEBUG:boto:auth_path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.043 DEBUG:boto:Method: GET 2023-09-29T22:06:37.043 DEBUG:boto:Path: /hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.043 DEBUG:boto:Data: 2023-09-29T22:06:37.043 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.044 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.044 DEBUG:boto:Params: {} 2023-09-29T22:06:37.044 DEBUG:boto:Token: None 2023-09-29T22:06:37.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/?versions 2023-09-29T22:06:37.044 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc= 2023-09-29T22:06:37.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc=', 'Content-Length': '0'} 2023-09-29T22:06:37.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a54045afa852624-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.048 DEBUG:boto:b'hcnphz-18999falseobjnulltrue2023-09-29T22:05:56.280Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:37.048 DEBUG:boto:path=/hcnphz-18/ 2023-09-29T22:06:37.048 DEBUG:boto:auth_path=/hcnphz-18/ 2023-09-29T22:06:37.048 DEBUG:boto:path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.049 DEBUG:boto:auth_path=/hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.049 DEBUG:boto:Method: GET 2023-09-29T22:06:37.049 DEBUG:boto:Path: /hcnphz-18/?versions&max-keys=999 2023-09-29T22:06:37.049 DEBUG:boto:Data: 2023-09-29T22:06:37.049 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.049 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.049 DEBUG:boto:Params: {} 2023-09-29T22:06:37.049 DEBUG:boto:Token: None 2023-09-29T22:06:37.050 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/?versions 2023-09-29T22:06:37.050 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc= 2023-09-29T22:06:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LHqduZ66QjGlIo7CHc01TAAeWDc=', 'Content-Length': '0'} 2023-09-29T22:06:37.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070ca49ac9e2e4d4c-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.054 DEBUG:boto:b'hcnphz-18999falseobjnulltrue2023-09-29T22:05:56.280Zrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:06:37.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2023-09-29T22:06:37.054 DEBUG:boto:path=/hcnphz-18/obj 2023-09-29T22:06:37.055 DEBUG:boto:auth_path=/hcnphz-18/obj 2023-09-29T22:06:37.055 DEBUG:boto:Method: HEAD 2023-09-29T22:06:37.055 DEBUG:boto:Path: /hcnphz-18/obj 2023-09-29T22:06:37.055 DEBUG:boto:Data: 2023-09-29T22:06:37.055 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.055 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.055 DEBUG:boto:Params: {} 2023-09-29T22:06:37.055 DEBUG:boto:Token: None 2023-09-29T22:06:37.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/obj 2023-09-29T22:06:37.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ktyLqUqMaYYuxsO5mkarN4E65qQ= 2023-09-29T22:06:37.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ktyLqUqMaYYuxsO5mkarN4E65qQ=', 'Content-Length': '0'} 2023-09-29T22:06:37.058 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000b8f9f683e1699915-0065174a6d-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.058 DEBUG:boto:path=/hcnphz-18/obj 2023-09-29T22:06:37.058 DEBUG:boto:auth_path=/hcnphz-18/obj 2023-09-29T22:06:37.058 DEBUG:boto:Method: HEAD 2023-09-29T22:06:37.058 DEBUG:boto:Path: /hcnphz-18/obj 2023-09-29T22:06:37.058 DEBUG:boto:Data: 2023-09-29T22:06:37.058 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.059 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.059 DEBUG:boto:Params: {} 2023-09-29T22:06:37.059 DEBUG:boto:Token: None 2023-09-29T22:06:37.059 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-18/obj 2023-09-29T22:06:37.059 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ktyLqUqMaYYuxsO5mkarN4E65qQ= 2023-09-29T22:06:37.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ktyLqUqMaYYuxsO5mkarN4E65qQ=', 'Content-Length': '0'} 2023-09-29T22:06:37.061 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c600f8823b75b72a-0065174a6d-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.062 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-18 zones={a2, a1} 2023-09-29T22:06:37.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2023-09-29T22:06:37.063 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-19 2023-09-29T22:06:37.063 DEBUG:boto:path=/hcnphz-19/ 2023-09-29T22:06:37.063 DEBUG:boto:auth_path=/hcnphz-19/ 2023-09-29T22:06:37.063 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.063 DEBUG:boto:Path: /hcnphz-19/ 2023-09-29T22:06:37.063 DEBUG:boto:Data: 2023-09-29T22:06:37.063 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.064 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.064 DEBUG:boto:Params: {} 2023-09-29T22:06:37.064 DEBUG:boto:Token: None 2023-09-29T22:06:37.064 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-19/ 2023-09-29T22:06:37.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KoiStAsVi+vPOgitEwi+1D4jwbs= 2023-09-29T22:06:37.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KoiStAsVi+vPOgitEwi+1D4jwbs=', 'Content-Length': '0'} 2023-09-29T22:06:37.089 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002d058bbc5a1c8b77-0065174a6d-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.090 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-20 2023-09-29T22:06:37.090 DEBUG:boto:path=/hcnphz-20/ 2023-09-29T22:06:37.090 DEBUG:boto:auth_path=/hcnphz-20/ 2023-09-29T22:06:37.090 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.090 DEBUG:boto:Path: /hcnphz-20/ 2023-09-29T22:06:37.090 DEBUG:boto:Data: 2023-09-29T22:06:37.091 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.091 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.091 DEBUG:boto:Params: {} 2023-09-29T22:06:37.091 DEBUG:boto:Token: None 2023-09-29T22:06:37.091 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-20/ 2023-09-29T22:06:37.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3U7SNgsJGSac2XcG0Wp1hY99Px4= 2023-09-29T22:06:37.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3U7SNgsJGSac2XcG0Wp1hY99Px4=', 'Content-Length': '0'} 2023-09-29T22:06:37.138 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5efcdcfcda5b5ef-0065174a6d-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.139 DEBUG:boto:Using access key provided by client. 2023-09-29T22:06:37.139 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:06:37.139 DEBUG:boto:Using access key provided by client. 2023-09-29T22:06:37.139 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:06:37.140 DEBUG:boto:Using access key provided by client. 2023-09-29T22:06:37.140 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:06:37.140 DEBUG:boto:Using access key provided by client. 2023-09-29T22:06:37.141 DEBUG:boto:Using secret key provided by client. 2023-09-29T22:06:37.141 INFO:rgw_multi.tests:create bucket zone=b1 name=hcnphz-21 2023-09-29T22:06:37.141 DEBUG:boto:path=/hcnphz-21/ 2023-09-29T22:06:37.141 DEBUG:boto:auth_path=/hcnphz-21/ 2023-09-29T22:06:37.141 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.141 DEBUG:boto:Path: /hcnphz-21/ 2023-09-29T22:06:37.141 DEBUG:boto:Data: 2023-09-29T22:06:37.141 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.142 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.142 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.142 DEBUG:boto:Params: {} 2023-09-29T22:06:37.142 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:06:37.142 DEBUG:boto:Token: None 2023-09-29T22:06:37.142 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-21/ 2023-09-29T22:06:37.142 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jGDk58YJ/B+eCf3N4KFtojn1Vjo= 2023-09-29T22:06:37.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jGDk58YJ/B+eCf3N4KFtojn1Vjo=', 'Content-Length': '0'} 2023-09-29T22:06:37.192 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fd60cde7475cd174-0065174a6d-1103-b1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.192 INFO:rgw_multi.tests:create bucket zone=b2 name=hcnphz-22 2023-09-29T22:06:37.192 DEBUG:boto:path=/hcnphz-22/ 2023-09-29T22:06:37.192 DEBUG:boto:auth_path=/hcnphz-22/ 2023-09-29T22:06:37.192 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.192 DEBUG:boto:Path: /hcnphz-22/ 2023-09-29T22:06:37.193 DEBUG:boto:Data: 2023-09-29T22:06:37.193 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.193 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.193 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.193 DEBUG:boto:Params: {} 2023-09-29T22:06:37.193 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:06:37.193 DEBUG:boto:Token: None 2023-09-29T22:06:37.193 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-22/ 2023-09-29T22:06:37.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9lvuUpoOvRNVSduJ/dqzSlsoSOc= 2023-09-29T22:06:37.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9lvuUpoOvRNVSduJ/dqzSlsoSOc=', 'Content-Length': '0'} 2023-09-29T22:06:37.238 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3d7bed6230815a4-0065174a6d-1109-b2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.238 DEBUG:boto:path=/hcnphz-19/ 2023-09-29T22:06:37.239 DEBUG:boto:auth_path=/hcnphz-19/ 2023-09-29T22:06:37.239 DEBUG:boto:path=/hcnphz-19/?versioning 2023-09-29T22:06:37.239 DEBUG:boto:auth_path=/hcnphz-19/?versioning 2023-09-29T22:06:37.239 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.239 DEBUG:boto:Path: /hcnphz-19/?versioning 2023-09-29T22:06:37.239 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:06:37.239 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.239 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.240 DEBUG:boto:Params: {} 2023-09-29T22:06:37.240 DEBUG:boto:Token: None 2023-09-29T22:06:37.240 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-19/?versioning 2023-09-29T22:06:37.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PvVKN4OUEGGdbYjBHGyj5xPnB8M= 2023-09-29T22:06:37.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PvVKN4OUEGGdbYjBHGyj5xPnB8M=', 'Content-Length': '228'} 2023-09-29T22:06:37.249 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004b957c26d7bf2c66-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.249 DEBUG:boto:path=/hcnphz-19/ 2023-09-29T22:06:37.250 DEBUG:boto:auth_path=/hcnphz-19/ 2023-09-29T22:06:37.250 DEBUG:boto:path=/hcnphz-19/?versioning 2023-09-29T22:06:37.250 DEBUG:boto:auth_path=/hcnphz-19/?versioning 2023-09-29T22:06:37.250 DEBUG:boto:Method: GET 2023-09-29T22:06:37.250 DEBUG:boto:Path: /hcnphz-19/?versioning 2023-09-29T22:06:37.250 DEBUG:boto:Data: 2023-09-29T22:06:37.250 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.250 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.251 DEBUG:boto:Params: {} 2023-09-29T22:06:37.251 DEBUG:boto:Token: None 2023-09-29T22:06:37.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-19/?versioning 2023-09-29T22:06:37.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aFKWMhH1oOcfc42kTtHKQOaOzjI= 2023-09-29T22:06:37.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aFKWMhH1oOcfc42kTtHKQOaOzjI=', 'Content-Length': '0'} 2023-09-29T22:06:37.253 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f9bcda1bd4373265-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.298 DEBUG:boto:EnabledDisabled 2023-09-29T22:06:37.299 DEBUG:boto:path=/hcnphz-20/ 2023-09-29T22:06:37.299 DEBUG:boto:auth_path=/hcnphz-20/ 2023-09-29T22:06:37.299 DEBUG:boto:path=/hcnphz-20/?versioning 2023-09-29T22:06:37.299 DEBUG:boto:auth_path=/hcnphz-20/?versioning 2023-09-29T22:06:37.300 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.300 DEBUG:boto:Path: /hcnphz-20/?versioning 2023-09-29T22:06:37.300 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:06:37.300 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.300 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.300 DEBUG:boto:Params: {} 2023-09-29T22:06:37.300 DEBUG:boto:Token: None 2023-09-29T22:06:37.301 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-20/?versioning 2023-09-29T22:06:37.301 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iqEIxJq4lc+ElJew1SUiK5vnpt0= 2023-09-29T22:06:37.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iqEIxJq4lc+ElJew1SUiK5vnpt0=', 'Content-Length': '228'} 2023-09-29T22:06:37.319 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000085494b8508d737b4-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.319 DEBUG:boto:path=/hcnphz-20/ 2023-09-29T22:06:37.320 DEBUG:boto:auth_path=/hcnphz-20/ 2023-09-29T22:06:37.320 DEBUG:boto:path=/hcnphz-20/?versioning 2023-09-29T22:06:37.320 DEBUG:boto:auth_path=/hcnphz-20/?versioning 2023-09-29T22:06:37.320 DEBUG:boto:Method: GET 2023-09-29T22:06:37.320 DEBUG:boto:Path: /hcnphz-20/?versioning 2023-09-29T22:06:37.320 DEBUG:boto:Data: 2023-09-29T22:06:37.320 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.320 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.320 DEBUG:boto:Params: {} 2023-09-29T22:06:37.321 DEBUG:boto:Token: None 2023-09-29T22:06:37.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-20/?versioning 2023-09-29T22:06:37.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WqoojgmK2iYziJMnB5QNWorwZN4= 2023-09-29T22:06:37.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WqoojgmK2iYziJMnB5QNWorwZN4=', 'Content-Length': '0'} 2023-09-29T22:06:37.323 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ce81ae6c0dd7214-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.323 DEBUG:boto:EnabledDisabled 2023-09-29T22:06:37.323 DEBUG:boto:path=/hcnphz-21/ 2023-09-29T22:06:37.323 DEBUG:boto:auth_path=/hcnphz-21/ 2023-09-29T22:06:37.323 DEBUG:boto:path=/hcnphz-21/?versioning 2023-09-29T22:06:37.323 DEBUG:boto:auth_path=/hcnphz-21/?versioning 2023-09-29T22:06:37.323 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.323 DEBUG:boto:Path: /hcnphz-21/?versioning 2023-09-29T22:06:37.324 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:06:37.324 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.324 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.324 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.324 DEBUG:boto:Params: {} 2023-09-29T22:06:37.324 DEBUG:boto:Token: None 2023-09-29T22:06:37.324 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-21/?versioning 2023-09-29T22:06:37.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:98yvnJq/Cmm8Cxozro4BCc6fVVY= 2023-09-29T22:06:37.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:98yvnJq/Cmm8Cxozro4BCc6fVVY=', 'Content-Length': '228'} 2023-09-29T22:06:37.342 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000081a4525a0fa62c89-0065174a6d-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.343 DEBUG:boto:path=/hcnphz-21/ 2023-09-29T22:06:37.343 DEBUG:boto:auth_path=/hcnphz-21/ 2023-09-29T22:06:37.343 DEBUG:boto:path=/hcnphz-21/?versioning 2023-09-29T22:06:37.343 DEBUG:boto:auth_path=/hcnphz-21/?versioning 2023-09-29T22:06:37.343 DEBUG:boto:Method: GET 2023-09-29T22:06:37.343 DEBUG:boto:Path: /hcnphz-21/?versioning 2023-09-29T22:06:37.343 DEBUG:boto:Data: 2023-09-29T22:06:37.343 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.343 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.344 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.344 DEBUG:boto:Params: {} 2023-09-29T22:06:37.344 DEBUG:boto:Token: None 2023-09-29T22:06:37.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-21/?versioning 2023-09-29T22:06:37.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Razrk9Q1IwsymicjXmJebXcn7PQ= 2023-09-29T22:06:37.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Razrk9Q1IwsymicjXmJebXcn7PQ=', 'Content-Length': '0'} 2023-09-29T22:06:37.346 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000edd9a97b22a873-0065174a6d-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.346 DEBUG:boto:EnabledDisabled 2023-09-29T22:06:37.346 DEBUG:boto:path=/hcnphz-22/ 2023-09-29T22:06:37.346 DEBUG:boto:auth_path=/hcnphz-22/ 2023-09-29T22:06:37.346 DEBUG:boto:path=/hcnphz-22/?versioning 2023-09-29T22:06:37.346 DEBUG:boto:auth_path=/hcnphz-22/?versioning 2023-09-29T22:06:37.347 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.347 DEBUG:boto:Path: /hcnphz-22/?versioning 2023-09-29T22:06:37.347 DEBUG:boto:Data: Enabled Disabled 2023-09-29T22:06:37.347 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.347 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.347 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.347 DEBUG:boto:Params: {} 2023-09-29T22:06:37.347 DEBUG:boto:Token: None 2023-09-29T22:06:37.347 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-22/?versioning 2023-09-29T22:06:37.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZWh6aOGXJ9HrqXSuMTVST8sLho0= 2023-09-29T22:06:37.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZWh6aOGXJ9HrqXSuMTVST8sLho0=', 'Content-Length': '228'} 2023-09-29T22:06:37.374 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006ae206e90ee0f03a-0065174a6d-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.374 DEBUG:boto:path=/hcnphz-22/ 2023-09-29T22:06:37.374 DEBUG:boto:auth_path=/hcnphz-22/ 2023-09-29T22:06:37.374 DEBUG:boto:path=/hcnphz-22/?versioning 2023-09-29T22:06:37.374 DEBUG:boto:auth_path=/hcnphz-22/?versioning 2023-09-29T22:06:37.374 DEBUG:boto:Method: GET 2023-09-29T22:06:37.374 DEBUG:boto:Path: /hcnphz-22/?versioning 2023-09-29T22:06:37.375 DEBUG:boto:Data: 2023-09-29T22:06:37.375 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.375 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.375 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.375 DEBUG:boto:Params: {} 2023-09-29T22:06:37.375 DEBUG:boto:Token: None 2023-09-29T22:06:37.375 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-22/?versioning 2023-09-29T22:06:37.375 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5ZrWAJNI9+8n93POLjOOOaJf10A= 2023-09-29T22:06:37.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5ZrWAJNI9+8n93POLjOOOaJf10A=', 'Content-Length': '0'} 2023-09-29T22:06:37.377 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ddbaa8ef9c6b194d-0065174a6d-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '192'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.377 DEBUG:boto:EnabledDisabled 2023-09-29T22:06:37.377 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2023-09-29T22:06:37.378 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-23 2023-09-29T22:06:37.378 DEBUG:boto:path=/hcnphz-23/ 2023-09-29T22:06:37.378 DEBUG:boto:auth_path=/hcnphz-23/ 2023-09-29T22:06:37.378 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.378 DEBUG:boto:Path: /hcnphz-23/ 2023-09-29T22:06:37.379 DEBUG:boto:Data: 2023-09-29T22:06:37.379 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.379 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.379 DEBUG:boto:Params: {} 2023-09-29T22:06:37.379 DEBUG:boto:Token: None 2023-09-29T22:06:37.379 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-23/ 2023-09-29T22:06:37.379 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+I0yPrh+jAnywELTI3Cs1zdF0+Y= 2023-09-29T22:06:37.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+I0yPrh+jAnywELTI3Cs1zdF0+Y=', 'Content-Length': '0'} 2023-09-29T22:06:37.410 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bd5888ee89ac6fd9-0065174a6d-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.411 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-24 2023-09-29T22:06:37.411 DEBUG:boto:path=/hcnphz-24/ 2023-09-29T22:06:37.411 DEBUG:boto:auth_path=/hcnphz-24/ 2023-09-29T22:06:37.411 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.411 DEBUG:boto:Path: /hcnphz-24/ 2023-09-29T22:06:37.411 DEBUG:boto:Data: 2023-09-29T22:06:37.411 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.411 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.412 DEBUG:boto:Params: {} 2023-09-29T22:06:37.412 DEBUG:boto:Token: None 2023-09-29T22:06:37.412 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-24/ 2023-09-29T22:06:37.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TiQFQw4mqrkbofczaYmA5hFq86A= 2023-09-29T22:06:37.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TiQFQw4mqrkbofczaYmA5hFq86A=', 'Content-Length': '0'} 2023-09-29T22:06:37.453 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b41f790e2c28996-0065174a6d-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.454 INFO:rgw_multi.tests:create bucket zone=b1 name=hcnphz-25 2023-09-29T22:06:37.454 DEBUG:boto:path=/hcnphz-25/ 2023-09-29T22:06:37.454 DEBUG:boto:auth_path=/hcnphz-25/ 2023-09-29T22:06:37.454 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.454 DEBUG:boto:Path: /hcnphz-25/ 2023-09-29T22:06:37.454 DEBUG:boto:Data: 2023-09-29T22:06:37.455 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.455 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.455 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.455 DEBUG:boto:Params: {} 2023-09-29T22:06:37.455 DEBUG:boto:Token: None 2023-09-29T22:06:37.455 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-25/ 2023-09-29T22:06:37.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f5211vX2BbDiDI6X86cDB2j+Ddc= 2023-09-29T22:06:37.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f5211vX2BbDiDI6X86cDB2j+Ddc=', 'Content-Length': '0'} 2023-09-29T22:06:37.503 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000064994d7186f326bd-0065174a6d-1103-b1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.503 INFO:rgw_multi.tests:create bucket zone=b2 name=hcnphz-26 2023-09-29T22:06:37.503 DEBUG:boto:path=/hcnphz-26/ 2023-09-29T22:06:37.503 DEBUG:boto:auth_path=/hcnphz-26/ 2023-09-29T22:06:37.504 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.504 DEBUG:boto:Path: /hcnphz-26/ 2023-09-29T22:06:37.504 DEBUG:boto:Data: 2023-09-29T22:06:37.504 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.504 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.504 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.504 DEBUG:boto:Params: {} 2023-09-29T22:06:37.504 DEBUG:boto:Token: None 2023-09-29T22:06:37.504 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-26/ 2023-09-29T22:06:37.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6AEnH31jO8vuWlaspSvxENRkIQQ= 2023-09-29T22:06:37.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6AEnH31jO8vuWlaspSvxENRkIQQ=', 'Content-Length': '0'} 2023-09-29T22:06:37.549 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000001c3f935d32c9d027-0065174a6d-1109-b2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.549 DEBUG:boto:path=/hcnphz-23/ 2023-09-29T22:06:37.549 DEBUG:boto:auth_path=/hcnphz-23/ 2023-09-29T22:06:37.550 DEBUG:boto:path=/hcnphz-23/?acl 2023-09-29T22:06:37.550 DEBUG:boto:auth_path=/hcnphz-23/?acl 2023-09-29T22:06:37.550 DEBUG:boto:Method: GET 2023-09-29T22:06:37.550 DEBUG:boto:Path: /hcnphz-23/?acl 2023-09-29T22:06:37.550 DEBUG:boto:Data: 2023-09-29T22:06:37.550 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.550 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.550 DEBUG:boto:Params: {} 2023-09-29T22:06:37.551 DEBUG:boto:Token: None 2023-09-29T22:06:37.551 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-23/?acl 2023-09-29T22:06:37.551 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0cPBTCdkWn31a6qciQI/CLpgFQc= 2023-09-29T22:06:37.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0cPBTCdkWn31a6qciQI/CLpgFQc=', 'Content-Length': '0'} 2023-09-29T22:06:37.552 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007ac2f43a3f564e93-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.553 DEBUG:boto:path=/hcnphz-23/ 2023-09-29T22:06:37.553 DEBUG:boto:auth_path=/hcnphz-23/ 2023-09-29T22:06:37.553 DEBUG:boto:path=/hcnphz-23/?acl 2023-09-29T22:06:37.554 DEBUG:boto:auth_path=/hcnphz-23/?acl 2023-09-29T22:06:37.554 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.554 DEBUG:boto:Path: /hcnphz-23/?acl 2023-09-29T22:06:37.554 DEBUG:boto:Data: 2023-09-29T22:06:37.554 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:06:37.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.555 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.555 DEBUG:boto:Params: {} 2023-09-29T22:06:37.555 DEBUG:boto:Token: None 2023-09-29T22:06:37.556 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT x-amz-acl:public-read /hcnphz-23/?acl 2023-09-29T22:06:37.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IDsTIjBa0Q5ADm650UkdTfjqlo0= 2023-09-29T22:06:37.556 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IDsTIjBa0Q5ADm650UkdTfjqlo0=', 'Content-Length': '0'} 2023-09-29T22:06:37.577 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000555927964b08d325-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.577 DEBUG:boto:path=/hcnphz-23/ 2023-09-29T22:06:37.578 DEBUG:boto:auth_path=/hcnphz-23/ 2023-09-29T22:06:37.578 DEBUG:boto:path=/hcnphz-23/?acl 2023-09-29T22:06:37.578 DEBUG:boto:auth_path=/hcnphz-23/?acl 2023-09-29T22:06:37.578 DEBUG:boto:Method: GET 2023-09-29T22:06:37.578 DEBUG:boto:Path: /hcnphz-23/?acl 2023-09-29T22:06:37.578 DEBUG:boto:Data: 2023-09-29T22:06:37.578 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.578 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.579 DEBUG:boto:Params: {} 2023-09-29T22:06:37.579 DEBUG:boto:Token: None 2023-09-29T22:06:37.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-23/?acl 2023-09-29T22:06:37.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0cPBTCdkWn31a6qciQI/CLpgFQc= 2023-09-29T22:06:37.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0cPBTCdkWn31a6qciQI/CLpgFQc=', 'Content-Length': '0'} 2023-09-29T22:06:37.581 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c3b75506e384409e-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.582 DEBUG:boto:path=/hcnphz-24/ 2023-09-29T22:06:37.582 DEBUG:boto:auth_path=/hcnphz-24/ 2023-09-29T22:06:37.582 DEBUG:boto:path=/hcnphz-24/?acl 2023-09-29T22:06:37.582 DEBUG:boto:auth_path=/hcnphz-24/?acl 2023-09-29T22:06:37.582 DEBUG:boto:Method: GET 2023-09-29T22:06:37.583 DEBUG:boto:Path: /hcnphz-24/?acl 2023-09-29T22:06:37.583 DEBUG:boto:Data: 2023-09-29T22:06:37.583 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.583 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.583 DEBUG:boto:Params: {} 2023-09-29T22:06:37.583 DEBUG:boto:Token: None 2023-09-29T22:06:37.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-24/?acl 2023-09-29T22:06:37.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E3+v2Z2HydveQBKYH2DUoRQ73Ps= 2023-09-29T22:06:37.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E3+v2Z2HydveQBKYH2DUoRQ73Ps=', 'Content-Length': '0'} 2023-09-29T22:06:37.585 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000164cd1f3c1a4e50e-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.585 DEBUG:boto:path=/hcnphz-24/ 2023-09-29T22:06:37.585 DEBUG:boto:auth_path=/hcnphz-24/ 2023-09-29T22:06:37.585 DEBUG:boto:path=/hcnphz-24/?acl 2023-09-29T22:06:37.586 DEBUG:boto:auth_path=/hcnphz-24/?acl 2023-09-29T22:06:37.586 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.586 DEBUG:boto:Path: /hcnphz-24/?acl 2023-09-29T22:06:37.586 DEBUG:boto:Data: 2023-09-29T22:06:37.586 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:06:37.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.586 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.586 DEBUG:boto:Params: {} 2023-09-29T22:06:37.586 DEBUG:boto:Token: None 2023-09-29T22:06:37.587 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT x-amz-acl:public-read /hcnphz-24/?acl 2023-09-29T22:06:37.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:61QMbyMqaRRYPRXxs6IKXCEEOUU= 2023-09-29T22:06:37.587 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:61QMbyMqaRRYPRXxs6IKXCEEOUU=', 'Content-Length': '0'} 2023-09-29T22:06:37.614 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005828963cc09e05a8-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.615 DEBUG:boto:path=/hcnphz-24/ 2023-09-29T22:06:37.615 DEBUG:boto:auth_path=/hcnphz-24/ 2023-09-29T22:06:37.615 DEBUG:boto:path=/hcnphz-24/?acl 2023-09-29T22:06:37.615 DEBUG:boto:auth_path=/hcnphz-24/?acl 2023-09-29T22:06:37.615 DEBUG:boto:Method: GET 2023-09-29T22:06:37.615 DEBUG:boto:Path: /hcnphz-24/?acl 2023-09-29T22:06:37.615 DEBUG:boto:Data: 2023-09-29T22:06:37.615 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.616 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.616 DEBUG:boto:Params: {} 2023-09-29T22:06:37.616 DEBUG:boto:Token: None 2023-09-29T22:06:37.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-24/?acl 2023-09-29T22:06:37.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E3+v2Z2HydveQBKYH2DUoRQ73Ps= 2023-09-29T22:06:37.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E3+v2Z2HydveQBKYH2DUoRQ73Ps=', 'Content-Length': '0'} 2023-09-29T22:06:37.617 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c67a123b56a0d220-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.618 DEBUG:boto:path=/hcnphz-25/ 2023-09-29T22:06:37.618 DEBUG:boto:auth_path=/hcnphz-25/ 2023-09-29T22:06:37.618 DEBUG:boto:path=/hcnphz-25/?acl 2023-09-29T22:06:37.618 DEBUG:boto:auth_path=/hcnphz-25/?acl 2023-09-29T22:06:37.618 DEBUG:boto:Method: GET 2023-09-29T22:06:37.619 DEBUG:boto:Path: /hcnphz-25/?acl 2023-09-29T22:06:37.619 DEBUG:boto:Data: 2023-09-29T22:06:37.619 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.619 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.619 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.619 DEBUG:boto:Params: {} 2023-09-29T22:06:37.620 DEBUG:boto:Token: None 2023-09-29T22:06:37.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-25/?acl 2023-09-29T22:06:37.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DW0QCIHUUKDkL7qsVJrRbtW/Qpg= 2023-09-29T22:06:37.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DW0QCIHUUKDkL7qsVJrRbtW/Qpg=', 'Content-Length': '0'} 2023-09-29T22:06:37.621 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b47bf7659b713a28-0065174a6d-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.622 DEBUG:boto:path=/hcnphz-25/ 2023-09-29T22:06:37.622 DEBUG:boto:auth_path=/hcnphz-25/ 2023-09-29T22:06:37.622 DEBUG:boto:path=/hcnphz-25/?acl 2023-09-29T22:06:37.622 DEBUG:boto:auth_path=/hcnphz-25/?acl 2023-09-29T22:06:37.622 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.622 DEBUG:boto:Path: /hcnphz-25/?acl 2023-09-29T22:06:37.623 DEBUG:boto:Data: 2023-09-29T22:06:37.623 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:06:37.623 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.623 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.623 DEBUG:boto:Params: {} 2023-09-29T22:06:37.623 DEBUG:boto:Token: None 2023-09-29T22:06:37.623 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT x-amz-acl:public-read /hcnphz-25/?acl 2023-09-29T22:06:37.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n8AmOOQIcxBtFiYb3UVPkObhc0c= 2023-09-29T22:06:37.623 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n8AmOOQIcxBtFiYb3UVPkObhc0c=', 'Content-Length': '0'} 2023-09-29T22:06:37.638 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002b9cc6ea551c2f51-0065174a6d-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.638 DEBUG:boto:path=/hcnphz-25/ 2023-09-29T22:06:37.638 DEBUG:boto:auth_path=/hcnphz-25/ 2023-09-29T22:06:37.638 DEBUG:boto:path=/hcnphz-25/?acl 2023-09-29T22:06:37.638 DEBUG:boto:auth_path=/hcnphz-25/?acl 2023-09-29T22:06:37.638 DEBUG:boto:Method: GET 2023-09-29T22:06:37.639 DEBUG:boto:Path: /hcnphz-25/?acl 2023-09-29T22:06:37.639 DEBUG:boto:Data: 2023-09-29T22:06:37.639 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.639 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.639 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.639 DEBUG:boto:Params: {} 2023-09-29T22:06:37.639 DEBUG:boto:Token: None 2023-09-29T22:06:37.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-25/?acl 2023-09-29T22:06:37.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DW0QCIHUUKDkL7qsVJrRbtW/Qpg= 2023-09-29T22:06:37.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DW0QCIHUUKDkL7qsVJrRbtW/Qpg=', 'Content-Length': '0'} 2023-09-29T22:06:37.641 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000860bbe68c1d6d8be-0065174a6d-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.641 DEBUG:boto:path=/hcnphz-26/ 2023-09-29T22:06:37.642 DEBUG:boto:auth_path=/hcnphz-26/ 2023-09-29T22:06:37.642 DEBUG:boto:path=/hcnphz-26/?acl 2023-09-29T22:06:37.642 DEBUG:boto:auth_path=/hcnphz-26/?acl 2023-09-29T22:06:37.642 DEBUG:boto:Method: GET 2023-09-29T22:06:37.642 DEBUG:boto:Path: /hcnphz-26/?acl 2023-09-29T22:06:37.642 DEBUG:boto:Data: 2023-09-29T22:06:37.642 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.642 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.642 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.643 DEBUG:boto:Params: {} 2023-09-29T22:06:37.643 DEBUG:boto:Token: None 2023-09-29T22:06:37.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-26/?acl 2023-09-29T22:06:37.643 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RaLlmp0imWShl3KHuUH0hqY+SwM= 2023-09-29T22:06:37.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RaLlmp0imWShl3KHuUH0hqY+SwM=', 'Content-Length': '0'} 2023-09-29T22:06:37.644 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000bca6c6ea60024dfb-0065174a6d-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '489'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.645 DEBUG:boto:path=/hcnphz-26/ 2023-09-29T22:06:37.645 DEBUG:boto:auth_path=/hcnphz-26/ 2023-09-29T22:06:37.645 DEBUG:boto:path=/hcnphz-26/?acl 2023-09-29T22:06:37.645 DEBUG:boto:auth_path=/hcnphz-26/?acl 2023-09-29T22:06:37.645 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.645 DEBUG:boto:Path: /hcnphz-26/?acl 2023-09-29T22:06:37.646 DEBUG:boto:Data: 2023-09-29T22:06:37.646 DEBUG:boto:Headers: {'x-amz-acl': 'public-read'} 2023-09-29T22:06:37.646 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.646 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.646 DEBUG:boto:Params: {} 2023-09-29T22:06:37.646 DEBUG:boto:Token: None 2023-09-29T22:06:37.646 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT x-amz-acl:public-read /hcnphz-26/?acl 2023-09-29T22:06:37.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7yWYcXDo9mVyZ5T1J2IOm9cLXk= 2023-09-29T22:06:37.647 DEBUG:boto:Final headers: {'x-amz-acl': 'public-read', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7yWYcXDo9mVyZ5T1J2IOm9cLXk=', 'Content-Length': '0'} 2023-09-29T22:06:37.664 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a3a2279a4725f58c-0065174a6d-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.664 DEBUG:boto:path=/hcnphz-26/ 2023-09-29T22:06:37.665 DEBUG:boto:auth_path=/hcnphz-26/ 2023-09-29T22:06:37.665 DEBUG:boto:path=/hcnphz-26/?acl 2023-09-29T22:06:37.665 DEBUG:boto:auth_path=/hcnphz-26/?acl 2023-09-29T22:06:37.665 DEBUG:boto:Method: GET 2023-09-29T22:06:37.665 DEBUG:boto:Path: /hcnphz-26/?acl 2023-09-29T22:06:37.665 DEBUG:boto:Data: 2023-09-29T22:06:37.665 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.665 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.665 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.666 DEBUG:boto:Params: {} 2023-09-29T22:06:37.666 DEBUG:boto:Token: None 2023-09-29T22:06:37.666 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-26/?acl 2023-09-29T22:06:37.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RaLlmp0imWShl3KHuUH0hqY+SwM= 2023-09-29T22:06:37.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RaLlmp0imWShl3KHuUH0hqY+SwM=', 'Content-Length': '0'} 2023-09-29T22:06:37.668 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004fd7db9589c45235-0065174a6d-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '681'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.668 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2023-09-29T22:06:37.669 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-27 2023-09-29T22:06:37.669 DEBUG:boto:path=/hcnphz-27/ 2023-09-29T22:06:37.669 DEBUG:boto:auth_path=/hcnphz-27/ 2023-09-29T22:06:37.669 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.669 DEBUG:boto:Path: /hcnphz-27/ 2023-09-29T22:06:37.670 DEBUG:boto:Data: 2023-09-29T22:06:37.670 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.670 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.670 DEBUG:boto:Params: {} 2023-09-29T22:06:37.670 DEBUG:boto:Token: None 2023-09-29T22:06:37.670 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-27/ 2023-09-29T22:06:37.671 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ApAPV8YPgYTdOPEFJTTptRixgrU= 2023-09-29T22:06:37.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ApAPV8YPgYTdOPEFJTTptRixgrU=', 'Content-Length': '0'} 2023-09-29T22:06:37.699 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b1d66a23204d5ab8-0065174a6d-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.699 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-28 2023-09-29T22:06:37.700 DEBUG:boto:path=/hcnphz-28/ 2023-09-29T22:06:37.700 DEBUG:boto:auth_path=/hcnphz-28/ 2023-09-29T22:06:37.700 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.700 DEBUG:boto:Path: /hcnphz-28/ 2023-09-29T22:06:37.700 DEBUG:boto:Data: 2023-09-29T22:06:37.700 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.700 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.700 DEBUG:boto:Params: {} 2023-09-29T22:06:37.701 DEBUG:boto:Token: None 2023-09-29T22:06:37.701 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-28/ 2023-09-29T22:06:37.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:plMeL/el92JBycrfRmwSnISekWc= 2023-09-29T22:06:37.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:plMeL/el92JBycrfRmwSnISekWc=', 'Content-Length': '0'} 2023-09-29T22:06:37.741 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a4a7919c57f5a12c-0065174a6d-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.741 INFO:rgw_multi.tests:create bucket zone=b1 name=hcnphz-29 2023-09-29T22:06:37.742 DEBUG:boto:path=/hcnphz-29/ 2023-09-29T22:06:37.742 DEBUG:boto:auth_path=/hcnphz-29/ 2023-09-29T22:06:37.742 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.743 DEBUG:boto:Path: /hcnphz-29/ 2023-09-29T22:06:37.743 DEBUG:boto:Data: 2023-09-29T22:06:37.743 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.744 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:37.744 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.744 DEBUG:boto:Params: {} 2023-09-29T22:06:37.744 DEBUG:boto:Token: None 2023-09-29T22:06:37.745 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-29/ 2023-09-29T22:06:37.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H5b6hhL/E4SbSRz7f62qaTPgRlE= 2023-09-29T22:06:37.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H5b6hhL/E4SbSRz7f62qaTPgRlE=', 'Content-Length': '0'} 2023-09-29T22:06:37.801 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005940fd620e73c679-0065174a6d-1103-b1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.801 INFO:rgw_multi.tests:create bucket zone=b2 name=hcnphz-30 2023-09-29T22:06:37.801 DEBUG:boto:path=/hcnphz-30/ 2023-09-29T22:06:37.801 DEBUG:boto:auth_path=/hcnphz-30/ 2023-09-29T22:06:37.802 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.802 DEBUG:boto:Path: /hcnphz-30/ 2023-09-29T22:06:37.802 DEBUG:boto:Data: 2023-09-29T22:06:37.802 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.802 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:37.802 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.802 DEBUG:boto:Params: {} 2023-09-29T22:06:37.802 DEBUG:boto:Token: None 2023-09-29T22:06:37.803 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-30/ 2023-09-29T22:06:37.803 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h/v7ALhUizHrxReN6Chm6QxcYfE= 2023-09-29T22:06:37.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h/v7ALhUizHrxReN6Chm6QxcYfE=', 'Content-Length': '0'} 2023-09-29T22:06:37.850 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009ffa179fb86423e3-0065174a6d-1109-b2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.851 DEBUG:boto:path=/hcnphz-27/ 2023-09-29T22:06:37.851 DEBUG:boto:auth_path=/hcnphz-27/ 2023-09-29T22:06:37.851 DEBUG:boto:path=/hcnphz-27/?cors 2023-09-29T22:06:37.851 DEBUG:boto:auth_path=/hcnphz-27/?cors 2023-09-29T22:06:37.851 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.851 DEBUG:boto:Path: /hcnphz-27/?cors 2023-09-29T22:06:37.851 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:06:37.851 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:06:37.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.852 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.852 DEBUG:boto:Params: {} 2023-09-29T22:06:37.852 DEBUG:boto:Token: None 2023-09-29T22:06:37.852 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-27/?cors 2023-09-29T22:06:37.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:33/MUXevpO8Nf49qXRRwmorxXj8= 2023-09-29T22:06:37.852 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:33/MUXevpO8Nf49qXRRwmorxXj8=', 'Content-Length': '218'} 2023-09-29T22:06:37.863 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b38e0d7f17faa913-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.863 DEBUG:boto:path=/hcnphz-27/ 2023-09-29T22:06:37.863 DEBUG:boto:auth_path=/hcnphz-27/ 2023-09-29T22:06:37.864 DEBUG:boto:path=/hcnphz-27/?cors 2023-09-29T22:06:37.864 DEBUG:boto:auth_path=/hcnphz-27/?cors 2023-09-29T22:06:37.864 DEBUG:boto:Method: GET 2023-09-29T22:06:37.864 DEBUG:boto:Path: /hcnphz-27/?cors 2023-09-29T22:06:37.864 DEBUG:boto:Data: 2023-09-29T22:06:37.864 DEBUG:boto:Headers: {} 2023-09-29T22:06:37.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:37.864 DEBUG:boto:Port: 8000 2023-09-29T22:06:37.864 DEBUG:boto:Params: {} 2023-09-29T22:06:37.865 DEBUG:boto:Token: None 2023-09-29T22:06:37.865 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-27/?cors 2023-09-29T22:06:37.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nwtq9GfaTJIsjPT2HRx1046EII0= 2023-09-29T22:06:37.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nwtq9GfaTJIsjPT2HRx1046EII0=', 'Content-Length': '0'} 2023-09-29T22:06:37.866 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000007de114ee90cb78e-0065174a6d-1122-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:06:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:37.867 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:06:37.867 DEBUG:boto:path=/hcnphz-28/ 2023-09-29T22:06:37.867 DEBUG:boto:auth_path=/hcnphz-28/ 2023-09-29T22:06:37.867 DEBUG:boto:path=/hcnphz-28/?cors 2023-09-29T22:06:37.867 DEBUG:boto:auth_path=/hcnphz-28/?cors 2023-09-29T22:06:37.868 DEBUG:boto:Method: PUT 2023-09-29T22:06:37.868 DEBUG:boto:Path: /hcnphz-28/?cors 2023-09-29T22:06:37.868 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:06:37.868 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:06:37.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:37.868 DEBUG:boto:Port: 8001 2023-09-29T22:06:37.868 DEBUG:boto:Params: {} 2023-09-29T22:06:37.868 DEBUG:boto:Token: None 2023-09-29T22:06:37.869 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:06:37 GMT /hcnphz-28/?cors 2023-09-29T22:06:37.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QvUjqzMyvp12HsNgB5Fm7hkQG9g= 2023-09-29T22:06:37.869 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QvUjqzMyvp12HsNgB5Fm7hkQG9g=', 'Content-Length': '218'} 2023-09-29T22:06:38.888 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007bb97f92afe8e0a5-0065174a6d-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:38.888 DEBUG:boto:path=/hcnphz-28/ 2023-09-29T22:06:38.888 DEBUG:boto:auth_path=/hcnphz-28/ 2023-09-29T22:06:38.888 DEBUG:boto:path=/hcnphz-28/?cors 2023-09-29T22:06:38.888 DEBUG:boto:auth_path=/hcnphz-28/?cors 2023-09-29T22:06:38.888 DEBUG:boto:Method: GET 2023-09-29T22:06:38.888 DEBUG:boto:Path: /hcnphz-28/?cors 2023-09-29T22:06:38.889 DEBUG:boto:Data: 2023-09-29T22:06:38.889 DEBUG:boto:Headers: {} 2023-09-29T22:06:38.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:38.889 DEBUG:boto:Port: 8001 2023-09-29T22:06:38.889 DEBUG:boto:Params: {} 2023-09-29T22:06:38.889 DEBUG:boto:Token: None 2023-09-29T22:06:38.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:38 GMT /hcnphz-28/?cors 2023-09-29T22:06:38.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6aF05RgEOaXVE45e5Vo8c0ryFD0= 2023-09-29T22:06:38.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6aF05RgEOaXVE45e5Vo8c0ryFD0=', 'Content-Length': '0'} 2023-09-29T22:06:38.891 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5b48526c653e7c9-0065174a6e-1126-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:06:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:38.934 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:06:38.935 DEBUG:boto:path=/hcnphz-29/ 2023-09-29T22:06:38.935 DEBUG:boto:auth_path=/hcnphz-29/ 2023-09-29T22:06:38.935 DEBUG:boto:path=/hcnphz-29/?cors 2023-09-29T22:06:38.935 DEBUG:boto:auth_path=/hcnphz-29/?cors 2023-09-29T22:06:38.935 DEBUG:boto:Method: PUT 2023-09-29T22:06:38.935 DEBUG:boto:Path: /hcnphz-29/?cors 2023-09-29T22:06:38.936 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:06:38.936 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:06:38.936 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:38.936 DEBUG:boto:Port: 8000 2023-09-29T22:06:38.936 DEBUG:boto:Params: {} 2023-09-29T22:06:38.936 DEBUG:boto:Token: None 2023-09-29T22:06:38.936 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:06:38 GMT /hcnphz-29/?cors 2023-09-29T22:06:38.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yRmphwBbqvW5tdrA72+pLwHre+s= 2023-09-29T22:06:38.937 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yRmphwBbqvW5tdrA72+pLwHre+s=', 'Content-Length': '218'} 2023-09-29T22:06:39.957 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f074e9c6556fefa9-0065174a6e-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:39.957 DEBUG:boto:path=/hcnphz-29/ 2023-09-29T22:06:39.957 DEBUG:boto:auth_path=/hcnphz-29/ 2023-09-29T22:06:39.957 DEBUG:boto:path=/hcnphz-29/?cors 2023-09-29T22:06:39.958 DEBUG:boto:auth_path=/hcnphz-29/?cors 2023-09-29T22:06:39.958 DEBUG:boto:Method: GET 2023-09-29T22:06:39.958 DEBUG:boto:Path: /hcnphz-29/?cors 2023-09-29T22:06:39.958 DEBUG:boto:Data: 2023-09-29T22:06:39.958 DEBUG:boto:Headers: {} 2023-09-29T22:06:39.958 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:06:39.958 DEBUG:boto:Port: 8000 2023-09-29T22:06:39.958 DEBUG:boto:Params: {} 2023-09-29T22:06:39.959 DEBUG:boto:Token: None 2023-09-29T22:06:39.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:39 GMT /hcnphz-29/?cors 2023-09-29T22:06:39.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xhTQjUf/but63o0D0JOM3pnR+SE= 2023-09-29T22:06:39.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xhTQjUf/but63o0D0JOM3pnR+SE=', 'Content-Length': '0'} 2023-09-29T22:06:39.960 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000bf0bef5254edfb7-0065174a6f-1103-b1'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:06:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:40.006 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:06:40.007 DEBUG:boto:path=/hcnphz-30/ 2023-09-29T22:06:40.007 DEBUG:boto:auth_path=/hcnphz-30/ 2023-09-29T22:06:40.007 DEBUG:boto:path=/hcnphz-30/?cors 2023-09-29T22:06:40.007 DEBUG:boto:auth_path=/hcnphz-30/?cors 2023-09-29T22:06:40.007 DEBUG:boto:Method: PUT 2023-09-29T22:06:40.008 DEBUG:boto:Path: /hcnphz-30/?cors 2023-09-29T22:06:40.008 DEBUG:boto:Data: DELETEhttps://www.example.com*3000 2023-09-29T22:06:40.008 DEBUG:boto:Headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml'} 2023-09-29T22:06:40.008 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:40.008 DEBUG:boto:Port: 8001 2023-09-29T22:06:40.008 DEBUG:boto:Params: {} 2023-09-29T22:06:40.008 DEBUG:boto:Token: None 2023-09-29T22:06:40.008 DEBUG:boto:StringToSign: PUT c5gEzAIzM7zcw8WVE+AbrQ== text/xml Fri, 29 Sep 2023 22:06:40 GMT /hcnphz-30/?cors 2023-09-29T22:06:40.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:voLtLstlpP++XN4/oAQFMNOEN3k= 2023-09-29T22:06:40.009 DEBUG:boto:Final headers: {'Content-MD5': 'c5gEzAIzM7zcw8WVE+AbrQ==', 'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:voLtLstlpP++XN4/oAQFMNOEN3k=', 'Content-Length': '218'} 2023-09-29T22:06:41.030 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ab825881c6fa970b-0065174a70-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.030 DEBUG:boto:path=/hcnphz-30/ 2023-09-29T22:06:41.030 DEBUG:boto:auth_path=/hcnphz-30/ 2023-09-29T22:06:41.030 DEBUG:boto:path=/hcnphz-30/?cors 2023-09-29T22:06:41.030 DEBUG:boto:auth_path=/hcnphz-30/?cors 2023-09-29T22:06:41.030 DEBUG:boto:Method: GET 2023-09-29T22:06:41.031 DEBUG:boto:Path: /hcnphz-30/?cors 2023-09-29T22:06:41.031 DEBUG:boto:Data: 2023-09-29T22:06:41.031 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.031 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:06:41.031 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.031 DEBUG:boto:Params: {} 2023-09-29T22:06:41.031 DEBUG:boto:Token: None 2023-09-29T22:06:41.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-30/?cors 2023-09-29T22:06:41.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LlWoRB0t4lzR68WsDVz343WopvM= 2023-09-29T22:06:41.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LlWoRB0t4lzR68WsDVz343WopvM=', 'Content-Length': '0'} 2023-09-29T22:06:41.033 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f1a1e62c3198d6b4-0065174a71-1109-b2'), ('Content-Type', 'application/xml'), ('Content-Length', '266'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.074 DEBUG:boto:b'DELETEhttps://www.example.com*3000' 2023-09-29T22:06:41.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2023-09-29T22:06:41.075 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-31 2023-09-29T22:06:41.076 DEBUG:boto:path=/hcnphz-31/ 2023-09-29T22:06:41.076 DEBUG:boto:auth_path=/hcnphz-31/ 2023-09-29T22:06:41.076 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.076 DEBUG:boto:Path: /hcnphz-31/ 2023-09-29T22:06:41.076 DEBUG:boto:Data: 2023-09-29T22:06:41.076 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.076 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.076 DEBUG:boto:Params: {} 2023-09-29T22:06:41.077 DEBUG:boto:Token: None 2023-09-29T22:06:41.077 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-31/ 2023-09-29T22:06:41.077 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lulGmIk8O8UQb3vE5OJt51XRIMA= 2023-09-29T22:06:41.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lulGmIk8O8UQb3vE5OJt51XRIMA=', 'Content-Length': '0'} 2023-09-29T22:06:41.106 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000abc9d3cdefbc60f2-0065174a71-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.106 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-32 2023-09-29T22:06:41.106 DEBUG:boto:path=/hcnphz-32/ 2023-09-29T22:06:41.106 DEBUG:boto:auth_path=/hcnphz-32/ 2023-09-29T22:06:41.106 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.107 DEBUG:boto:Path: /hcnphz-32/ 2023-09-29T22:06:41.107 DEBUG:boto:Data: 2023-09-29T22:06:41.107 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.107 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.107 DEBUG:boto:Params: {} 2023-09-29T22:06:41.107 DEBUG:boto:Token: None 2023-09-29T22:06:41.107 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-32/ 2023-09-29T22:06:41.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MNvFugD/xS3rOqvU6lIyiAfZ5RE= 2023-09-29T22:06:41.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MNvFugD/xS3rOqvU6lIyiAfZ5RE=', 'Content-Length': '0'} 2023-09-29T22:06:41.161 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000096b7b5f513395df5-0065174a71-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.161 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.162 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:41.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:41.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:41.253 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:06:41.278 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:06:41.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.278 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:06:41.278 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:06:41.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.134731_205.1", 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:41.134731Z" 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:41.279 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:06:41.280 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.280 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:06:41.281 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025201.134731_205.1', 2: '', 3: ''} 2023-09-29T22:06:41.281 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:06:41.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.282 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:41.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:41.323 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:41.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:41.377+0000 7f097cfd4f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:06:41.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:41.381+0000 7f097cfd4f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:06:41.403 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:41.403 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:41.405 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.134731_205.1", 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.134731Z", 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:41.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:41.408 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:41.409 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:06:41.410 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:06:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.411 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:41Z" 2023-09-29T22:06:41.411 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:41.412 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.134731_205.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.134731Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:06:41Z" } 2023-09-29T22:06:41.412 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.134731_205.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.134731Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:06:41.412 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.134731_205.1', 2: '', 3: ''} 2023-09-29T22:06:41.412 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.134731_205.1', 2: '', 3: ''} 2023-09-29T22:06:41.412 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:06:41.413 DEBUG:boto:path=/hcnphz-31/ 2023-09-29T22:06:41.413 DEBUG:boto:auth_path=/hcnphz-31/ 2023-09-29T22:06:41.413 DEBUG:boto:Method: HEAD 2023-09-29T22:06:41.413 DEBUG:boto:Path: /hcnphz-31/ 2023-09-29T22:06:41.413 DEBUG:boto:Data: 2023-09-29T22:06:41.413 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.413 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.413 DEBUG:boto:Params: {} 2023-09-29T22:06:41.414 DEBUG:boto:Token: None 2023-09-29T22:06:41.414 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-31/ 2023-09-29T22:06:41.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Relf3UceZgbxf0G7sPyyv343BZU= 2023-09-29T22:06:41.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Relf3UceZgbxf0G7sPyyv343BZU=', 'Content-Length': '0'} 2023-09-29T22:06:41.417 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc7d8177758b05b2-0065174a71-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.418 DEBUG:boto:path=/hcnphz-31/foo 2023-09-29T22:06:41.418 DEBUG:boto:auth_path=/hcnphz-31/foo 2023-09-29T22:06:41.418 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.418 DEBUG:boto:Path: /hcnphz-31/foo 2023-09-29T22:06:41.418 DEBUG:boto:Data: 2023-09-29T22:06:41.418 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:06:41.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.419 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.419 DEBUG:boto:Params: {} 2023-09-29T22:06:41.419 DEBUG:boto:Token: None 2023-09-29T22:06:41.419 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-31/foo 2023-09-29T22:06:41.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9d3LW+EBIjkcFghN26d137Udtwk= 2023-09-29T22:06:41.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9d3LW+EBIjkcFghN26d137Udtwk='} 2023-09-29T22:06:41.432 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c60a1d04ebc94e12-0065174a71-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.432 DEBUG:boto:path=/hcnphz-31/ 2023-09-29T22:06:41.432 DEBUG:boto:auth_path=/hcnphz-31/ 2023-09-29T22:06:41.432 DEBUG:boto:Method: DELETE 2023-09-29T22:06:41.433 DEBUG:boto:Path: /hcnphz-31/ 2023-09-29T22:06:41.433 DEBUG:boto:Data: 2023-09-29T22:06:41.433 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.433 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.433 DEBUG:boto:Params: {} 2023-09-29T22:06:41.433 DEBUG:boto:Token: None 2023-09-29T22:06:41.433 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-31/ 2023-09-29T22:06:41.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HlqJZhZsZ37w1mFnamIaxCGXZRE= 2023-09-29T22:06:41.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HlqJZhZsZ37w1mFnamIaxCGXZRE=', 'Content-Length': '0'} 2023-09-29T22:06:41.447 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000f687b61e35700f54-0065174a71-1122-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.491 DEBUG:boto:path=/hcnphz-32/ 2023-09-29T22:06:41.491 DEBUG:boto:auth_path=/hcnphz-32/ 2023-09-29T22:06:41.491 DEBUG:boto:Method: HEAD 2023-09-29T22:06:41.491 DEBUG:boto:Path: /hcnphz-32/ 2023-09-29T22:06:41.491 DEBUG:boto:Data: 2023-09-29T22:06:41.491 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.491 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.492 DEBUG:boto:Params: {} 2023-09-29T22:06:41.492 DEBUG:boto:Token: None 2023-09-29T22:06:41.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-32/ 2023-09-29T22:06:41.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SIvGkweLRSAqglJFrxpUmGBAwb0= 2023-09-29T22:06:41.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SIvGkweLRSAqglJFrxpUmGBAwb0=', 'Content-Length': '0'} 2023-09-29T22:06:41.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001ae2406d0c5ec3bb-0065174a71-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.496 DEBUG:boto:path=/hcnphz-32/foo 2023-09-29T22:06:41.496 DEBUG:boto:auth_path=/hcnphz-32/foo 2023-09-29T22:06:41.496 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.496 DEBUG:boto:Path: /hcnphz-32/foo 2023-09-29T22:06:41.496 DEBUG:boto:Data: 2023-09-29T22:06:41.496 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:06:41.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.497 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.497 DEBUG:boto:Params: {} 2023-09-29T22:06:41.497 DEBUG:boto:Token: None 2023-09-29T22:06:41.497 DEBUG:boto:StringToSign: PUT N7UdGUp1E+RbVvZSTy1R8g== application/octet-stream Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-32/foo 2023-09-29T22:06:41.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VfvnswquIkTensWUy5zjTp3uhk4= 2023-09-29T22:06:41.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VfvnswquIkTensWUy5zjTp3uhk4='} 2023-09-29T22:06:41.507 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"37b51d194a7513e45b56f6524f2d51f2"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e2dc419c4946e88-0065174a71-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.507 DEBUG:boto:path=/hcnphz-32/ 2023-09-29T22:06:41.507 DEBUG:boto:auth_path=/hcnphz-32/ 2023-09-29T22:06:41.507 DEBUG:boto:Method: DELETE 2023-09-29T22:06:41.507 DEBUG:boto:Path: /hcnphz-32/ 2023-09-29T22:06:41.508 DEBUG:boto:Data: 2023-09-29T22:06:41.508 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.508 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.508 DEBUG:boto:Params: {} 2023-09-29T22:06:41.508 DEBUG:boto:Token: None 2023-09-29T22:06:41.508 DEBUG:boto:StringToSign: DELETE Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-32/ 2023-09-29T22:06:41.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ujmjtcjKXNdTyTw0ZySa6qbsaHs= 2023-09-29T22:06:41.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ujmjtcjKXNdTyTw0ZySa6qbsaHs=', 'Content-Length': '0'} 2023-09-29T22:06:41.521 DEBUG:boto:Response headers: [('Content-Length', '224'), ('x-amz-request-id', 'tx00000524cff84b10404d0-0065174a71-1126-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.522 DEBUG:boto:path=/hcnphz-31/ 2023-09-29T22:06:41.522 DEBUG:boto:auth_path=/hcnphz-31/ 2023-09-29T22:06:41.522 DEBUG:boto:Method: HEAD 2023-09-29T22:06:41.522 DEBUG:boto:Path: /hcnphz-31/ 2023-09-29T22:06:41.522 DEBUG:boto:Data: 2023-09-29T22:06:41.522 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.523 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.523 DEBUG:boto:Params: {} 2023-09-29T22:06:41.523 DEBUG:boto:Token: None 2023-09-29T22:06:41.523 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-31/ 2023-09-29T22:06:41.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Relf3UceZgbxf0G7sPyyv343BZU= 2023-09-29T22:06:41.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Relf3UceZgbxf0G7sPyyv343BZU=', 'Content-Length': '0'} 2023-09-29T22:06:41.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e81dfd197132f619-0065174a71-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.526 DEBUG:boto:path=/hcnphz-32/ 2023-09-29T22:06:41.527 DEBUG:boto:auth_path=/hcnphz-32/ 2023-09-29T22:06:41.527 DEBUG:boto:Method: HEAD 2023-09-29T22:06:41.527 DEBUG:boto:Path: /hcnphz-32/ 2023-09-29T22:06:41.527 DEBUG:boto:Data: 2023-09-29T22:06:41.527 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.527 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.527 DEBUG:boto:Params: {} 2023-09-29T22:06:41.527 DEBUG:boto:Token: None 2023-09-29T22:06:41.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-32/ 2023-09-29T22:06:41.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SIvGkweLRSAqglJFrxpUmGBAwb0= 2023-09-29T22:06:41.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SIvGkweLRSAqglJFrxpUmGBAwb0=', 'Content-Length': '0'} 2023-09-29T22:06:41.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000efc7ccbc8d896c13-0065174a71-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2023-09-29T22:06:41.532 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... SKIP: test_multi_period_incremental_sync skipped. Requires 3 or more zones in master zonegroup. 2023-09-29T22:06:41.533 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-33 2023-09-29T22:06:41.533 DEBUG:boto:path=/hcnphz-33/ 2023-09-29T22:06:41.533 DEBUG:boto:auth_path=/hcnphz-33/ 2023-09-29T22:06:41.533 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.533 DEBUG:boto:Path: /hcnphz-33/ 2023-09-29T22:06:41.533 DEBUG:boto:Data: 2023-09-29T22:06:41.533 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.533 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.534 DEBUG:boto:Params: {} 2023-09-29T22:06:41.534 DEBUG:boto:Token: None 2023-09-29T22:06:41.534 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-33/ 2023-09-29T22:06:41.534 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aquQyNBRx3T5A77d+HlVsFXjXJk= 2023-09-29T22:06:41.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aquQyNBRx3T5A77d+HlVsFXjXJk=', 'Content-Length': '0'} 2023-09-29T22:06:41.560 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d3d10252ae9e4d1a-0065174a71-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.560 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-34 2023-09-29T22:06:41.560 DEBUG:boto:path=/hcnphz-34/ 2023-09-29T22:06:41.561 DEBUG:boto:auth_path=/hcnphz-34/ 2023-09-29T22:06:41.561 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.561 DEBUG:boto:Path: /hcnphz-34/ 2023-09-29T22:06:41.561 DEBUG:boto:Data: 2023-09-29T22:06:41.561 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.561 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.561 DEBUG:boto:Params: {} 2023-09-29T22:06:41.561 DEBUG:boto:Token: None 2023-09-29T22:06:41.562 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-34/ 2023-09-29T22:06:41.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vs1/e3uOjRI5QXZW3IuzJ7YV6QA= 2023-09-29T22:06:41.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vs1/e3uOjRI5QXZW3IuzJ7YV6QA=', 'Content-Length': '0'} 2023-09-29T22:06:41.602 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f2e938328f2fe165-0065174a71-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.602 DEBUG:boto:path=/hcnphz-33/ 2023-09-29T22:06:41.602 DEBUG:boto:auth_path=/hcnphz-33/ 2023-09-29T22:06:41.602 DEBUG:boto:Method: HEAD 2023-09-29T22:06:41.602 DEBUG:boto:Path: /hcnphz-33/ 2023-09-29T22:06:41.602 DEBUG:boto:Data: 2023-09-29T22:06:41.602 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.603 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.603 DEBUG:boto:Params: {} 2023-09-29T22:06:41.603 DEBUG:boto:Token: None 2023-09-29T22:06:41.603 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-33/ 2023-09-29T22:06:41.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xm2UIiNUczyuxWbov/OAgvTPbJk= 2023-09-29T22:06:41.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xm2UIiNUczyuxWbov/OAgvTPbJk=', 'Content-Length': '0'} 2023-09-29T22:06:41.606 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000113c2a46c506e0da-0065174a71-1122-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.606 DEBUG:boto:path=/hcnphz-33/key 2023-09-29T22:06:41.607 DEBUG:boto:auth_path=/hcnphz-33/key 2023-09-29T22:06:41.607 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.607 DEBUG:boto:Path: /hcnphz-33/key 2023-09-29T22:06:41.607 DEBUG:boto:Data: 2023-09-29T22:06:41.607 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-29T22:06:41.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:06:41.607 DEBUG:boto:Port: 8000 2023-09-29T22:06:41.607 DEBUG:boto:Params: {} 2023-09-29T22:06:41.607 DEBUG:boto:Token: None 2023-09-29T22:06:41.608 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-33/key 2023-09-29T22:06:41.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nnhAM4NK081mSWx5JL5HOh8u2b0= 2023-09-29T22:06:41.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nnhAM4NK081mSWx5JL5HOh8u2b0='} 2023-09-29T22:06:41.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f39c2638d2b6037e-0065174a71-1122-a1'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.651 DEBUG:boto:path=/hcnphz-34/ 2023-09-29T22:06:41.651 DEBUG:boto:auth_path=/hcnphz-34/ 2023-09-29T22:06:41.651 DEBUG:boto:Method: HEAD 2023-09-29T22:06:41.651 DEBUG:boto:Path: /hcnphz-34/ 2023-09-29T22:06:41.651 DEBUG:boto:Data: 2023-09-29T22:06:41.651 DEBUG:boto:Headers: {} 2023-09-29T22:06:41.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.652 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.652 DEBUG:boto:Params: {} 2023-09-29T22:06:41.652 DEBUG:boto:Token: None 2023-09-29T22:06:41.652 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-34/ 2023-09-29T22:06:41.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+tAqNgO2jnw5vwxr2eOegd49XSA= 2023-09-29T22:06:41.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+tAqNgO2jnw5vwxr2eOegd49XSA=', 'Content-Length': '0'} 2023-09-29T22:06:41.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059a4c203b8e983b4-0065174a71-1126-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.656 DEBUG:boto:path=/hcnphz-34/key 2023-09-29T22:06:41.657 DEBUG:boto:auth_path=/hcnphz-34/key 2023-09-29T22:06:41.657 DEBUG:boto:Method: PUT 2023-09-29T22:06:41.657 DEBUG:boto:Path: /hcnphz-34/key 2023-09-29T22:06:41.657 DEBUG:boto:Data: 2023-09-29T22:06:41.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue'} 2023-09-29T22:06:41.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:06:41.657 DEBUG:boto:Port: 8001 2023-09-29T22:06:41.657 DEBUG:boto:Params: {} 2023-09-29T22:06:41.658 DEBUG:boto:Token: None 2023-09-29T22:06:41.658 DEBUG:boto:StringToSign: PUT hBotaJrYa9FhFEdFPCLG/A== application/octet-stream Fri, 29 Sep 2023 22:06:41 GMT /hcnphz-34/key 2023-09-29T22:06:41.658 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MUGPAEux2So9QgHHN/gdEi+XOTM= 2023-09-29T22:06:41.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'hBotaJrYa9FhFEdFPCLG/A==', 'Content-Length': '4', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:06:41 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MUGPAEux2So9QgHHN/gdEi+XOTM='} 2023-09-29T22:06:41.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"841a2d689ad86bd1611447453c22c6fc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007ed035ff33d324ad-0065174a71-1126-a2'), ('Date', 'Fri, 29 Sep 2023 22:06:41 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:06:41.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.707 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:41.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:41.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:41.802 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:06:41.829 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:41.588363Z" 2023-09-29T22:06:41.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:41.831 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:06:41.832 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:06:41.833 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:06:41.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:41.833 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:41.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:41.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:41.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:41.977+0000 7f64bbdf5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:06:41.977 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:06:41.977+0000 7f64bbdf5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:06:42.002 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:42.002 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.003 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:42.004 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.005 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.006 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:06:42.007 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.008 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:06:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:06:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.009 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:41Z" 2023-09-29T22:06:42.009 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:42.010 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:06:41Z" } 2023-09-29T22:06:42.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:06:42.010 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:06:42.010 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:06:42.011 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:06:42.011 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:06:42.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:42.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:42.011 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:42.059 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:42.059 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:42.145 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:06:42.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:41.618816Z" 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:41.432673Z" 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.147 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:06:42.148 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:42.148 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:06:42.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:42.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:42.149 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:42.195 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:42.284 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:42.284 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:42.284 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:42.284 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:42.285 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:11.920565Z" 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.286 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:37.608065Z" 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:06:42.289 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:42Z" 2023-09-29T22:06:42.290 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:42.291 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:11.920565Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:30.832179Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:57:37.608065Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:06:42Z" } 2023-09-29T22:06:42.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:11.920565Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:30.832179Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:57:37.608065Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:06:42.292 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:42.292 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003324', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:06:42.292 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003465 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-09-29T22:06:47.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:47.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:47.293 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:47.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:47.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:47.418 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:47.418 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:47.418 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:47.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003324", 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:11.920565Z" 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:47.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:47.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:47.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000929", 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:37.608065Z" 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:47.423 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:47.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:47Z" 2023-09-29T22:06:47.425 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:47.426 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003324", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:11.920565Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:30.832179Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000929", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T21:57:37.608065Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:06:47Z" } 2023-09-29T22:06:47.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003324', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:11.920565Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:30.832179Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000929', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T21:57:37.608065Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:06:47.427 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:47.427 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003324', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000000929', 3: ''} 2023-09-29T22:06:47.427 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000003465 target=00000000000000000000:00000000000000003324, shard=2 master=00000000000000000000:00000000000000001068 target=00000000000000000000:00000000000000000929 2023-09-29T22:06:52.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:52.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:52.429 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:52.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:52.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:52.561 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:52.561 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:52.561 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003465", 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.618816Z" 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.563 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:52.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.432673Z" 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.565 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.566 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:52Z" 2023-09-29T22:06:52.567 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:52.568 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003465", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.618816Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002482", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:30.832179Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.432673Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:06:52Z" } 2023-09-29T22:06:52.568 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003465', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.618816Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002482', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:30.832179Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.432673Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:06:52.568 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:52.568 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003465', 1: '00000000000000000000:00000000000000002482', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:52.569 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:06:52.569 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:06:52.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:52.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:52.569 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:52.614 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:52.704 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:06:52.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.704 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004734", 2023-09-29T22:06:52.704 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:51.924166Z" 2023-09-29T22:06:52.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:50.915176Z" 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.706 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:52.706 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:06:52.706 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.706 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:06:52.707 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:06:52.707 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:06:52.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:52.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:52.707 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:52.755 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:52.755 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:06:52.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.840 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.671507Z" 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.841 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.842 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.508201Z" 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:52.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:52.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:52Z" 2023-09-29T22:06:52.845 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:52.846 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.671507Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:10.866065Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.508201Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:06:52Z" } 2023-09-29T22:06:52.846 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.671507Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:10.866065Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.508201Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:06:52.846 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:06:52.847 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:52.847 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004734 target=00000000000000000000:00000000000000004593, shard=2 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-09-29T22:06:57.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:57.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:06:57.848 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:06:57.897 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:06:57.898 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:06:57.981 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:06:57.981 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:06:57.981 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:06:57.981 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:57.982 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.671507Z" 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:06:57.983 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:57.984 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.508201Z" 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:57.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:57.986 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:06:57Z" 2023-09-29T22:06:57.987 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:06:57.988 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.671507Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:10.866065Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.508201Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:06:57Z" } 2023-09-29T22:06:57.988 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.671507Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:10.866065Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.508201Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:06:57.989 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:06:57.989 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:06:57.989 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004734 target=00000000000000000000:00000000000000004593, shard=2 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-09-29T22:07:02.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:02.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:02.990 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:03.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:03.039 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:03.120 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:07:03.120 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:03.121 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.671507Z" 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:03.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:03.123 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:03.124 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.508201Z" 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:03.125 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:07:03.126 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:07:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:03.127 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:07:03Z" 2023-09-29T22:07:03.127 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:07:03.128 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.671507Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:10.866065Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.508201Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:07:03Z" } 2023-09-29T22:07:03.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.671507Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:10.866065Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.508201Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:07:03.128 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:07:03.128 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:07:03.128 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004734 target=00000000000000000000:00000000000000004593, shard=2 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-09-29T22:07:08.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:08.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:08.129 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:08.177 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:08.177 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:08.257 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:08.258 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.671507Z" 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:08.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001068", 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:08.260 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.508201Z" 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:07:08.261 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:07:08.262 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:07:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:08.263 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:07:08Z" 2023-09-29T22:07:08.263 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:07:08.264 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.671507Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:10.866065Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001068", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.508201Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:07:08Z" } 2023-09-29T22:07:08.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.671507Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:10.866065Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001068', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.508201Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:07:08.264 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:07:08.264 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004593', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001068', 3: ''} 2023-09-29T22:07:08.264 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=0 master=00000000000000000000:00000000000000004734 target=00000000000000000000:00000000000000004593, shard=2 master=00000000000000000000:00000000000000001210 target=00000000000000000000:00000000000000001068 2023-09-29T22:07:13.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.266 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:13.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:13.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:13.396 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004734", 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:51.924166Z" 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:13.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001918", 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:01:10.866065Z" 2023-09-29T22:07:13.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:13.400 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:50.915176Z" 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:07:13.401 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:07:13.402 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:07:13.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.403 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:07:13Z" 2023-09-29T22:07:13.403 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:07:13.404 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004734", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:51.924166Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:01:10.866065Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:50.915176Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:07:13Z" } 2023-09-29T22:07:13.404 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004734', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:51.924166Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:01:10.866065Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:50.915176Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:07:13.404 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:07:13.404 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000001918', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:07:13.404 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:07:13.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.405 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:13.444 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:13.445 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:13.537 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:51.973838Z" 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:07:13.538 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:50.894391Z" 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:13.539 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:07:13.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:13.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:13.541 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:07:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:13.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:13.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.684 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:13.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:13.729 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:13.814 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:07:13.817 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.817 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:13.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:13.864 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:13.946 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:07:13.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:13.950 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:13.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:13.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.073 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:07:14.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.076 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:14.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.116 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.204 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:07:14.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:14.204 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-09-29T22:07:14.204 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:51.973838Z" 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002482", 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:01:30.832179Z" 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:50.894391Z" 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:14.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:14.206 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:07:14.206 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:07:14.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:14.206 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:07:14.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:14.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:14.207 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:07:14.247 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.248 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.347 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '0', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.347 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 0 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:14.392 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.473 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:07:14.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '1', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.476 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 1 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.521 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.605 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:07:14.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--shard-id', '2', '--max-entries', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:07:14.608 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --shard-id 2 --max-entries 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:07:14.652 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.652 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.736 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:07:14.739 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2023-09-29T22:07:14.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:14.741 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:14.741 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=false --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:07:14.783 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.783 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:14.825 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:07:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:07:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-09-29T22:07:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T22:07:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:07:14.828 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-09-29T22:07:14.829 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-09-29T22:07:14.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:07:14.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:14.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:07:14.832 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:07:14.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:07:14.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:07:15.011 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T22:07:15.063 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:07:15.063 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:07:15.063 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5, 2023-09-29T22:07:15.063 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:07:15.063 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:07:15.064 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:07:15.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:07:15.066 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": false, 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:07:15.067 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:07:15.068 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:07:15.069 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:07:15.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.071 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:07:15.073 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.074 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:07:15.074 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:07:15.074 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:07:15.074 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:07:15.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.074 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:07:15.075 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:07:15.076 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:07:15.076 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:07:15.076 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:07:15.076 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:07:15.076 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:07:15.084 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:07:15.085 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:07:15.086 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:07:15.086 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:07:15.086 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:07:15.086 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:07:15.088 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:07:15.088 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.088 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.088 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.088 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:07:15.088 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:07:15.089 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:07:15.090 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.092 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.093 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:07:15.094 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:07:15.095 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:07:15.096 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:07:15.097 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:07:15.099 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to false 2023-09-29T22:08:15.099 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-35 2023-09-29T22:08:15.100 DEBUG:boto:path=/hcnphz-35/ 2023-09-29T22:08:15.100 DEBUG:boto:auth_path=/hcnphz-35/ 2023-09-29T22:08:15.100 DEBUG:boto:Method: PUT 2023-09-29T22:08:15.100 DEBUG:boto:Path: /hcnphz-35/ 2023-09-29T22:08:15.100 DEBUG:boto:Data: 2023-09-29T22:08:15.100 DEBUG:boto:Headers: {} 2023-09-29T22:08:15.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:08:15.101 DEBUG:boto:Port: 8000 2023-09-29T22:08:15.101 DEBUG:boto:Params: {} 2023-09-29T22:08:15.101 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:08:15.101 DEBUG:boto:Token: None 2023-09-29T22:08:15.102 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:08:15 GMT /hcnphz-35/ 2023-09-29T22:08:15.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GvPZ3p+rw5swAcf42UTSqLM6w+w= 2023-09-29T22:08:15.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:08:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GvPZ3p+rw5swAcf42UTSqLM6w+w=', 'Content-Length': '0'} 2023-09-29T22:08:15.139 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d9db6a0c40d9ac1-0065174acf-14f8-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:08:15.139 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:08:15.139 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:08:15.140 DEBUG:boto:Method: PUT 2023-09-29T22:08:15.140 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:08:15.140 DEBUG:boto:Data: 2023-09-29T22:08:15.140 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:08:15.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:08:15.140 DEBUG:boto:Port: 8000 2023-09-29T22:08:15.140 DEBUG:boto:Params: {} 2023-09-29T22:08:15.140 DEBUG:boto:Token: None 2023-09-29T22:08:15.141 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 29 Sep 2023 22:08:15 GMT /hcnphz-35/testredirect 2023-09-29T22:08:15.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mtF9fCIp5VLZzmLXOxPW3MSOUpg= 2023-09-29T22:08:15.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:08:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mtF9fCIp5VLZzmLXOxPW3MSOUpg='} 2023-09-29T22:08:15.155 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae987fc40178faef-0065174acf-14f8-a1'), ('Date', 'Fri, 29 Sep 2023 22:08:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:08:15.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:15.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:15.156 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:15.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:15.199 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:15.259 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:08:15.137960Z" 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:08:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:15.288 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:08:15.289 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:15.289 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:08:15.289 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:15.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:15.289 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:15.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:15.393 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:15.390+0000 7f66abb9af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:15.395 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:15.394+0000 7f66abb9af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:15.419 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:15.424 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.425 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:15.426 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:15.426 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:15.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:15.426 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:15Z" 2023-09-29T22:08:15.426 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:15.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:15Z" } 2023-09-29T22:08:15.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:15.427 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:15.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:15.428 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:20.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:20.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:20.429 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:20.476 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:20.476 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:20.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:20.526+0000 7f9b7fcf7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:20.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:20.526+0000 7f9b7fcf7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:20.553 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:20.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:20.555 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:20.556 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:20.557 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:20.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:20Z" 2023-09-29T22:08:20.561 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:20.562 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:20Z" } 2023-09-29T22:08:20.562 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:20.562 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:20.562 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:20.563 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:25.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:25.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:25.564 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:25.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:25.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:25.667 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:25.666+0000 7fd530228f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:25.668 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:25.666+0000 7fd530228f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:25.693 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:25.693 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:25.693 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:25.693 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:25.693 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:25.693 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:25.694 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:25.695 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:25.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:25.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:25Z" 2023-09-29T22:08:25.700 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:25.701 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:25Z" } 2023-09-29T22:08:25.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:25.701 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:25.701 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:25.701 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:30.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:30.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:30.703 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:30.749 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:30.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:30.802+0000 7f6a53668f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:30.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:30.802+0000 7f6a53668f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:30.832 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:30.832 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:30.832 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:30.832 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:30.833 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:30.833 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:30.833 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:30.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:30.833 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:30.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:30.834 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:30.834 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:30.834 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:30.834 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:30.834 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:30.835 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:30.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:30.835 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:30.835 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:30.835 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:30.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:30.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:30.836 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:30.836 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:30.836 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:30.836 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:30.837 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:30.837 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:30.837 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:30.837 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:30.837 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:30.837 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:30.838 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:30.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:30.838 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:30.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:30.838 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:30.839 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:30.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:30.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:30.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:30.840 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:30.840 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:30.841 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:30.841 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:30.841 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:30.841 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:30.841 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:30.841 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:30.842 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:30.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:30.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:30.842 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:30.843 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:30Z" 2023-09-29T22:08:30.844 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:30.845 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:30Z" } 2023-09-29T22:08:30.845 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:30.845 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:30.846 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:30.846 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:35.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:35.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:35.847 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:35.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:35.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:35.950+0000 7f2288233f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:35.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:35.954+0000 7f2288233f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:35.981 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:35.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:35.983 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:35.984 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:35.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:35.986 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:35.987 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:35.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:35.989 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:35Z" 2023-09-29T22:08:35.989 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:35.990 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:35Z" } 2023-09-29T22:08:35.990 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:35.990 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:35.990 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:35.990 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:40.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:40.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:40.991 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:41.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:41.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:41.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:41.090+0000 7ff68e34cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:41.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:41.094+0000 7ff68e34cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:41.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:41.117 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:41.118 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:41.119 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:41.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:41.121 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:41.122 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:41Z" 2023-09-29T22:08:41.123 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:41.123 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:41Z" } 2023-09-29T22:08:41.123 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:41.124 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:41.124 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:41.124 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:46.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:46.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:46.125 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:46.170 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:46.171 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:46.227 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:46.226+0000 7fbe7cb2df40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:46.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:46.226+0000 7fbe7cb2df40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:46.254 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:46.254 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:46.254 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:46.254 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:46.254 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:46.254 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:46.255 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:46.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:46.255 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:46.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:46.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:46.257 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:46.257 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:46.257 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:46.257 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:46.257 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:46.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:46.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:46.260 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:46.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:46.262 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:46Z" 2023-09-29T22:08:46.263 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:46.264 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:46Z" } 2023-09-29T22:08:46.264 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:46.265 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:46.265 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:46.265 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:51.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:51.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:51.267 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:51.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:51.313 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:51.368 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:51.365+0000 7fbaa425bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:51.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:51.365+0000 7fbaa425bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:51.393 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:51.393 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:51.393 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:51.393 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:51.393 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:51.394 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:51.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:51.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:51.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:51.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:51.399 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:51Z" 2023-09-29T22:08:51.400 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:51.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:51Z" } 2023-09-29T22:08:51.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:51.401 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:51.401 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:51.401 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:08:56.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:56.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:08:56.403 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:08:56.448 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:08:56.449 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:08:56.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:56.501+0000 7fc508f6ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:56.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:08:56.505+0000 7fc508f6ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:08:56.533 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:08:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:08:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:08:56.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:08:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:08:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:08:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:08:56.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:08:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:08:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:56.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:56.536 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:08:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:08:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:56.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:08:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:56.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:56.539 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:08:56.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:56.541 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:08:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:08:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:08:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:08:56.542 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:08:56Z" 2023-09-29T22:08:56.542 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:08:56.543 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:08:56Z" } 2023-09-29T22:08:56.543 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:08:56.543 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:08:56.544 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:08:56.544 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:09:01.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:01.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:01.545 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:01.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:01.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:01.646 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:01.641+0000 7f26f3e6af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:01.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:01.645+0000 7f26f3e6af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:01.673 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:09:01.673 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:09:01.673 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:09:01.673 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:09:01.673 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:09:01.673 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:09:01.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:01.675 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:01.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:01.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:01.678 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:09:01Z" 2023-09-29T22:09:01.679 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:09:01.680 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:09:01Z" } 2023-09-29T22:09:01.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:09:01.681 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:09:01.681 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:09:01.681 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:09:06.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:06.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:06.682 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:06.731 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:06.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:06.785+0000 7f670c123f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:06.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:06.785+0000 7f670c123f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:06.810 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:06.811 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:06.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:09:06.813 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:06.814 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:06.815 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:06.816 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:09:06Z" 2023-09-29T22:09:06.817 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:09:06.818 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:09:06Z" } 2023-09-29T22:09:06.818 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:09:06.819 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:09:06.819 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:09:06.819 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:09:11.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:11.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:11.820 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:11.866 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:11.866 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:11.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:11.925+0000 7f1177bbdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:11.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:11.925+0000 7f1177bbdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:11.952 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:11.953 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:11.954 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025201.588363_213.1", 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:11.955 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:41.588363Z", 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:11.956 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:09:11.957 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:11.958 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:09:11Z" 2023-09-29T22:09:11.959 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:09:11.960 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025201.588363_213.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:41.588363Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:09:11Z" } 2023-09-29T22:09:11.960 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025201.588363_213.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:41.588363Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:09:11.961 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:09:11.961 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025201.588363_213.1', 2: '', 3: ''} 2023-09-29T22:09:11.961 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025295.137960_229.1 target=1_1696025201.588363_213.1 2023-09-29T22:09:16.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:16.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:09:16.963 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:09:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:17.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:17.061+0000 7f8965ccdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:17.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:09:17.061+0000 7f8965ccdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:09:17.082 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:09:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:09:17.082 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:09:17.083 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:09:17.084 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:17.085 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:17.086 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:09:17.088 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:09:17Z" 2023-09-29T22:09:17.089 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:09:17.090 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:09:17Z" } 2023-09-29T22:09:17.091 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:09:17.091 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:09:17.091 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:09:17.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:09:17.092 DEBUG:boto:path=/hcnphz-35/ 2023-09-29T22:09:17.092 DEBUG:boto:auth_path=/hcnphz-35/ 2023-09-29T22:09:17.092 DEBUG:boto:Method: HEAD 2023-09-29T22:09:17.092 DEBUG:boto:Path: /hcnphz-35/ 2023-09-29T22:09:17.092 DEBUG:boto:Data: 2023-09-29T22:09:17.092 DEBUG:boto:Headers: {} 2023-09-29T22:09:17.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:09:17.092 DEBUG:boto:Port: 8001 2023-09-29T22:09:17.092 DEBUG:boto:Params: {} 2023-09-29T22:09:17.093 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:09:17.093 DEBUG:boto:Token: None 2023-09-29T22:09:17.093 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:09:17 GMT /hcnphz-35/ 2023-09-29T22:09:17.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aFfvRrlIXp4shN3o2vFpobGZLsM= 2023-09-29T22:09:17.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:09:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aFfvRrlIXp4shN3o2vFpobGZLsM=', 'Content-Length': '0'} 2023-09-29T22:09:17.102 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6c35e53cdc20f88-0065174b0d-14fc-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:09:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:09:17.102 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:09:17.103 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:09:17.103 DEBUG:boto:Method: HEAD 2023-09-29T22:09:17.103 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:09:17.103 DEBUG:boto:Data: 2023-09-29T22:09:17.103 DEBUG:boto:Headers: {} 2023-09-29T22:09:17.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:09:17.103 DEBUG:boto:Port: 8001 2023-09-29T22:09:17.103 DEBUG:boto:Params: {} 2023-09-29T22:09:17.103 DEBUG:boto:Token: None 2023-09-29T22:09:17.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:09:17 GMT /hcnphz-35/testredirect 2023-09-29T22:09:17.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GcF8N6Yle3ueWp2OsSWg5Xhrs2s= 2023-09-29T22:09:17.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:09:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GcF8N6Yle3ueWp2OsSWg5Xhrs2s=', 'Content-Length': '0'} 2023-09-29T22:09:17.106 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000007d4ca6af7175c5ff-0065174b0d-14fc-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:09:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:09:17.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:17.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:17.107 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:17.151 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:17.152 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:17.191 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:09:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:09:17.191 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-09-29T22:09:17.192 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T22:09:17.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-09-29T22:09:17.194 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.195 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:09:17.196 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-09-29T22:09:17.196 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:09:17.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:17.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:09:17.197 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:09:17.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:09:17.240 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:09:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T22:09:17.385 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:09:17.385 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6, 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:09:17.386 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:09:17.387 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:17.388 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": false, 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:17.390 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:09:17.394 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:09:17.395 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:09:17.396 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:09:17.407 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:09:17.408 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.409 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:09:17.410 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.411 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:09:17.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:09:17.413 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:09:17.414 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:09:17.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.416 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:09:17.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:09:17.418 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:09:17.419 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T22:10:17.420 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.421 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.421 DEBUG:boto:Method: HEAD 2023-09-29T22:10:17.421 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.421 DEBUG:boto:Data: 2023-09-29T22:10:17.421 DEBUG:boto:Headers: {} 2023-09-29T22:10:17.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:10:17.421 DEBUG:boto:Port: 8001 2023-09-29T22:10:17.421 DEBUG:boto:Params: {} 2023-09-29T22:10:17.422 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:10:17.422 DEBUG:boto:Token: None 2023-09-29T22:10:17.422 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UysyGWg+A6PRjxvBNt1AmEiyc0s= 2023-09-29T22:10:17.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UysyGWg+A6PRjxvBNt1AmEiyc0s=', 'Content-Length': '0'} 2023-09-29T22:10:17.431 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/hcnphz-35/testredirect?'), ('x-amz-request-id', 'tx00000dba4bd20ca6a0053-0065174b49-1580-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.432 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/hcnphz-35/testredirect 2023-09-29T22:10:17.432 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': '8000'} 2023-09-29T22:10:17.432 DEBUG:boto:Token: None 2023-09-29T22:10:17.432 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UysyGWg+A6PRjxvBNt1AmEiyc0s= 2023-09-29T22:10:17.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UysyGWg+A6PRjxvBNt1AmEiyc0s=', 'Content-Length': '0'} 2023-09-29T22:10:17.440 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:08:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000e45f0173a1384245-0065174b49-1584-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.440 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.441 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.441 DEBUG:boto:Method: GET 2023-09-29T22:10:17.441 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.441 DEBUG:boto:Data: 2023-09-29T22:10:17.441 DEBUG:boto:Headers: {} 2023-09-29T22:10:17.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:10:17.441 DEBUG:boto:Port: 8001 2023-09-29T22:10:17.441 DEBUG:boto:Params: {} 2023-09-29T22:10:17.441 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:10:17.442 DEBUG:boto:Token: None 2023-09-29T22:10:17.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.445 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/hcnphz-35/testredirect?'), ('x-amz-request-id', 'tx000003c0c3b511de49af3-0065174b49-1580-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.446 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/hcnphz-35/testredirect 2023-09-29T22:10:17.446 DEBUG:boto:Token: None 2023-09-29T22:10:17.446 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.446 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.449 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:08:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx00000ff65b8c7e0f999eb-0065174b49-1584-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.450 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.450 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.450 DEBUG:boto:Method: PUT 2023-09-29T22:10:17.450 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.450 DEBUG:boto:Data: 2023-09-29T22:10:17.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:10:17.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:10:17.450 DEBUG:boto:Port: 8000 2023-09-29T22:10:17.451 DEBUG:boto:Params: {} 2023-09-29T22:10:17.451 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:10:17.451 DEBUG:boto:Token: None 2023-09-29T22:10:17.451 DEBUG:boto:StringToSign: PUT VpBzljOcorCZvRIkX5Nt3A== application/octet-stream Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.451 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YUAnYgFBtx56mh8Jp9APWDf34PI= 2023-09-29T22:10:17.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'VpBzljOcorCZvRIkX5Nt3A==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YUAnYgFBtx56mh8Jp9APWDf34PI='} 2023-09-29T22:10:17.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5cbbb75edd4ed7c-0065174b49-1584-a1'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.473 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.473 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.473 DEBUG:boto:Method: GET 2023-09-29T22:10:17.473 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.473 DEBUG:boto:Data: 2023-09-29T22:10:17.473 DEBUG:boto:Headers: {} 2023-09-29T22:10:17.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:10:17.473 DEBUG:boto:Port: 8001 2023-09-29T22:10:17.474 DEBUG:boto:Params: {} 2023-09-29T22:10:17.474 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:10:17.474 DEBUG:boto:Token: None 2023-09-29T22:10:17.474 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.478 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/hcnphz-35/testredirect?'), ('x-amz-request-id', 'tx00000fa7eaba7ba7869a4-0065174b49-1580-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.478 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/hcnphz-35/testredirect 2023-09-29T22:10:17.478 DEBUG:boto:Token: None 2023-09-29T22:10:17.478 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.481 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"56907396339ca2b099bd12245f936ddc"'), ('x-amz-request-id', 'tx000000d3e1058964e5250-0065174b49-1584-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.481 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.481 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.481 DEBUG:boto:Method: PUT 2023-09-29T22:10:17.482 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.482 DEBUG:boto:Data: 2023-09-29T22:10:17.482 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:10:17.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:10:17.482 DEBUG:boto:Port: 8000 2023-09-29T22:10:17.482 DEBUG:boto:Params: {} 2023-09-29T22:10:17.482 DEBUG:boto:Token: None 2023-09-29T22:10:17.482 DEBUG:boto:StringToSign: PUT uk9S5NXZfBvPq4jGr+LM5g== application/octet-stream Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.482 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lv2OjR6RSKDfRP6vnoigFgv6wQA= 2023-09-29T22:10:17.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'uk9S5NXZfBvPq4jGr+LM5g==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lv2OjR6RSKDfRP6vnoigFgv6wQA='} 2023-09-29T22:10:17.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009bc3ea34c77d3da3-0065174b49-1584-a1'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.494 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.494 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.494 DEBUG:boto:Method: GET 2023-09-29T22:10:17.495 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.495 DEBUG:boto:Data: 2023-09-29T22:10:17.495 DEBUG:boto:Headers: {} 2023-09-29T22:10:17.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:10:17.495 DEBUG:boto:Port: 8001 2023-09-29T22:10:17.495 DEBUG:boto:Params: {} 2023-09-29T22:10:17.495 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:10:17.495 DEBUG:boto:Token: None 2023-09-29T22:10:17.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.499 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/hcnphz-35/testredirect?'), ('x-amz-request-id', 'tx000009a6ffa45f1729430-0065174b49-1580-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.499 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/hcnphz-35/testredirect 2023-09-29T22:10:17.499 DEBUG:boto:Token: None 2023-09-29T22:10:17.499 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.502 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"ba4f52e4d5d97c1bcfab88c6afe2cce6"'), ('x-amz-request-id', 'tx0000003fc0ce4a174ed49-0065174b49-1584-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.502 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.503 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.503 DEBUG:boto:Method: PUT 2023-09-29T22:10:17.503 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.503 DEBUG:boto:Data: 2023-09-29T22:10:17.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:10:17.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:10:17.503 DEBUG:boto:Port: 8000 2023-09-29T22:10:17.503 DEBUG:boto:Params: {} 2023-09-29T22:10:17.503 DEBUG:boto:Token: None 2023-09-29T22:10:17.504 DEBUG:boto:StringToSign: PUT 6aKmvaJMgaK7Z2rfDZvVeg== application/octet-stream Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JsV/kNrDIoP8Iv+aRnZqqWDs0Mw= 2023-09-29T22:10:17.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '6aKmvaJMgaK7Z2rfDZvVeg==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JsV/kNrDIoP8Iv+aRnZqqWDs0Mw='} 2023-09-29T22:10:17.515 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fc05e805df7d755-0065174b49-1584-a1'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.515 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.515 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.515 DEBUG:boto:Method: GET 2023-09-29T22:10:17.516 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.516 DEBUG:boto:Data: 2023-09-29T22:10:17.516 DEBUG:boto:Headers: {} 2023-09-29T22:10:17.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:10:17.516 DEBUG:boto:Port: 8001 2023-09-29T22:10:17.516 DEBUG:boto:Params: {} 2023-09-29T22:10:17.516 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:10:17.517 DEBUG:boto:Token: None 2023-09-29T22:10:17.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.520 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/hcnphz-35/testredirect?'), ('x-amz-request-id', 'tx000005a8c6a3706f23c5f-0065174b49-1580-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.520 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/hcnphz-35/testredirect 2023-09-29T22:10:17.520 DEBUG:boto:Token: None 2023-09-29T22:10:17.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.523 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"e9a2a6bda24c81a2bb676adf0d9bd57a"'), ('x-amz-request-id', 'tx000006b0cfb821675efa0-0065174b49-1584-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.523 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.523 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.523 DEBUG:boto:Method: PUT 2023-09-29T22:10:17.524 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.524 DEBUG:boto:Data: 2023-09-29T22:10:17.524 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:10:17.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:10:17.524 DEBUG:boto:Port: 8000 2023-09-29T22:10:17.524 DEBUG:boto:Params: {} 2023-09-29T22:10:17.524 DEBUG:boto:Token: None 2023-09-29T22:10:17.524 DEBUG:boto:StringToSign: PUT GbNgNQ5MAukCtgY+rsT5Xw== application/octet-stream Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a3ZSAn1fRF0KnMhl6ggCjTYgwk8= 2023-09-29T22:10:17.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'GbNgNQ5MAukCtgY+rsT5Xw==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a3ZSAn1fRF0KnMhl6ggCjTYgwk8='} 2023-09-29T22:10:17.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c5b471a07c692e7-0065174b49-1584-a1'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.571 DEBUG:boto:path=/hcnphz-35/testredirect 2023-09-29T22:10:17.571 DEBUG:boto:auth_path=/hcnphz-35/testredirect 2023-09-29T22:10:17.571 DEBUG:boto:Method: GET 2023-09-29T22:10:17.571 DEBUG:boto:Path: /hcnphz-35/testredirect 2023-09-29T22:10:17.571 DEBUG:boto:Data: 2023-09-29T22:10:17.571 DEBUG:boto:Headers: {} 2023-09-29T22:10:17.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:10:17.573 DEBUG:boto:Port: 8001 2023-09-29T22:10:17.573 DEBUG:boto:Params: {} 2023-09-29T22:10:17.573 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:10:17.573 DEBUG:boto:Token: None 2023-09-29T22:10:17.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.578 DEBUG:boto:Response headers: [('Location', 'http://smithi112.front.sepia.ceph.com:8000/hcnphz-35/testredirect?'), ('x-amz-request-id', 'tx0000038d3ac47be596266-0065174b49-1580-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.578 DEBUG:boto:Redirecting: http://smithi112.front.sepia.ceph.com/hcnphz-35/testredirect 2023-09-29T22:10:17.578 DEBUG:boto:Token: None 2023-09-29T22:10:17.578 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:10:17 GMT /hcnphz-35/testredirect 2023-09-29T22:10:17.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE= 2023-09-29T22:10:17.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:10:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERTOtsn2AKm8p9R1/2W4x21LHmE=', 'Content-Length': '0'} 2023-09-29T22:10:17.581 DEBUG:boto:Response headers: [('Content-Length', '512'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"19b360350e4c02e902b6063eaec4f95f"'), ('x-amz-request-id', 'tx00000ca885acad1e33e44-0065174b49-1584-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:10:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:10:17.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:10:17.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:10:17.583 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --sync-from-all=true --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:10:17.627 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:17.627 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:17.664 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-09-29T22:10:17.665 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T22:10:17.666 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-09-29T22:10:17.667 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:10:17.668 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-09-29T22:10:17.669 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:10:17.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:10:17.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:10:17.670 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:10:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:10:17.714 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:10:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T22:10:17.853 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:10:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7, 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:10:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:10:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:10:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:10:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.859 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.860 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:10:17.862 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:10:17.863 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:10:17.864 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:10:17.875 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.876 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:10:17.877 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:10:17.878 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.879 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:10:17.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:10:17.882 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:10:17.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:10:17.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:10:17.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:10:17.886 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:10:17.887 INFO:rgw_multi.tests:Set sync_from_all flag on zone a2 to true 2023-09-29T22:11:17.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:11:18.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:11:18.185 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --redirect-zone= --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:11:18.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:18.631 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "a2.rgw.meta:root", 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "a2.rgw.control", 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "a2.rgw.log:gc", 2023-09-29T22:11:18.633 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "a2.rgw.log:lc", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "a2.rgw.log", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "a2.rgw.log:intent", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "a2.rgw.log:usage", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "a2.rgw.meta:roles", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "a2.rgw.log:reshard", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "a2.rgw.meta:users.keys", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "a2.rgw.meta:users.email", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "a2.rgw.meta:users.swift", 2023-09-29T22:11:18.634 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "a2.rgw.meta:users.uid", 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "a2.rgw.otp", 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T22:11:18.635 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "a2.rgw.buckets.index", 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "a2.rgw.buckets.data", 2023-09-29T22:11:18.636 INFO:teuthology.orchestra.run.smithi112.stdout: "compression_type": "random" 2023-09-29T22:11:18.637 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.638 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "a2.rgw.buckets.non-ec", 2023-09-29T22:11:18.638 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T22:11:18.638 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T22:11:18.638 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.638 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.639 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.639 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:11:18.639 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "a2.rgw.log:notif" 2023-09-29T22:11:18.639 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:11:18.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:11:18.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:11:18.640 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:11:18.684 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:11:18.684 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:11:18.773 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T22:11:18.827 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:11:18.860 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:11:18.860 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8, 2023-09-29T22:11:18.860 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:11:18.861 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:11:18.861 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:11:18.861 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:11:18.861 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:11:18.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.861 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:11:18.862 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.863 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:11:18.864 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:11:18.865 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:11:18.866 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:11:18.867 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:11:18.868 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:11:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:11:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:11:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:11:18.869 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:11:18.870 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.871 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:11:18.872 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:11:18.873 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:11:18.873 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:11:18.873 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:11:18.873 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:11:18.874 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:11:18.874 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:11:18.875 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:11:18.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:11:18.878 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:11:18.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.880 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:11:18.880 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:11:18.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:11:18.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:11:18.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:11:18.882 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:11:18.882 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:11:18.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.882 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:11:18.883 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:11:18.884 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:11:18.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.886 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:11:18.887 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:11:18.888 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:11:18.888 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:11:18.888 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:11:18.888 INFO:rgw_multi.tests:Set redirect_zone zone a2 to "" 2023-09-29T22:12:18.889 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2023-09-29T22:12:18.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:18.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:18.891 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:18.935 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:18.935 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:18.994 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:12:19.019 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:08:15.137960Z" 2023-09-29T22:12:19.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.021 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:12:19.022 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:19.022 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:12:19.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:19.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:19.023 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:19.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:19.066 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:19.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:19.120+0000 7f42a5672f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:19.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:19.124+0000 7f42a5672f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:19.149 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:19.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:19.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:19.152 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.153 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.154 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:19Z" 2023-09-29T22:12:19.155 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:19.156 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:19Z" } 2023-09-29T22:12:19.156 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:19.156 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:19.157 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:19.157 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:12:19.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:19.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--access-key', '4CZJCBDK5Q5RRW49', '--secret', 'yNwmuqEvJINfrtIqdSm6917HHqtYG0a9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:19.157 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --access-key 4CZJCBDK5Q5RRW49 --secret yNwmuqEvJINfrtIqdSm6917HHqtYG0a9 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:19.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:19.205 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:19.235 INFO:teuthology.orchestra.run.smithi112.stderr:NOTICE: set zone's realm_id=afcff763-1d9e-43af-a674-cef7f31c7813 2023-09-29T22:12:19.275 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "d6d0895d-bfac-4d7d-b54f-19a320ff3024", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "remove.rgw.meta:root", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "remove.rgw.control", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "remove.rgw.log:gc", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "remove.rgw.log:lc", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "remove.rgw.log", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-09-29T22:12:19.276 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "remove.rgw.otp", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "4CZJCBDK5Q5RRW49", 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "yNwmuqEvJINfrtIqdSm6917HHqtYG0a9" 2023-09-29T22:12:19.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "remove.rgw.buckets.index", 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "remove.rgw.buckets.data" 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "remove.rgw.log:notif" 2023-09-29T22:12:19.279 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:19.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:19.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:19.280 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone remove --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:19.325 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:19.350 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:19.348+0000 7efc2d2eaf40 0 period (b4e33480-bfb8-4364-b1d7-649c06bc8c56 does not have zone d6d0895d-bfac-4d7d-b54f-19a320ff3024 configured 2023-09-29T22:12:25.690 INFO:teuthology.orchestra.run.smithi112.stderr:Sending period to new master zone a75fe7d4-5635-4f57-8059-26c1aacd3a0f 2023-09-29T22:12:25.734 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9, 2023-09-29T22:12:25.734 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:12:25.735 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:25.736 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.737 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.739 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "d6d0895d-bfac-4d7d-b54f-19a320ff3024", 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.741 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:25.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:25.743 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:12:25.744 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.745 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.746 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.748 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.749 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.750 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.751 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:25.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:25.753 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:25.753 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:25.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.753 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:25.753 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:25.755 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "d6d0895d-bfac-4d7d-b54f-19a320ff3024", 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 836533914 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.756 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.757 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:25.758 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:25.759 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.760 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:12:25.761 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:25.762 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:25.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:25.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:25.763 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:25.803 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:25.803 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:25.836 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:12:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:12:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:12:25.840 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:25.841 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.842 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:25.843 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:25.844 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:25.845 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:25.845 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:25.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:25.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:25.847 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup a --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:25.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:25.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:25.923 INFO:teuthology.orchestra.run.smithi112.stderr:zone name remove not found in zonegroup a 2023-09-29T22:12:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-09-29T22:12:25.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:25.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:25.932 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:25.971 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:25.972 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.051 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.145 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.146 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:26.172+0000 7fb78b4b9f40 0 WARNING: can't generate connection for zone d6d0895d-bfac-4d7d-b54f-19a320ff3024 id remove: no endpoints defined 2023-09-29T22:12:26.235 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:26.235 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10, 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:12:26.236 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.237 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.238 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.240 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.241 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:26.242 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:26.243 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.244 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:12:26.245 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.246 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.253 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.254 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.255 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:26.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.257 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:12:26.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:26.260 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:26.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.262 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:26.263 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:26.264 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:26.265 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2023-09-29T22:12:26.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.266 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.339 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b62b505c-0187-4580-825c-a34f34c56062", 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove_zg", 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "remove_zg", 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:26.340 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "", 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [], 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [] 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:26.341 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.342 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:26.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.344 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:26.432+0000 7fb5688dff40 0 NOTICE: promoted remove as new master_zone of zonegroup remove_zg 2023-09-29T22:12:26.439 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:26.439 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "ac5d235f-b80e-4c2e-9dda-29b5a6a1887d", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "domain_root": "remove.rgw.meta:root", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "control_pool": "remove.rgw.control", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "gc_pool": "remove.rgw.log:gc", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "lc_pool": "remove.rgw.log:lc", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "log_pool": "remove.rgw.log", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "intent_log_pool": "remove.rgw.log:intent", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "usage_log_pool": "remove.rgw.log:usage", 2023-09-29T22:12:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "roles_pool": "remove.rgw.meta:roles", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "reshard_pool": "remove.rgw.log:reshard", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "otp_pool": "remove.rgw.otp", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "system_key": { 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "access_key": "", 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "secret_key": "" 2023-09-29T22:12:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_pools": [ 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "default-placement", 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "index_pool": "remove.rgw.buckets.index", 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": { 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD": { 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "data_pool": "remove.rgw.buckets.data" 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "index_type": 0, 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "inline_data": true 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "notif_pool": "remove.rgw.log:notif" 2023-09-29T22:12:26.443 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:26.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.445 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.533 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.658 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11, 2023-09-29T22:12:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:12:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:12:26.658 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.659 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:12:26.660 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.661 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.662 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.663 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:26.664 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:26.665 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:12:26.666 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:26.667 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.668 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.669 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.670 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.671 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.672 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b62b505c-0187-4580-825c-a34f34c56062", 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove_zg", 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "remove_zg", 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "ac5d235f-b80e-4c2e-9dda-29b5a6a1887d", 2023-09-29T22:12:26.674 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "ac5d235f-b80e-4c2e-9dda-29b5a6a1887d", 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "remove", 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [], 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": false, 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:26.675 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.676 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:26.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:12:26.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:26.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "ac5d235f-b80e-4c2e-9dda-29b5a6a1887d", 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3349916353 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.680 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:12:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:26.681 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:12:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:12:26.683 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:26.684 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.685 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:26.686 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:26.687 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:26.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.689 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:26.780+0000 7f6692789f40 0 NOTICE: removed master_zone of zonegroup remove_zg 2023-09-29T22:12:26.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:26.780+0000 7f6692789f40 0 Removed zone from zonegroup remove_zg 2023-09-29T22:12:26.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.793 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.877 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:26.876+0000 7f5459e68f40 0 ERROR: zonegroup remove_zg has a non existent master zone 2023-09-29T22:12:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-09-29T22:12:26.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.885 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:26.924 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:26.924 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:26.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:12:26.957 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:12:27.042 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:27.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:27.137 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:27.137 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:27.137 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12, 2023-09-29T22:12:27.138 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T22:12:27.138 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T22:12:27.138 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T22:12:27.138 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:27.138 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T22:12:27.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.139 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T22:12:27.139 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T22:12:27.139 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:27.139 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T22:12:27.139 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:27.140 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:27.140 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.140 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:27.140 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:27.140 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:27.140 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.141 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:27.142 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:27.144 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:27.145 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:27.146 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.146 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.146 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:27.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.146 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:27.147 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:27.147 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:27.147 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:27.147 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.147 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.148 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:27.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:27.148 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:27.148 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:27.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.148 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:27.149 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:27.149 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.149 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T22:12:27.149 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T22:12:27.150 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T22:12:27.150 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T22:12:27.150 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:27.150 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:27.150 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.150 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T22:12:27.151 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T22:12:27.151 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:27.151 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T22:12:27.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.151 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:27.151 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T22:12:27.152 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:27.152 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T22:12:27.152 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.152 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:27.152 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:27.152 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:27.153 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:27.153 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:27.153 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:27.153 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:27.153 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:27.153 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:27.154 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:27.157 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:27.157 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.158 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:27.158 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T22:12:27.158 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T22:12:27.158 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T22:12:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T22:12:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T22:12:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T22:12:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T22:12:27.159 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T22:12:27.160 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T22:12:27.160 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T22:12:27.160 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T22:12:27.160 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T22:12:27.160 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T22:12:27.161 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:27.161 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.161 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.161 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.161 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T22:12:27.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.162 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T22:12:27.162 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T22:12:27.162 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T22:12:27.162 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T22:12:27.162 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T22:12:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T22:12:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T22:12:27.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T22:12:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T22:12:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T22:12:27.164 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T22:12:27.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.165 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:12:27.165 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T22:12:27.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.165 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T22:12:27.166 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T22:12:27.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.166 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T22:12:27.166 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T22:12:27.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:27.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:27.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T22:12:27.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.167 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:27.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:27.168 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:27.169 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:27.169 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:27.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.169 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T22:12:27.169 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T22:12:27.170 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T22:12:27.170 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T22:12:27.170 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T22:12:27.170 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T22:12:27.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.170 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T22:12:27.171 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:27.171 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:27.171 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:27.171 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:27.171 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:27.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:27.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T22:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T22:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T22:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T22:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T22:12:27.173 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T22:12:27.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:27.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:27.174 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T22:12:27.174 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T22:12:27.174 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:27.174 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:27.176 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2023-09-29T22:12:27.176 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-36 2023-09-29T22:12:27.177 DEBUG:boto:path=/hcnphz-36/ 2023-09-29T22:12:27.177 DEBUG:boto:auth_path=/hcnphz-36/ 2023-09-29T22:12:27.177 DEBUG:boto:Method: PUT 2023-09-29T22:12:27.177 DEBUG:boto:Path: /hcnphz-36/ 2023-09-29T22:12:27.177 DEBUG:boto:Data: 2023-09-29T22:12:27.177 DEBUG:boto:Headers: {} 2023-09-29T22:12:27.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:12:27.178 DEBUG:boto:Port: 8000 2023-09-29T22:12:27.178 DEBUG:boto:Params: {} 2023-09-29T22:12:27.182 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:12:27.182 DEBUG:boto:Token: None 2023-09-29T22:12:27.183 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:27 GMT /hcnphz-36/ 2023-09-29T22:12:27.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C0xMkd9ro84Zoj/WmvOrpGcwYfY= 2023-09-29T22:12:27.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C0xMkd9ro84Zoj/WmvOrpGcwYfY=', 'Content-Length': '0'} 2023-09-29T22:12:27.706 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000037660de63c9db440-0065174bcb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:27.706 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-37 2023-09-29T22:12:27.706 DEBUG:boto:path=/hcnphz-37/ 2023-09-29T22:12:27.706 DEBUG:boto:auth_path=/hcnphz-37/ 2023-09-29T22:12:27.706 DEBUG:boto:Method: PUT 2023-09-29T22:12:27.707 DEBUG:boto:Path: /hcnphz-37/ 2023-09-29T22:12:27.707 DEBUG:boto:Data: 2023-09-29T22:12:27.707 DEBUG:boto:Headers: {} 2023-09-29T22:12:27.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:12:27.707 DEBUG:boto:Port: 8001 2023-09-29T22:12:27.707 DEBUG:boto:Params: {} 2023-09-29T22:12:27.708 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:12:27.708 DEBUG:boto:Token: None 2023-09-29T22:12:27.708 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:27 GMT /hcnphz-37/ 2023-09-29T22:12:27.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zyCvWzBdL9XinceHceCqlklG3YU= 2023-09-29T22:12:27.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zyCvWzBdL9XinceHceCqlklG3YU=', 'Content-Length': '0'} 2023-09-29T22:12:27.751 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000025fa564b1485fcee-0065174bcb-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:27.751 INFO:rgw_multi.tests:create bucket zone=b1 name=hcnphz-38 2023-09-29T22:12:27.751 DEBUG:boto:path=/hcnphz-38/ 2023-09-29T22:12:27.751 DEBUG:boto:auth_path=/hcnphz-38/ 2023-09-29T22:12:27.752 DEBUG:boto:Method: PUT 2023-09-29T22:12:27.752 DEBUG:boto:Path: /hcnphz-38/ 2023-09-29T22:12:27.752 DEBUG:boto:Data: 2023-09-29T22:12:27.752 DEBUG:boto:Headers: {} 2023-09-29T22:12:27.752 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:12:27.752 DEBUG:boto:Port: 8000 2023-09-29T22:12:27.752 DEBUG:boto:Params: {} 2023-09-29T22:12:27.752 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:12:27.753 DEBUG:boto:Token: None 2023-09-29T22:12:27.753 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:27 GMT /hcnphz-38/ 2023-09-29T22:12:27.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kc4KbFlQodwYqQVGgaDS8pIu6b0= 2023-09-29T22:12:27.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kc4KbFlQodwYqQVGgaDS8pIu6b0=', 'Content-Length': '0'} 2023-09-29T22:12:27.992 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007b01f1170cf03c10-0065174bcb-116b-b1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:27.992 INFO:rgw_multi.tests:create bucket zone=b2 name=hcnphz-39 2023-09-29T22:12:27.992 DEBUG:boto:path=/hcnphz-39/ 2023-09-29T22:12:27.992 DEBUG:boto:auth_path=/hcnphz-39/ 2023-09-29T22:12:27.992 DEBUG:boto:Method: PUT 2023-09-29T22:12:27.993 DEBUG:boto:Path: /hcnphz-39/ 2023-09-29T22:12:27.993 DEBUG:boto:Data: 2023-09-29T22:12:27.993 DEBUG:boto:Headers: {} 2023-09-29T22:12:27.993 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:12:27.993 DEBUG:boto:Port: 8001 2023-09-29T22:12:27.993 DEBUG:boto:Params: {} 2023-09-29T22:12:27.993 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:12:27.993 DEBUG:boto:Token: None 2023-09-29T22:12:27.994 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:27 GMT /hcnphz-39/ 2023-09-29T22:12:27.994 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:meg7VXQ+JejdP+3Hcj5rMKRk+lQ= 2023-09-29T22:12:27.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:meg7VXQ+JejdP+3Hcj5rMKRk+lQ=', 'Content-Length': '0'} 2023-09-29T22:12:28.039 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e5d069a694527b7e-0065174bcb-1167-b2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.040 DEBUG:boto:path=/hcnphz-36/ 2023-09-29T22:12:28.040 DEBUG:boto:auth_path=/hcnphz-36/ 2023-09-29T22:12:28.040 DEBUG:boto:path=/hcnphz-36/?website 2023-09-29T22:12:28.040 DEBUG:boto:auth_path=/hcnphz-36/?website 2023-09-29T22:12:28.040 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.040 DEBUG:boto:Path: /hcnphz-36/?website 2023-09-29T22:12:28.040 DEBUG:boto:Data: index.htmlerror.html 2023-09-29T22:12:28.040 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:12:28.041 DEBUG:boto:Port: 8000 2023-09-29T22:12:28.041 DEBUG:boto:Params: {} 2023-09-29T22:12:28.041 DEBUG:boto:Token: None 2023-09-29T22:12:28.041 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-36/?website 2023-09-29T22:12:28.042 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZmA/jkN1asIjXkNe7utc4zVzT5c= 2023-09-29T22:12:28.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZmA/jkN1asIjXkNe7utc4zVzT5c=', 'Content-Length': '241'} 2023-09-29T22:12:28.043 DEBUG:boto:Response headers: [('Content-Length', '192'), ('x-amz-request-id', 'tx0000029d64c05c0d9784c-0065174bcc-1648-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.043 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... SKIP: test_set_bucket_website skipped. Requires rgw_enable_static_website = 1. 2023-09-29T22:12:28.044 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-40 2023-09-29T22:12:28.044 DEBUG:boto:path=/hcnphz-40/ 2023-09-29T22:12:28.044 DEBUG:boto:auth_path=/hcnphz-40/ 2023-09-29T22:12:28.044 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.044 DEBUG:boto:Path: /hcnphz-40/ 2023-09-29T22:12:28.045 DEBUG:boto:Data: 2023-09-29T22:12:28.045 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:12:28.045 DEBUG:boto:Port: 8000 2023-09-29T22:12:28.045 DEBUG:boto:Params: {} 2023-09-29T22:12:28.045 DEBUG:boto:Token: None 2023-09-29T22:12:28.045 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-40/ 2023-09-29T22:12:28.046 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxRcwsvYl8lBx1WxVy7s+7GsiW4= 2023-09-29T22:12:28.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxRcwsvYl8lBx1WxVy7s+7GsiW4=', 'Content-Length': '0'} 2023-09-29T22:12:28.069 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000756e34f856f83a07-0065174bcc-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.070 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-41 2023-09-29T22:12:28.070 DEBUG:boto:path=/hcnphz-41/ 2023-09-29T22:12:28.070 DEBUG:boto:auth_path=/hcnphz-41/ 2023-09-29T22:12:28.070 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.070 DEBUG:boto:Path: /hcnphz-41/ 2023-09-29T22:12:28.071 DEBUG:boto:Data: 2023-09-29T22:12:28.071 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:12:28.071 DEBUG:boto:Port: 8001 2023-09-29T22:12:28.071 DEBUG:boto:Params: {} 2023-09-29T22:12:28.071 DEBUG:boto:Token: None 2023-09-29T22:12:28.071 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-41/ 2023-09-29T22:12:28.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f/R8CXl6asDuKR5811GMYfyj/sc= 2023-09-29T22:12:28.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f/R8CXl6asDuKR5811GMYfyj/sc=', 'Content-Length': '0'} 2023-09-29T22:12:28.108 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000e687968e85402f4a-0065174bcc-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.109 INFO:rgw_multi.tests:create bucket zone=b1 name=hcnphz-42 2023-09-29T22:12:28.109 DEBUG:boto:path=/hcnphz-42/ 2023-09-29T22:12:28.109 DEBUG:boto:auth_path=/hcnphz-42/ 2023-09-29T22:12:28.109 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.109 DEBUG:boto:Path: /hcnphz-42/ 2023-09-29T22:12:28.109 DEBUG:boto:Data: 2023-09-29T22:12:28.109 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.110 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:12:28.110 DEBUG:boto:Port: 8000 2023-09-29T22:12:28.110 DEBUG:boto:Params: {} 2023-09-29T22:12:28.110 DEBUG:boto:Token: None 2023-09-29T22:12:28.110 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-42/ 2023-09-29T22:12:28.110 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IMPK8fFdzaC95J/8J8XG/eOOcgg= 2023-09-29T22:12:28.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IMPK8fFdzaC95J/8J8XG/eOOcgg=', 'Content-Length': '0'} 2023-09-29T22:12:28.152 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000054f0f204d97fe22b-0065174bcc-116b-b1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.152 INFO:rgw_multi.tests:create bucket zone=b2 name=hcnphz-43 2023-09-29T22:12:28.152 DEBUG:boto:path=/hcnphz-43/ 2023-09-29T22:12:28.153 DEBUG:boto:auth_path=/hcnphz-43/ 2023-09-29T22:12:28.153 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.153 DEBUG:boto:Path: /hcnphz-43/ 2023-09-29T22:12:28.153 DEBUG:boto:Data: 2023-09-29T22:12:28.153 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.153 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:12:28.153 DEBUG:boto:Port: 8001 2023-09-29T22:12:28.153 DEBUG:boto:Params: {} 2023-09-29T22:12:28.153 DEBUG:boto:Token: None 2023-09-29T22:12:28.154 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-43/ 2023-09-29T22:12:28.154 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OmdcgTAJ+pAg/SRpuCTiqEhr3xc= 2023-09-29T22:12:28.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OmdcgTAJ+pAg/SRpuCTiqEhr3xc=', 'Content-Length': '0'} 2023-09-29T22:12:28.196 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000004c74d5a5631ac75c-0065174bcc-1167-b2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.197 DEBUG:boto:path=/hcnphz-40/ 2023-09-29T22:12:28.197 DEBUG:boto:auth_path=/hcnphz-40/ 2023-09-29T22:12:28.197 DEBUG:boto:path=/hcnphz-40/?policy 2023-09-29T22:12:28.197 DEBUG:boto:auth_path=/hcnphz-40/?policy 2023-09-29T22:12:28.197 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.198 DEBUG:boto:Path: /hcnphz-40/?policy 2023-09-29T22:12:28.198 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:12:28.198 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:12:28.198 DEBUG:boto:Port: 8000 2023-09-29T22:12:28.198 DEBUG:boto:Params: {} 2023-09-29T22:12:28.199 DEBUG:boto:Token: None 2023-09-29T22:12:28.199 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-40/?policy 2023-09-29T22:12:28.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e+Mw4pLf95DOAEy2M4QfQPh5W18= 2023-09-29T22:12:28.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e+Mw4pLf95DOAEy2M4QfQPh5W18=', 'Content-Length': '97'} 2023-09-29T22:12:28.212 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000b7aef7b3a17c6d-0065174bcc-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.212 DEBUG:boto:path=/hcnphz-40/ 2023-09-29T22:12:28.212 DEBUG:boto:auth_path=/hcnphz-40/ 2023-09-29T22:12:28.212 DEBUG:boto:path=/hcnphz-40/?policy 2023-09-29T22:12:28.213 DEBUG:boto:auth_path=/hcnphz-40/?policy 2023-09-29T22:12:28.213 DEBUG:boto:Method: GET 2023-09-29T22:12:28.213 DEBUG:boto:Path: /hcnphz-40/?policy 2023-09-29T22:12:28.213 DEBUG:boto:Data: 2023-09-29T22:12:28.213 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:12:28.214 DEBUG:boto:Port: 8000 2023-09-29T22:12:28.214 DEBUG:boto:Params: {} 2023-09-29T22:12:28.214 DEBUG:boto:Token: None 2023-09-29T22:12:28.214 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-40/?policy 2023-09-29T22:12:28.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+b4TyM8nAv1MCPhWVSIwijIO6wc= 2023-09-29T22:12:28.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+b4TyM8nAv1MCPhWVSIwijIO6wc=', 'Content-Length': '0'} 2023-09-29T22:12:28.216 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005435747f32124854-0065174bcc-1648-a1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:12:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:28.216 DEBUG:boto:path=/hcnphz-41/ 2023-09-29T22:12:28.216 DEBUG:boto:auth_path=/hcnphz-41/ 2023-09-29T22:12:28.217 DEBUG:boto:path=/hcnphz-41/?policy 2023-09-29T22:12:28.217 DEBUG:boto:auth_path=/hcnphz-41/?policy 2023-09-29T22:12:28.217 DEBUG:boto:Method: PUT 2023-09-29T22:12:28.217 DEBUG:boto:Path: /hcnphz-41/?policy 2023-09-29T22:12:28.217 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:12:28.218 DEBUG:boto:Headers: {} 2023-09-29T22:12:28.218 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:12:28.218 DEBUG:boto:Port: 8001 2023-09-29T22:12:28.218 DEBUG:boto:Params: {} 2023-09-29T22:12:28.218 DEBUG:boto:Token: None 2023-09-29T22:12:28.219 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:28 GMT /hcnphz-41/?policy 2023-09-29T22:12:28.219 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yYOB35RjtReRGkbMwgPtJN6bDZg= 2023-09-29T22:12:28.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yYOB35RjtReRGkbMwgPtJN6bDZg=', 'Content-Length': '97'} 2023-09-29T22:12:29.238 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009d66bd7692a91665-0065174bcc-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:29.238 DEBUG:boto:path=/hcnphz-41/ 2023-09-29T22:12:29.238 DEBUG:boto:auth_path=/hcnphz-41/ 2023-09-29T22:12:29.238 DEBUG:boto:path=/hcnphz-41/?policy 2023-09-29T22:12:29.238 DEBUG:boto:auth_path=/hcnphz-41/?policy 2023-09-29T22:12:29.239 DEBUG:boto:Method: GET 2023-09-29T22:12:29.239 DEBUG:boto:Path: /hcnphz-41/?policy 2023-09-29T22:12:29.239 DEBUG:boto:Data: 2023-09-29T22:12:29.239 DEBUG:boto:Headers: {} 2023-09-29T22:12:29.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:12:29.239 DEBUG:boto:Port: 8001 2023-09-29T22:12:29.239 DEBUG:boto:Params: {} 2023-09-29T22:12:29.239 DEBUG:boto:Token: None 2023-09-29T22:12:29.239 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:29 GMT /hcnphz-41/?policy 2023-09-29T22:12:29.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6XDNxEu6D4WfgSjihZnJ3WObxlQ= 2023-09-29T22:12:29.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6XDNxEu6D4WfgSjihZnJ3WObxlQ=', 'Content-Length': '0'} 2023-09-29T22:12:29.241 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c5378706ec28f7ce-0065174bcd-1642-a2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:12:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:29.241 DEBUG:boto:path=/hcnphz-42/ 2023-09-29T22:12:29.241 DEBUG:boto:auth_path=/hcnphz-42/ 2023-09-29T22:12:29.242 DEBUG:boto:path=/hcnphz-42/?policy 2023-09-29T22:12:29.242 DEBUG:boto:auth_path=/hcnphz-42/?policy 2023-09-29T22:12:29.242 DEBUG:boto:Method: PUT 2023-09-29T22:12:29.242 DEBUG:boto:Path: /hcnphz-42/?policy 2023-09-29T22:12:29.242 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:12:29.242 DEBUG:boto:Headers: {} 2023-09-29T22:12:29.242 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:12:29.242 DEBUG:boto:Port: 8000 2023-09-29T22:12:29.242 DEBUG:boto:Params: {} 2023-09-29T22:12:29.243 DEBUG:boto:Token: None 2023-09-29T22:12:29.243 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:29 GMT /hcnphz-42/?policy 2023-09-29T22:12:29.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hPHdH142Xsd8D6amMd8D2xXiSdw= 2023-09-29T22:12:29.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPHdH142Xsd8D6amMd8D2xXiSdw=', 'Content-Length': '97'} 2023-09-29T22:12:30.264 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006875bea572997fbb-0065174bcd-116b-b1'), ('Date', 'Fri, 29 Sep 2023 22:12:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:30.264 DEBUG:boto:path=/hcnphz-42/ 2023-09-29T22:12:30.265 DEBUG:boto:auth_path=/hcnphz-42/ 2023-09-29T22:12:30.265 DEBUG:boto:path=/hcnphz-42/?policy 2023-09-29T22:12:30.265 DEBUG:boto:auth_path=/hcnphz-42/?policy 2023-09-29T22:12:30.265 DEBUG:boto:Method: GET 2023-09-29T22:12:30.265 DEBUG:boto:Path: /hcnphz-42/?policy 2023-09-29T22:12:30.265 DEBUG:boto:Data: 2023-09-29T22:12:30.265 DEBUG:boto:Headers: {} 2023-09-29T22:12:30.265 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8000 2023-09-29T22:12:30.265 DEBUG:boto:Port: 8000 2023-09-29T22:12:30.266 DEBUG:boto:Params: {} 2023-09-29T22:12:30.266 DEBUG:boto:Token: None 2023-09-29T22:12:30.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:30 GMT /hcnphz-42/?policy 2023-09-29T22:12:30.266 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/sKlu9dL3lj6ky+W5RPXM2KIX8k= 2023-09-29T22:12:30.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/sKlu9dL3lj6ky+W5RPXM2KIX8k=', 'Content-Length': '0'} 2023-09-29T22:12:30.268 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005d94eae74141c401-0065174bce-116b-b1'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:12:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:30.268 DEBUG:boto:path=/hcnphz-43/ 2023-09-29T22:12:30.268 DEBUG:boto:auth_path=/hcnphz-43/ 2023-09-29T22:12:30.268 DEBUG:boto:path=/hcnphz-43/?policy 2023-09-29T22:12:30.268 DEBUG:boto:auth_path=/hcnphz-43/?policy 2023-09-29T22:12:30.268 DEBUG:boto:Method: PUT 2023-09-29T22:12:30.268 DEBUG:boto:Path: /hcnphz-43/?policy 2023-09-29T22:12:30.269 DEBUG:boto:Data: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": "*" }] } 2023-09-29T22:12:30.269 DEBUG:boto:Headers: {} 2023-09-29T22:12:30.269 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:12:30.269 DEBUG:boto:Port: 8001 2023-09-29T22:12:30.269 DEBUG:boto:Params: {} 2023-09-29T22:12:30.269 DEBUG:boto:Token: None 2023-09-29T22:12:30.269 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:30 GMT /hcnphz-43/?policy 2023-09-29T22:12:30.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0kh3WORsDtHUSqflZiUyVL24RxE= 2023-09-29T22:12:30.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0kh3WORsDtHUSqflZiUyVL24RxE=', 'Content-Length': '97'} 2023-09-29T22:12:31.289 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000efa8148b491d1ac9-0065174bce-1167-b2'), ('Date', 'Fri, 29 Sep 2023 22:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:31.289 DEBUG:boto:path=/hcnphz-43/ 2023-09-29T22:12:31.289 DEBUG:boto:auth_path=/hcnphz-43/ 2023-09-29T22:12:31.290 DEBUG:boto:path=/hcnphz-43/?policy 2023-09-29T22:12:31.290 DEBUG:boto:auth_path=/hcnphz-43/?policy 2023-09-29T22:12:31.290 DEBUG:boto:Method: GET 2023-09-29T22:12:31.290 DEBUG:boto:Path: /hcnphz-43/?policy 2023-09-29T22:12:31.290 DEBUG:boto:Data: 2023-09-29T22:12:31.290 DEBUG:boto:Headers: {} 2023-09-29T22:12:31.290 DEBUG:boto:Host: smithi131.front.sepia.ceph.com:8001 2023-09-29T22:12:31.290 DEBUG:boto:Port: 8001 2023-09-29T22:12:31.290 DEBUG:boto:Params: {} 2023-09-29T22:12:31.291 DEBUG:boto:Token: None 2023-09-29T22:12:31.291 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:12:31 GMT /hcnphz-43/?policy 2023-09-29T22:12:31.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KDfKe4zzQmEeTFLlPBBbC1zyNmY= 2023-09-29T22:12:31.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KDfKe4zzQmEeTFLlPBBbC1zyNmY=', 'Content-Length': '0'} 2023-09-29T22:12:31.293 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000855e9a73f66455e0-0065174bcf-1167-b2'), ('Content-Type', 'application/json'), ('Content-Length', '97'), ('Date', 'Fri, 29 Sep 2023 22:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:31.293 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_policy ... ok 2023-09-29T22:12:31.294 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-44 2023-09-29T22:12:31.294 DEBUG:boto:path=/hcnphz-44/ 2023-09-29T22:12:31.294 DEBUG:boto:auth_path=/hcnphz-44/ 2023-09-29T22:12:31.294 DEBUG:boto:Method: PUT 2023-09-29T22:12:31.294 DEBUG:boto:Path: /hcnphz-44/ 2023-09-29T22:12:31.294 DEBUG:boto:Data: 2023-09-29T22:12:31.294 DEBUG:boto:Headers: {} 2023-09-29T22:12:31.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:12:31.294 DEBUG:boto:Port: 8000 2023-09-29T22:12:31.294 DEBUG:boto:Params: {} 2023-09-29T22:12:31.297 DEBUG:boto:Token: None 2023-09-29T22:12:31.297 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:31 GMT /hcnphz-44/ 2023-09-29T22:12:31.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LLneqjbOaxGr4f6qXVXxNIwL+AA= 2023-09-29T22:12:31.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LLneqjbOaxGr4f6qXVXxNIwL+AA=', 'Content-Length': '0'} 2023-09-29T22:12:31.326 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000ef0cf7653c5bf988-0065174bcf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:31.326 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-45 2023-09-29T22:12:31.326 DEBUG:boto:path=/hcnphz-45/ 2023-09-29T22:12:31.326 DEBUG:boto:auth_path=/hcnphz-45/ 2023-09-29T22:12:31.326 DEBUG:boto:Method: PUT 2023-09-29T22:12:31.327 DEBUG:boto:Path: /hcnphz-45/ 2023-09-29T22:12:31.327 DEBUG:boto:Data: 2023-09-29T22:12:31.327 DEBUG:boto:Headers: {} 2023-09-29T22:12:31.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:12:31.327 DEBUG:boto:Port: 8001 2023-09-29T22:12:31.327 DEBUG:boto:Params: {} 2023-09-29T22:12:31.327 DEBUG:boto:Token: None 2023-09-29T22:12:31.327 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:12:31 GMT /hcnphz-45/ 2023-09-29T22:12:31.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E9Or3s9A+4U6/ZHQvYRjUotogVk= 2023-09-29T22:12:31.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:12:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E9Or3s9A+4U6/ZHQvYRjUotogVk=', 'Content-Length': '0'} 2023-09-29T22:12:31.371 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f27d2210a09e1522-0065174bcf-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:12:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:12:31.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:31.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:31.371 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:31.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:31.414 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:31.471 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:12:31.496 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:12:31.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025551.357019_354.1", 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:12:31.357019Z" 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:31.498 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:12:31.501 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:31.501 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:12:31.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:31.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:31.502 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:31.545 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:31.545 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:31.601 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:31.600+0000 7fcf82cdaf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:31.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:31.604+0000 7fcf82cdaf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:31.628 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:31.628 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:31.629 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.630 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:31.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:31.632 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:31.633 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.634 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:31.635 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:31Z" 2023-09-29T22:12:31.635 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:31.636 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:31Z" } 2023-09-29T22:12:31.636 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:31.636 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:31.636 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:31.637 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:12:36.637 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:36.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:36.638 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:36.682 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:36.683 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:36.739 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:36.735+0000 7f5929228f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:36.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:36.739+0000 7f5929228f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:36.766 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:36.766 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:36.766 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:36.766 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:36.767 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:36.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:36.769 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:36.770 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:36.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:36.772 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:36.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:36.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:36.775 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:36Z" 2023-09-29T22:12:36.775 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:36.776 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:36Z" } 2023-09-29T22:12:36.777 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:36.777 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:36.777 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:36.777 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:12:41.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:41.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:41.779 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:41.825 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:41.825 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:41.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:41.879+0000 7f4d2929af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:41.883 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:41.883+0000 7f4d2929af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:41.907 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:41.907 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:41.907 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:41.907 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:41.907 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:41.908 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:41.909 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:41.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:41.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:41.912 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:41Z" 2023-09-29T22:12:41.913 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:41.914 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:41Z" } 2023-09-29T22:12:41.915 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:41.915 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:41.915 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:41.915 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:12:46.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:46.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:46.916 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:46.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:46.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:47.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:47.015+0000 7ff31d1e8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:47.020 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:47.019+0000 7ff31d1e8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:47.046 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:47Z" 2023-09-29T22:12:47.053 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:47.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:47Z" } 2023-09-29T22:12:47.054 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:47.054 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:47.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:47.055 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:12:52.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:52.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:52.057 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:52.101 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:52.101 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:52.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:52.151+0000 7f589a5a7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:52.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:52.151+0000 7f589a5a7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:52.177 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:52.177 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:52.177 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:52.177 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:52.177 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:52.177 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:52.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:52.179 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:52.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:52.181 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:52.182 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:52.183 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:52.184 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:52.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:52.184 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:52Z" 2023-09-29T22:12:52.184 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:52.185 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:52Z" } 2023-09-29T22:12:52.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:52.185 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:52.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:52.186 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:12:57.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:57.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:12:57.188 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:12:57.235 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:12:57.236 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:12:57.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:57.287+0000 7ff3c024ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:57.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:12:57.291+0000 7ff3c024ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:12:57.317 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:12:57.317 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:12:57.317 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:12:57.318 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:57.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:12:57.320 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:57.321 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:12:57.322 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:12:57.323 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:12:57Z" 2023-09-29T22:12:57.324 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:12:57.325 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:12:57Z" } 2023-09-29T22:12:57.325 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:12:57.326 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:12:57.326 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:12:57.326 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:02.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:02.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:02.327 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:02.375 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:02.375 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:02.429 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:02.427+0000 7f9374e88f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:02.431 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:02.431+0000 7f9374e88f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:02.456 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:02.456 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:02.456 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:02.456 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:02.456 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:02.456 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:02.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:02.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:02.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:02.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:02.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:02.463 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:02Z" 2023-09-29T22:13:02.463 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:02.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:02Z" } 2023-09-29T22:13:02.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:02.464 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:02.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:02.464 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:07.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:07.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:07.466 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:07.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:07.512 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:07.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:07.563+0000 7fb864038f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:07.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:07.563+0000 7fb864038f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:07.591 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:07.591 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:07.591 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:07.591 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:07.591 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:07.591 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:07.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:07.593 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:07.594 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:07.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:07.596 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:07.597 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:07Z" 2023-09-29T22:13:07.598 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:07.599 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:07Z" } 2023-09-29T22:13:07.599 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:07.600 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:07.600 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:07.600 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:12.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:12.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:12.601 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:12.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:12.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:12.700 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:12.698+0000 7fdf5cb15f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:12.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:12.698+0000 7fdf5cb15f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:12.726 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:12.726 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:12.726 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:12.726 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:12.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:12.728 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:12.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:12.730 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:12.731 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:12.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:12.733 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:12Z" 2023-09-29T22:13:12.733 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:12.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:12Z" } 2023-09-29T22:13:12.734 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:12.734 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:12.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:12.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:17.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:17.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:17.735 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:17.828 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:17.826+0000 7f677f857f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:17.829 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:17.826+0000 7f677f857f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:17.851 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:17.851 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:17.852 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:17.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:17.854 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:17.855 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:17.856 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:17.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:17.858 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:17Z" 2023-09-29T22:13:17.858 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:17.859 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:17Z" } 2023-09-29T22:13:17.859 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:17.859 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:17.859 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:17.860 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:22.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:22.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:22.861 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:22.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:22.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:22.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:22.962+0000 7f00f354ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:22.965 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:22.962+0000 7f00f354ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:22.992 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:22.992 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:22.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:22.994 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:22.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:22.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:22.997 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:22.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:22.999 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:22Z" 2023-09-29T22:13:22.999 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:23.000 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:22Z" } 2023-09-29T22:13:23.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:23.000 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:23.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:23.000 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:28.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:28.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:28.002 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:28.049 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:28.050 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:28.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:28.090+0000 7f0108397f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:28.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:28.094+0000 7f0108397f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:28.119 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:28.120 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:28.121 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:28.122 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:28.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:28.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:28.125 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:28.126 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:28.127 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:28Z" 2023-09-29T22:13:28.127 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:28.128 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:28Z" } 2023-09-29T22:13:28.128 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:28.128 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:28.128 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:28.128 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:33.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:33.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:33.130 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:33.177 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:33.177 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:33.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:33.230+0000 7f279fb05f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:33.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:33.234+0000 7f279fb05f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:33.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:33.261 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:33.262 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:33.263 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:33.264 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:33.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:33Z" 2023-09-29T22:13:33.266 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:33.267 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:33Z" } 2023-09-29T22:13:33.268 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:33.268 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:33.268 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:33.268 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:38.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:38.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:38.270 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:38.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:38.314 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:38.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:38.366+0000 7f52c1dd5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:38.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:38.370+0000 7f52c1dd5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:38.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:38.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:38.399 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:38.400 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:38.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:38.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:38.403 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:38Z" 2023-09-29T22:13:38.404 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:38.404 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:38Z" } 2023-09-29T22:13:38.405 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:38.405 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:38.405 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:38.405 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:43.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:43.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:43.407 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:43.452 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:43.453 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:43.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:43.505+0000 7f24c2734f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:43.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:43.509+0000 7f24c2734f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:43.536 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:43.536 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:43.536 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:43.536 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:43.536 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:43.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:43.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:43.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:43.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:43.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:43.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:43.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:43.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:43.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:43.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:43.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:43.543 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:43.543 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:43.543 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:43.543 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:43.543 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:43.543 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:43.544 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:43.545 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:43.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:43.545 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:43Z" 2023-09-29T22:13:43.545 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:43.546 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:43Z" } 2023-09-29T22:13:43.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:43.547 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:43.547 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:43.547 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:48.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:48.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:48.549 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:48.594 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:48.594 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:48.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:48.645+0000 7f854986ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:48.650 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:48.645+0000 7f854986ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:48.674 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:48.674 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:48.674 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:48.674 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:48.674 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:48.675 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:48.675 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:48.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:48.675 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:48.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:48.675 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:48.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:48.676 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:48.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:48.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:48.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:48.677 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:48.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:48.679 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:48.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:48.681 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:48.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:48.682 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:48Z" 2023-09-29T22:13:48.682 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:48.683 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:48Z" } 2023-09-29T22:13:48.683 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:48.683 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:48.683 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:48.684 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:53.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:53.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:53.685 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:53.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:53.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:53.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:53.781+0000 7f33920d0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:53.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:53.785+0000 7f33920d0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:53.813 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:53.814 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:53.815 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:53.816 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:53.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:53.818 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:53.819 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:53.820 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:53Z" 2023-09-29T22:13:53.820 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:53.821 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:53Z" } 2023-09-29T22:13:53.821 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:53.822 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:53.822 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:53.822 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:13:58.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:58.823 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:13:58.824 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:13:58.871 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:13:58.871 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:13:58.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:58.925+0000 7ff96beaff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:58.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:13:58.929+0000 7ff96beaff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:13:58.952 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:13:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:13:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:13:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:13:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:13:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:13:58.953 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:13:58.954 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:58.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:58.956 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:13:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:58.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:58.958 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:13:58.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:13:58.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:13:58Z" 2023-09-29T22:13:58.961 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:13:58.962 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:13:58Z" } 2023-09-29T22:13:58.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:13:58.963 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:13:58.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:13:58.963 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:14:03.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:03.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:03.964 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:04.011 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:04.011 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:04.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:04.065+0000 7f97fb39df40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:04.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:04.065+0000 7f97fb39df40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:04.090 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:04.090 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:04.090 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:04.091 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:04.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:04.093 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:04.094 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:04.095 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:04.096 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:04.097 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:04Z" 2023-09-29T22:14:04.098 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:04.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:04Z" } 2023-09-29T22:14:04.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:04.099 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:04.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:14:04.099 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:14:09.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:09.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:09.100 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:09.147 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:09.147 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:09.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:09.201+0000 7f37079cbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:09.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:09.201+0000 7f37079cbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:09.229 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:09.229 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:09.229 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:09.229 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:09.229 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:09.229 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:09.230 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:09.234 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:09Z" 2023-09-29T22:14:09.235 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:09.236 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:09Z" } 2023-09-29T22:14:09.237 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:09.237 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:09.237 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:14:09.237 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:14:14.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:14.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:14.239 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:14.285 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:14.285 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:14.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:14.337+0000 7f1169533f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:14.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:14.341+0000 7f1169533f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:14.365 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:14.366 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:14.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:14.368 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:14.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:14.370 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:14:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:14.371 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:14:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:14.372 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:14.373 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:14.374 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:14.375 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:14.376 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:14.377 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:14.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:14.379 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:14Z" 2023-09-29T22:14:14.380 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:14.381 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:14Z" } 2023-09-29T22:14:14.381 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:14.381 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:14.382 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:14:14.382 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:14:19.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:19.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:19.384 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:19.429 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:19.429 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:19.483 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:19.484+0000 7f8fd6e70f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:19.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:19.484+0000 7f8fd6e70f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:19.510 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:19.510 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:19.510 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:19.511 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:19.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:14:19.513 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:19.514 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:19.515 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:19.516 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:19Z" 2023-09-29T22:14:19.517 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:19.518 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:19Z" } 2023-09-29T22:14:19.518 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:19.519 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:19.519 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:14:19.519 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:14:24.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:24.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:24.521 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:24.566 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:24.566 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:24.624 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:24.624+0000 7f4717bdbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:24.625 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:24.624+0000 7f4717bdbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:24.652 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:24.653 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:24.654 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:24.655 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025295.137960_229.1", 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:08:15.137960Z", 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:24.656 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:24.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:24.658 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:24Z" 2023-09-29T22:14:24.659 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:24.660 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025295.137960_229.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:08:15.137960Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:24Z" } 2023-09-29T22:14:24.661 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025295.137960_229.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:08:15.137960Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:24.661 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:24.661 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025295.137960_229.1', 2: '', 3: ''} 2023-09-29T22:14:24.661 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025295.137960_229.1 2023-09-29T22:14:29.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:29.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:29.663 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:29.710 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:29.711 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:29.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:29.764+0000 7fb7cb25bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:29.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:29.768+0000 7fb7cb25bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:29.792 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:29.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:29.794 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:29.795 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025547.691852_296.1", 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:27.691852Z", 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:29.796 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:29.797 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:29.798 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:29.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:29.800 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:29Z" 2023-09-29T22:14:29.800 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:29.800 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025547.691852_296.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:27.691852Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:29Z" } 2023-09-29T22:14:29.801 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025547.691852_296.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:27.691852Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:29.801 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:29.801 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025547.691852_296.1', 2: '', 3: ''} 2023-09-29T22:14:29.801 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025547.691852_296.1 2023-09-29T22:14:34.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:34.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:34.803 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:34.849 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:34.849 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:34.907 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:34.908+0000 7f4375538f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:34.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:34.908+0000 7f4375538f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:34.936 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:34.936 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:34.936 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:34.936 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:34.936 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:34.937 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:34.938 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025547.728566_301.1", 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:27.728566Z", 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:34.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:34.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:34.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:34.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:34.943 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:34.943 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:34.943 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:34.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:34.943 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:34Z" 2023-09-29T22:14:34.943 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:34.944 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025547.728566_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:27.728566Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:34Z" } 2023-09-29T22:14:34.944 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025547.728566_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:27.728566Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:34.944 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:34.945 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025547.728566_301.1', 2: '', 3: ''} 2023-09-29T22:14:34.945 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025547.728566_301.1 2023-09-29T22:14:39.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:39.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:39.946 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:39.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:39.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:40.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:40.044+0000 7fbe9fbd0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:40.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:40.048+0000 7fbe9fbd0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:40.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:40.071 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025547.728566_301.1", 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:40.072 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:27.728566Z", 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:40.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:40.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:40.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:40Z" 2023-09-29T22:14:40.076 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:40.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025547.728566_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:27.728566Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:40Z" } 2023-09-29T22:14:40.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025547.728566_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:27.728566Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:40.077 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:40.078 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025547.728566_301.1', 2: '', 3: ''} 2023-09-29T22:14:40.078 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025547.728566_301.1 2023-09-29T22:14:45.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:45.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:45.080 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:45.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:45.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:45.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:45.180+0000 7f4b612faf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:45.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:45.184+0000 7f4b612faf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:45.210 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:45.210 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:45.211 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:45.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025547.728566_301.1", 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:27.728566Z", 2023-09-29T22:14:45.213 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:45.214 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:45.215 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:45.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:45.217 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:45.217 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:45.217 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:45.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:45.217 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:45Z" 2023-09-29T22:14:45.217 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:45.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025547.728566_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:27.728566Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:45Z" } 2023-09-29T22:14:45.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025547.728566_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:27.728566Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:45.218 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:45.219 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025547.728566_301.1', 2: '', 3: ''} 2023-09-29T22:14:45.219 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025547.728566_301.1 2023-09-29T22:14:50.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:50.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:50.220 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:50.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:50.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:50.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:50.312+0000 7f46df044f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:50.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:50.316+0000 7f46df044f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:50.340 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:50.340 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:50.341 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:50.342 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025547.969783_315.1", 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:27.969783Z", 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:50.343 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:50.344 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:50.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:50Z" 2023-09-29T22:14:50.346 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:50.347 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025547.969783_315.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:27.969783Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:50Z" } 2023-09-29T22:14:50.347 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025547.969783_315.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:27.969783Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:50.347 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:50.348 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025547.969783_315.1', 2: '', 3: ''} 2023-09-29T22:14:50.348 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025547.969783_315.1 2023-09-29T22:14:55.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:55.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:14:55.349 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:14:55.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:14:55.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:14:55.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:55.447+0000 7fa413eb9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:55.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:14:55.451+0000 7fa413eb9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:14:55.478 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:55.479 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:55.480 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025547.728566_301.1", 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:55.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:27.728566Z", 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:55.482 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:14:55.483 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:55.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:14:55Z" 2023-09-29T22:14:55.485 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:14:55.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025547.728566_301.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:27.728566Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:14:55Z" } 2023-09-29T22:14:55.486 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025547.728566_301.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:27.728566Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:14:55.487 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:14:55.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025547.728566_301.1', 2: '', 3: ''} 2023-09-29T22:14:55.487 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696025551.357019_354.1 target=1_1696025547.728566_301.1 2023-09-29T22:15:00.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:00.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:00.488 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:00.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:00.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:00.584 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:00.583+0000 7f3311305f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:00.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:00.583+0000 7f3311305f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:00.611 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:15:00.612 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:15:00.613 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025551.357019_354.1", 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:00.614 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:12:31.357019Z", 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:00.615 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:00.616 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:00.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:15:00Z" 2023-09-29T22:15:00.618 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:15:00.619 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025551.357019_354.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:12:31.357019Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:15:00Z" } 2023-09-29T22:15:00.620 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025551.357019_354.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:12:31.357019Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:15:00.620 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:15:00.620 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025551.357019_354.1', 2: '', 3: ''} 2023-09-29T22:15:00.620 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:15:00.620 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hcnphz-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:15:00.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hcnphz-44', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:15:00.621 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hcnphz-44 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:15:00.708 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:00.708 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:00.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hcnphz-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:15:00.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hcnphz-45', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:15:00.845 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hcnphz-45 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:15:00.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:00.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:01.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.022 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:01.061 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:01.061 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:01.158 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000078", 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.history", 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:9", 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:9", 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:9", 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000070", 2023-09-29T22:15:01.159 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000104", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:0", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000026", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:3", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:7", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.1.retry", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:2", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:6", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:9", 2023-09-29T22:15:01.160 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16", 2023-09-29T22:15:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.8", 2023-09-29T22:15:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000028", 2023-09-29T22:15:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:7", 2023-09-29T22:15:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:8", 2023-09-29T22:15:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:1", 2023-09-29T22:15:01.161 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:2", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:4", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1.wmTAQRDpBJ+.0", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:0", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:4", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000040", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:7", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:9", 2023-09-29T22:15:01.162 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:0", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.6", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:0", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:3", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:5", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:7", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000015", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:5", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000069", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000095", 2023-09-29T22:15:01.163 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000003", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000047", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:6", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.1", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:7", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:5", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000035", 2023-09-29T22:15:01.164 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000037", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:1", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:6", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000024", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:0", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000000", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000031", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:5", 2023-09-29T22:15:01.165 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:0", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:3", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:8", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:8", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:0", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:6", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "data_loggenerations_metadata", 2023-09-29T22:15:01.166 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000076", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000113", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:2", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000005", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000011", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000058", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000052", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:10", 2023-09-29T22:15:01.167 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:10", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000088", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000080", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000090", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:10", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:2", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000110", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:3", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000096", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:6", 2023-09-29T22:15:01.168 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:2", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:6", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:2", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000087", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:2", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:1", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000008", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8", 2023-09-29T22:15:01.169 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000006", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000029", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:0", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000089", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000022", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:1", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:10", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:5", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:10", 2023-09-29T22:15:01.170 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:2", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog.trim", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000016", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:4", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:10", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000041", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:6", 2023-09-29T22:15:01.171 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000018", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000092", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.4", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:6", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:1", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000014", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000112", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000007", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000021", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:5", 2023-09-29T22:15:01.172 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000064", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000071", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:5", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:2", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000074", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:6", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000081", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000009", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000121", 2023-09-29T22:15:01.173 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:4", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000125", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000082", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:9", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:1", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:9", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000105", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000059", 2023-09-29T22:15:01.174 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:4", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:3", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:2", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:10", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000077", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000032", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:10", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:9", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:2", 2023-09-29T22:15:01.175 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000053", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000091", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:3", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:3", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:7", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000065", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:3", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000083", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000010", 2023-09-29T22:15:01.176 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.2", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.7", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.3", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2.ye9KGP7RuQj.0", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:1", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:9", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:6", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:5", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:8", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000045", 2023-09-29T22:15:01.177 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000002", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:7", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000116", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:9", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:8", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:0", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000034", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:3", 2023-09-29T22:15:01.178 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:9", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000101", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0.HxjUuT3Xj64.0", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:8", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:3", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:9", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000079", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000049", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000117", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:6", 2023-09-29T22:15:01.179 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000044", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:0", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:7", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000066", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:4", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000068", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000085", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:10", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000073", 2023-09-29T22:15:01.180 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000038", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:4", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:9", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:3", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:5", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000118", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:5", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000036", 2023-09-29T22:15:01.181 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:0", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:4", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000103", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:0", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:6", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:5", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000119", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:1", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000098", 2023-09-29T22:15:01.182 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:10", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000027", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000019", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:8", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000039", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000100", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:10", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000093", 2023-09-29T22:15:01.183 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:2", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:7", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:4", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:1", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.3", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000004", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000063", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000122", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:2", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000057", 2023-09-29T22:15:01.184 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:8", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000054", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000114", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.2", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:7", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3.J39ibWIUtFz.0", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:6", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:1", 2023-09-29T22:15:01.185 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000012", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:7", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000084", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000043", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:1", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:4", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:0", 2023-09-29T22:15:01.186 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.b4e33480-bfb8-4364-b1d7-649c06bc8c56.0", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:9", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:1", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:5", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000111", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000048", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:4", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:8", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000020", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:1", 2023-09-29T22:15:01.187 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:4", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:8", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:10", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000099", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000056", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:0", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:5", 2023-09-29T22:15:01.188 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000072", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000062", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:1", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:5", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000115", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.1", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000033", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:7", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:9", 2023-09-29T22:15:01.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000025", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:2", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000123", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000108", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000094", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000060", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:7", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.0.retry", 2023-09-29T22:15:01.190 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:5", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000109", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:1", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000013", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:4", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:9", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:9", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000051", 2023-09-29T22:15:01.191 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000106", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:4", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000017", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:8", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:4", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000023", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:6", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:3", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:3", 2023-09-29T22:15:01.192 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000107", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000046", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:10", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:8", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:10", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:10", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:5", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18", 2023-09-29T22:15:01.193 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000067", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:8", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:8", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.9", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000086", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.5", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:1", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:7", 2023-09-29T22:15:01.194 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:10", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:1", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:7", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.0", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:6", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:3", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:8", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:8", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:6", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:4", 2023-09-29T22:15:01.195 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000050", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000102", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:3", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000124", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:4", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000001", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000126", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.272b60f1-3a4d-4009-a171-ab410395ba9a.2.retry", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:5", 2023-09-29T22:15:01.196 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000055", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:2", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:0", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000061", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:7", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:10", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:0", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:6", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000097", 2023-09-29T22:15:01.197 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.b4e33480-bfb8-4364-b1d7-649c06bc8c56.1", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:3", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000042", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:0", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:3", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:8", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:2", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000075", 2023-09-29T22:15:01.198 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000120", 2023-09-29T22:15:01.199 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000030", 2023-09-29T22:15:01.199 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:7", 2023-09-29T22:15:01.199 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:2" 2023-09-29T22:15:01.199 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:15:01.199 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T22:15:01.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.201 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:01.244 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:01.244 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:01.338 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:15:01.338 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000078", 2023-09-29T22:15:01.338 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.history", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:6", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:8", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000070", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:3", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000104", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000026", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:4", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:1", 2023-09-29T22:15:01.339 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:9", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.8", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000028", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:8", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:10", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:7", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:1", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:7", 2023-09-29T22:15:01.340 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000040", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.6", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:9", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:7", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:1", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:2", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000015", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000069", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000095", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000003", 2023-09-29T22:15:01.341 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000047", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.2", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:3", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:8", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:10", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000035", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000037", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:0", 2023-09-29T22:15:01.342 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:9", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:8", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:7", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:2", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000024", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000000", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:2", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000031", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:2", 2023-09-29T22:15:01.343 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:8", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.3", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:2", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:4", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1.XhlWSn3tYXk.0", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:2", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:2", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "data_loggenerations_metadata", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:1", 2023-09-29T22:15:01.344 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:6", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:7", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:8", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:10", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:7", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000076", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000113", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:0", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000005", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000011", 2023-09-29T22:15:01.345 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000058", 2023-09-29T22:15:01.346 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000052", 2023-09-29T22:15:01.346 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:8", 2023-09-29T22:15:01.346 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:0", 2023-09-29T22:15:01.346 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7", 2023-09-29T22:15:01.346 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:9", 2023-09-29T22:15:01.346 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:9", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:7", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:6", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:4", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000088", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:4", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000080", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000090", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000110", 2023-09-29T22:15:01.347 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:2", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000096", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000087", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000008", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:2", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000006", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000029", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000089", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:1", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000022", 2023-09-29T22:15:01.348 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:5", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:5", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:9", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:6", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:9", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog.trim", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000016", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:9", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:8", 2023-09-29T22:15:01.349 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000041", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:1", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000018", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000092", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.2.retry", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.4", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:6", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:10", 2023-09-29T22:15:01.350 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000014", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000112", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000007", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000021", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:6", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:1", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:8", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.16", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:0", 2023-09-29T22:15:01.351 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:10", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000064", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:6", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000071", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000074", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:8", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:5", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:9", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000081", 2023-09-29T22:15:01.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000009", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000121", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.1", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:7", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:3", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0.eYgEAQAZRtY.0", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.17", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000125", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000082", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:6", 2023-09-29T22:15:01.353 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:6", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:1", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:10", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000105", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000059", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:6", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000077", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000032", 2023-09-29T22:15:01.354 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:6", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:0", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.1", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:2", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:9", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000053", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000091", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000065", 2023-09-29T22:15:01.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000083", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000010", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:1", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.7", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.3", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:9", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:3", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.0.retry", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:1", 2023-09-29T22:15:01.356 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:3", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:9", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.2", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:4", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:6", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:4", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:7", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000045", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000002", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.12", 2023-09-29T22:15:01.357 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000116", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000034", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000101", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.13", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.1", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:8", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:8", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000079", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000049", 2023-09-29T22:15:01.358 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000117", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000044", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:5", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:4", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.14", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000066", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:2", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000068", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000085", 2023-09-29T22:15:01.359 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000073", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:9", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000038", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:5", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:3", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.10", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000118", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000036", 2023-09-29T22:15:01.360 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:0", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:5", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.3", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000103", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:7", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.3", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:3", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:5", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:10", 2023-09-29T22:15:01.361 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000119", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:5", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:6", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000098", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:7", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status.shard.0", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:3", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:5", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000027", 2023-09-29T22:15:01.362 INFO:teuthology.orchestra.run.smithi112.stdout: "mdlog.sync-status", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000019", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:7", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:8", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000039", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:0", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:3", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:10", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000100", 2023-09-29T22:15:01.363 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:5", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:0", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:7", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:0", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:10", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000093", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:10", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:10", 2023-09-29T22:15:01.364 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000004", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000063", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000122", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000057", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:7", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:3", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000054", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000114", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.2", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:7", 2023-09-29T22:15:01.365 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.0", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:1", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000012", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000084", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000043", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:1", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.11", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.b4e33480-bfb8-4364-b1d7-649c06bc8c56.0", 2023-09-29T22:15:01.366 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:10", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2.m/2efXPggJQ.0", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:2", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:0", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000111", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000048", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:0", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000020", 2023-09-29T22:15:01.367 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:5", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:2", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:9", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000099", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000056", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:4", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:2", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:8", 2023-09-29T22:15:01.368 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000072", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:4", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000062", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000115", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.1", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000033", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:6", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.0", 2023-09-29T22:15:01.369 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:10", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000025", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000123", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:4", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000108", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000094", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000060", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14", 2023-09-29T22:15:01.370 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000109", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.15", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:0", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000013", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:7", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:3", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:1", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:10", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:5", 2023-09-29T22:15:01.371 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:8", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:3", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000051", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000106", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:10", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000017", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-14:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.14:3", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:2", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-34:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.34:7", 2023-09-29T22:15:01.372 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000023", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:0", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000107", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000046", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-10:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.10:9", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:1", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:1", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:0", 2023-09-29T22:15:01.373 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:8", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:3", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "data_log.2", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000067", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:4", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:9", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-17:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.17:2", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:4", 2023-09-29T22:15:01.374 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.9", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000086", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:3", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "sync.error-log.5", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:5", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:6", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:10", 2023-09-29T22:15:01.375 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:5", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:3", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:1", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:6", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:6", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-18:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.18:4", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-35:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5371.1:1", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9", 2023-09-29T22:15:01.377 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:4", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000050", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000102", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000124", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:5", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000001", 2023-09-29T22:15:01.378 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:0", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000126", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000055", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-8:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.8:0", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000061", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:5", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-13:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.13:8", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.shard.a75fe7d4-5635-4f57-8059-26c1aacd3a0f.1.retry", 2023-09-29T22:15:01.379 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000097", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "datalog.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "meta.log.b4e33480-bfb8-4364-b1d7-649c06bc8c56.1", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-31:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.31:9", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-32:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.32:4", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-16:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.16:5", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000042", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-12:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.12:10", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-11:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.11:4", 2023-09-29T22:15:01.380 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-15:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.15:0", 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-33:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.33:2", 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000075", 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000120", 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: "obj_delete_at_hint.0000000030", 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-7:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.7:4", 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-9:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.9:3" 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:15:01.381 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-09-29T22:15:01.382 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:15:01.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.383 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:01.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:01.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:01.559 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:15:01.559 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.559 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004449", 2023-09-29T22:15:01.559 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:15:00.980951Z" 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003745", 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:15:00.983231Z" 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.560 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:15:00.984954Z" 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000933", 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:15:00.978970Z" 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:01.561 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:15:01.562 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003745', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-09-29T22:15:01.562 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:15:01.563 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:01.563 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:01.608 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:01.609 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:01.692 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:15:01.692 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:15:01.692 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:15:01.692 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:15:01.692 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:15:01.692 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:01.693 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:51.973838Z" 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:01.694 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:10:17.934408Z" 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:15:01.695 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:50.894391Z" 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:01.696 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:01.697 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:15:01Z" 2023-09-29T22:15:01.698 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:15:01.699 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:51.973838Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:10:17.934408Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:50.894391Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:15:01Z" } 2023-09-29T22:15:01.700 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:51.973838Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:10:17.934408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:50.894391Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:15:01.700 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003745', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-09-29T22:15:01.700 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000002903', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:15:01.700 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000004449 target=00000000000000000000:00000000000000003606, shard=1 master=00000000000000000000:00000000000000003745 target=00000000000000000000:00000000000000002903, shard=2 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=3 master=00000000000000000000:00000000000000000933 target= 2023-09-29T22:15:06.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:06.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:06.701 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:06.749 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:06.749 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:06.837 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003606", 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:06.838 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:51.973838Z" 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002903", 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:06.839 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:10:17.934408Z" 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:06.840 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:50.894391Z" 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:15:06.841 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:15:06.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:15:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:15:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:15:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:06.843 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:15:06Z" 2023-09-29T22:15:06.843 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:15:06.844 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003606", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:51.973838Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002903", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:10:17.934408Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:50.894391Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:15:06Z" } 2023-09-29T22:15:06.844 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003606', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:51.973838Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002903', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:10:17.934408Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:50.894391Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:15:06.844 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003745', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-09-29T22:15:06.845 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000003606', 1: '00000000000000000000:00000000000000002903', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:15:06.845 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000004449 target=00000000000000000000:00000000000000003606, shard=1 master=00000000000000000000:00000000000000003745 target=00000000000000000000:00000000000000002903, shard=2 master=00000000000000000000:00000000000000002055 target=00000000000000000000:00000000000000001210, shard=3 master=00000000000000000000:00000000000000000933 target= 2023-09-29T22:15:11.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:11.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:11.846 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:11.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:11.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:11.979 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004449", 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:11.980 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:15:00.980951Z" 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000003745", 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:11.981 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:15:00.983231Z" 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002055", 2023-09-29T22:15:11.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:15:00.984954Z" 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:11.983 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000000933", 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:15:00.978970Z" 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:11.984 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:15:11.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:15:11.985 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:15:11.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:11.985 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:15:11Z" 2023-09-29T22:15:11.985 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:15:11.986 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004449", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:00.980951Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000003745", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:00.983231Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002055", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:00.984954Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000000933", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:00.978970Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:15:11Z" } 2023-09-29T22:15:11.986 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004449', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:00.980951Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000003745', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:00.983231Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002055', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:00.984954Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000000933', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:00.978970Z'}}] 2023-09-29T22:15:11.986 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003745', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-09-29T22:15:11.986 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004449', 1: '00000000000000000000:00000000000000003745', 2: '00000000000000000000:00000000000000002055', 3: '00000000000000000000:00000000000000000933'} 2023-09-29T22:15:11.987 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:15:11.987 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:15:11.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:11.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:11.987 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:12.032 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:12.032 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:12.114 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:15:12.115 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.115 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004734", 2023-09-29T22:15:12.115 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:51.924166Z" 2023-09-29T22:15:12.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.116 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002059", 2023-09-29T22:15:12.116 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:10:18.179225Z" 2023-09-29T22:15:12.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.117 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:15:12.117 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:06:50.915176Z" 2023-09-29T22:15:12.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.118 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.118 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:12.119 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:15:12.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.119 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:15:12.120 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000002059', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:15:12.120 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:15:12.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:12.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:12.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:12.167 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:12.167 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:12.250 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:15:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:15:12.250 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.251 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004734", 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:51.924166Z" 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:15:12.252 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002059", 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:10:18.179225Z" 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.253 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001210", 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:06:50.915176Z" 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.255 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000" 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.256 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:15:12Z" 2023-09-29T22:15:12.257 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:15:12.258 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004734", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:51.924166Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002059", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:10:18.179225Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001210", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:06:50.915176Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:15:12Z" } 2023-09-29T22:15:12.258 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004734', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:51.924166Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002059', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:10:18.179225Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001210', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:06:50.915176Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000'}}] 2023-09-29T22:15:12.258 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000002059', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:15:12.258 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000004734', 1: '00000000000000000000:00000000000000002059', 2: '00000000000000000000:00000000000000001210', 3: ''} 2023-09-29T22:15:12.258 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:15:12.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2023-09-29T22:15:12.260 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-46 2023-09-29T22:15:12.260 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:15:12.260 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:15:12.261 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.261 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:15:12.261 DEBUG:boto:Data: 2023-09-29T22:15:12.261 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.261 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.261 DEBUG:boto:Params: {} 2023-09-29T22:15:12.261 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:15:12.262 DEBUG:boto:Token: None 2023-09-29T22:15:12.262 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/ 2023-09-29T22:15:12.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UcLnuBAAARc8LzkAe2eIVwbDm8g= 2023-09-29T22:15:12.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UcLnuBAAARc8LzkAe2eIVwbDm8g=', 'Content-Length': '0'} 2023-09-29T22:15:12.292 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000016590dbc07c584ea-0065174c70-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.292 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-47 2023-09-29T22:15:12.293 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:15:12.293 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:15:12.293 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.293 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:15:12.293 DEBUG:boto:Data: 2023-09-29T22:15:12.293 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.293 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.293 DEBUG:boto:Params: {} 2023-09-29T22:15:12.294 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:15:12.294 DEBUG:boto:Token: None 2023-09-29T22:15:12.294 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/ 2023-09-29T22:15:12.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DMW0HlQQnpUs/vboAfuYS9vr2h8= 2023-09-29T22:15:12.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DMW0HlQQnpUs/vboAfuYS9vr2h8=', 'Content-Length': '0'} 2023-09-29T22:15:12.339 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c1a230d9f7e4c279-0065174c70-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.340 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:15:12.340 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:15:12.340 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.340 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:15:12.340 DEBUG:boto:Data: 2023-09-29T22:15:12.340 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.340 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.341 DEBUG:boto:Params: {} 2023-09-29T22:15:12.341 DEBUG:boto:Token: None 2023-09-29T22:15:12.341 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/ 2023-09-29T22:15:12.341 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU= 2023-09-29T22:15:12.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU=', 'Content-Length': '0'} 2023-09-29T22:15:12.344 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000006a8042266af21f3-0065174c70-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.344 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:15:12.344 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:15:12.344 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.345 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:15:12.345 DEBUG:boto:Data: 2023-09-29T22:15:12.345 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.345 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.345 DEBUG:boto:Params: {} 2023-09-29T22:15:12.345 DEBUG:boto:Token: None 2023-09-29T22:15:12.345 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/obj1 2023-09-29T22:15:12.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Sg5RgrCirb1pDbvh9WrvMIQkPI= 2023-09-29T22:15:12.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Sg5RgrCirb1pDbvh9WrvMIQkPI='} 2023-09-29T22:15:12.356 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006247ab42155bf421-0065174c70-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.356 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:15:12.357 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:15:12.357 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.357 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:15:12.357 DEBUG:boto:Data: 2023-09-29T22:15:12.357 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.357 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.357 DEBUG:boto:Params: {} 2023-09-29T22:15:12.357 DEBUG:boto:Token: None 2023-09-29T22:15:12.358 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/ 2023-09-29T22:15:12.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU= 2023-09-29T22:15:12.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU=', 'Content-Length': '0'} 2023-09-29T22:15:12.361 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000393c1c034db6a184-0065174c70-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.362 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:15:12.362 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:15:12.362 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.362 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:15:12.362 DEBUG:boto:Data: 2023-09-29T22:15:12.362 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.363 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.363 DEBUG:boto:Params: {} 2023-09-29T22:15:12.363 DEBUG:boto:Token: None 2023-09-29T22:15:12.363 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/obj2 2023-09-29T22:15:12.363 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VIevHg1BUID3avPvb9HP9xsED2w= 2023-09-29T22:15:12.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VIevHg1BUID3avPvb9HP9xsED2w='} 2023-09-29T22:15:12.384 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000095854b066df17c1-0065174c70-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.384 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:15:12.384 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:15:12.384 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.384 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:15:12.384 DEBUG:boto:Data: 2023-09-29T22:15:12.384 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.385 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.385 DEBUG:boto:Params: {} 2023-09-29T22:15:12.385 DEBUG:boto:Token: None 2023-09-29T22:15:12.385 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/ 2023-09-29T22:15:12.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU= 2023-09-29T22:15:12.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU=', 'Content-Length': '0'} 2023-09-29T22:15:12.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a568d7047678bd0-0065174c70-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.388 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:15:12.388 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:15:12.388 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.389 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:15:12.389 DEBUG:boto:Data: 2023-09-29T22:15:12.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.389 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.389 DEBUG:boto:Params: {} 2023-09-29T22:15:12.389 DEBUG:boto:Token: None 2023-09-29T22:15:12.389 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/obj3 2023-09-29T22:15:12.390 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uty1Tj4lK9WbWn5QUIF2DUTCcbM= 2023-09-29T22:15:12.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uty1Tj4lK9WbWn5QUIF2DUTCcbM='} 2023-09-29T22:15:12.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7ac704fd555374f-0065174c70-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.435 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:15:12.435 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:15:12.435 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.435 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:15:12.435 DEBUG:boto:Data: 2023-09-29T22:15:12.435 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.436 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.436 DEBUG:boto:Params: {} 2023-09-29T22:15:12.436 DEBUG:boto:Token: None 2023-09-29T22:15:12.436 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/ 2023-09-29T22:15:12.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU= 2023-09-29T22:15:12.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QmbFpkLFoql9xrmz/t9/VoNl1SU=', 'Content-Length': '0'} 2023-09-29T22:15:12.439 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e51e4d13bc36ff49-0065174c70-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.439 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:15:12.440 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:15:12.440 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.440 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:15:12.440 DEBUG:boto:Data: 2023-09-29T22:15:12.440 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:15:12.440 DEBUG:boto:Port: 8000 2023-09-29T22:15:12.440 DEBUG:boto:Params: {} 2023-09-29T22:15:12.440 DEBUG:boto:Token: None 2023-09-29T22:15:12.441 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-46/obj4 2023-09-29T22:15:12.441 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DXGJsdIoy50EPLQOzfAj+Jf2/sU= 2023-09-29T22:15:12.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DXGJsdIoy50EPLQOzfAj+Jf2/sU='} 2023-09-29T22:15:12.452 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000085583bfdb2366418-0065174c70-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.453 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:15:12.453 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:15:12.453 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.453 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:15:12.453 DEBUG:boto:Data: 2023-09-29T22:15:12.453 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.454 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.454 DEBUG:boto:Params: {} 2023-09-29T22:15:12.454 DEBUG:boto:Token: None 2023-09-29T22:15:12.454 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/ 2023-09-29T22:15:12.454 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A= 2023-09-29T22:15:12.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A=', 'Content-Length': '0'} 2023-09-29T22:15:12.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009023cd7a7c81744d-0065174c70-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.458 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:15:12.458 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:15:12.458 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.458 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:15:12.458 DEBUG:boto:Data: 2023-09-29T22:15:12.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.459 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.459 DEBUG:boto:Params: {} 2023-09-29T22:15:12.459 DEBUG:boto:Token: None 2023-09-29T22:15:12.459 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/obj1 2023-09-29T22:15:12.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lopo8pURJrkJh7IVYMZRQ2ZkfPE= 2023-09-29T22:15:12.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lopo8pURJrkJh7IVYMZRQ2ZkfPE='} 2023-09-29T22:15:12.472 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000133a33c9d753abc3-0065174c70-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.472 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:15:12.473 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:15:12.473 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.473 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:15:12.473 DEBUG:boto:Data: 2023-09-29T22:15:12.473 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.473 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.473 DEBUG:boto:Params: {} 2023-09-29T22:15:12.473 DEBUG:boto:Token: None 2023-09-29T22:15:12.474 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/ 2023-09-29T22:15:12.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A= 2023-09-29T22:15:12.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A=', 'Content-Length': '0'} 2023-09-29T22:15:12.477 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6e2cd00ec7a94aa-0065174c70-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.477 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:15:12.477 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:15:12.478 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.478 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:15:12.478 DEBUG:boto:Data: 2023-09-29T22:15:12.478 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.478 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.478 DEBUG:boto:Params: {} 2023-09-29T22:15:12.479 DEBUG:boto:Token: None 2023-09-29T22:15:12.479 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/obj2 2023-09-29T22:15:12.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnCnXFo9GBit74/zDnDKTVT+4EM= 2023-09-29T22:15:12.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnCnXFo9GBit74/zDnDKTVT+4EM='} 2023-09-29T22:15:12.490 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8cded6b6c080209-0065174c70-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.490 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:15:12.490 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:15:12.491 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.491 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:15:12.491 DEBUG:boto:Data: 2023-09-29T22:15:12.491 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.491 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.492 DEBUG:boto:Params: {} 2023-09-29T22:15:12.492 DEBUG:boto:Token: None 2023-09-29T22:15:12.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/ 2023-09-29T22:15:12.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A= 2023-09-29T22:15:12.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A=', 'Content-Length': '0'} 2023-09-29T22:15:12.495 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000072e7db63dc4b5503-0065174c70-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.496 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:15:12.496 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:15:12.496 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.496 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:15:12.496 DEBUG:boto:Data: 2023-09-29T22:15:12.497 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.497 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.497 DEBUG:boto:Params: {} 2023-09-29T22:15:12.497 DEBUG:boto:Token: None 2023-09-29T22:15:12.497 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/obj3 2023-09-29T22:15:12.498 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yihIMIHDsEVrDdMRXnzfSGLFk2U= 2023-09-29T22:15:12.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yihIMIHDsEVrDdMRXnzfSGLFk2U='} 2023-09-29T22:15:12.542 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da040c32976cce36-0065174c70-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.543 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:15:12.543 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:15:12.543 DEBUG:boto:Method: HEAD 2023-09-29T22:15:12.543 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:15:12.543 DEBUG:boto:Data: 2023-09-29T22:15:12.544 DEBUG:boto:Headers: {} 2023-09-29T22:15:12.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.544 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.544 DEBUG:boto:Params: {} 2023-09-29T22:15:12.544 DEBUG:boto:Token: None 2023-09-29T22:15:12.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/ 2023-09-29T22:15:12.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A= 2023-09-29T22:15:12.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7JX73ZddLJc7WtoSU57mRMUyg1A=', 'Content-Length': '0'} 2023-09-29T22:15:12.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005846dba9c6e4de49-0065174c70-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.548 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:15:12.548 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:15:12.548 DEBUG:boto:Method: PUT 2023-09-29T22:15:12.548 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:15:12.549 DEBUG:boto:Data: 2023-09-29T22:15:12.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:15:12.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:15:12.549 DEBUG:boto:Port: 8001 2023-09-29T22:15:12.549 DEBUG:boto:Params: {} 2023-09-29T22:15:12.549 DEBUG:boto:Token: None 2023-09-29T22:15:12.549 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:15:12 GMT /hcnphz-47/obj4 2023-09-29T22:15:12.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WiCaxBfuZxJbx7xi5UrnFSldtJA= 2023-09-29T22:15:12.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:15:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WiCaxBfuZxJbx7xi5UrnFSldtJA='} 2023-09-29T22:15:12.560 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003000af63b9f351cb-0065174c70-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:15:12.560 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:12.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:12.561 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:12.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:12.606 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:12.664 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:15:12.687 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:15:12.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025712.324858_366.1", 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:15:12.324858Z" 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.689 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:15:12.690 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025712.324858_366.1', 2: '', 3: ''} 2023-09-29T22:15:12.691 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:15:12.691 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:12.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:15:12.691 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:15:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:12.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:12.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:12.791+0000 7f1b65b07f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:12.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:12.791+0000 7f1b65b07f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:15:12.819 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:15:12.819 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.820 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:12.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025712.324858_366.1", 2023-09-29T22:15:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:15:12.324858Z", 2023-09-29T22:15:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:15:12.826 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:15:12.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:15:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:15:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:15:12.828 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:15:12Z" 2023-09-29T22:15:12.828 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:15:12.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025712.324858_366.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:12.324858Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:15:12Z" } 2023-09-29T22:15:12.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025712.324858_366.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:12.324858Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:15:12.829 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025712.324858_366.1', 2: '', 3: ''} 2023-09-29T22:15:12.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025712.324858_366.1', 2: '', 3: ''} 2023-09-29T22:15:12.829 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:15:12.830 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-46 2023-09-29T22:15:12.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:12.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:12.830 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:15:12.923 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:12.924 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:12.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:12.987+0000 7f5b752fef40 1 waiting to reach incremental sync.. 2023-09-29T22:15:17.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:17.987+0000 7f5b752fef40 1 waiting to reach incremental sync.. 2023-09-29T22:15:18.986 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:15:18.980+0000 7efd4dc3b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:15:18.986 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:15:18.980+0000 7efd46c2d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:15:18.987 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:15:18.984+0000 7efd4ec3d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:15:18.987 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:15:18.984+0000 7efd4742e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:15:18.987 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:15:18.984+0000 7efd4b436700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:15:18.987 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:15:18.984+0000 7efd49c33700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:15:22.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:22.991+0000 7f5b752fef40 1 waiting to reach incremental sync.. 2023-09-29T22:15:27.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:27.990+0000 7f5b752fef40 1 waiting to reach incremental sync.. 2023-09-29T22:15:32.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:32.994+0000 7f5b752fef40 1 bucket sync caught up with source: 2023-09-29T22:15:32.997 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.25.6, , 00000000001.23.6, , 00000000001.31.6, , , 00000000001.27.6, , , ] 2023-09-29T22:15:32.997 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.25.6, , 00000000001.23.6, , 00000000001.31.6, , , 00000000001.27.6, , , ] 2023-09-29T22:15:32.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:32.994+0000 7f5b752fef40 0 bucket checkpoint complete 2023-09-29T22:15:33.026 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-46 2023-09-29T22:15:33.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:33.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:15:33.026 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:15:33.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:15:33.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:15:33.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:33.138+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:15:38.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:38.138+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:15:43.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:43.142+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:15:48.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:48.142+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:15:53.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:53.142+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:15:58.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:15:58.142+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:03.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:03.145+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:08.149 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:08.149+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:13.150 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:13.145+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:18.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:18.149+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:23.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:23.153+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:28.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:28.153+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:33.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:33.153+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:38.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:38.153+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:43.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:43.156+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:48.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:48.156+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:53.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:53.156+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:16:58.159 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:16:58.156+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:03.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:03.160+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:08.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:08.160+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:13.163 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:13.160+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:18.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:18.163+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:23.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:23.163+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:28.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:28.163+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:33.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:33.167+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:38.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:38.167+0000 7f40890c7f40 1 waiting to reach incremental sync.. 2023-09-29T22:17:43.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:43.171+0000 7f40890c7f40 1 bucket sync caught up with source: 2023-09-29T22:17:43.173 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.70.6, , 00000000001.71.6, , 00000000001.97.6, , , 00000000001.501.6, , , ] 2023-09-29T22:17:43.173 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.70.6, , 00000000001.71.6, , 00000000001.97.6, , , 00000000001.501.6, , , ] 2023-09-29T22:17:43.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:43.171+0000 7f40890c7f40 0 bucket checkpoint complete 2023-09-29T22:17:43.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-46 zones={a2, a1} 2023-09-29T22:17:43.203 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.203 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.204 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.204 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:17:43.204 DEBUG:boto:Data: 2023-09-29T22:17:43.204 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.204 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.204 DEBUG:boto:Params: {} 2023-09-29T22:17:43.205 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:17:43.205 DEBUG:boto:Token: None 2023-09-29T22:17:43.205 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:040RudCh0iosxFZO/PLgLm1vSEk= 2023-09-29T22:17:43.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:040RudCh0iosxFZO/PLgLm1vSEk=', 'Content-Length': '0'} 2023-09-29T22:17:43.211 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fbf8e8f8352e55ed-0065174d07-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.211 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.211 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.212 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.212 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:17:43.212 DEBUG:boto:Data: 2023-09-29T22:17:43.212 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.212 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.212 DEBUG:boto:Params: {} 2023-09-29T22:17:43.212 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:17:43.213 DEBUG:boto:Token: None 2023-09-29T22:17:43.213 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:040RudCh0iosxFZO/PLgLm1vSEk= 2023-09-29T22:17:43.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:040RudCh0iosxFZO/PLgLm1vSEk=', 'Content-Length': '0'} 2023-09-29T22:17:43.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed38920b05199da4-0065174d07-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.218 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:43.218 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.218 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.218 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.218 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.219 DEBUG:boto:Method: GET 2023-09-29T22:17:43.219 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.219 DEBUG:boto:Data: 2023-09-29T22:17:43.219 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.219 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.219 DEBUG:boto:Params: {} 2023-09-29T22:17:43.219 DEBUG:boto:Token: None 2023-09-29T22:17:43.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/?versions 2023-09-29T22:17:43.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c= 2023-09-29T22:17:43.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c=', 'Content-Length': '0'} 2023-09-29T22:17:43.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd07eebab37888bc-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.224 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:17:43.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:17:43.225 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:17:43.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:17:43.226 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:43.226 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.226 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.226 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.226 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.226 DEBUG:boto:Method: GET 2023-09-29T22:17:43.226 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.226 DEBUG:boto:Data: 2023-09-29T22:17:43.227 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.227 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.227 DEBUG:boto:Params: {} 2023-09-29T22:17:43.227 DEBUG:boto:Token: None 2023-09-29T22:17:43.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/?versions 2023-09-29T22:17:43.227 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c= 2023-09-29T22:17:43.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c=', 'Content-Length': '0'} 2023-09-29T22:17:43.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000413734a219e7738-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.232 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:17:43.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:17:43.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:17:43.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:17:43.232 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.233 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.233 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.233 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.233 DEBUG:boto:Method: GET 2023-09-29T22:17:43.233 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.233 DEBUG:boto:Data: 2023-09-29T22:17:43.233 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.233 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.234 DEBUG:boto:Params: {} 2023-09-29T22:17:43.234 DEBUG:boto:Token: None 2023-09-29T22:17:43.234 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/?versions 2023-09-29T22:17:43.234 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c= 2023-09-29T22:17:43.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c=', 'Content-Length': '0'} 2023-09-29T22:17:43.238 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f0fa20bba71bf73-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.238 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.238 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.239 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.239 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.239 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.239 DEBUG:boto:Method: GET 2023-09-29T22:17:43.239 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:17:43.239 DEBUG:boto:Data: 2023-09-29T22:17:43.239 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.240 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.240 DEBUG:boto:Params: {} 2023-09-29T22:17:43.240 DEBUG:boto:Token: None 2023-09-29T22:17:43.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/?versions 2023-09-29T22:17:43.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c= 2023-09-29T22:17:43.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zv4so8xPH8ioxpdEIBwujPqGE4c=', 'Content-Length': '0'} 2023-09-29T22:17:43.244 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000359e651f8d8913e6-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.244 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:17:43.245 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.245 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.245 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.245 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.245 DEBUG:boto:Method: GET 2023-09-29T22:17:43.245 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.245 DEBUG:boto:Data: 2023-09-29T22:17:43.245 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.245 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.246 DEBUG:boto:Params: {} 2023-09-29T22:17:43.246 DEBUG:boto:Token: None 2023-09-29T22:17:43.246 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE= 2023-09-29T22:17:43.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE=', 'Content-Length': '0'} 2023-09-29T22:17:43.248 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030304fc726ca993f-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.249 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.249 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.249 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.249 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.249 DEBUG:boto:Method: GET 2023-09-29T22:17:43.249 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.250 DEBUG:boto:Data: 2023-09-29T22:17:43.250 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.250 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.250 DEBUG:boto:Params: {} 2023-09-29T22:17:43.250 DEBUG:boto:Token: None 2023-09-29T22:17:43.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE= 2023-09-29T22:17:43.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE=', 'Content-Length': '0'} 2023-09-29T22:17:43.253 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002efc8fea52a74cc7-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.253 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.253 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.253 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.253 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.254 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.254 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.254 DEBUG:boto:Data: 2023-09-29T22:17:43.254 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.254 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.254 DEBUG:boto:Params: {} 2023-09-29T22:17:43.254 DEBUG:boto:Token: None 2023-09-29T22:17:43.254 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.255 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8WMg8z1473uWn3dIlkDnxjICTdA= 2023-09-29T22:17:43.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8WMg8z1473uWn3dIlkDnxjICTdA=', 'Content-Length': '0'} 2023-09-29T22:17:43.257 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005cdb59d31660977b-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.257 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.258 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.258 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.258 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.258 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.258 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.258 DEBUG:boto:Data: 2023-09-29T22:17:43.258 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.259 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.259 DEBUG:boto:Params: {} 2023-09-29T22:17:43.259 DEBUG:boto:Token: None 2023-09-29T22:17:43.259 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.259 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8WMg8z1473uWn3dIlkDnxjICTdA= 2023-09-29T22:17:43.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8WMg8z1473uWn3dIlkDnxjICTdA=', 'Content-Length': '0'} 2023-09-29T22:17:43.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000050c2696454223dfc-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:17:43.262 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.262 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.262 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.262 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.263 DEBUG:boto:Method: GET 2023-09-29T22:17:43.263 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.263 DEBUG:boto:Data: 2023-09-29T22:17:43.263 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.263 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.263 DEBUG:boto:Params: {} 2023-09-29T22:17:43.263 DEBUG:boto:Token: None 2023-09-29T22:17:43.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE= 2023-09-29T22:17:43.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE=', 'Content-Length': '0'} 2023-09-29T22:17:43.266 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079b66e7b8946e749-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.267 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.267 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.267 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.267 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.267 DEBUG:boto:Method: GET 2023-09-29T22:17:43.267 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.267 DEBUG:boto:Data: 2023-09-29T22:17:43.267 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.268 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.268 DEBUG:boto:Params: {} 2023-09-29T22:17:43.268 DEBUG:boto:Token: None 2023-09-29T22:17:43.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:17:43.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE= 2023-09-29T22:17:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iu9wEvFPlOVJPpXG4G/rGy/Y4ZE=', 'Content-Length': '0'} 2023-09-29T22:17:43.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000012618c7d3b172948-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.272 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.272 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.272 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.272 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.272 DEBUG:boto:Method: GET 2023-09-29T22:17:43.272 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.272 DEBUG:boto:Data: 2023-09-29T22:17:43.272 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.273 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.273 DEBUG:boto:Params: {} 2023-09-29T22:17:43.273 DEBUG:boto:Token: None 2023-09-29T22:17:43.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094401fdaaad86844-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.326 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.327 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.327 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.327 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.327 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.327 DEBUG:boto:Method: GET 2023-09-29T22:17:43.328 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.328 DEBUG:boto:Data: 2023-09-29T22:17:43.328 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.328 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.328 DEBUG:boto:Params: {} 2023-09-29T22:17:43.328 DEBUG:boto:Token: None 2023-09-29T22:17:43.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c3da5ce55ac5076-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.334 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.335 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.335 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.335 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.335 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:17:43.335 DEBUG:boto:Data: 2023-09-29T22:17:43.335 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.335 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.336 DEBUG:boto:Params: {} 2023-09-29T22:17:43.336 DEBUG:boto:Token: None 2023-09-29T22:17:43.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1 2023-09-29T22:17:43.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PlUxgccpZ/ryjoJBLV7GR5DwCso= 2023-09-29T22:17:43.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PlUxgccpZ/ryjoJBLV7GR5DwCso=', 'Content-Length': '0'} 2023-09-29T22:17:43.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e83cee0e4f71ba65-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.339 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.339 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.339 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.339 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:17:43.339 DEBUG:boto:Data: 2023-09-29T22:17:43.339 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.339 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.340 DEBUG:boto:Params: {} 2023-09-29T22:17:43.340 DEBUG:boto:Token: None 2023-09-29T22:17:43.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1 2023-09-29T22:17:43.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PlUxgccpZ/ryjoJBLV7GR5DwCso= 2023-09-29T22:17:43.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PlUxgccpZ/ryjoJBLV7GR5DwCso=', 'Content-Length': '0'} 2023-09-29T22:17:43.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000389443e2b99985f3-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:17:43.343 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.343 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.343 DEBUG:boto:Method: GET 2023-09-29T22:17:43.343 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:17:43.343 DEBUG:boto:Data: 2023-09-29T22:17:43.344 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.344 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.344 DEBUG:boto:Params: {} 2023-09-29T22:17:43.344 DEBUG:boto:Token: None 2023-09-29T22:17:43.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1 2023-09-29T22:17:43.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8KhLHICZFYC2SFjSylcOQXktqK0= 2023-09-29T22:17:43.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8KhLHICZFYC2SFjSylcOQXktqK0=', 'Content-Length': '0'} 2023-09-29T22:17:43.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc743b6293fff42d-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.347 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:17:43.347 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:17:43.347 DEBUG:boto:Method: GET 2023-09-29T22:17:43.347 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:17:43.347 DEBUG:boto:Data: 2023-09-29T22:17:43.348 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.348 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.348 DEBUG:boto:Params: {} 2023-09-29T22:17:43.348 DEBUG:boto:Token: None 2023-09-29T22:17:43.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj1 2023-09-29T22:17:43.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8KhLHICZFYC2SFjSylcOQXktqK0= 2023-09-29T22:17:43.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8KhLHICZFYC2SFjSylcOQXktqK0=', 'Content-Length': '0'} 2023-09-29T22:17:43.351 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bf09a855a9f17c01-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.394 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.395 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.395 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.395 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.395 DEBUG:boto:Method: GET 2023-09-29T22:17:43.395 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.395 DEBUG:boto:Data: 2023-09-29T22:17:43.395 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.395 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.396 DEBUG:boto:Params: {} 2023-09-29T22:17:43.396 DEBUG:boto:Token: None 2023-09-29T22:17:43.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.401 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016a8a69dda035c90-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.401 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.401 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.401 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.401 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.402 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.402 DEBUG:boto:Method: GET 2023-09-29T22:17:43.402 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:17:43.402 DEBUG:boto:Data: 2023-09-29T22:17:43.402 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.402 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.402 DEBUG:boto:Params: {} 2023-09-29T22:17:43.402 DEBUG:boto:Token: None 2023-09-29T22:17:43.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.403 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aaad179de92169ee-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.407 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:17:43.408 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.408 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.408 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.408 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.408 DEBUG:boto:Method: GET 2023-09-29T22:17:43.408 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.408 DEBUG:boto:Data: 2023-09-29T22:17:43.408 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.408 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.409 DEBUG:boto:Params: {} 2023-09-29T22:17:43.409 DEBUG:boto:Token: None 2023-09-29T22:17:43.409 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk= 2023-09-29T22:17:43.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk=', 'Content-Length': '0'} 2023-09-29T22:17:43.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa14004d65f27a8c-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.454 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.455 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.455 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.455 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.455 DEBUG:boto:Method: GET 2023-09-29T22:17:43.455 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.455 DEBUG:boto:Data: 2023-09-29T22:17:43.455 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.455 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.455 DEBUG:boto:Params: {} 2023-09-29T22:17:43.456 DEBUG:boto:Token: None 2023-09-29T22:17:43.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk= 2023-09-29T22:17:43.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk=', 'Content-Length': '0'} 2023-09-29T22:17:43.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008a172ddf472eb8a5-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.459 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.459 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.459 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.459 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.459 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.460 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.460 DEBUG:boto:Data: 2023-09-29T22:17:43.460 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.460 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.460 DEBUG:boto:Params: {} 2023-09-29T22:17:43.460 DEBUG:boto:Token: None 2023-09-29T22:17:43.460 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LYfl2ysrMP8pNfW6OeeMFynJ78s= 2023-09-29T22:17:43.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LYfl2ysrMP8pNfW6OeeMFynJ78s=', 'Content-Length': '0'} 2023-09-29T22:17:43.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb38dedab87d4989-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.463 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.463 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.463 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.464 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.464 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.464 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.464 DEBUG:boto:Data: 2023-09-29T22:17:43.464 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.464 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.464 DEBUG:boto:Params: {} 2023-09-29T22:17:43.464 DEBUG:boto:Token: None 2023-09-29T22:17:43.464 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LYfl2ysrMP8pNfW6OeeMFynJ78s= 2023-09-29T22:17:43.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LYfl2ysrMP8pNfW6OeeMFynJ78s=', 'Content-Length': '0'} 2023-09-29T22:17:43.467 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000acc9764989c68c2b-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:17:43.468 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.468 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.468 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.468 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.468 DEBUG:boto:Method: GET 2023-09-29T22:17:43.468 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.468 DEBUG:boto:Data: 2023-09-29T22:17:43.468 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.469 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.469 DEBUG:boto:Params: {} 2023-09-29T22:17:43.469 DEBUG:boto:Token: None 2023-09-29T22:17:43.469 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.469 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk= 2023-09-29T22:17:43.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk=', 'Content-Length': '0'} 2023-09-29T22:17:43.472 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000611bf28c5f336c51-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.472 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.472 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.472 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.472 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.472 DEBUG:boto:Method: GET 2023-09-29T22:17:43.473 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.473 DEBUG:boto:Data: 2023-09-29T22:17:43.473 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.473 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.473 DEBUG:boto:Params: {} 2023-09-29T22:17:43.473 DEBUG:boto:Token: None 2023-09-29T22:17:43.473 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:17:43.473 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk= 2023-09-29T22:17:43.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KT1A8FuBYrXdHsbXGHoAN/phBvk=', 'Content-Length': '0'} 2023-09-29T22:17:43.476 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000073c4dbaeb353cc94-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.518 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.519 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.519 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.519 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.519 DEBUG:boto:Method: GET 2023-09-29T22:17:43.519 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.519 DEBUG:boto:Data: 2023-09-29T22:17:43.519 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.519 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.519 DEBUG:boto:Params: {} 2023-09-29T22:17:43.520 DEBUG:boto:Token: None 2023-09-29T22:17:43.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000055a62794c84a91d-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.525 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.525 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.525 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.525 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.526 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.526 DEBUG:boto:Method: GET 2023-09-29T22:17:43.526 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.526 DEBUG:boto:Data: 2023-09-29T22:17:43.526 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.526 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.526 DEBUG:boto:Params: {} 2023-09-29T22:17:43.526 DEBUG:boto:Token: None 2023-09-29T22:17:43.527 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035dafeda21b7f38d-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.532 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.532 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.532 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.532 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.533 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:17:43.533 DEBUG:boto:Data: 2023-09-29T22:17:43.533 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.533 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.533 DEBUG:boto:Params: {} 2023-09-29T22:17:43.533 DEBUG:boto:Token: None 2023-09-29T22:17:43.533 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2 2023-09-29T22:17:43.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wmJ9FKFfUNq7qADrfLcVkeksNAc= 2023-09-29T22:17:43.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wmJ9FKFfUNq7qADrfLcVkeksNAc=', 'Content-Length': '0'} 2023-09-29T22:17:43.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c19fae81ae780d07-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.536 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.536 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.536 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.536 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:17:43.536 DEBUG:boto:Data: 2023-09-29T22:17:43.536 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.537 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.537 DEBUG:boto:Params: {} 2023-09-29T22:17:43.537 DEBUG:boto:Token: None 2023-09-29T22:17:43.537 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2 2023-09-29T22:17:43.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wmJ9FKFfUNq7qADrfLcVkeksNAc= 2023-09-29T22:17:43.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wmJ9FKFfUNq7qADrfLcVkeksNAc=', 'Content-Length': '0'} 2023-09-29T22:17:43.540 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eb03173703c99e18-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:17:43.540 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.540 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.540 DEBUG:boto:Method: GET 2023-09-29T22:17:43.540 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:17:43.541 DEBUG:boto:Data: 2023-09-29T22:17:43.541 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.541 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.541 DEBUG:boto:Params: {} 2023-09-29T22:17:43.541 DEBUG:boto:Token: None 2023-09-29T22:17:43.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2 2023-09-29T22:17:43.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nx9uYAkN5aUYbsriH5pEgKEXIqA= 2023-09-29T22:17:43.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nx9uYAkN5aUYbsriH5pEgKEXIqA=', 'Content-Length': '0'} 2023-09-29T22:17:43.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008542dc183bf63b2b-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.590 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:17:43.590 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:17:43.591 DEBUG:boto:Method: GET 2023-09-29T22:17:43.591 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:17:43.591 DEBUG:boto:Data: 2023-09-29T22:17:43.591 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.591 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.591 DEBUG:boto:Params: {} 2023-09-29T22:17:43.591 DEBUG:boto:Token: None 2023-09-29T22:17:43.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj2 2023-09-29T22:17:43.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nx9uYAkN5aUYbsriH5pEgKEXIqA= 2023-09-29T22:17:43.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nx9uYAkN5aUYbsriH5pEgKEXIqA=', 'Content-Length': '0'} 2023-09-29T22:17:43.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002f84bffe2a9f2d75-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.594 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.595 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.595 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.595 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.595 DEBUG:boto:Method: GET 2023-09-29T22:17:43.595 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.595 DEBUG:boto:Data: 2023-09-29T22:17:43.595 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.595 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.595 DEBUG:boto:Params: {} 2023-09-29T22:17:43.596 DEBUG:boto:Token: None 2023-09-29T22:17:43.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a4f960d14cbcc75-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.601 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.602 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.602 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.602 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.602 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.602 DEBUG:boto:Method: GET 2023-09-29T22:17:43.602 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:17:43.602 DEBUG:boto:Data: 2023-09-29T22:17:43.602 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.603 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.603 DEBUG:boto:Params: {} 2023-09-29T22:17:43.603 DEBUG:boto:Token: None 2023-09-29T22:17:43.603 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001333403946ae5277-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.650 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:17:43.651 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.651 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.651 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.651 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.651 DEBUG:boto:Method: GET 2023-09-29T22:17:43.651 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.651 DEBUG:boto:Data: 2023-09-29T22:17:43.651 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.652 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.652 DEBUG:boto:Params: {} 2023-09-29T22:17:43.652 DEBUG:boto:Token: None 2023-09-29T22:17:43.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4= 2023-09-29T22:17:43.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4=', 'Content-Length': '0'} 2023-09-29T22:17:43.655 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000881715f4a6eacd4f-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.655 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.656 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.656 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.656 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.656 DEBUG:boto:Method: GET 2023-09-29T22:17:43.656 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.656 DEBUG:boto:Data: 2023-09-29T22:17:43.656 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.656 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.656 DEBUG:boto:Params: {} 2023-09-29T22:17:43.657 DEBUG:boto:Token: None 2023-09-29T22:17:43.657 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4= 2023-09-29T22:17:43.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4=', 'Content-Length': '0'} 2023-09-29T22:17:43.660 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cf94b3ba98f7c8d2-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.660 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.660 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.660 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.660 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.660 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.661 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.661 DEBUG:boto:Data: 2023-09-29T22:17:43.661 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.661 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.661 DEBUG:boto:Params: {} 2023-09-29T22:17:43.661 DEBUG:boto:Token: None 2023-09-29T22:17:43.661 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LTOwDSlpadex6qVO6B6DC+7QOwo= 2023-09-29T22:17:43.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LTOwDSlpadex6qVO6B6DC+7QOwo=', 'Content-Length': '0'} 2023-09-29T22:17:43.664 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2624d04bdd6448d-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.664 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.664 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.665 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.665 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.665 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.665 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.665 DEBUG:boto:Data: 2023-09-29T22:17:43.665 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.665 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.665 DEBUG:boto:Params: {} 2023-09-29T22:17:43.665 DEBUG:boto:Token: None 2023-09-29T22:17:43.666 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LTOwDSlpadex6qVO6B6DC+7QOwo= 2023-09-29T22:17:43.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LTOwDSlpadex6qVO6B6DC+7QOwo=', 'Content-Length': '0'} 2023-09-29T22:17:43.668 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000de01605560b50d6c-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:17:43.669 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.669 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.669 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.669 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.669 DEBUG:boto:Method: GET 2023-09-29T22:17:43.669 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.669 DEBUG:boto:Data: 2023-09-29T22:17:43.669 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.670 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.670 DEBUG:boto:Params: {} 2023-09-29T22:17:43.670 DEBUG:boto:Token: None 2023-09-29T22:17:43.670 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4= 2023-09-29T22:17:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4=', 'Content-Length': '0'} 2023-09-29T22:17:43.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003fed1bcd4fc4f6c5-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.714 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.714 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.715 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.715 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.715 DEBUG:boto:Method: GET 2023-09-29T22:17:43.715 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.715 DEBUG:boto:Data: 2023-09-29T22:17:43.715 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.715 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.716 DEBUG:boto:Params: {} 2023-09-29T22:17:43.716 DEBUG:boto:Token: None 2023-09-29T22:17:43.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:17:43.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4= 2023-09-29T22:17:43.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bNi09JR5W+EbbSvthvBOFF+4bM4=', 'Content-Length': '0'} 2023-09-29T22:17:43.719 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009eb215ce80a56783-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.719 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.720 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.720 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.720 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.720 DEBUG:boto:Method: GET 2023-09-29T22:17:43.720 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.720 DEBUG:boto:Data: 2023-09-29T22:17:43.720 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.721 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.721 DEBUG:boto:Params: {} 2023-09-29T22:17:43.721 DEBUG:boto:Token: None 2023-09-29T22:17:43.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.726 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000846373eda9145e7d-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.727 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.727 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.727 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.727 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.728 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.728 DEBUG:boto:Method: GET 2023-09-29T22:17:43.728 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.728 DEBUG:boto:Data: 2023-09-29T22:17:43.728 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.729 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.729 DEBUG:boto:Params: {} 2023-09-29T22:17:43.729 DEBUG:boto:Token: None 2023-09-29T22:17:43.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e52c14e4cd72b4c1-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.778 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.779 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.779 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.779 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.779 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:17:43.779 DEBUG:boto:Data: 2023-09-29T22:17:43.779 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.780 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.780 DEBUG:boto:Params: {} 2023-09-29T22:17:43.780 DEBUG:boto:Token: None 2023-09-29T22:17:43.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3 2023-09-29T22:17:43.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C6Sf4Btkouc3Tmk4D8DX5sZWTI4= 2023-09-29T22:17:43.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C6Sf4Btkouc3Tmk4D8DX5sZWTI4=', 'Content-Length': '0'} 2023-09-29T22:17:43.783 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000967b22ae02597631-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.784 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.784 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.784 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.784 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:17:43.784 DEBUG:boto:Data: 2023-09-29T22:17:43.784 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.784 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.785 DEBUG:boto:Params: {} 2023-09-29T22:17:43.785 DEBUG:boto:Token: None 2023-09-29T22:17:43.785 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3 2023-09-29T22:17:43.785 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C6Sf4Btkouc3Tmk4D8DX5sZWTI4= 2023-09-29T22:17:43.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C6Sf4Btkouc3Tmk4D8DX5sZWTI4=', 'Content-Length': '0'} 2023-09-29T22:17:43.788 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd74ff54427d2d83-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:17:43.788 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.788 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.788 DEBUG:boto:Method: GET 2023-09-29T22:17:43.789 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:17:43.789 DEBUG:boto:Data: 2023-09-29T22:17:43.789 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.789 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.789 DEBUG:boto:Params: {} 2023-09-29T22:17:43.789 DEBUG:boto:Token: None 2023-09-29T22:17:43.789 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3 2023-09-29T22:17:43.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iYO/yhzazfixfCzGPXqYr90XUlc= 2023-09-29T22:17:43.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iYO/yhzazfixfCzGPXqYr90XUlc=', 'Content-Length': '0'} 2023-09-29T22:17:43.792 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000df0faa567eb4f5cb-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.793 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:17:43.793 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:17:43.793 DEBUG:boto:Method: GET 2023-09-29T22:17:43.793 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:17:43.793 DEBUG:boto:Data: 2023-09-29T22:17:43.793 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.794 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.794 DEBUG:boto:Params: {} 2023-09-29T22:17:43.794 DEBUG:boto:Token: None 2023-09-29T22:17:43.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj3 2023-09-29T22:17:43.794 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iYO/yhzazfixfCzGPXqYr90XUlc= 2023-09-29T22:17:43.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iYO/yhzazfixfCzGPXqYr90XUlc=', 'Content-Length': '0'} 2023-09-29T22:17:43.797 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e029e4923c168045-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.798 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.798 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.798 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.798 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.798 DEBUG:boto:Method: GET 2023-09-29T22:17:43.798 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.798 DEBUG:boto:Data: 2023-09-29T22:17:43.798 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.799 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.799 DEBUG:boto:Params: {} 2023-09-29T22:17:43.799 DEBUG:boto:Token: None 2023-09-29T22:17:43.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.799 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f98cbe1d27fbdf23-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.805 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.805 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.805 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.805 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.805 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.806 DEBUG:boto:Method: GET 2023-09-29T22:17:43.806 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:17:43.806 DEBUG:boto:Data: 2023-09-29T22:17:43.806 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.806 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.806 DEBUG:boto:Params: {} 2023-09-29T22:17:43.806 DEBUG:boto:Token: None 2023-09-29T22:17:43.806 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.810 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045bb1d03e7f07ba1-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.810 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:17:43.811 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.811 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.811 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.811 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.811 DEBUG:boto:Method: GET 2023-09-29T22:17:43.811 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.812 DEBUG:boto:Data: 2023-09-29T22:17:43.812 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.812 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.812 DEBUG:boto:Params: {} 2023-09-29T22:17:43.812 DEBUG:boto:Token: None 2023-09-29T22:17:43.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY= 2023-09-29T22:17:43.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY=', 'Content-Length': '0'} 2023-09-29T22:17:43.815 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000019fbdc4264840f06-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.858 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.859 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.859 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.859 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.859 DEBUG:boto:Method: GET 2023-09-29T22:17:43.859 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.859 DEBUG:boto:Data: 2023-09-29T22:17:43.859 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.859 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.860 DEBUG:boto:Params: {} 2023-09-29T22:17:43.860 DEBUG:boto:Token: None 2023-09-29T22:17:43.860 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY= 2023-09-29T22:17:43.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY=', 'Content-Length': '0'} 2023-09-29T22:17:43.863 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aaa00dcab955583d-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.863 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.863 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.864 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.864 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.864 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.864 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.864 DEBUG:boto:Data: 2023-09-29T22:17:43.864 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.864 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.864 DEBUG:boto:Params: {} 2023-09-29T22:17:43.865 DEBUG:boto:Token: None 2023-09-29T22:17:43.865 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMrkH8anxFK/3EwZtbvmF4AwY9o= 2023-09-29T22:17:43.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMrkH8anxFK/3EwZtbvmF4AwY9o=', 'Content-Length': '0'} 2023-09-29T22:17:43.867 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058ea499b97653b3f-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.868 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.868 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.868 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.868 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.868 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.868 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.868 DEBUG:boto:Data: 2023-09-29T22:17:43.868 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.869 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.869 DEBUG:boto:Params: {} 2023-09-29T22:17:43.869 DEBUG:boto:Token: None 2023-09-29T22:17:43.869 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMrkH8anxFK/3EwZtbvmF4AwY9o= 2023-09-29T22:17:43.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMrkH8anxFK/3EwZtbvmF4AwY9o=', 'Content-Length': '0'} 2023-09-29T22:17:43.872 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014fdb16fe321f18e-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:17:43.873 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.873 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.873 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.873 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.873 DEBUG:boto:Method: GET 2023-09-29T22:17:43.873 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.874 DEBUG:boto:Data: 2023-09-29T22:17:43.874 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.874 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.874 DEBUG:boto:Params: {} 2023-09-29T22:17:43.874 DEBUG:boto:Token: None 2023-09-29T22:17:43.874 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.874 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY= 2023-09-29T22:17:43.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY=', 'Content-Length': '0'} 2023-09-29T22:17:43.877 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d996742662f0b64e-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.877 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.877 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.878 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.878 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.878 DEBUG:boto:Method: GET 2023-09-29T22:17:43.878 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.878 DEBUG:boto:Data: 2023-09-29T22:17:43.878 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.878 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.878 DEBUG:boto:Params: {} 2023-09-29T22:17:43.879 DEBUG:boto:Token: None 2023-09-29T22:17:43.879 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:17:43.879 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY= 2023-09-29T22:17:43.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vS89ZsaXeVeI7uuPAvm6SrdMRuY=', 'Content-Length': '0'} 2023-09-29T22:17:43.881 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c00db2c98676203b-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.926 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.927 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.927 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.927 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.927 DEBUG:boto:Method: GET 2023-09-29T22:17:43.927 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.927 DEBUG:boto:Data: 2023-09-29T22:17:43.927 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.927 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.928 DEBUG:boto:Params: {} 2023-09-29T22:17:43.928 DEBUG:boto:Token: None 2023-09-29T22:17:43.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.933 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a75c34f08fc117e0-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.933 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.934 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.934 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.934 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.934 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.934 DEBUG:boto:Method: GET 2023-09-29T22:17:43.934 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.934 DEBUG:boto:Data: 2023-09-29T22:17:43.935 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.935 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.935 DEBUG:boto:Params: {} 2023-09-29T22:17:43.935 DEBUG:boto:Token: None 2023-09-29T22:17:43.935 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/ 2023-09-29T22:17:43.935 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E= 2023-09-29T22:17:43.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCTHEBjeVd+3ID1OEo8qli/AF6E=', 'Content-Length': '0'} 2023-09-29T22:17:43.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012bfe1041b9c6c09-0065174d07-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.940 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:43.940 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.941 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.941 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.941 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:17:43.941 DEBUG:boto:Data: 2023-09-29T22:17:43.941 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.941 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.941 DEBUG:boto:Params: {} 2023-09-29T22:17:43.941 DEBUG:boto:Token: None 2023-09-29T22:17:43.942 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4 2023-09-29T22:17:43.942 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xe0G6RdRtKtsW6uRzVmP5LbxLoY= 2023-09-29T22:17:43.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xe0G6RdRtKtsW6uRzVmP5LbxLoY=', 'Content-Length': '0'} 2023-09-29T22:17:43.944 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe1a6061f2f5d91b-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.944 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.944 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.944 DEBUG:boto:Method: HEAD 2023-09-29T22:17:43.944 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:17:43.944 DEBUG:boto:Data: 2023-09-29T22:17:43.944 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.945 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.945 DEBUG:boto:Params: {} 2023-09-29T22:17:43.945 DEBUG:boto:Token: None 2023-09-29T22:17:43.945 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4 2023-09-29T22:17:43.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xe0G6RdRtKtsW6uRzVmP5LbxLoY= 2023-09-29T22:17:43.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xe0G6RdRtKtsW6uRzVmP5LbxLoY=', 'Content-Length': '0'} 2023-09-29T22:17:43.947 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b06aee61b6f9409-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:17:43.947 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.948 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.948 DEBUG:boto:Method: GET 2023-09-29T22:17:43.948 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:17:43.948 DEBUG:boto:Data: 2023-09-29T22:17:43.948 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:43.948 DEBUG:boto:Port: 8001 2023-09-29T22:17:43.948 DEBUG:boto:Params: {} 2023-09-29T22:17:43.948 DEBUG:boto:Token: None 2023-09-29T22:17:43.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4 2023-09-29T22:17:43.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iHKZZGyomuymfWSnVgZVk3o0UsQ= 2023-09-29T22:17:43.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iHKZZGyomuymfWSnVgZVk3o0UsQ=', 'Content-Length': '0'} 2023-09-29T22:17:43.951 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f53b0e2c9b56ac0a-0065174d07-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.994 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:17:43.995 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:17:43.995 DEBUG:boto:Method: GET 2023-09-29T22:17:43.995 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:17:43.995 DEBUG:boto:Data: 2023-09-29T22:17:43.995 DEBUG:boto:Headers: {} 2023-09-29T22:17:43.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:43.995 DEBUG:boto:Port: 8000 2023-09-29T22:17:43.995 DEBUG:boto:Params: {} 2023-09-29T22:17:43.995 DEBUG:boto:Token: None 2023-09-29T22:17:43.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:43 GMT /hcnphz-46/obj4 2023-09-29T22:17:43.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iHKZZGyomuymfWSnVgZVk3o0UsQ= 2023-09-29T22:17:43.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iHKZZGyomuymfWSnVgZVk3o0UsQ=', 'Content-Length': '0'} 2023-09-29T22:17:43.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042f27a73eec1cb26-0065174d07-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:43.999 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:43.999 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:43.999 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:43.999 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:44.000 DEBUG:boto:Method: GET 2023-09-29T22:17:44.000 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:17:44.000 DEBUG:boto:Data: 2023-09-29T22:17:44.000 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.000 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.000 DEBUG:boto:Params: {} 2023-09-29T22:17:44.000 DEBUG:boto:Token: None 2023-09-29T22:17:44.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-46/ 2023-09-29T22:17:44.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P/WnOHGL5NOA1LeQN0EbGQyWW7k= 2023-09-29T22:17:44.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P/WnOHGL5NOA1LeQN0EbGQyWW7k=', 'Content-Length': '0'} 2023-09-29T22:17:44.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002912c970fc83bb17-0065174d07-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.005 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.006 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:44.006 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:44.006 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:44.006 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:17:44.006 DEBUG:boto:Method: GET 2023-09-29T22:17:44.007 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:17:44.007 DEBUG:boto:Data: 2023-09-29T22:17:44.007 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.007 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.008 DEBUG:boto:Params: {} 2023-09-29T22:17:44.008 DEBUG:boto:Token: None 2023-09-29T22:17:44.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-46/ 2023-09-29T22:17:44.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P/WnOHGL5NOA1LeQN0EbGQyWW7k= 2023-09-29T22:17:44.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P/WnOHGL5NOA1LeQN0EbGQyWW7k=', 'Content-Length': '0'} 2023-09-29T22:17:44.012 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5204a8d417f70ea-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.054 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.055 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-46 zones={a2, a1} 2023-09-29T22:17:44.055 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-47 2023-09-29T22:17:44.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:44.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:44.056 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:44.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:44.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:44.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:44.163+0000 7f8e140c6f40 1 bucket sync caught up with source: 2023-09-29T22:17:44.166 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.33.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-29T22:17:44.166 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.33.6, , 00000000001.26.6, , 00000000001.22.6, , , 00000000001.22.6, , , ] 2023-09-29T22:17:44.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:44.163+0000 7f8e140c6f40 0 bucket checkpoint complete 2023-09-29T22:17:44.193 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-47 2023-09-29T22:17:44.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:44.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:44.193 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:44.245 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:44.245 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:44.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:44.307+0000 7f179a5b9f40 1 bucket sync caught up with source: 2023-09-29T22:17:44.310 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.95.6, , 00000000001.168.6, , 00000000001.115.6, , , 00000000001.138.6, , , ] 2023-09-29T22:17:44.310 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.95.6, , 00000000001.168.6, , 00000000001.115.6, , , 00000000001.138.6, , , ] 2023-09-29T22:17:44.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:44.307+0000 7f179a5b9f40 0 bucket checkpoint complete 2023-09-29T22:17:44.336 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-47 zones={a2, a1} 2023-09-29T22:17:44.337 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.337 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.337 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.337 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:17:44.337 DEBUG:boto:Data: 2023-09-29T22:17:44.337 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.337 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.337 DEBUG:boto:Params: {} 2023-09-29T22:17:44.338 DEBUG:boto:Token: None 2023-09-29T22:17:44.338 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LlaWbA/YEF7aVRdAkq9TGsmJdP4= 2023-09-29T22:17:44.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LlaWbA/YEF7aVRdAkq9TGsmJdP4=', 'Content-Length': '0'} 2023-09-29T22:17:44.342 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002aa28d600155acc7-0065174d08-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.343 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.343 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.343 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.343 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:17:44.343 DEBUG:boto:Data: 2023-09-29T22:17:44.343 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.343 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.344 DEBUG:boto:Params: {} 2023-09-29T22:17:44.344 DEBUG:boto:Token: None 2023-09-29T22:17:44.344 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LlaWbA/YEF7aVRdAkq9TGsmJdP4= 2023-09-29T22:17:44.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LlaWbA/YEF7aVRdAkq9TGsmJdP4=', 'Content-Length': '0'} 2023-09-29T22:17:44.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a6a39fece62539a0-0065174d08-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.348 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:17:44.348 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.348 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.348 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.348 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.348 DEBUG:boto:Method: GET 2023-09-29T22:17:44.348 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.348 DEBUG:boto:Data: 2023-09-29T22:17:44.348 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.349 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.349 DEBUG:boto:Params: {} 2023-09-29T22:17:44.349 DEBUG:boto:Token: None 2023-09-29T22:17:44.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/?versions 2023-09-29T22:17:44.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ= 2023-09-29T22:17:44.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ=', 'Content-Length': '0'} 2023-09-29T22:17:44.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ec0fa63a3eaf4b2-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.353 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:17:44.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:17:44.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:17:44.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:17:44.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:17:44.355 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.355 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.355 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.355 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.355 DEBUG:boto:Method: GET 2023-09-29T22:17:44.355 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.355 DEBUG:boto:Data: 2023-09-29T22:17:44.355 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.355 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.356 DEBUG:boto:Params: {} 2023-09-29T22:17:44.356 DEBUG:boto:Token: None 2023-09-29T22:17:44.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/?versions 2023-09-29T22:17:44.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ= 2023-09-29T22:17:44.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ=', 'Content-Length': '0'} 2023-09-29T22:17:44.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d959c3dfc331ae2f-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.360 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:17:44.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:17:44.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:17:44.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:17:44.361 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.361 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.361 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.361 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.361 DEBUG:boto:Method: GET 2023-09-29T22:17:44.361 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.361 DEBUG:boto:Data: 2023-09-29T22:17:44.361 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.361 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.362 DEBUG:boto:Params: {} 2023-09-29T22:17:44.362 DEBUG:boto:Token: None 2023-09-29T22:17:44.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/?versions 2023-09-29T22:17:44.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ= 2023-09-29T22:17:44.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ=', 'Content-Length': '0'} 2023-09-29T22:17:44.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000beb5c14922754c8e-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.366 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.367 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.368 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.368 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.368 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.368 DEBUG:boto:Method: GET 2023-09-29T22:17:44.368 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:17:44.369 DEBUG:boto:Data: 2023-09-29T22:17:44.369 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.369 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.369 DEBUG:boto:Params: {} 2023-09-29T22:17:44.370 DEBUG:boto:Token: None 2023-09-29T22:17:44.370 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/?versions 2023-09-29T22:17:44.370 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ= 2023-09-29T22:17:44.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XX0ExX11P2T9xENnNpnZzoeWgwQ=', 'Content-Length': '0'} 2023-09-29T22:17:44.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010ab12d9dc1953c3-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.375 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:17:44.376 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.376 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.377 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.377 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.377 DEBUG:boto:Method: GET 2023-09-29T22:17:44.377 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.377 DEBUG:boto:Data: 2023-09-29T22:17:44.378 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.378 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.378 DEBUG:boto:Params: {} 2023-09-29T22:17:44.379 DEBUG:boto:Token: None 2023-09-29T22:17:44.379 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.379 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw= 2023-09-29T22:17:44.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw=', 'Content-Length': '0'} 2023-09-29T22:17:44.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004cd5c03d972a5faf-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.383 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.383 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.383 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.383 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.384 DEBUG:boto:Method: GET 2023-09-29T22:17:44.384 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.384 DEBUG:boto:Data: 2023-09-29T22:17:44.384 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.385 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.385 DEBUG:boto:Params: {} 2023-09-29T22:17:44.385 DEBUG:boto:Token: None 2023-09-29T22:17:44.385 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw= 2023-09-29T22:17:44.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw=', 'Content-Length': '0'} 2023-09-29T22:17:44.389 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000003d55143443f0bf2-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.430 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.431 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.431 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.431 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.431 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.431 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.431 DEBUG:boto:Data: 2023-09-29T22:17:44.432 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.432 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.432 DEBUG:boto:Params: {} 2023-09-29T22:17:44.432 DEBUG:boto:Token: None 2023-09-29T22:17:44.433 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rkiqK4yqittjivPnHunolUSnNBI= 2023-09-29T22:17:44.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rkiqK4yqittjivPnHunolUSnNBI=', 'Content-Length': '0'} 2023-09-29T22:17:44.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000453dc835a1508d95-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.436 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.437 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.437 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.437 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.437 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.437 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.437 DEBUG:boto:Data: 2023-09-29T22:17:44.437 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.437 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.438 DEBUG:boto:Params: {} 2023-09-29T22:17:44.438 DEBUG:boto:Token: None 2023-09-29T22:17:44.438 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rkiqK4yqittjivPnHunolUSnNBI= 2023-09-29T22:17:44.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rkiqK4yqittjivPnHunolUSnNBI=', 'Content-Length': '0'} 2023-09-29T22:17:44.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b67132bf29b3722f-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:17:44.442 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.442 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.443 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.443 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.443 DEBUG:boto:Method: GET 2023-09-29T22:17:44.443 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.443 DEBUG:boto:Data: 2023-09-29T22:17:44.443 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.444 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.444 DEBUG:boto:Params: {} 2023-09-29T22:17:44.444 DEBUG:boto:Token: None 2023-09-29T22:17:44.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw= 2023-09-29T22:17:44.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw=', 'Content-Length': '0'} 2023-09-29T22:17:44.448 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001366c6dcb378e86f-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.490 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.491 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.491 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.491 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.491 DEBUG:boto:Method: GET 2023-09-29T22:17:44.491 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.491 DEBUG:boto:Data: 2023-09-29T22:17:44.491 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.491 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.492 DEBUG:boto:Params: {} 2023-09-29T22:17:44.492 DEBUG:boto:Token: None 2023-09-29T22:17:44.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:17:44.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw= 2023-09-29T22:17:44.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JF2Hug71KGHHIP/FH3MWjyPERtw=', 'Content-Length': '0'} 2023-09-29T22:17:44.495 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000404c0d3980c187ab-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.495 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.496 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.496 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.496 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.496 DEBUG:boto:Method: GET 2023-09-29T22:17:44.496 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.496 DEBUG:boto:Data: 2023-09-29T22:17:44.496 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.496 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.497 DEBUG:boto:Params: {} 2023-09-29T22:17:44.497 DEBUG:boto:Token: None 2023-09-29T22:17:44.497 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.502 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed973c7d11af38f9-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.502 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.502 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.503 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.503 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.503 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.503 DEBUG:boto:Method: GET 2023-09-29T22:17:44.503 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.503 DEBUG:boto:Data: 2023-09-29T22:17:44.503 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.504 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.504 DEBUG:boto:Params: {} 2023-09-29T22:17:44.504 DEBUG:boto:Token: None 2023-09-29T22:17:44.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.509 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007177f0cdace68088-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.509 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.509 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.509 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.510 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.510 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:17:44.510 DEBUG:boto:Data: 2023-09-29T22:17:44.510 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.510 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.510 DEBUG:boto:Params: {} 2023-09-29T22:17:44.510 DEBUG:boto:Token: None 2023-09-29T22:17:44.511 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1 2023-09-29T22:17:44.511 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgPipmI8VW/Dh8Y47NvpcAwauVE= 2023-09-29T22:17:44.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgPipmI8VW/Dh8Y47NvpcAwauVE=', 'Content-Length': '0'} 2023-09-29T22:17:44.513 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4ff2b7a65060e59-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.513 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.514 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.514 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.514 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:17:44.514 DEBUG:boto:Data: 2023-09-29T22:17:44.514 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.514 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.514 DEBUG:boto:Params: {} 2023-09-29T22:17:44.514 DEBUG:boto:Token: None 2023-09-29T22:17:44.515 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1 2023-09-29T22:17:44.515 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgPipmI8VW/Dh8Y47NvpcAwauVE= 2023-09-29T22:17:44.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgPipmI8VW/Dh8Y47NvpcAwauVE=', 'Content-Length': '0'} 2023-09-29T22:17:44.517 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069ad018693cbf904-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:17:44.518 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.518 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.518 DEBUG:boto:Method: GET 2023-09-29T22:17:44.518 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:17:44.518 DEBUG:boto:Data: 2023-09-29T22:17:44.518 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.518 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.519 DEBUG:boto:Params: {} 2023-09-29T22:17:44.519 DEBUG:boto:Token: None 2023-09-29T22:17:44.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1 2023-09-29T22:17:44.519 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vnM7f/IVqPfystFxdAis/KPXR/M= 2023-09-29T22:17:44.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vnM7f/IVqPfystFxdAis/KPXR/M=', 'Content-Length': '0'} 2023-09-29T22:17:44.521 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000919aa46138d387ee-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.522 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:17:44.522 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:17:44.522 DEBUG:boto:Method: GET 2023-09-29T22:17:44.522 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:17:44.522 DEBUG:boto:Data: 2023-09-29T22:17:44.522 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.523 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.523 DEBUG:boto:Params: {} 2023-09-29T22:17:44.523 DEBUG:boto:Token: None 2023-09-29T22:17:44.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj1 2023-09-29T22:17:44.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vnM7f/IVqPfystFxdAis/KPXR/M= 2023-09-29T22:17:44.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vnM7f/IVqPfystFxdAis/KPXR/M=', 'Content-Length': '0'} 2023-09-29T22:17:44.526 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000225d6de5be3edb62-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.570 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.571 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.571 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.571 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.571 DEBUG:boto:Method: GET 2023-09-29T22:17:44.571 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.571 DEBUG:boto:Data: 2023-09-29T22:17:44.571 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.571 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.572 DEBUG:boto:Params: {} 2023-09-29T22:17:44.572 DEBUG:boto:Token: None 2023-09-29T22:17:44.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d71eb596e5d02fcb-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.577 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.578 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.578 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.578 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.579 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.579 DEBUG:boto:Method: GET 2023-09-29T22:17:44.579 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:17:44.579 DEBUG:boto:Data: 2023-09-29T22:17:44.580 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.580 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.580 DEBUG:boto:Params: {} 2023-09-29T22:17:44.581 DEBUG:boto:Token: None 2023-09-29T22:17:44.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1bcb72a524a233a-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.587 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:17:44.588 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.588 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.588 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.588 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.589 DEBUG:boto:Method: GET 2023-09-29T22:17:44.589 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.589 DEBUG:boto:Data: 2023-09-29T22:17:44.589 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.590 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.590 DEBUG:boto:Params: {} 2023-09-29T22:17:44.590 DEBUG:boto:Token: None 2023-09-29T22:17:44.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo= 2023-09-29T22:17:44.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo=', 'Content-Length': '0'} 2023-09-29T22:17:44.594 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009c9828f6a0f4a891-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.634 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.635 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.635 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.635 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.635 DEBUG:boto:Method: GET 2023-09-29T22:17:44.635 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.635 DEBUG:boto:Data: 2023-09-29T22:17:44.635 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.636 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.636 DEBUG:boto:Params: {} 2023-09-29T22:17:44.636 DEBUG:boto:Token: None 2023-09-29T22:17:44.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo= 2023-09-29T22:17:44.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo=', 'Content-Length': '0'} 2023-09-29T22:17:44.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009d6d53a6f36b104b-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.639 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.640 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.640 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.640 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.640 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.640 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.640 DEBUG:boto:Data: 2023-09-29T22:17:44.640 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.640 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.640 DEBUG:boto:Params: {} 2023-09-29T22:17:44.641 DEBUG:boto:Token: None 2023-09-29T22:17:44.641 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AduyDc4jCfb27w7kKaFy1MsItuo= 2023-09-29T22:17:44.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AduyDc4jCfb27w7kKaFy1MsItuo=', 'Content-Length': '0'} 2023-09-29T22:17:44.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c52ae56371cb93fe-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.644 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.644 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.644 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.644 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.644 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.644 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.644 DEBUG:boto:Data: 2023-09-29T22:17:44.644 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.645 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.645 DEBUG:boto:Params: {} 2023-09-29T22:17:44.645 DEBUG:boto:Token: None 2023-09-29T22:17:44.645 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AduyDc4jCfb27w7kKaFy1MsItuo= 2023-09-29T22:17:44.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AduyDc4jCfb27w7kKaFy1MsItuo=', 'Content-Length': '0'} 2023-09-29T22:17:44.648 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003bb6d31a50e055b5-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:17:44.648 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.648 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.648 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.649 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.649 DEBUG:boto:Method: GET 2023-09-29T22:17:44.649 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.649 DEBUG:boto:Data: 2023-09-29T22:17:44.649 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.649 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.649 DEBUG:boto:Params: {} 2023-09-29T22:17:44.650 DEBUG:boto:Token: None 2023-09-29T22:17:44.650 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.650 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo= 2023-09-29T22:17:44.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo=', 'Content-Length': '0'} 2023-09-29T22:17:44.653 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023b4d512db4eb6c4-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.653 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.653 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.654 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.654 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.654 DEBUG:boto:Method: GET 2023-09-29T22:17:44.654 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.655 DEBUG:boto:Data: 2023-09-29T22:17:44.655 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.655 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.655 DEBUG:boto:Params: {} 2023-09-29T22:17:44.656 DEBUG:boto:Token: None 2023-09-29T22:17:44.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:17:44.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo= 2023-09-29T22:17:44.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mJzEzKIs1YeXN9rLo/YAsEjSTHo=', 'Content-Length': '0'} 2023-09-29T22:17:44.659 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002019545bbd7dd92b-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.702 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.703 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.703 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.703 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.703 DEBUG:boto:Method: GET 2023-09-29T22:17:44.703 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.703 DEBUG:boto:Data: 2023-09-29T22:17:44.704 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.704 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.704 DEBUG:boto:Params: {} 2023-09-29T22:17:44.704 DEBUG:boto:Token: None 2023-09-29T22:17:44.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7e6827ffd901ad-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.709 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.710 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.710 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.710 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.710 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.711 DEBUG:boto:Method: GET 2023-09-29T22:17:44.711 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.711 DEBUG:boto:Data: 2023-09-29T22:17:44.711 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.711 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.711 DEBUG:boto:Params: {} 2023-09-29T22:17:44.711 DEBUG:boto:Token: None 2023-09-29T22:17:44.711 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031ea05d52ea26510-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.717 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.717 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.717 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.717 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.717 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:17:44.718 DEBUG:boto:Data: 2023-09-29T22:17:44.718 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.718 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.718 DEBUG:boto:Params: {} 2023-09-29T22:17:44.718 DEBUG:boto:Token: None 2023-09-29T22:17:44.719 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2 2023-09-29T22:17:44.719 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mioQJwdGQMR/G1tFVFwh3PHnKrU= 2023-09-29T22:17:44.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mioQJwdGQMR/G1tFVFwh3PHnKrU=', 'Content-Length': '0'} 2023-09-29T22:17:44.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f669d409e21c37c8-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.721 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.721 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.721 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.722 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:17:44.722 DEBUG:boto:Data: 2023-09-29T22:17:44.722 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.722 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.722 DEBUG:boto:Params: {} 2023-09-29T22:17:44.722 DEBUG:boto:Token: None 2023-09-29T22:17:44.723 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2 2023-09-29T22:17:44.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mioQJwdGQMR/G1tFVFwh3PHnKrU= 2023-09-29T22:17:44.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mioQJwdGQMR/G1tFVFwh3PHnKrU=', 'Content-Length': '0'} 2023-09-29T22:17:44.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7f81cf4848c35a8-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:17:44.725 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.725 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.726 DEBUG:boto:Method: GET 2023-09-29T22:17:44.726 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:17:44.726 DEBUG:boto:Data: 2023-09-29T22:17:44.726 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.726 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.726 DEBUG:boto:Params: {} 2023-09-29T22:17:44.726 DEBUG:boto:Token: None 2023-09-29T22:17:44.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2 2023-09-29T22:17:44.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u1cQXsMAgsup+V0oUXSLoC3JT04= 2023-09-29T22:17:44.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u1cQXsMAgsup+V0oUXSLoC3JT04=', 'Content-Length': '0'} 2023-09-29T22:17:44.729 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014aebde465af286a-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.770 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:17:44.771 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:17:44.771 DEBUG:boto:Method: GET 2023-09-29T22:17:44.771 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:17:44.771 DEBUG:boto:Data: 2023-09-29T22:17:44.771 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.771 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.771 DEBUG:boto:Params: {} 2023-09-29T22:17:44.772 DEBUG:boto:Token: None 2023-09-29T22:17:44.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj2 2023-09-29T22:17:44.772 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u1cQXsMAgsup+V0oUXSLoC3JT04= 2023-09-29T22:17:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u1cQXsMAgsup+V0oUXSLoC3JT04=', 'Content-Length': '0'} 2023-09-29T22:17:44.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f033a817ddf6a775-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.775 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.775 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.775 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.775 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.776 DEBUG:boto:Method: GET 2023-09-29T22:17:44.776 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.776 DEBUG:boto:Data: 2023-09-29T22:17:44.776 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.776 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.776 DEBUG:boto:Params: {} 2023-09-29T22:17:44.776 DEBUG:boto:Token: None 2023-09-29T22:17:44.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b506621c982b28f-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.782 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.783 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.783 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.784 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.784 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.784 DEBUG:boto:Method: GET 2023-09-29T22:17:44.784 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:17:44.784 DEBUG:boto:Data: 2023-09-29T22:17:44.785 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.785 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.785 DEBUG:boto:Params: {} 2023-09-29T22:17:44.786 DEBUG:boto:Token: None 2023-09-29T22:17:44.786 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8c59743e5c4c4f-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.838 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:17:44.839 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.839 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.839 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.840 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.840 DEBUG:boto:Method: GET 2023-09-29T22:17:44.840 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.840 DEBUG:boto:Data: 2023-09-29T22:17:44.840 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.840 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.840 DEBUG:boto:Params: {} 2023-09-29T22:17:44.840 DEBUG:boto:Token: None 2023-09-29T22:17:44.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.841 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE= 2023-09-29T22:17:44.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE=', 'Content-Length': '0'} 2023-09-29T22:17:44.844 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003aef6281889b7dae-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.844 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.844 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.844 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.845 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.845 DEBUG:boto:Method: GET 2023-09-29T22:17:44.845 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.845 DEBUG:boto:Data: 2023-09-29T22:17:44.845 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.846 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.846 DEBUG:boto:Params: {} 2023-09-29T22:17:44.846 DEBUG:boto:Token: None 2023-09-29T22:17:44.846 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.846 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE= 2023-09-29T22:17:44.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE=', 'Content-Length': '0'} 2023-09-29T22:17:44.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001fca1545994e1f7-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.850 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.850 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.850 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.850 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.850 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.850 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.851 DEBUG:boto:Data: 2023-09-29T22:17:44.851 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.851 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.851 DEBUG:boto:Params: {} 2023-09-29T22:17:44.851 DEBUG:boto:Token: None 2023-09-29T22:17:44.851 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3CbGk3WI6JFJYlDuUlXjPzbQ9dk= 2023-09-29T22:17:44.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3CbGk3WI6JFJYlDuUlXjPzbQ9dk=', 'Content-Length': '0'} 2023-09-29T22:17:44.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000913fd12ea47fa378-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.855 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.855 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.855 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.855 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.855 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.855 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.855 DEBUG:boto:Data: 2023-09-29T22:17:44.855 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.856 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.856 DEBUG:boto:Params: {} 2023-09-29T22:17:44.856 DEBUG:boto:Token: None 2023-09-29T22:17:44.856 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3CbGk3WI6JFJYlDuUlXjPzbQ9dk= 2023-09-29T22:17:44.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3CbGk3WI6JFJYlDuUlXjPzbQ9dk=', 'Content-Length': '0'} 2023-09-29T22:17:44.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f81db166ea60cd69-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:17:44.860 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.860 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.860 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.860 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.860 DEBUG:boto:Method: GET 2023-09-29T22:17:44.860 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.861 DEBUG:boto:Data: 2023-09-29T22:17:44.861 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.861 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.861 DEBUG:boto:Params: {} 2023-09-29T22:17:44.861 DEBUG:boto:Token: None 2023-09-29T22:17:44.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.862 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE= 2023-09-29T22:17:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE=', 'Content-Length': '0'} 2023-09-29T22:17:44.864 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000290a576327b59902-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.910 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.911 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.911 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.911 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.911 DEBUG:boto:Method: GET 2023-09-29T22:17:44.911 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.911 DEBUG:boto:Data: 2023-09-29T22:17:44.911 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.912 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.912 DEBUG:boto:Params: {} 2023-09-29T22:17:44.912 DEBUG:boto:Token: None 2023-09-29T22:17:44.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:17:44.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE= 2023-09-29T22:17:44.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uXlPWiJ+zDomcWkEa6tYWk6UoWE=', 'Content-Length': '0'} 2023-09-29T22:17:44.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000070e657c68185460-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.916 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.916 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.916 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.916 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.916 DEBUG:boto:Method: GET 2023-09-29T22:17:44.916 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.917 DEBUG:boto:Data: 2023-09-29T22:17:44.917 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.917 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.917 DEBUG:boto:Params: {} 2023-09-29T22:17:44.917 DEBUG:boto:Token: None 2023-09-29T22:17:44.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.922 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f77afeecfa04c77e-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.923 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.923 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.923 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.923 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.923 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.924 DEBUG:boto:Method: GET 2023-09-29T22:17:44.924 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.924 DEBUG:boto:Data: 2023-09-29T22:17:44.924 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.924 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.924 DEBUG:boto:Params: {} 2023-09-29T22:17:44.924 DEBUG:boto:Token: None 2023-09-29T22:17:44.925 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.925 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002396d45ef846a843-0065174d08-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.930 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:44.931 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.931 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.931 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.931 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:17:44.931 DEBUG:boto:Data: 2023-09-29T22:17:44.931 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.931 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.931 DEBUG:boto:Params: {} 2023-09-29T22:17:44.932 DEBUG:boto:Token: None 2023-09-29T22:17:44.932 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3 2023-09-29T22:17:44.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wnyjrl9xH2m9z4+5LWqODkvQgR8= 2023-09-29T22:17:44.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wnyjrl9xH2m9z4+5LWqODkvQgR8=', 'Content-Length': '0'} 2023-09-29T22:17:44.934 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009da3013b7ac23aef-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.937 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.937 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.937 DEBUG:boto:Method: HEAD 2023-09-29T22:17:44.937 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:17:44.937 DEBUG:boto:Data: 2023-09-29T22:17:44.937 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.938 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.938 DEBUG:boto:Params: {} 2023-09-29T22:17:44.938 DEBUG:boto:Token: None 2023-09-29T22:17:44.938 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3 2023-09-29T22:17:44.938 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wnyjrl9xH2m9z4+5LWqODkvQgR8= 2023-09-29T22:17:44.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wnyjrl9xH2m9z4+5LWqODkvQgR8=', 'Content-Length': '0'} 2023-09-29T22:17:44.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001086b0523329404d-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:17:44.941 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.941 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.941 DEBUG:boto:Method: GET 2023-09-29T22:17:44.941 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:17:44.941 DEBUG:boto:Data: 2023-09-29T22:17:44.942 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.942 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.942 DEBUG:boto:Params: {} 2023-09-29T22:17:44.942 DEBUG:boto:Token: None 2023-09-29T22:17:44.942 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3 2023-09-29T22:17:44.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:72hfIlgnojQoGKlix/8aK1nXWhM= 2023-09-29T22:17:44.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:72hfIlgnojQoGKlix/8aK1nXWhM=', 'Content-Length': '0'} 2023-09-29T22:17:44.945 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bce91be664abd317-0065174d08-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.945 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:17:44.945 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:17:44.945 DEBUG:boto:Method: GET 2023-09-29T22:17:44.946 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:17:44.946 DEBUG:boto:Data: 2023-09-29T22:17:44.946 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:44.946 DEBUG:boto:Port: 8000 2023-09-29T22:17:44.946 DEBUG:boto:Params: {} 2023-09-29T22:17:44.946 DEBUG:boto:Token: None 2023-09-29T22:17:44.946 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/obj3 2023-09-29T22:17:44.947 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:72hfIlgnojQoGKlix/8aK1nXWhM= 2023-09-29T22:17:44.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:72hfIlgnojQoGKlix/8aK1nXWhM=', 'Content-Length': '0'} 2023-09-29T22:17:44.949 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000749d81dbcefcdd74-0065174d08-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:44.991 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:44.991 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:44.991 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.991 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.992 DEBUG:boto:Method: GET 2023-09-29T22:17:44.992 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:17:44.992 DEBUG:boto:Data: 2023-09-29T22:17:44.992 DEBUG:boto:Headers: {} 2023-09-29T22:17:44.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:44.993 DEBUG:boto:Port: 8001 2023-09-29T22:17:44.993 DEBUG:boto:Params: {} 2023-09-29T22:17:44.993 DEBUG:boto:Token: None 2023-09-29T22:17:44.993 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:44 GMT /hcnphz-47/ 2023-09-29T22:17:44.994 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4= 2023-09-29T22:17:44.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IU6USqdvWoT73hgCHr/t9WQ91o4=', 'Content-Length': '0'} 2023-09-29T22:17:44.999 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000febdd17fc516e66f-0065174d08-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.000 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:45.000 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:45.001 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:45.001 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:45.001 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:17:45.001 DEBUG:boto:Method: GET 2023-09-29T22:17:45.001 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:17:45.002 DEBUG:boto:Data: 2023-09-29T22:17:45.002 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.002 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.002 DEBUG:boto:Params: {} 2023-09-29T22:17:45.002 DEBUG:boto:Token: None 2023-09-29T22:17:45.003 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/ 2023-09-29T22:17:45.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8= 2023-09-29T22:17:45.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8=', 'Content-Length': '0'} 2023-09-29T22:17:45.007 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000495b803b071293e5-0065174d09-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.007 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:45.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:17:45.008 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.008 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.008 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.008 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.009 DEBUG:boto:Method: GET 2023-09-29T22:17:45.009 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.009 DEBUG:boto:Data: 2023-09-29T22:17:45.009 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.010 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.010 DEBUG:boto:Params: {} 2023-09-29T22:17:45.010 DEBUG:boto:Token: None 2023-09-29T22:17:45.010 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U= 2023-09-29T22:17:45.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U=', 'Content-Length': '0'} 2023-09-29T22:17:45.013 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017119a43ad07bcf8-0065174d09-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.058 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.059 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.059 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.059 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.059 DEBUG:boto:Method: GET 2023-09-29T22:17:45.059 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.060 DEBUG:boto:Data: 2023-09-29T22:17:45.060 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.060 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.060 DEBUG:boto:Params: {} 2023-09-29T22:17:45.060 DEBUG:boto:Token: None 2023-09-29T22:17:45.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U= 2023-09-29T22:17:45.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U=', 'Content-Length': '0'} 2023-09-29T22:17:45.064 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b70d0248921329f-0065174d09-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.065 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.065 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.065 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.065 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.065 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.066 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.066 DEBUG:boto:Data: 2023-09-29T22:17:45.066 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.066 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.066 DEBUG:boto:Params: {} 2023-09-29T22:17:45.067 DEBUG:boto:Token: None 2023-09-29T22:17:45.067 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.067 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:93OUqCW5t6VuXf6IGa62rMk90s0= 2023-09-29T22:17:45.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:93OUqCW5t6VuXf6IGa62rMk90s0=', 'Content-Length': '0'} 2023-09-29T22:17:45.070 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000e6587c11dd15134-0065174d09-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.070 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.070 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.071 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.071 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.071 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.071 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.072 DEBUG:boto:Data: 2023-09-29T22:17:45.072 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.072 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.073 DEBUG:boto:Params: {} 2023-09-29T22:17:45.073 DEBUG:boto:Token: None 2023-09-29T22:17:45.073 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.074 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:93OUqCW5t6VuXf6IGa62rMk90s0= 2023-09-29T22:17:45.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:93OUqCW5t6VuXf6IGa62rMk90s0=', 'Content-Length': '0'} 2023-09-29T22:17:45.077 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097420f1c86d65a6e-0065174d09-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:17:45.077 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.077 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.077 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.077 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.078 DEBUG:boto:Method: GET 2023-09-29T22:17:45.078 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.078 DEBUG:boto:Data: 2023-09-29T22:17:45.078 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.078 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.078 DEBUG:boto:Params: {} 2023-09-29T22:17:45.078 DEBUG:boto:Token: None 2023-09-29T22:17:45.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.079 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U= 2023-09-29T22:17:45.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U=', 'Content-Length': '0'} 2023-09-29T22:17:45.081 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000182450960fbc67c5-0065174d09-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.082 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.082 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.082 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.082 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.082 DEBUG:boto:Method: GET 2023-09-29T22:17:45.082 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.082 DEBUG:boto:Data: 2023-09-29T22:17:45.083 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.083 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.083 DEBUG:boto:Params: {} 2023-09-29T22:17:45.083 DEBUG:boto:Token: None 2023-09-29T22:17:45.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:17:45.083 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U= 2023-09-29T22:17:45.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nI0t6byiHMg0toFAvs4CUImEM+U=', 'Content-Length': '0'} 2023-09-29T22:17:45.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000055e329e4921b683e-0065174d09-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.087 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:45.087 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:45.087 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.087 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.087 DEBUG:boto:Method: GET 2023-09-29T22:17:45.087 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.087 DEBUG:boto:Data: 2023-09-29T22:17:45.087 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.088 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.088 DEBUG:boto:Params: {} 2023-09-29T22:17:45.088 DEBUG:boto:Token: None 2023-09-29T22:17:45.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/ 2023-09-29T22:17:45.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8= 2023-09-29T22:17:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8=', 'Content-Length': '0'} 2023-09-29T22:17:45.093 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010e7e60e8dc23eae-0065174d09-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.094 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:45.094 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:45.094 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:45.094 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.095 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.095 DEBUG:boto:Method: GET 2023-09-29T22:17:45.095 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.095 DEBUG:boto:Data: 2023-09-29T22:17:45.095 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.095 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.095 DEBUG:boto:Params: {} 2023-09-29T22:17:45.095 DEBUG:boto:Token: None 2023-09-29T22:17:45.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/ 2023-09-29T22:17:45.096 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8= 2023-09-29T22:17:45.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8=', 'Content-Length': '0'} 2023-09-29T22:17:45.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018e09d71a17f341f-0065174d09-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.146 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:45.147 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.147 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.147 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.147 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:17:45.147 DEBUG:boto:Data: 2023-09-29T22:17:45.148 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.148 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.148 DEBUG:boto:Params: {} 2023-09-29T22:17:45.149 DEBUG:boto:Token: None 2023-09-29T22:17:45.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4 2023-09-29T22:17:45.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1gQxnnq5veQ1zzqE0qbFV3cyaI8= 2023-09-29T22:17:45.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1gQxnnq5veQ1zzqE0qbFV3cyaI8=', 'Content-Length': '0'} 2023-09-29T22:17:45.152 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f14cf66a228177e-0065174d09-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.152 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.152 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.152 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.152 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:17:45.153 DEBUG:boto:Data: 2023-09-29T22:17:45.153 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.153 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.153 DEBUG:boto:Params: {} 2023-09-29T22:17:45.153 DEBUG:boto:Token: None 2023-09-29T22:17:45.153 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4 2023-09-29T22:17:45.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1gQxnnq5veQ1zzqE0qbFV3cyaI8= 2023-09-29T22:17:45.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1gQxnnq5veQ1zzqE0qbFV3cyaI8=', 'Content-Length': '0'} 2023-09-29T22:17:45.156 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002d9066879035abb3-0065174d09-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:17:45.157 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.157 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.157 DEBUG:boto:Method: GET 2023-09-29T22:17:45.157 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:17:45.157 DEBUG:boto:Data: 2023-09-29T22:17:45.157 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.157 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.157 DEBUG:boto:Params: {} 2023-09-29T22:17:45.157 DEBUG:boto:Token: None 2023-09-29T22:17:45.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4 2023-09-29T22:17:45.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jfYpd3UrFyLKTwzGL6Pv+BMbXhw= 2023-09-29T22:17:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jfYpd3UrFyLKTwzGL6Pv+BMbXhw=', 'Content-Length': '0'} 2023-09-29T22:17:45.160 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000381f4b5a3c9bd486-0065174d09-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.161 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:17:45.161 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:17:45.161 DEBUG:boto:Method: GET 2023-09-29T22:17:45.161 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:17:45.161 DEBUG:boto:Data: 2023-09-29T22:17:45.161 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.162 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.162 DEBUG:boto:Params: {} 2023-09-29T22:17:45.162 DEBUG:boto:Token: None 2023-09-29T22:17:45.162 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/obj4 2023-09-29T22:17:45.162 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jfYpd3UrFyLKTwzGL6Pv+BMbXhw= 2023-09-29T22:17:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jfYpd3UrFyLKTwzGL6Pv+BMbXhw=', 'Content-Length': '0'} 2023-09-29T22:17:45.165 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9e444e84b4254b2-0065174d09-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.165 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:45.165 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:45.165 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.165 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.166 DEBUG:boto:Method: GET 2023-09-29T22:17:45.166 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.166 DEBUG:boto:Data: 2023-09-29T22:17:45.166 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:45.166 DEBUG:boto:Port: 8001 2023-09-29T22:17:45.166 DEBUG:boto:Params: {} 2023-09-29T22:17:45.166 DEBUG:boto:Token: None 2023-09-29T22:17:45.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/ 2023-09-29T22:17:45.167 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8= 2023-09-29T22:17:45.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8=', 'Content-Length': '0'} 2023-09-29T22:17:45.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006de6775133aea962-0065174d09-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.172 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:45.172 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:45.172 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:45.172 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.172 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.173 DEBUG:boto:Method: GET 2023-09-29T22:17:45.173 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:17:45.173 DEBUG:boto:Data: 2023-09-29T22:17:45.173 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.173 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.173 DEBUG:boto:Params: {} 2023-09-29T22:17:45.173 DEBUG:boto:Token: None 2023-09-29T22:17:45.173 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-47/ 2023-09-29T22:17:45.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8= 2023-09-29T22:17:45.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mBpQVnDvQBJJHsi5jENO3KACav8=', 'Content-Length': '0'} 2023-09-29T22:17:45.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000167ef274f6c44d8b-0065174d09-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.179 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:17:45.179 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-47 zones={a2, a1} 2023-09-29T22:17:45.180 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hcnphz-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hcnphz-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.180 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hcnphz-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:17:45.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:45.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:45.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hcnphz-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hcnphz-46', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.364 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hcnphz-46 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:17:45.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:45.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:45.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hcnphz-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hcnphz-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.538 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hcnphz-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:17:45.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:45.585 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:45.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hcnphz-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hcnphz-47', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:17:45.718 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hcnphz-47 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:17:45.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:45.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:45.895 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:45.895 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:45.895 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.895 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:17:45.895 DEBUG:boto:Data: 2023-09-29T22:17:45.895 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.896 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.896 DEBUG:boto:Params: {} 2023-09-29T22:17:45.896 DEBUG:boto:Token: None 2023-09-29T22:17:45.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-46/ 2023-09-29T22:17:45.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yCkIwiDprif4Gq0m+EL3+gREt+8= 2023-09-29T22:17:45.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCkIwiDprif4Gq0m+EL3+gREt+8=', 'Content-Length': '0'} 2023-09-29T22:17:45.900 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba5076e7881da1b7-0065174d09-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.900 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:17:45.900 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:17:45.901 DEBUG:boto:Method: PUT 2023-09-29T22:17:45.901 DEBUG:boto:Path: /hcnphz-46/obj5 2023-09-29T22:17:45.901 DEBUG:boto:Data: 2023-09-29T22:17:45.901 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:45.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.901 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.901 DEBUG:boto:Params: {} 2023-09-29T22:17:45.901 DEBUG:boto:Token: None 2023-09-29T22:17:45.901 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-46/obj5 2023-09-29T22:17:45.902 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AmEGy0THVx1TiouF9wsluFuazdc= 2023-09-29T22:17:45.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AmEGy0THVx1TiouF9wsluFuazdc='} 2023-09-29T22:17:45.950 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d27a60eff6fe3dc2-0065174d09-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.951 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:45.951 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:45.951 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.951 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:17:45.951 DEBUG:boto:Data: 2023-09-29T22:17:45.951 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.952 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.952 DEBUG:boto:Params: {} 2023-09-29T22:17:45.952 DEBUG:boto:Token: None 2023-09-29T22:17:45.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-46/ 2023-09-29T22:17:45.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yCkIwiDprif4Gq0m+EL3+gREt+8= 2023-09-29T22:17:45.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCkIwiDprif4Gq0m+EL3+gREt+8=', 'Content-Length': '0'} 2023-09-29T22:17:45.957 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e86dbd3d5792ab2d-0065174d09-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.957 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:17:45.957 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:17:45.958 DEBUG:boto:Method: PUT 2023-09-29T22:17:45.958 DEBUG:boto:Path: /hcnphz-46/obj6 2023-09-29T22:17:45.958 DEBUG:boto:Data: 2023-09-29T22:17:45.958 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:45.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.958 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.958 DEBUG:boto:Params: {} 2023-09-29T22:17:45.958 DEBUG:boto:Token: None 2023-09-29T22:17:45.959 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-46/obj6 2023-09-29T22:17:45.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RRt1nLON+kVzREEgtMRxYTIoDlo= 2023-09-29T22:17:45.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RRt1nLON+kVzREEgtMRxYTIoDlo='} 2023-09-29T22:17:45.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008c7d94095f1002e5-0065174d09-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.970 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:45.970 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:45.970 DEBUG:boto:Method: HEAD 2023-09-29T22:17:45.970 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:17:45.971 DEBUG:boto:Data: 2023-09-29T22:17:45.971 DEBUG:boto:Headers: {} 2023-09-29T22:17:45.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.971 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.971 DEBUG:boto:Params: {} 2023-09-29T22:17:45.971 DEBUG:boto:Token: None 2023-09-29T22:17:45.971 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-46/ 2023-09-29T22:17:45.971 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yCkIwiDprif4Gq0m+EL3+gREt+8= 2023-09-29T22:17:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCkIwiDprif4Gq0m+EL3+gREt+8=', 'Content-Length': '0'} 2023-09-29T22:17:45.974 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cef13e68c4af2a1c-0065174d09-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:45.974 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:17:45.975 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:17:45.975 DEBUG:boto:Method: PUT 2023-09-29T22:17:45.975 DEBUG:boto:Path: /hcnphz-46/obj7 2023-09-29T22:17:45.975 DEBUG:boto:Data: 2023-09-29T22:17:45.975 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:45.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:45.975 DEBUG:boto:Port: 8000 2023-09-29T22:17:45.975 DEBUG:boto:Params: {} 2023-09-29T22:17:45.975 DEBUG:boto:Token: None 2023-09-29T22:17:45.976 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:45 GMT /hcnphz-46/obj7 2023-09-29T22:17:45.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggdH5JpcvnCU9dRRPLofFJ8en84= 2023-09-29T22:17:45.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggdH5JpcvnCU9dRRPLofFJ8en84='} 2023-09-29T22:17:46.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000533b0cd2c6a309f6-0065174d09-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.019 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:17:46.019 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:17:46.019 DEBUG:boto:Method: HEAD 2023-09-29T22:17:46.019 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:17:46.019 DEBUG:boto:Data: 2023-09-29T22:17:46.019 DEBUG:boto:Headers: {} 2023-09-29T22:17:46.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:46.020 DEBUG:boto:Port: 8000 2023-09-29T22:17:46.020 DEBUG:boto:Params: {} 2023-09-29T22:17:46.020 DEBUG:boto:Token: None 2023-09-29T22:17:46.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-46/ 2023-09-29T22:17:46.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clQcGWnwApdasRTT5rql/BHJRMo= 2023-09-29T22:17:46.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clQcGWnwApdasRTT5rql/BHJRMo=', 'Content-Length': '0'} 2023-09-29T22:17:46.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd52913cfa330204-0065174d0a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.024 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:17:46.024 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:17:46.024 DEBUG:boto:Method: PUT 2023-09-29T22:17:46.024 DEBUG:boto:Path: /hcnphz-46/obj8 2023-09-29T22:17:46.024 DEBUG:boto:Data: 2023-09-29T22:17:46.024 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:46.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:17:46.025 DEBUG:boto:Port: 8000 2023-09-29T22:17:46.025 DEBUG:boto:Params: {} 2023-09-29T22:17:46.025 DEBUG:boto:Token: None 2023-09-29T22:17:46.025 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-46/obj8 2023-09-29T22:17:46.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q7qR2t+85DUxr1lFgkFADCUWgd0= 2023-09-29T22:17:46.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q7qR2t+85DUxr1lFgkFADCUWgd0='} 2023-09-29T22:17:46.036 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d59099fdba9ce8f9-0065174d0a-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.036 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:46.037 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:46.037 DEBUG:boto:Method: HEAD 2023-09-29T22:17:46.037 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:17:46.037 DEBUG:boto:Data: 2023-09-29T22:17:46.037 DEBUG:boto:Headers: {} 2023-09-29T22:17:46.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.037 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.037 DEBUG:boto:Params: {} 2023-09-29T22:17:46.037 DEBUG:boto:Token: None 2023-09-29T22:17:46.038 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/ 2023-09-29T22:17:46.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg= 2023-09-29T22:17:46.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg=', 'Content-Length': '0'} 2023-09-29T22:17:46.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a1a41d5db47d5df-0065174d0a-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.041 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:17:46.041 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:17:46.041 DEBUG:boto:Method: PUT 2023-09-29T22:17:46.041 DEBUG:boto:Path: /hcnphz-47/obj5 2023-09-29T22:17:46.041 DEBUG:boto:Data: 2023-09-29T22:17:46.041 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:46.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.042 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.042 DEBUG:boto:Params: {} 2023-09-29T22:17:46.042 DEBUG:boto:Token: None 2023-09-29T22:17:46.042 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/obj5 2023-09-29T22:17:46.042 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ec8ntVzQVDMXzKYyWw9t+GqnqaY= 2023-09-29T22:17:46.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ec8ntVzQVDMXzKYyWw9t+GqnqaY='} 2023-09-29T22:17:46.060 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000062379b75d8a1bd73-0065174d0a-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.060 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:46.060 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:46.060 DEBUG:boto:Method: HEAD 2023-09-29T22:17:46.061 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:17:46.061 DEBUG:boto:Data: 2023-09-29T22:17:46.061 DEBUG:boto:Headers: {} 2023-09-29T22:17:46.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.061 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.061 DEBUG:boto:Params: {} 2023-09-29T22:17:46.061 DEBUG:boto:Token: None 2023-09-29T22:17:46.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/ 2023-09-29T22:17:46.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg= 2023-09-29T22:17:46.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg=', 'Content-Length': '0'} 2023-09-29T22:17:46.064 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006967a8cf035a6d32-0065174d0a-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.064 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:17:46.064 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:17:46.065 DEBUG:boto:Method: PUT 2023-09-29T22:17:46.065 DEBUG:boto:Path: /hcnphz-47/obj6 2023-09-29T22:17:46.065 DEBUG:boto:Data: 2023-09-29T22:17:46.065 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:46.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.065 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.065 DEBUG:boto:Params: {} 2023-09-29T22:17:46.065 DEBUG:boto:Token: None 2023-09-29T22:17:46.065 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/obj6 2023-09-29T22:17:46.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:knoyfAA7gRWAkguE7K3mnWBor/o= 2023-09-29T22:17:46.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:knoyfAA7gRWAkguE7K3mnWBor/o='} 2023-09-29T22:17:46.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000504496da4d088f2-0065174d0a-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.115 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:46.115 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:46.115 DEBUG:boto:Method: HEAD 2023-09-29T22:17:46.115 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:17:46.115 DEBUG:boto:Data: 2023-09-29T22:17:46.115 DEBUG:boto:Headers: {} 2023-09-29T22:17:46.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.116 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.116 DEBUG:boto:Params: {} 2023-09-29T22:17:46.116 DEBUG:boto:Token: None 2023-09-29T22:17:46.116 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/ 2023-09-29T22:17:46.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg= 2023-09-29T22:17:46.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg=', 'Content-Length': '0'} 2023-09-29T22:17:46.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006aa4ff20c2411686-0065174d0a-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.120 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:17:46.120 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:17:46.120 DEBUG:boto:Method: PUT 2023-09-29T22:17:46.120 DEBUG:boto:Path: /hcnphz-47/obj7 2023-09-29T22:17:46.121 DEBUG:boto:Data: 2023-09-29T22:17:46.121 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:46.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.121 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.121 DEBUG:boto:Params: {} 2023-09-29T22:17:46.121 DEBUG:boto:Token: None 2023-09-29T22:17:46.121 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/obj7 2023-09-29T22:17:46.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i/4S7gIvZi8+0QXOxIJH8OPVKsM= 2023-09-29T22:17:46.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i/4S7gIvZi8+0QXOxIJH8OPVKsM='} 2023-09-29T22:17:46.132 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000118a6289626d2be3-0065174d0a-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.132 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:17:46.132 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:17:46.132 DEBUG:boto:Method: HEAD 2023-09-29T22:17:46.132 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:17:46.132 DEBUG:boto:Data: 2023-09-29T22:17:46.133 DEBUG:boto:Headers: {} 2023-09-29T22:17:46.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.133 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.133 DEBUG:boto:Params: {} 2023-09-29T22:17:46.133 DEBUG:boto:Token: None 2023-09-29T22:17:46.133 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/ 2023-09-29T22:17:46.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg= 2023-09-29T22:17:46.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1gR/Ufwn5uzamU6iHDYv+cZ/ETg=', 'Content-Length': '0'} 2023-09-29T22:17:46.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c6a7ec115a7c25c-0065174d0a-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.136 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:17:46.136 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:17:46.137 DEBUG:boto:Method: PUT 2023-09-29T22:17:46.137 DEBUG:boto:Path: /hcnphz-47/obj8 2023-09-29T22:17:46.137 DEBUG:boto:Data: 2023-09-29T22:17:46.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:17:46.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:17:46.137 DEBUG:boto:Port: 8001 2023-09-29T22:17:46.137 DEBUG:boto:Params: {} 2023-09-29T22:17:46.137 DEBUG:boto:Token: None 2023-09-29T22:17:46.137 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:17:46 GMT /hcnphz-47/obj8 2023-09-29T22:17:46.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UB8iVzharuhPvnWQxkzuiZHn9hw= 2023-09-29T22:17:46.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:17:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UB8iVzharuhPvnWQxkzuiZHn9hw='} 2023-09-29T22:17:46.183 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000828711e8d9377822-0065174d0a-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:17:46.183 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-46 2023-09-29T22:17:46.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:46.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:46.183 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-46 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:46.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:46.237 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:46.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:46.299+0000 7f929da62f40 1 waiting for incremental sync to catch up: 2023-09-29T22:17:46.300 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.25.6, , 00000000001.23.6, , 00000000001.31.6, , , 00000000001.27.6, , , ] 2023-09-29T22:17:46.301 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.35.6, 00000000002.22.1, 00000000004.33.6, 00000000002.22.1, 00000000004.39.6, 00000000002.31.1, 00000000002.23.1, 00000000004.31.6, 00000000002.23.1, 00000000002.24.1, 00000000002.23.1] 2023-09-29T22:17:51.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:51.299+0000 7f929da62f40 1 bucket sync caught up with source: 2023-09-29T22:17:51.303 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.35.6, 00000000002.22.1, 00000000004.33.6, 00000000002.22.1, 00000000004.39.6, 00000000002.31.1, 00000000002.23.1, 00000000004.31.6, 00000000002.23.1, 00000000002.24.1, 00000000002.23.1] 2023-09-29T22:17:51.303 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.35.6, 00000000002.22.1, 00000000004.33.6, 00000000002.22.1, 00000000004.39.6, 00000000002.31.1, 00000000002.23.1, 00000000004.31.6, 00000000002.23.1, 00000000002.24.1, 00000000002.23.1] 2023-09-29T22:17:51.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:51.299+0000 7f929da62f40 0 bucket checkpoint complete 2023-09-29T22:17:51.333 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-46 2023-09-29T22:17:51.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:51.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-46', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:17:51.333 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-46 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:17:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:17:51.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:17:51.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:51.447+0000 7fac6de31f40 1 waiting for incremental sync to catch up: 2023-09-29T22:17:51.450 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.70.6, , 00000000001.71.6, , 00000000001.97.6, , , 00000000001.501.6, , , ] 2023-09-29T22:17:51.450 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.75.6, , 00000000002.74.6, , 00000000002.101.6, , , 00000000002.503.6, , , ] 2023-09-29T22:17:56.452 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:17:56.450+0000 7fac6de31f40 1 waiting for incremental sync to catch up: 2023-09-29T22:17:56.453 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.70.6, , 00000000001.71.6, , 00000000001.97.6, , , 00000000001.501.6, , , ] 2023-09-29T22:17:56.453 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.75.6, , 00000000002.74.6, , 00000000002.101.6, , , 00000000002.503.6, , , ] 2023-09-29T22:18:01.455 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:01.450+0000 7fac6de31f40 1 bucket sync caught up with source: 2023-09-29T22:18:02.218 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.75.6, , 00000000002.74.6, , 00000000002.101.6, , , 00000000002.503.6, , , ] 2023-09-29T22:18:02.219 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.75.6, , 00000000002.74.6, , 00000000002.101.6, , , 00000000002.503.6, , , ] 2023-09-29T22:18:02.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:01.450+0000 7fac6de31f40 0 bucket checkpoint complete 2023-09-29T22:18:02.221 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-46 zones={a2, a1} 2023-09-29T22:18:02.221 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.221 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.221 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.222 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:18:02.222 DEBUG:boto:Data: 2023-09-29T22:18:02.222 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.223 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.223 DEBUG:boto:Params: {} 2023-09-29T22:18:02.223 DEBUG:boto:Token: None 2023-09-29T22:18:02.223 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.223 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ScXdx7GlEKaOs17RBdFSXpgPGSg= 2023-09-29T22:18:02.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ScXdx7GlEKaOs17RBdFSXpgPGSg=', 'Content-Length': '0'} 2023-09-29T22:18:02.228 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0d988ccefe63b71-0065174d1a-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.228 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.228 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.228 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.228 DEBUG:boto:Path: /hcnphz-46/ 2023-09-29T22:18:02.229 DEBUG:boto:Data: 2023-09-29T22:18:02.229 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.229 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.229 DEBUG:boto:Params: {} 2023-09-29T22:18:02.229 DEBUG:boto:Token: None 2023-09-29T22:18:02.229 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ScXdx7GlEKaOs17RBdFSXpgPGSg= 2023-09-29T22:18:02.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ScXdx7GlEKaOs17RBdFSXpgPGSg=', 'Content-Length': '0'} 2023-09-29T22:18:02.233 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5b994e95edafd87-0065174d1a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.234 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:18:02.234 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.234 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.234 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.234 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.234 DEBUG:boto:Method: GET 2023-09-29T22:18:02.234 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.234 DEBUG:boto:Data: 2023-09-29T22:18:02.235 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.235 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.235 DEBUG:boto:Params: {} 2023-09-29T22:18:02.235 DEBUG:boto:Token: None 2023-09-29T22:18:02.235 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/?versions 2023-09-29T22:18:02.235 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM= 2023-09-29T22:18:02.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM=', 'Content-Length': '0'} 2023-09-29T22:18:02.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98691bc3d7d059a-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.315 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:18:02.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:18:02.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:18:02.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:18:02.316 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:18:02.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:18:02.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:18:02.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:18:02.317 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:18:02.317 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.317 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.317 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.317 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.318 DEBUG:boto:Method: GET 2023-09-29T22:18:02.318 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.318 DEBUG:boto:Data: 2023-09-29T22:18:02.318 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.318 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.318 DEBUG:boto:Params: {} 2023-09-29T22:18:02.318 DEBUG:boto:Token: None 2023-09-29T22:18:02.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/?versions 2023-09-29T22:18:02.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM= 2023-09-29T22:18:02.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM=', 'Content-Length': '0'} 2023-09-29T22:18:02.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000480985d2b23cb6d5-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.324 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:18:02.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:18:02.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:18:02.326 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.326 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.326 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.326 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.326 DEBUG:boto:Method: GET 2023-09-29T22:18:02.326 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.326 DEBUG:boto:Data: 2023-09-29T22:18:02.327 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.327 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.327 DEBUG:boto:Params: {} 2023-09-29T22:18:02.327 DEBUG:boto:Token: None 2023-09-29T22:18:02.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/?versions 2023-09-29T22:18:02.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM= 2023-09-29T22:18:02.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM=', 'Content-Length': '0'} 2023-09-29T22:18:02.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080c3ad395a3858a-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.373 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.374 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.374 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.374 DEBUG:boto:path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.374 DEBUG:boto:auth_path=/hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.374 DEBUG:boto:Method: GET 2023-09-29T22:18:02.374 DEBUG:boto:Path: /hcnphz-46/?versions&max-keys=999 2023-09-29T22:18:02.374 DEBUG:boto:Data: 2023-09-29T22:18:02.374 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.375 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.375 DEBUG:boto:Params: {} 2023-09-29T22:18:02.375 DEBUG:boto:Token: None 2023-09-29T22:18:02.375 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/?versions 2023-09-29T22:18:02.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM= 2023-09-29T22:18:02.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JiZo2nufAeucQj8mRNrDrPFpSVM=', 'Content-Length': '0'} 2023-09-29T22:18:02.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f426e3107972e0d7-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.381 DEBUG:boto:b'hcnphz-46999falseobj1nulltrue2023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:18:02.382 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.382 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.382 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.382 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.382 DEBUG:boto:Method: GET 2023-09-29T22:18:02.383 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.383 DEBUG:boto:Data: 2023-09-29T22:18:02.383 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.383 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.383 DEBUG:boto:Params: {} 2023-09-29T22:18:02.383 DEBUG:boto:Token: None 2023-09-29T22:18:02.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.383 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w= 2023-09-29T22:18:02.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w=', 'Content-Length': '0'} 2023-09-29T22:18:02.386 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030e5f0664c7ca695-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.386 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.387 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.387 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.387 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.387 DEBUG:boto:Method: GET 2023-09-29T22:18:02.387 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.387 DEBUG:boto:Data: 2023-09-29T22:18:02.387 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.387 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.388 DEBUG:boto:Params: {} 2023-09-29T22:18:02.388 DEBUG:boto:Token: None 2023-09-29T22:18:02.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w= 2023-09-29T22:18:02.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w=', 'Content-Length': '0'} 2023-09-29T22:18:02.391 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d7c3e0fa1ac26412-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.553 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.553 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.553 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.553 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.553 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.553 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.554 DEBUG:boto:Data: 2023-09-29T22:18:02.554 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.554 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.554 DEBUG:boto:Params: {} 2023-09-29T22:18:02.554 DEBUG:boto:Token: None 2023-09-29T22:18:02.554 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.554 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IKNW+OxHC/WgoiWN3fDCGygRO+U= 2023-09-29T22:18:02.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IKNW+OxHC/WgoiWN3fDCGygRO+U=', 'Content-Length': '0'} 2023-09-29T22:18:02.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000adf5c8710ea0834a-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.557 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.558 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.558 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.558 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.558 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.558 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.558 DEBUG:boto:Data: 2023-09-29T22:18:02.558 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.559 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.559 DEBUG:boto:Params: {} 2023-09-29T22:18:02.560 DEBUG:boto:Token: None 2023-09-29T22:18:02.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IKNW+OxHC/WgoiWN3fDCGygRO+U= 2023-09-29T22:18:02.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IKNW+OxHC/WgoiWN3fDCGygRO+U=', 'Content-Length': '0'} 2023-09-29T22:18:02.565 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000937d0ec9a0b9c592-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:18:02.565 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.565 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.565 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.566 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.566 DEBUG:boto:Method: GET 2023-09-29T22:18:02.566 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.566 DEBUG:boto:Data: 2023-09-29T22:18:02.566 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.566 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.566 DEBUG:boto:Params: {} 2023-09-29T22:18:02.567 DEBUG:boto:Token: None 2023-09-29T22:18:02.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w= 2023-09-29T22:18:02.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w=', 'Content-Length': '0'} 2023-09-29T22:18:02.570 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000097c8509de12a8ee7-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.570 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.570 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.570 DEBUG:boto:path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.573 DEBUG:boto:auth_path=/hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.574 DEBUG:boto:Method: GET 2023-09-29T22:18:02.574 DEBUG:boto:Path: /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.574 DEBUG:boto:Data: 2023-09-29T22:18:02.574 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.574 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.574 DEBUG:boto:Params: {} 2023-09-29T22:18:02.575 DEBUG:boto:Token: None 2023-09-29T22:18:02.575 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1?versionId=null 2023-09-29T22:18:02.575 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w= 2023-09-29T22:18:02.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BmdaiiDoDaBVS4TDAtrw4ASyP9w=', 'Content-Length': '0'} 2023-09-29T22:18:02.578 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000977f984895d18565-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.578 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.578 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.578 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.579 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.579 DEBUG:boto:Method: GET 2023-09-29T22:18:02.579 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.579 DEBUG:boto:Data: 2023-09-29T22:18:02.579 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.579 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.579 DEBUG:boto:Params: {} 2023-09-29T22:18:02.579 DEBUG:boto:Token: None 2023-09-29T22:18:02.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9bdf20f43c860a2-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.585 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.585 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.585 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.585 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.585 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.586 DEBUG:boto:Method: GET 2023-09-29T22:18:02.586 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.586 DEBUG:boto:Data: 2023-09-29T22:18:02.586 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.586 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.586 DEBUG:boto:Params: {} 2023-09-29T22:18:02.586 DEBUG:boto:Token: None 2023-09-29T22:18:02.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.591 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f415df27bf8a14e-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.591 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.592 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.592 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.592 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.592 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:18:02.592 DEBUG:boto:Data: 2023-09-29T22:18:02.592 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.593 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.593 DEBUG:boto:Params: {} 2023-09-29T22:18:02.593 DEBUG:boto:Token: None 2023-09-29T22:18:02.593 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1 2023-09-29T22:18:02.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DpyodxQ5PPpmVeXC9iN4t4el/RI= 2023-09-29T22:18:02.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpyodxQ5PPpmVeXC9iN4t4el/RI=', 'Content-Length': '0'} 2023-09-29T22:18:02.595 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000057749a9b72d3a3c0-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.596 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.596 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.596 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.596 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:18:02.596 DEBUG:boto:Data: 2023-09-29T22:18:02.596 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.596 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.597 DEBUG:boto:Params: {} 2023-09-29T22:18:02.597 DEBUG:boto:Token: None 2023-09-29T22:18:02.597 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1 2023-09-29T22:18:02.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DpyodxQ5PPpmVeXC9iN4t4el/RI= 2023-09-29T22:18:02.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpyodxQ5PPpmVeXC9iN4t4el/RI=', 'Content-Length': '0'} 2023-09-29T22:18:02.599 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f7b218d7bc57a1e-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:18:02.600 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.600 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.600 DEBUG:boto:Method: GET 2023-09-29T22:18:02.600 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:18:02.600 DEBUG:boto:Data: 2023-09-29T22:18:02.600 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.601 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.601 DEBUG:boto:Params: {} 2023-09-29T22:18:02.601 DEBUG:boto:Token: None 2023-09-29T22:18:02.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1 2023-09-29T22:18:02.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ZSLKkTsjn4sfCtscvf3ZhV6nAY= 2023-09-29T22:18:02.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ZSLKkTsjn4sfCtscvf3ZhV6nAY=', 'Content-Length': '0'} 2023-09-29T22:18:02.604 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f966032d4b3b2db-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.646 DEBUG:boto:path=/hcnphz-46/obj1 2023-09-29T22:18:02.647 DEBUG:boto:auth_path=/hcnphz-46/obj1 2023-09-29T22:18:02.647 DEBUG:boto:Method: GET 2023-09-29T22:18:02.647 DEBUG:boto:Path: /hcnphz-46/obj1 2023-09-29T22:18:02.647 DEBUG:boto:Data: 2023-09-29T22:18:02.647 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.647 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.647 DEBUG:boto:Params: {} 2023-09-29T22:18:02.648 DEBUG:boto:Token: None 2023-09-29T22:18:02.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj1 2023-09-29T22:18:02.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ZSLKkTsjn4sfCtscvf3ZhV6nAY= 2023-09-29T22:18:02.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ZSLKkTsjn4sfCtscvf3ZhV6nAY=', 'Content-Length': '0'} 2023-09-29T22:18:02.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004d29da9ab8f9617-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.651 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.651 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.651 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.652 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.652 DEBUG:boto:Method: GET 2023-09-29T22:18:02.652 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.652 DEBUG:boto:Data: 2023-09-29T22:18:02.652 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.652 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.653 DEBUG:boto:Params: {} 2023-09-29T22:18:02.653 DEBUG:boto:Token: None 2023-09-29T22:18:02.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e208dc0edf2ec1ad-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.659 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.659 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.659 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.659 DEBUG:boto:path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.659 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.660 DEBUG:boto:Method: GET 2023-09-29T22:18:02.660 DEBUG:boto:Path: /hcnphz-46/?prefix=obj1 2023-09-29T22:18:02.660 DEBUG:boto:Data: 2023-09-29T22:18:02.660 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.660 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.660 DEBUG:boto:Params: {} 2023-09-29T22:18:02.660 DEBUG:boto:Token: None 2023-09-29T22:18:02.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca01feabe9e9016a-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.706 DEBUG:boto:b'hcnphz-46obj11000falseobj12023-09-29T22:15:12.350Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:18:02.707 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.707 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.707 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.707 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.708 DEBUG:boto:Method: GET 2023-09-29T22:18:02.708 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.708 DEBUG:boto:Data: 2023-09-29T22:18:02.708 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.708 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.708 DEBUG:boto:Params: {} 2023-09-29T22:18:02.708 DEBUG:boto:Token: None 2023-09-29T22:18:02.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs= 2023-09-29T22:18:02.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs=', 'Content-Length': '0'} 2023-09-29T22:18:02.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000564d62728bc06bb7-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.712 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.712 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.712 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.712 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.712 DEBUG:boto:Method: GET 2023-09-29T22:18:02.712 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.713 DEBUG:boto:Data: 2023-09-29T22:18:02.713 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.713 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.713 DEBUG:boto:Params: {} 2023-09-29T22:18:02.713 DEBUG:boto:Token: None 2023-09-29T22:18:02.713 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs= 2023-09-29T22:18:02.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs=', 'Content-Length': '0'} 2023-09-29T22:18:02.716 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010cc94260af297dd-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.716 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.717 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.717 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.717 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.717 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.717 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.717 DEBUG:boto:Data: 2023-09-29T22:18:02.717 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.717 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.718 DEBUG:boto:Params: {} 2023-09-29T22:18:02.718 DEBUG:boto:Token: None 2023-09-29T22:18:02.718 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z96sfTAj1aUx4NwgQvWHxR4S+jY= 2023-09-29T22:18:02.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z96sfTAj1aUx4NwgQvWHxR4S+jY=', 'Content-Length': '0'} 2023-09-29T22:18:02.721 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000033c1573ee5892d98-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.721 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.721 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.721 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.721 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.721 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.721 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.722 DEBUG:boto:Data: 2023-09-29T22:18:02.722 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.722 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.722 DEBUG:boto:Params: {} 2023-09-29T22:18:02.722 DEBUG:boto:Token: None 2023-09-29T22:18:02.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z96sfTAj1aUx4NwgQvWHxR4S+jY= 2023-09-29T22:18:02.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z96sfTAj1aUx4NwgQvWHxR4S+jY=', 'Content-Length': '0'} 2023-09-29T22:18:02.725 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005569525115929c9e-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:18:02.726 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.726 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.726 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.726 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.726 DEBUG:boto:Method: GET 2023-09-29T22:18:02.726 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.727 DEBUG:boto:Data: 2023-09-29T22:18:02.727 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.727 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.727 DEBUG:boto:Params: {} 2023-09-29T22:18:02.727 DEBUG:boto:Token: None 2023-09-29T22:18:02.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs= 2023-09-29T22:18:02.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs=', 'Content-Length': '0'} 2023-09-29T22:18:02.730 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002608d95f05cb1cc5-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.731 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.731 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.731 DEBUG:boto:path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.731 DEBUG:boto:auth_path=/hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.731 DEBUG:boto:Method: GET 2023-09-29T22:18:02.731 DEBUG:boto:Path: /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.731 DEBUG:boto:Data: 2023-09-29T22:18:02.731 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.732 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.732 DEBUG:boto:Params: {} 2023-09-29T22:18:02.732 DEBUG:boto:Token: None 2023-09-29T22:18:02.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2?versionId=null 2023-09-29T22:18:02.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs= 2023-09-29T22:18:02.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:84ddD232LnhZsaKQFjsZlVqX2Fs=', 'Content-Length': '0'} 2023-09-29T22:18:02.735 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058a6cf0d7c2d23ee-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.735 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.736 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.736 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.736 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.736 DEBUG:boto:Method: GET 2023-09-29T22:18:02.736 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.736 DEBUG:boto:Data: 2023-09-29T22:18:02.736 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.736 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.737 DEBUG:boto:Params: {} 2023-09-29T22:18:02.737 DEBUG:boto:Token: None 2023-09-29T22:18:02.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.737 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ba373ee8199c88-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.742 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.742 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.743 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.743 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.743 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.743 DEBUG:boto:Method: GET 2023-09-29T22:18:02.743 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.743 DEBUG:boto:Data: 2023-09-29T22:18:02.743 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.743 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.743 DEBUG:boto:Params: {} 2023-09-29T22:18:02.744 DEBUG:boto:Token: None 2023-09-29T22:18:02.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cba0728176fcbf4a-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.794 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.795 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.795 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.795 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.795 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:18:02.795 DEBUG:boto:Data: 2023-09-29T22:18:02.795 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.796 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.796 DEBUG:boto:Params: {} 2023-09-29T22:18:02.796 DEBUG:boto:Token: None 2023-09-29T22:18:02.796 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2 2023-09-29T22:18:02.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V8lAd6LIomBUKHss4CZq19nw5gw= 2023-09-29T22:18:02.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V8lAd6LIomBUKHss4CZq19nw5gw=', 'Content-Length': '0'} 2023-09-29T22:18:02.799 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b0ab913adcfcdb4-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.799 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.799 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.799 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.799 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:18:02.800 DEBUG:boto:Data: 2023-09-29T22:18:02.800 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.800 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.800 DEBUG:boto:Params: {} 2023-09-29T22:18:02.800 DEBUG:boto:Token: None 2023-09-29T22:18:02.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2 2023-09-29T22:18:02.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V8lAd6LIomBUKHss4CZq19nw5gw= 2023-09-29T22:18:02.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V8lAd6LIomBUKHss4CZq19nw5gw=', 'Content-Length': '0'} 2023-09-29T22:18:02.802 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003454cb8a84ab6e82-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:18:02.803 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.803 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.803 DEBUG:boto:Method: GET 2023-09-29T22:18:02.803 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:18:02.803 DEBUG:boto:Data: 2023-09-29T22:18:02.803 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.803 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.803 DEBUG:boto:Params: {} 2023-09-29T22:18:02.804 DEBUG:boto:Token: None 2023-09-29T22:18:02.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2 2023-09-29T22:18:02.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+AAD7WMK3P32BKCqJT3rXVKMREo= 2023-09-29T22:18:02.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+AAD7WMK3P32BKCqJT3rXVKMREo=', 'Content-Length': '0'} 2023-09-29T22:18:02.806 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fee82ee5ff211816-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.850 DEBUG:boto:path=/hcnphz-46/obj2 2023-09-29T22:18:02.851 DEBUG:boto:auth_path=/hcnphz-46/obj2 2023-09-29T22:18:02.851 DEBUG:boto:Method: GET 2023-09-29T22:18:02.851 DEBUG:boto:Path: /hcnphz-46/obj2 2023-09-29T22:18:02.851 DEBUG:boto:Data: 2023-09-29T22:18:02.851 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.851 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.851 DEBUG:boto:Params: {} 2023-09-29T22:18:02.852 DEBUG:boto:Token: None 2023-09-29T22:18:02.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj2 2023-09-29T22:18:02.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+AAD7WMK3P32BKCqJT3rXVKMREo= 2023-09-29T22:18:02.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+AAD7WMK3P32BKCqJT3rXVKMREo=', 'Content-Length': '0'} 2023-09-29T22:18:02.855 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9b2d6f311b3367f-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.855 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.855 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.855 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.856 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.856 DEBUG:boto:Method: GET 2023-09-29T22:18:02.856 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.856 DEBUG:boto:Data: 2023-09-29T22:18:02.856 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.856 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.857 DEBUG:boto:Params: {} 2023-09-29T22:18:02.857 DEBUG:boto:Token: None 2023-09-29T22:18:02.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000837a7711ac56b1f6-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.863 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.863 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.863 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.864 DEBUG:boto:path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.864 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.864 DEBUG:boto:Method: GET 2023-09-29T22:18:02.864 DEBUG:boto:Path: /hcnphz-46/?prefix=obj2 2023-09-29T22:18:02.864 DEBUG:boto:Data: 2023-09-29T22:18:02.864 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.864 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.864 DEBUG:boto:Params: {} 2023-09-29T22:18:02.865 DEBUG:boto:Token: None 2023-09-29T22:18:02.865 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edfef209419057db-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.869 DEBUG:boto:b'hcnphz-46obj21000falseobj22023-09-29T22:15:12.368Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:18:02.870 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:02.870 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:02.871 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.871 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.871 DEBUG:boto:Method: GET 2023-09-29T22:18:02.871 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.871 DEBUG:boto:Data: 2023-09-29T22:18:02.871 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.871 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.871 DEBUG:boto:Params: {} 2023-09-29T22:18:02.872 DEBUG:boto:Token: None 2023-09-29T22:18:02.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw= 2023-09-29T22:18:02.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw=', 'Content-Length': '0'} 2023-09-29T22:18:02.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fcf5162c893503f7-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.875 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:02.875 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:02.875 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.875 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.875 DEBUG:boto:Method: GET 2023-09-29T22:18:02.875 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.876 DEBUG:boto:Data: 2023-09-29T22:18:02.876 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.876 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.876 DEBUG:boto:Params: {} 2023-09-29T22:18:02.876 DEBUG:boto:Token: None 2023-09-29T22:18:02.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw= 2023-09-29T22:18:02.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw=', 'Content-Length': '0'} 2023-09-29T22:18:02.879 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff9d5edceb0c5b36-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.926 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:02.927 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:02.927 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.927 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.927 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.927 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.927 DEBUG:boto:Data: 2023-09-29T22:18:02.927 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.928 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.928 DEBUG:boto:Params: {} 2023-09-29T22:18:02.928 DEBUG:boto:Token: None 2023-09-29T22:18:02.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jLpl4RnnM2+Za7A+qm9P8ZnHtnQ= 2023-09-29T22:18:02.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jLpl4RnnM2+Za7A+qm9P8ZnHtnQ=', 'Content-Length': '0'} 2023-09-29T22:18:02.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fe110bd0a9fd3428-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.932 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:02.932 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:02.932 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.932 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.932 DEBUG:boto:Method: HEAD 2023-09-29T22:18:02.932 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.932 DEBUG:boto:Data: 2023-09-29T22:18:02.932 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.933 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.933 DEBUG:boto:Params: {} 2023-09-29T22:18:02.933 DEBUG:boto:Token: None 2023-09-29T22:18:02.933 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.933 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jLpl4RnnM2+Za7A+qm9P8ZnHtnQ= 2023-09-29T22:18:02.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jLpl4RnnM2+Za7A+qm9P8ZnHtnQ=', 'Content-Length': '0'} 2023-09-29T22:18:02.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e37e9a530b94d07a-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:18:02.936 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:02.936 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:02.936 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.936 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.937 DEBUG:boto:Method: GET 2023-09-29T22:18:02.937 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.937 DEBUG:boto:Data: 2023-09-29T22:18:02.937 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.937 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.937 DEBUG:boto:Params: {} 2023-09-29T22:18:02.937 DEBUG:boto:Token: None 2023-09-29T22:18:02.937 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.938 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw= 2023-09-29T22:18:02.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw=', 'Content-Length': '0'} 2023-09-29T22:18:02.940 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e32067df3ca13b2-0065174d1a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.982 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:02.983 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:02.983 DEBUG:boto:path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.983 DEBUG:boto:auth_path=/hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.983 DEBUG:boto:Method: GET 2023-09-29T22:18:02.983 DEBUG:boto:Path: /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.983 DEBUG:boto:Data: 2023-09-29T22:18:02.984 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.984 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.984 DEBUG:boto:Params: {} 2023-09-29T22:18:02.984 DEBUG:boto:Token: None 2023-09-29T22:18:02.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/obj3?versionId=null 2023-09-29T22:18:02.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw= 2023-09-29T22:18:02.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HH4W+hhncbqA2SAzoNfYHSn3Irw=', 'Content-Length': '0'} 2023-09-29T22:18:02.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000098cb13141579280c-0065174d1a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.988 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.988 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.988 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:02.988 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:02.988 DEBUG:boto:Method: GET 2023-09-29T22:18:02.988 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:18:02.988 DEBUG:boto:Data: 2023-09-29T22:18:02.989 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:02.989 DEBUG:boto:Port: 8001 2023-09-29T22:18:02.989 DEBUG:boto:Params: {} 2023-09-29T22:18:02.989 DEBUG:boto:Token: None 2023-09-29T22:18:02.989 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:02.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ab5db2907b8145e-0065174d1a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:02.994 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:02.995 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:02.995 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:02.996 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:02.996 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:02.996 DEBUG:boto:Method: GET 2023-09-29T22:18:02.996 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:18:02.997 DEBUG:boto:Data: 2023-09-29T22:18:02.997 DEBUG:boto:Headers: {} 2023-09-29T22:18:02.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:02.997 DEBUG:boto:Port: 8000 2023-09-29T22:18:02.997 DEBUG:boto:Params: {} 2023-09-29T22:18:02.998 DEBUG:boto:Token: None 2023-09-29T22:18:02.998 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:02 GMT /hcnphz-46/ 2023-09-29T22:18:02.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk= 2023-09-29T22:18:02.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/n8758NqrlF+CNnzZkeV2gcTWLk=', 'Content-Length': '0'} 2023-09-29T22:18:03.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3299988339a51fc-0065174d1a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.003 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.003 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:03.004 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:03.004 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.004 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:18:03.004 DEBUG:boto:Data: 2023-09-29T22:18:03.004 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.004 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.004 DEBUG:boto:Params: {} 2023-09-29T22:18:03.004 DEBUG:boto:Token: None 2023-09-29T22:18:03.005 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj3 2023-09-29T22:18:03.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h8Z48Uo1hZKm6KnGuaa8yySmHNw= 2023-09-29T22:18:03.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h8Z48Uo1hZKm6KnGuaa8yySmHNw=', 'Content-Length': '0'} 2023-09-29T22:18:03.007 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5952cd2d311b570-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.007 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:03.008 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:03.008 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.008 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:18:03.008 DEBUG:boto:Data: 2023-09-29T22:18:03.008 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.008 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.008 DEBUG:boto:Params: {} 2023-09-29T22:18:03.008 DEBUG:boto:Token: None 2023-09-29T22:18:03.009 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj3 2023-09-29T22:18:03.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h8Z48Uo1hZKm6KnGuaa8yySmHNw= 2023-09-29T22:18:03.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h8Z48Uo1hZKm6KnGuaa8yySmHNw=', 'Content-Length': '0'} 2023-09-29T22:18:03.011 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002bf194ad52ea7306-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:18:03.012 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:03.012 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:03.012 DEBUG:boto:Method: GET 2023-09-29T22:18:03.012 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:18:03.012 DEBUG:boto:Data: 2023-09-29T22:18:03.012 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.012 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.012 DEBUG:boto:Params: {} 2023-09-29T22:18:03.013 DEBUG:boto:Token: None 2023-09-29T22:18:03.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj3 2023-09-29T22:18:03.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L0WB//uKLoedTXDEZA1Ce6GlEL4= 2023-09-29T22:18:03.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L0WB//uKLoedTXDEZA1Ce6GlEL4=', 'Content-Length': '0'} 2023-09-29T22:18:03.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a43604e31c26f31-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.062 DEBUG:boto:path=/hcnphz-46/obj3 2023-09-29T22:18:03.063 DEBUG:boto:auth_path=/hcnphz-46/obj3 2023-09-29T22:18:03.063 DEBUG:boto:Method: GET 2023-09-29T22:18:03.063 DEBUG:boto:Path: /hcnphz-46/obj3 2023-09-29T22:18:03.063 DEBUG:boto:Data: 2023-09-29T22:18:03.063 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.063 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.063 DEBUG:boto:Params: {} 2023-09-29T22:18:03.063 DEBUG:boto:Token: None 2023-09-29T22:18:03.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj3 2023-09-29T22:18:03.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L0WB//uKLoedTXDEZA1Ce6GlEL4= 2023-09-29T22:18:03.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L0WB//uKLoedTXDEZA1Ce6GlEL4=', 'Content-Length': '0'} 2023-09-29T22:18:03.066 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f47ee1fa1a540d30-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.067 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.067 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.067 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:03.067 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:03.067 DEBUG:boto:Method: GET 2023-09-29T22:18:03.067 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:18:03.067 DEBUG:boto:Data: 2023-09-29T22:18:03.067 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.068 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.068 DEBUG:boto:Params: {} 2023-09-29T22:18:03.068 DEBUG:boto:Token: None 2023-09-29T22:18:03.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.074 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a87402bb0ef13508-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.074 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.075 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.075 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.075 DEBUG:boto:path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:03.075 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj3 2023-09-29T22:18:03.075 DEBUG:boto:Method: GET 2023-09-29T22:18:03.076 DEBUG:boto:Path: /hcnphz-46/?prefix=obj3 2023-09-29T22:18:03.076 DEBUG:boto:Data: 2023-09-29T22:18:03.076 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.076 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.076 DEBUG:boto:Params: {} 2023-09-29T22:18:03.076 DEBUG:boto:Token: None 2023-09-29T22:18:03.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.081 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006b9fb6a9d6a74d5-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.081 DEBUG:boto:b'hcnphz-46obj31000falseobj32023-09-29T22:15:12.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:18:03.082 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.082 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.082 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.082 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.083 DEBUG:boto:Method: GET 2023-09-29T22:18:03.083 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.083 DEBUG:boto:Data: 2023-09-29T22:18:03.083 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.083 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.083 DEBUG:boto:Params: {} 2023-09-29T22:18:03.083 DEBUG:boto:Token: None 2023-09-29T22:18:03.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE= 2023-09-29T22:18:03.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE=', 'Content-Length': '0'} 2023-09-29T22:18:03.086 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d06ea8a0e04a87d3-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.086 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.086 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.086 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.086 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.087 DEBUG:boto:Method: GET 2023-09-29T22:18:03.087 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.087 DEBUG:boto:Data: 2023-09-29T22:18:03.087 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.087 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.087 DEBUG:boto:Params: {} 2023-09-29T22:18:03.087 DEBUG:boto:Token: None 2023-09-29T22:18:03.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE= 2023-09-29T22:18:03.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE=', 'Content-Length': '0'} 2023-09-29T22:18:03.090 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000958ae2784c0b8544-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.090 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.090 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.090 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.091 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.091 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.091 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.091 DEBUG:boto:Data: 2023-09-29T22:18:03.091 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.091 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.091 DEBUG:boto:Params: {} 2023-09-29T22:18:03.091 DEBUG:boto:Token: None 2023-09-29T22:18:03.092 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DTDXTYji10OL+6GnBoCaNzEm7KU= 2023-09-29T22:18:03.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DTDXTYji10OL+6GnBoCaNzEm7KU=', 'Content-Length': '0'} 2023-09-29T22:18:03.094 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aac65e8e5fbcfd0d-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.094 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.094 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.094 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.094 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.095 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.095 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.095 DEBUG:boto:Data: 2023-09-29T22:18:03.095 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.095 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.095 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.095 DEBUG:boto:Params: {} 2023-09-29T22:18:03.095 DEBUG:boto:Token: None 2023-09-29T22:18:03.095 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.096 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DTDXTYji10OL+6GnBoCaNzEm7KU= 2023-09-29T22:18:03.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DTDXTYji10OL+6GnBoCaNzEm7KU=', 'Content-Length': '0'} 2023-09-29T22:18:03.098 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025692b5621943287-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:18:03.098 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.098 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.098 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.098 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.099 DEBUG:boto:Method: GET 2023-09-29T22:18:03.099 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.099 DEBUG:boto:Data: 2023-09-29T22:18:03.099 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.099 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.099 DEBUG:boto:Params: {} 2023-09-29T22:18:03.099 DEBUG:boto:Token: None 2023-09-29T22:18:03.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE= 2023-09-29T22:18:03.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE=', 'Content-Length': '0'} 2023-09-29T22:18:03.102 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000551747ec0289ec18-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.102 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.103 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.103 DEBUG:boto:path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.103 DEBUG:boto:auth_path=/hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.103 DEBUG:boto:Method: GET 2023-09-29T22:18:03.103 DEBUG:boto:Path: /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.103 DEBUG:boto:Data: 2023-09-29T22:18:03.103 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.103 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.104 DEBUG:boto:Params: {} 2023-09-29T22:18:03.104 DEBUG:boto:Token: None 2023-09-29T22:18:03.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4?versionId=null 2023-09-29T22:18:03.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE= 2023-09-29T22:18:03.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rZMJryfmHZP4SB6OUOzUAIP1cEE=', 'Content-Length': '0'} 2023-09-29T22:18:03.106 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9c1463398cd711c-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.107 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.107 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.107 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.107 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.107 DEBUG:boto:Method: GET 2023-09-29T22:18:03.107 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.107 DEBUG:boto:Data: 2023-09-29T22:18:03.107 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.108 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.108 DEBUG:boto:Params: {} 2023-09-29T22:18:03.108 DEBUG:boto:Token: None 2023-09-29T22:18:03.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.112 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000541e107e3322dcf2-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.154 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.155 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.155 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.155 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.155 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.156 DEBUG:boto:Method: GET 2023-09-29T22:18:03.156 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.156 DEBUG:boto:Data: 2023-09-29T22:18:03.156 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.156 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.156 DEBUG:boto:Params: {} 2023-09-29T22:18:03.156 DEBUG:boto:Token: None 2023-09-29T22:18:03.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.162 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c10e254e0b31090f-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.162 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.163 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.163 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.163 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.163 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:18:03.163 DEBUG:boto:Data: 2023-09-29T22:18:03.163 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.164 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.164 DEBUG:boto:Params: {} 2023-09-29T22:18:03.164 DEBUG:boto:Token: None 2023-09-29T22:18:03.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4 2023-09-29T22:18:03.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h9Us8WdxK6YPCq9ERKnIecGD3/I= 2023-09-29T22:18:03.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h9Us8WdxK6YPCq9ERKnIecGD3/I=', 'Content-Length': '0'} 2023-09-29T22:18:03.167 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cb1d122916b59bae-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.167 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.167 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.167 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.167 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:18:03.167 DEBUG:boto:Data: 2023-09-29T22:18:03.167 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.168 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.168 DEBUG:boto:Params: {} 2023-09-29T22:18:03.168 DEBUG:boto:Token: None 2023-09-29T22:18:03.168 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4 2023-09-29T22:18:03.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h9Us8WdxK6YPCq9ERKnIecGD3/I= 2023-09-29T22:18:03.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h9Us8WdxK6YPCq9ERKnIecGD3/I=', 'Content-Length': '0'} 2023-09-29T22:18:03.171 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007240a52f1b388a05-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:18:03.171 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.171 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.171 DEBUG:boto:Method: GET 2023-09-29T22:18:03.171 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:18:03.172 DEBUG:boto:Data: 2023-09-29T22:18:03.172 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.172 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.172 DEBUG:boto:Params: {} 2023-09-29T22:18:03.172 DEBUG:boto:Token: None 2023-09-29T22:18:03.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4 2023-09-29T22:18:03.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V5dgmSBPUuibY5RsS6pv7sRxK6Q= 2023-09-29T22:18:03.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V5dgmSBPUuibY5RsS6pv7sRxK6Q=', 'Content-Length': '0'} 2023-09-29T22:18:03.175 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b9a06c48bfe4953-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.175 DEBUG:boto:path=/hcnphz-46/obj4 2023-09-29T22:18:03.175 DEBUG:boto:auth_path=/hcnphz-46/obj4 2023-09-29T22:18:03.175 DEBUG:boto:Method: GET 2023-09-29T22:18:03.176 DEBUG:boto:Path: /hcnphz-46/obj4 2023-09-29T22:18:03.176 DEBUG:boto:Data: 2023-09-29T22:18:03.176 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.176 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.176 DEBUG:boto:Params: {} 2023-09-29T22:18:03.176 DEBUG:boto:Token: None 2023-09-29T22:18:03.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj4 2023-09-29T22:18:03.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V5dgmSBPUuibY5RsS6pv7sRxK6Q= 2023-09-29T22:18:03.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V5dgmSBPUuibY5RsS6pv7sRxK6Q=', 'Content-Length': '0'} 2023-09-29T22:18:03.179 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aadcc599dcf1a8f7-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.222 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.223 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.223 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.223 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.223 DEBUG:boto:Method: GET 2023-09-29T22:18:03.223 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.223 DEBUG:boto:Data: 2023-09-29T22:18:03.223 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.224 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.224 DEBUG:boto:Params: {} 2023-09-29T22:18:03.224 DEBUG:boto:Token: None 2023-09-29T22:18:03.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000982c647f3444b2fd-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.230 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.230 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.230 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.231 DEBUG:boto:path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.231 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.231 DEBUG:boto:Method: GET 2023-09-29T22:18:03.231 DEBUG:boto:Path: /hcnphz-46/?prefix=obj4 2023-09-29T22:18:03.231 DEBUG:boto:Data: 2023-09-29T22:18:03.231 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.231 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.231 DEBUG:boto:Params: {} 2023-09-29T22:18:03.232 DEBUG:boto:Token: None 2023-09-29T22:18:03.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e0ad39439689414-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.237 DEBUG:boto:b'hcnphz-46obj41000falseobj42023-09-29T22:15:12.445Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.238 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:18:03.238 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.238 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.238 DEBUG:boto:path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.239 DEBUG:boto:auth_path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.239 DEBUG:boto:Method: GET 2023-09-29T22:18:03.239 DEBUG:boto:Path: /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.239 DEBUG:boto:Data: 2023-09-29T22:18:03.240 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.240 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.240 DEBUG:boto:Params: {} 2023-09-29T22:18:03.240 DEBUG:boto:Token: None 2023-09-29T22:18:03.241 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0= 2023-09-29T22:18:03.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0=', 'Content-Length': '0'} 2023-09-29T22:18:03.244 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000150a6a0de9482947-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.290 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.291 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.291 DEBUG:boto:path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.291 DEBUG:boto:auth_path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.291 DEBUG:boto:Method: GET 2023-09-29T22:18:03.291 DEBUG:boto:Path: /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.291 DEBUG:boto:Data: 2023-09-29T22:18:03.291 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.292 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.292 DEBUG:boto:Params: {} 2023-09-29T22:18:03.292 DEBUG:boto:Token: None 2023-09-29T22:18:03.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0= 2023-09-29T22:18:03.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0=', 'Content-Length': '0'} 2023-09-29T22:18:03.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a9c9a2d142b5e04-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.296 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.296 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.296 DEBUG:boto:path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.296 DEBUG:boto:auth_path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.296 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.297 DEBUG:boto:Path: /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.297 DEBUG:boto:Data: 2023-09-29T22:18:03.297 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.297 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.298 DEBUG:boto:Params: {} 2023-09-29T22:18:03.298 DEBUG:boto:Token: None 2023-09-29T22:18:03.298 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hKXz0St9TJxNCBF8USfU9EB2XiY= 2023-09-29T22:18:03.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hKXz0St9TJxNCBF8USfU9EB2XiY=', 'Content-Length': '0'} 2023-09-29T22:18:03.302 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef9e48fe431f1cfd-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.302 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.302 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.302 DEBUG:boto:path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.302 DEBUG:boto:auth_path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.302 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.302 DEBUG:boto:Path: /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.303 DEBUG:boto:Data: 2023-09-29T22:18:03.303 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.303 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.303 DEBUG:boto:Params: {} 2023-09-29T22:18:03.303 DEBUG:boto:Token: None 2023-09-29T22:18:03.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hKXz0St9TJxNCBF8USfU9EB2XiY= 2023-09-29T22:18:03.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hKXz0St9TJxNCBF8USfU9EB2XiY=', 'Content-Length': '0'} 2023-09-29T22:18:03.306 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dab7600f0f2ce22b-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:18:03.307 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.307 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.307 DEBUG:boto:path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.307 DEBUG:boto:auth_path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.307 DEBUG:boto:Method: GET 2023-09-29T22:18:03.308 DEBUG:boto:Path: /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.308 DEBUG:boto:Data: 2023-09-29T22:18:03.308 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.308 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.309 DEBUG:boto:Params: {} 2023-09-29T22:18:03.309 DEBUG:boto:Token: None 2023-09-29T22:18:03.309 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.309 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0= 2023-09-29T22:18:03.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0=', 'Content-Length': '0'} 2023-09-29T22:18:03.312 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff5d140ef22f5168-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.313 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.313 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.313 DEBUG:boto:path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.313 DEBUG:boto:auth_path=/hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.313 DEBUG:boto:Method: GET 2023-09-29T22:18:03.313 DEBUG:boto:Path: /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.313 DEBUG:boto:Data: 2023-09-29T22:18:03.313 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.314 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.314 DEBUG:boto:Params: {} 2023-09-29T22:18:03.314 DEBUG:boto:Token: None 2023-09-29T22:18:03.314 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5?versionId=null 2023-09-29T22:18:03.314 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0= 2023-09-29T22:18:03.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HYGCk/A1Uh0ChOXZ6C1PTNTfHh0=', 'Content-Length': '0'} 2023-09-29T22:18:03.317 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da9022b76f716038-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.317 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.317 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.318 DEBUG:boto:path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.318 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.318 DEBUG:boto:Method: GET 2023-09-29T22:18:03.318 DEBUG:boto:Path: /hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.318 DEBUG:boto:Data: 2023-09-29T22:18:03.319 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.319 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.319 DEBUG:boto:Params: {} 2023-09-29T22:18:03.319 DEBUG:boto:Token: None 2023-09-29T22:18:03.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b83115d55c5f5a56-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.325 DEBUG:boto:b'hcnphz-46obj51000falseobj52023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.326 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.326 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.326 DEBUG:boto:path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.326 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.326 DEBUG:boto:Method: GET 2023-09-29T22:18:03.327 DEBUG:boto:Path: /hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.327 DEBUG:boto:Data: 2023-09-29T22:18:03.327 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.327 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.327 DEBUG:boto:Params: {} 2023-09-29T22:18:03.328 DEBUG:boto:Token: None 2023-09-29T22:18:03.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d80229458bc1eca5-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.333 DEBUG:boto:b'hcnphz-46obj51000falseobj52023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.334 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.334 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.334 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.334 DEBUG:boto:Path: /hcnphz-46/obj5 2023-09-29T22:18:03.334 DEBUG:boto:Data: 2023-09-29T22:18:03.334 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.335 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.335 DEBUG:boto:Params: {} 2023-09-29T22:18:03.335 DEBUG:boto:Token: None 2023-09-29T22:18:03.335 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5 2023-09-29T22:18:03.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HXH7oKVbqs1C7DSsjk7CjcAHqDw= 2023-09-29T22:18:03.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HXH7oKVbqs1C7DSsjk7CjcAHqDw=', 'Content-Length': '0'} 2023-09-29T22:18:03.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e088f450fcc683c-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.339 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.339 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.339 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.339 DEBUG:boto:Path: /hcnphz-46/obj5 2023-09-29T22:18:03.339 DEBUG:boto:Data: 2023-09-29T22:18:03.339 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.340 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.340 DEBUG:boto:Params: {} 2023-09-29T22:18:03.340 DEBUG:boto:Token: None 2023-09-29T22:18:03.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5 2023-09-29T22:18:03.341 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HXH7oKVbqs1C7DSsjk7CjcAHqDw= 2023-09-29T22:18:03.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HXH7oKVbqs1C7DSsjk7CjcAHqDw=', 'Content-Length': '0'} 2023-09-29T22:18:03.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c068cb893e7c749-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:18:03.344 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.344 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.344 DEBUG:boto:Method: GET 2023-09-29T22:18:03.345 DEBUG:boto:Path: /hcnphz-46/obj5 2023-09-29T22:18:03.345 DEBUG:boto:Data: 2023-09-29T22:18:03.345 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.345 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.345 DEBUG:boto:Params: {} 2023-09-29T22:18:03.346 DEBUG:boto:Token: None 2023-09-29T22:18:03.346 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5 2023-09-29T22:18:03.346 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AIS/kx9EN6frRHjHsW8ev+LTLUk= 2023-09-29T22:18:03.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AIS/kx9EN6frRHjHsW8ev+LTLUk=', 'Content-Length': '0'} 2023-09-29T22:18:03.349 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5aad6488f0c65b3-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.391 DEBUG:boto:path=/hcnphz-46/obj5 2023-09-29T22:18:03.394 DEBUG:boto:auth_path=/hcnphz-46/obj5 2023-09-29T22:18:03.394 DEBUG:boto:Method: GET 2023-09-29T22:18:03.394 DEBUG:boto:Path: /hcnphz-46/obj5 2023-09-29T22:18:03.394 DEBUG:boto:Data: 2023-09-29T22:18:03.394 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.394 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.394 DEBUG:boto:Params: {} 2023-09-29T22:18:03.395 DEBUG:boto:Token: None 2023-09-29T22:18:03.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj5 2023-09-29T22:18:03.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AIS/kx9EN6frRHjHsW8ev+LTLUk= 2023-09-29T22:18:03.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AIS/kx9EN6frRHjHsW8ev+LTLUk=', 'Content-Length': '0'} 2023-09-29T22:18:03.398 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000757f03f175bd7912-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.398 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.398 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.399 DEBUG:boto:path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.399 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.399 DEBUG:boto:Method: GET 2023-09-29T22:18:03.399 DEBUG:boto:Path: /hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.399 DEBUG:boto:Data: 2023-09-29T22:18:03.399 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.399 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.399 DEBUG:boto:Params: {} 2023-09-29T22:18:03.399 DEBUG:boto:Token: None 2023-09-29T22:18:03.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000257458ef5d24abcc-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.405 DEBUG:boto:b'hcnphz-46obj51000falseobj52023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.405 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.405 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.405 DEBUG:boto:path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.406 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.406 DEBUG:boto:Method: GET 2023-09-29T22:18:03.406 DEBUG:boto:Path: /hcnphz-46/?prefix=obj5 2023-09-29T22:18:03.406 DEBUG:boto:Data: 2023-09-29T22:18:03.406 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.406 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.406 DEBUG:boto:Params: {} 2023-09-29T22:18:03.407 DEBUG:boto:Token: None 2023-09-29T22:18:03.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.407 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006779580a812284ea-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.412 DEBUG:boto:b'hcnphz-46obj51000falseobj52023-09-29T22:17:45.906Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:18:03.413 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.413 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.413 DEBUG:boto:path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.413 DEBUG:boto:auth_path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.413 DEBUG:boto:Method: GET 2023-09-29T22:18:03.413 DEBUG:boto:Path: /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.413 DEBUG:boto:Data: 2023-09-29T22:18:03.413 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.414 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.414 DEBUG:boto:Params: {} 2023-09-29T22:18:03.414 DEBUG:boto:Token: None 2023-09-29T22:18:03.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo= 2023-09-29T22:18:03.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo=', 'Content-Length': '0'} 2023-09-29T22:18:03.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006d85f2d333fbe293-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.418 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.418 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.418 DEBUG:boto:path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.418 DEBUG:boto:auth_path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.418 DEBUG:boto:Method: GET 2023-09-29T22:18:03.419 DEBUG:boto:Path: /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.419 DEBUG:boto:Data: 2023-09-29T22:18:03.419 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.419 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.419 DEBUG:boto:Params: {} 2023-09-29T22:18:03.419 DEBUG:boto:Token: None 2023-09-29T22:18:03.419 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo= 2023-09-29T22:18:03.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo=', 'Content-Length': '0'} 2023-09-29T22:18:03.422 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008ce573c9a495f0cf-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.423 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.423 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.423 DEBUG:boto:path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.423 DEBUG:boto:auth_path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.423 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.423 DEBUG:boto:Path: /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.423 DEBUG:boto:Data: 2023-09-29T22:18:03.424 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.424 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.424 DEBUG:boto:Params: {} 2023-09-29T22:18:03.424 DEBUG:boto:Token: None 2023-09-29T22:18:03.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cdY5Ev5PrD4SRTjMvec7wM9e2PY= 2023-09-29T22:18:03.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cdY5Ev5PrD4SRTjMvec7wM9e2PY=', 'Content-Length': '0'} 2023-09-29T22:18:03.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0ac9dd206ab6ad3-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.427 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.427 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.428 DEBUG:boto:path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.428 DEBUG:boto:auth_path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.428 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.428 DEBUG:boto:Path: /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.428 DEBUG:boto:Data: 2023-09-29T22:18:03.428 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.428 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.428 DEBUG:boto:Params: {} 2023-09-29T22:18:03.429 DEBUG:boto:Token: None 2023-09-29T22:18:03.429 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cdY5Ev5PrD4SRTjMvec7wM9e2PY= 2023-09-29T22:18:03.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cdY5Ev5PrD4SRTjMvec7wM9e2PY=', 'Content-Length': '0'} 2023-09-29T22:18:03.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079344ecfcd74fcb1-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:18:03.432 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.432 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.432 DEBUG:boto:path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.432 DEBUG:boto:auth_path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.433 DEBUG:boto:Method: GET 2023-09-29T22:18:03.433 DEBUG:boto:Path: /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.433 DEBUG:boto:Data: 2023-09-29T22:18:03.433 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.433 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.433 DEBUG:boto:Params: {} 2023-09-29T22:18:03.433 DEBUG:boto:Token: None 2023-09-29T22:18:03.434 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo= 2023-09-29T22:18:03.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo=', 'Content-Length': '0'} 2023-09-29T22:18:03.436 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8ae5d2d1c18c739-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.478 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.479 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.479 DEBUG:boto:path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.479 DEBUG:boto:auth_path=/hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.479 DEBUG:boto:Method: GET 2023-09-29T22:18:03.479 DEBUG:boto:Path: /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.479 DEBUG:boto:Data: 2023-09-29T22:18:03.479 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.480 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.480 DEBUG:boto:Params: {} 2023-09-29T22:18:03.480 DEBUG:boto:Token: None 2023-09-29T22:18:03.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6?versionId=null 2023-09-29T22:18:03.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo= 2023-09-29T22:18:03.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bIwC6CTu4VwXBWyp0dj3iWU4iAo=', 'Content-Length': '0'} 2023-09-29T22:18:03.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004fb7dda2a19190d6-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.483 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.484 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.484 DEBUG:boto:path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.484 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.484 DEBUG:boto:Method: GET 2023-09-29T22:18:03.484 DEBUG:boto:Path: /hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.484 DEBUG:boto:Data: 2023-09-29T22:18:03.484 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.484 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.485 DEBUG:boto:Params: {} 2023-09-29T22:18:03.485 DEBUG:boto:Token: None 2023-09-29T22:18:03.485 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000650ab332aecc53fb-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.490 DEBUG:boto:b'hcnphz-46obj61000falseobj62023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.490 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.490 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.491 DEBUG:boto:path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.491 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.491 DEBUG:boto:Method: GET 2023-09-29T22:18:03.491 DEBUG:boto:Path: /hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.491 DEBUG:boto:Data: 2023-09-29T22:18:03.491 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.491 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.491 DEBUG:boto:Params: {} 2023-09-29T22:18:03.492 DEBUG:boto:Token: None 2023-09-29T22:18:03.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000050d712d56cd04d6-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.542 DEBUG:boto:b'hcnphz-46obj61000falseobj62023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.543 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.543 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.543 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.543 DEBUG:boto:Path: /hcnphz-46/obj6 2023-09-29T22:18:03.543 DEBUG:boto:Data: 2023-09-29T22:18:03.543 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.544 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.544 DEBUG:boto:Params: {} 2023-09-29T22:18:03.544 DEBUG:boto:Token: None 2023-09-29T22:18:03.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6 2023-09-29T22:18:03.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hCGWe/XWnXIy6qL6VRTDrScJsrs= 2023-09-29T22:18:03.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hCGWe/XWnXIy6qL6VRTDrScJsrs=', 'Content-Length': '0'} 2023-09-29T22:18:03.547 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b50faab55ae95be-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.547 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.548 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.548 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.548 DEBUG:boto:Path: /hcnphz-46/obj6 2023-09-29T22:18:03.548 DEBUG:boto:Data: 2023-09-29T22:18:03.548 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.548 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.548 DEBUG:boto:Params: {} 2023-09-29T22:18:03.549 DEBUG:boto:Token: None 2023-09-29T22:18:03.549 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6 2023-09-29T22:18:03.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hCGWe/XWnXIy6qL6VRTDrScJsrs= 2023-09-29T22:18:03.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hCGWe/XWnXIy6qL6VRTDrScJsrs=', 'Content-Length': '0'} 2023-09-29T22:18:03.552 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab22672b92bfd1cd-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:18:03.552 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.552 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.552 DEBUG:boto:Method: GET 2023-09-29T22:18:03.552 DEBUG:boto:Path: /hcnphz-46/obj6 2023-09-29T22:18:03.552 DEBUG:boto:Data: 2023-09-29T22:18:03.553 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.553 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.553 DEBUG:boto:Params: {} 2023-09-29T22:18:03.553 DEBUG:boto:Token: None 2023-09-29T22:18:03.553 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6 2023-09-29T22:18:03.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N2C7S0UWXKbJTFuve3khYTun65E= 2023-09-29T22:18:03.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N2C7S0UWXKbJTFuve3khYTun65E=', 'Content-Length': '0'} 2023-09-29T22:18:03.556 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000247223917132653d-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.556 DEBUG:boto:path=/hcnphz-46/obj6 2023-09-29T22:18:03.556 DEBUG:boto:auth_path=/hcnphz-46/obj6 2023-09-29T22:18:03.557 DEBUG:boto:Method: GET 2023-09-29T22:18:03.557 DEBUG:boto:Path: /hcnphz-46/obj6 2023-09-29T22:18:03.557 DEBUG:boto:Data: 2023-09-29T22:18:03.557 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.557 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.557 DEBUG:boto:Params: {} 2023-09-29T22:18:03.557 DEBUG:boto:Token: None 2023-09-29T22:18:03.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj6 2023-09-29T22:18:03.558 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N2C7S0UWXKbJTFuve3khYTun65E= 2023-09-29T22:18:03.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N2C7S0UWXKbJTFuve3khYTun65E=', 'Content-Length': '0'} 2023-09-29T22:18:03.561 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008f016a280fbd85ca-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.561 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.561 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.561 DEBUG:boto:path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.561 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.562 DEBUG:boto:Method: GET 2023-09-29T22:18:03.562 DEBUG:boto:Path: /hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.562 DEBUG:boto:Data: 2023-09-29T22:18:03.562 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.562 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.562 DEBUG:boto:Params: {} 2023-09-29T22:18:03.562 DEBUG:boto:Token: None 2023-09-29T22:18:03.562 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.563 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000653b284988b8f1d3-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.614 DEBUG:boto:b'hcnphz-46obj61000falseobj62023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.615 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.615 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.615 DEBUG:boto:path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.615 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.615 DEBUG:boto:Method: GET 2023-09-29T22:18:03.616 DEBUG:boto:Path: /hcnphz-46/?prefix=obj6 2023-09-29T22:18:03.616 DEBUG:boto:Data: 2023-09-29T22:18:03.616 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.616 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.616 DEBUG:boto:Params: {} 2023-09-29T22:18:03.616 DEBUG:boto:Token: None 2023-09-29T22:18:03.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.623 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f61f8f714b8a59e7-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.623 DEBUG:boto:b'hcnphz-46obj61000falseobj62023-09-29T22:17:45.963Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:18:03.624 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.624 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.624 DEBUG:boto:path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.624 DEBUG:boto:auth_path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.624 DEBUG:boto:Method: GET 2023-09-29T22:18:03.624 DEBUG:boto:Path: /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.624 DEBUG:boto:Data: 2023-09-29T22:18:03.624 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.625 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.625 DEBUG:boto:Params: {} 2023-09-29T22:18:03.625 DEBUG:boto:Token: None 2023-09-29T22:18:03.625 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.625 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto= 2023-09-29T22:18:03.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto=', 'Content-Length': '0'} 2023-09-29T22:18:03.628 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a88188d37e7881ab-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.628 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.629 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.629 DEBUG:boto:path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.629 DEBUG:boto:auth_path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.629 DEBUG:boto:Method: GET 2023-09-29T22:18:03.629 DEBUG:boto:Path: /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.629 DEBUG:boto:Data: 2023-09-29T22:18:03.629 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.629 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.630 DEBUG:boto:Params: {} 2023-09-29T22:18:03.630 DEBUG:boto:Token: None 2023-09-29T22:18:03.630 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.630 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto= 2023-09-29T22:18:03.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto=', 'Content-Length': '0'} 2023-09-29T22:18:03.633 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f544fefb3495a59-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.674 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.675 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.675 DEBUG:boto:path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.675 DEBUG:boto:auth_path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.675 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.675 DEBUG:boto:Path: /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.675 DEBUG:boto:Data: 2023-09-29T22:18:03.675 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.676 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.676 DEBUG:boto:Params: {} 2023-09-29T22:18:03.676 DEBUG:boto:Token: None 2023-09-29T22:18:03.676 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JyJVUz1mf6OdOUJcSRENJGvYi5o= 2023-09-29T22:18:03.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JyJVUz1mf6OdOUJcSRENJGvYi5o=', 'Content-Length': '0'} 2023-09-29T22:18:03.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072af2076443dc3a2-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.679 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.680 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.680 DEBUG:boto:path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.680 DEBUG:boto:auth_path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.680 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.680 DEBUG:boto:Path: /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.680 DEBUG:boto:Data: 2023-09-29T22:18:03.680 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.681 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.681 DEBUG:boto:Params: {} 2023-09-29T22:18:03.681 DEBUG:boto:Token: None 2023-09-29T22:18:03.681 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JyJVUz1mf6OdOUJcSRENJGvYi5o= 2023-09-29T22:18:03.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JyJVUz1mf6OdOUJcSRENJGvYi5o=', 'Content-Length': '0'} 2023-09-29T22:18:03.684 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3201c1cd5ae0e5f-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:18:03.684 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.684 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.685 DEBUG:boto:path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.685 DEBUG:boto:auth_path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.685 DEBUG:boto:Method: GET 2023-09-29T22:18:03.685 DEBUG:boto:Path: /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.685 DEBUG:boto:Data: 2023-09-29T22:18:03.685 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.685 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.685 DEBUG:boto:Params: {} 2023-09-29T22:18:03.686 DEBUG:boto:Token: None 2023-09-29T22:18:03.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto= 2023-09-29T22:18:03.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto=', 'Content-Length': '0'} 2023-09-29T22:18:03.689 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000201f7c2c65a50cf6-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.689 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.689 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.690 DEBUG:boto:path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.690 DEBUG:boto:auth_path=/hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.690 DEBUG:boto:Method: GET 2023-09-29T22:18:03.690 DEBUG:boto:Path: /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.690 DEBUG:boto:Data: 2023-09-29T22:18:03.690 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.690 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.690 DEBUG:boto:Params: {} 2023-09-29T22:18:03.691 DEBUG:boto:Token: None 2023-09-29T22:18:03.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7?versionId=null 2023-09-29T22:18:03.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto= 2023-09-29T22:18:03.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+kLjNCzFPmhMWRs28q4mF6Y3Pto=', 'Content-Length': '0'} 2023-09-29T22:18:03.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000082565baedde31d2c-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.695 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.695 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.695 DEBUG:boto:path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.695 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.696 DEBUG:boto:Method: GET 2023-09-29T22:18:03.696 DEBUG:boto:Path: /hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.696 DEBUG:boto:Data: 2023-09-29T22:18:03.696 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.696 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.697 DEBUG:boto:Params: {} 2023-09-29T22:18:03.697 DEBUG:boto:Token: None 2023-09-29T22:18:03.697 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.697 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009179636b2af8deb0-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.702 DEBUG:boto:b'hcnphz-46obj71000falseobj72023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.703 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.703 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.703 DEBUG:boto:path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.703 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.703 DEBUG:boto:Method: GET 2023-09-29T22:18:03.703 DEBUG:boto:Path: /hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.703 DEBUG:boto:Data: 2023-09-29T22:18:03.703 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.704 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.704 DEBUG:boto:Params: {} 2023-09-29T22:18:03.704 DEBUG:boto:Token: None 2023-09-29T22:18:03.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000172d30fbd3217644-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.709 DEBUG:boto:b'hcnphz-46obj71000falseobj72023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.710 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.710 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.710 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.710 DEBUG:boto:Path: /hcnphz-46/obj7 2023-09-29T22:18:03.710 DEBUG:boto:Data: 2023-09-29T22:18:03.710 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.711 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.711 DEBUG:boto:Params: {} 2023-09-29T22:18:03.711 DEBUG:boto:Token: None 2023-09-29T22:18:03.711 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7 2023-09-29T22:18:03.711 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tX/ghgWIX/1qGfw7BJZIBOUWDwA= 2023-09-29T22:18:03.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tX/ghgWIX/1qGfw7BJZIBOUWDwA=', 'Content-Length': '0'} 2023-09-29T22:18:03.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cff1a8eec55087df-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.714 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.714 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.714 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.714 DEBUG:boto:Path: /hcnphz-46/obj7 2023-09-29T22:18:03.714 DEBUG:boto:Data: 2023-09-29T22:18:03.714 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.715 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.715 DEBUG:boto:Params: {} 2023-09-29T22:18:03.715 DEBUG:boto:Token: None 2023-09-29T22:18:03.715 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7 2023-09-29T22:18:03.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tX/ghgWIX/1qGfw7BJZIBOUWDwA= 2023-09-29T22:18:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tX/ghgWIX/1qGfw7BJZIBOUWDwA=', 'Content-Length': '0'} 2023-09-29T22:18:03.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009af3279c3790f338-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:18:03.718 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.718 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.719 DEBUG:boto:Method: GET 2023-09-29T22:18:03.719 DEBUG:boto:Path: /hcnphz-46/obj7 2023-09-29T22:18:03.719 DEBUG:boto:Data: 2023-09-29T22:18:03.719 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.719 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.719 DEBUG:boto:Params: {} 2023-09-29T22:18:03.719 DEBUG:boto:Token: None 2023-09-29T22:18:03.719 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7 2023-09-29T22:18:03.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mpOYdhBkVKjFUi7N/vwaiWh3kY4= 2023-09-29T22:18:03.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mpOYdhBkVKjFUi7N/vwaiWh3kY4=', 'Content-Length': '0'} 2023-09-29T22:18:03.722 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e107da4f90ff8270-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.762 DEBUG:boto:path=/hcnphz-46/obj7 2023-09-29T22:18:03.763 DEBUG:boto:auth_path=/hcnphz-46/obj7 2023-09-29T22:18:03.763 DEBUG:boto:Method: GET 2023-09-29T22:18:03.763 DEBUG:boto:Path: /hcnphz-46/obj7 2023-09-29T22:18:03.763 DEBUG:boto:Data: 2023-09-29T22:18:03.763 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.763 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.763 DEBUG:boto:Params: {} 2023-09-29T22:18:03.764 DEBUG:boto:Token: None 2023-09-29T22:18:03.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj7 2023-09-29T22:18:03.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mpOYdhBkVKjFUi7N/vwaiWh3kY4= 2023-09-29T22:18:03.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mpOYdhBkVKjFUi7N/vwaiWh3kY4=', 'Content-Length': '0'} 2023-09-29T22:18:03.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000022db3a654241373e-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.767 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.767 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.767 DEBUG:boto:path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.767 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.767 DEBUG:boto:Method: GET 2023-09-29T22:18:03.767 DEBUG:boto:Path: /hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.768 DEBUG:boto:Data: 2023-09-29T22:18:03.768 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.768 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.768 DEBUG:boto:Params: {} 2023-09-29T22:18:03.768 DEBUG:boto:Token: None 2023-09-29T22:18:03.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f8db93ca4fff1e-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.773 DEBUG:boto:b'hcnphz-46obj71000falseobj72023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.774 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.774 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.774 DEBUG:boto:path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.774 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.774 DEBUG:boto:Method: GET 2023-09-29T22:18:03.774 DEBUG:boto:Path: /hcnphz-46/?prefix=obj7 2023-09-29T22:18:03.774 DEBUG:boto:Data: 2023-09-29T22:18:03.775 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.775 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.775 DEBUG:boto:Params: {} 2023-09-29T22:18:03.775 DEBUG:boto:Token: None 2023-09-29T22:18:03.775 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.775 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068fba3981d75c5dd-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.781 DEBUG:boto:b'hcnphz-46obj71000falseobj72023-09-29T22:17:45.979Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:18:03.782 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.782 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.782 DEBUG:boto:path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.782 DEBUG:boto:auth_path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.782 DEBUG:boto:Method: GET 2023-09-29T22:18:03.782 DEBUG:boto:Path: /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.782 DEBUG:boto:Data: 2023-09-29T22:18:03.782 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.783 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.783 DEBUG:boto:Params: {} 2023-09-29T22:18:03.783 DEBUG:boto:Token: None 2023-09-29T22:18:03.783 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.783 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w= 2023-09-29T22:18:03.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w=', 'Content-Length': '0'} 2023-09-29T22:18:03.786 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa97dadf04e81da2-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.786 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.786 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.787 DEBUG:boto:path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.787 DEBUG:boto:auth_path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.787 DEBUG:boto:Method: GET 2023-09-29T22:18:03.787 DEBUG:boto:Path: /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.787 DEBUG:boto:Data: 2023-09-29T22:18:03.787 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.787 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.788 DEBUG:boto:Params: {} 2023-09-29T22:18:03.788 DEBUG:boto:Token: None 2023-09-29T22:18:03.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w= 2023-09-29T22:18:03.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w=', 'Content-Length': '0'} 2023-09-29T22:18:03.790 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009948bf7a09053009-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.791 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.791 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.791 DEBUG:boto:path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.791 DEBUG:boto:auth_path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.791 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.791 DEBUG:boto:Path: /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.792 DEBUG:boto:Data: 2023-09-29T22:18:03.792 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.792 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.792 DEBUG:boto:Params: {} 2023-09-29T22:18:03.792 DEBUG:boto:Token: None 2023-09-29T22:18:03.792 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XljCfiwdYEGTW1DrWbHhcChL0OI= 2023-09-29T22:18:03.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XljCfiwdYEGTW1DrWbHhcChL0OI=', 'Content-Length': '0'} 2023-09-29T22:18:03.795 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014783fa2f35324c3-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.796 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.796 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.796 DEBUG:boto:path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.796 DEBUG:boto:auth_path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.796 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.796 DEBUG:boto:Path: /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.796 DEBUG:boto:Data: 2023-09-29T22:18:03.796 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.797 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.797 DEBUG:boto:Params: {} 2023-09-29T22:18:03.797 DEBUG:boto:Token: None 2023-09-29T22:18:03.797 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XljCfiwdYEGTW1DrWbHhcChL0OI= 2023-09-29T22:18:03.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XljCfiwdYEGTW1DrWbHhcChL0OI=', 'Content-Length': '0'} 2023-09-29T22:18:03.801 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc16cd15c98cfc9a-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:18:03.802 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.802 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.802 DEBUG:boto:path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.802 DEBUG:boto:auth_path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.802 DEBUG:boto:Method: GET 2023-09-29T22:18:03.802 DEBUG:boto:Path: /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.803 DEBUG:boto:Data: 2023-09-29T22:18:03.803 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.803 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.803 DEBUG:boto:Params: {} 2023-09-29T22:18:03.803 DEBUG:boto:Token: None 2023-09-29T22:18:03.803 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w= 2023-09-29T22:18:03.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w=', 'Content-Length': '0'} 2023-09-29T22:18:03.807 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000afd87c821c4d4523-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.854 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.855 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.855 DEBUG:boto:path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.855 DEBUG:boto:auth_path=/hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.855 DEBUG:boto:Method: GET 2023-09-29T22:18:03.855 DEBUG:boto:Path: /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.855 DEBUG:boto:Data: 2023-09-29T22:18:03.855 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.856 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.856 DEBUG:boto:Params: {} 2023-09-29T22:18:03.856 DEBUG:boto:Token: None 2023-09-29T22:18:03.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8?versionId=null 2023-09-29T22:18:03.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w= 2023-09-29T22:18:03.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7P9k/LNBC3eXrY3JjbPN7pOIR5w=', 'Content-Length': '0'} 2023-09-29T22:18:03.859 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081776aa18817c786-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.860 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.860 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.860 DEBUG:boto:path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.860 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.860 DEBUG:boto:Method: GET 2023-09-29T22:18:03.860 DEBUG:boto:Path: /hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.861 DEBUG:boto:Data: 2023-09-29T22:18:03.861 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.861 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.861 DEBUG:boto:Params: {} 2023-09-29T22:18:03.861 DEBUG:boto:Token: None 2023-09-29T22:18:03.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000011b35453e1ff7ba-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.866 DEBUG:boto:b'hcnphz-46obj81000falseobj82023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.867 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.867 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.867 DEBUG:boto:path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.867 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.867 DEBUG:boto:Method: GET 2023-09-29T22:18:03.868 DEBUG:boto:Path: /hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.868 DEBUG:boto:Data: 2023-09-29T22:18:03.868 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.868 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.868 DEBUG:boto:Params: {} 2023-09-29T22:18:03.868 DEBUG:boto:Token: None 2023-09-29T22:18:03.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037b5eee5b3d77ced-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.874 DEBUG:boto:b'hcnphz-46obj81000falseobj82023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.874 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.874 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.874 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.875 DEBUG:boto:Path: /hcnphz-46/obj8 2023-09-29T22:18:03.875 DEBUG:boto:Data: 2023-09-29T22:18:03.875 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.875 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.875 DEBUG:boto:Params: {} 2023-09-29T22:18:03.875 DEBUG:boto:Token: None 2023-09-29T22:18:03.876 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8 2023-09-29T22:18:03.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NvTCofuATkNNeMhu/FiiaB10REc= 2023-09-29T22:18:03.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NvTCofuATkNNeMhu/FiiaB10REc=', 'Content-Length': '0'} 2023-09-29T22:18:03.878 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fbb640210599904-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.878 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.878 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.879 DEBUG:boto:Method: HEAD 2023-09-29T22:18:03.879 DEBUG:boto:Path: /hcnphz-46/obj8 2023-09-29T22:18:03.879 DEBUG:boto:Data: 2023-09-29T22:18:03.879 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.879 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.879 DEBUG:boto:Params: {} 2023-09-29T22:18:03.879 DEBUG:boto:Token: None 2023-09-29T22:18:03.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8 2023-09-29T22:18:03.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NvTCofuATkNNeMhu/FiiaB10REc= 2023-09-29T22:18:03.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NvTCofuATkNNeMhu/FiiaB10REc=', 'Content-Length': '0'} 2023-09-29T22:18:03.882 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa7b7ad8f79149c9-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:18:03.882 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.882 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.883 DEBUG:boto:Method: GET 2023-09-29T22:18:03.883 DEBUG:boto:Path: /hcnphz-46/obj8 2023-09-29T22:18:03.883 DEBUG:boto:Data: 2023-09-29T22:18:03.883 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.883 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.883 DEBUG:boto:Params: {} 2023-09-29T22:18:03.883 DEBUG:boto:Token: None 2023-09-29T22:18:03.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8 2023-09-29T22:18:03.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CcRwcccGLbZLZMPCa7dj5Vus47A= 2023-09-29T22:18:03.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CcRwcccGLbZLZMPCa7dj5Vus47A=', 'Content-Length': '0'} 2023-09-29T22:18:03.886 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-46:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.11'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbff5c8c04303916-0065174d1b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.886 DEBUG:boto:path=/hcnphz-46/obj8 2023-09-29T22:18:03.886 DEBUG:boto:auth_path=/hcnphz-46/obj8 2023-09-29T22:18:03.886 DEBUG:boto:Method: GET 2023-09-29T22:18:03.887 DEBUG:boto:Path: /hcnphz-46/obj8 2023-09-29T22:18:03.887 DEBUG:boto:Data: 2023-09-29T22:18:03.887 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.887 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.887 DEBUG:boto:Params: {} 2023-09-29T22:18:03.887 DEBUG:boto:Token: None 2023-09-29T22:18:03.887 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/obj8 2023-09-29T22:18:03.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CcRwcccGLbZLZMPCa7dj5Vus47A= 2023-09-29T22:18:03.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CcRwcccGLbZLZMPCa7dj5Vus47A=', 'Content-Length': '0'} 2023-09-29T22:18:03.890 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d6857e0cd21a834a-0065174d1b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.890 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.891 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.891 DEBUG:boto:path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.891 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.891 DEBUG:boto:Method: GET 2023-09-29T22:18:03.891 DEBUG:boto:Path: /hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.891 DEBUG:boto:Data: 2023-09-29T22:18:03.891 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:03.892 DEBUG:boto:Port: 8001 2023-09-29T22:18:03.892 DEBUG:boto:Params: {} 2023-09-29T22:18:03.892 DEBUG:boto:Token: None 2023-09-29T22:18:03.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005fa15677b3e3cf60-0065174d1b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.897 DEBUG:boto:b'hcnphz-46obj81000falseobj82023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.897 DEBUG:boto:path=/hcnphz-46/ 2023-09-29T22:18:03.898 DEBUG:boto:auth_path=/hcnphz-46/ 2023-09-29T22:18:03.898 DEBUG:boto:path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.898 DEBUG:boto:auth_path=/hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.898 DEBUG:boto:Method: GET 2023-09-29T22:18:03.898 DEBUG:boto:Path: /hcnphz-46/?prefix=obj8 2023-09-29T22:18:03.898 DEBUG:boto:Data: 2023-09-29T22:18:03.898 DEBUG:boto:Headers: {} 2023-09-29T22:18:03.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:03.899 DEBUG:boto:Port: 8000 2023-09-29T22:18:03.899 DEBUG:boto:Params: {} 2023-09-29T22:18:03.899 DEBUG:boto:Token: None 2023-09-29T22:18:03.899 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:03 GMT /hcnphz-46/ 2023-09-29T22:18:03.899 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII= 2023-09-29T22:18:03.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVlASvmIOtbbE3HgZd9nx3eSjII=', 'Content-Length': '0'} 2023-09-29T22:18:03.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7db6d6f4e702873-0065174d1b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:03.905 DEBUG:boto:b'hcnphz-46obj81000falseobj82023-09-29T22:17:46.029Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:03.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-46 zones={a2, a1} 2023-09-29T22:18:03.905 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-47 2023-09-29T22:18:03.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:03.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:03.906 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-47 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:03.957 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:03.958 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:04.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:04.014+0000 7f29d74d4f40 1 waiting for incremental sync to catch up: 2023-09-29T22:18:04.019 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.37.1, 00000000002.29.1, 00000000003.30.1, 00000000002.19.1, 00000000003.24.1, 00000000002.25.1, 00000000002.36.1, 00000000003.26.1, 00000000002.24.1, 00000000002.23.1, 00000000002.20.1] 2023-09-29T22:18:04.019 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.41.6, 00000000002.29.1, 00000000004.32.6, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.36.1, 00000000004.28.6, 00000000002.24.1, 00000000002.23.1, 00000000002.20.1] 2023-09-29T22:18:09.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:09.018+0000 7f29d74d4f40 1 waiting for incremental sync to catch up: 2023-09-29T22:18:09.022 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.37.1, 00000000002.29.1, 00000000003.30.1, 00000000002.19.1, 00000000003.24.1, 00000000002.25.1, 00000000002.36.1, 00000000003.26.1, 00000000002.24.1, 00000000002.23.1, 00000000002.20.1] 2023-09-29T22:18:09.022 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.41.6, 00000000002.29.1, 00000000004.32.6, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.36.1, 00000000004.28.6, 00000000002.24.1, 00000000002.23.1, 00000000002.20.1] 2023-09-29T22:18:14.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:14.022+0000 7f29d74d4f40 1 bucket sync caught up with source: 2023-09-29T22:18:14.024 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.41.6, 00000000002.29.1, 00000000004.32.6, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.36.1, 00000000004.28.6, 00000000002.24.1, 00000000002.23.1, 00000000002.20.1] 2023-09-29T22:18:14.024 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.41.6, 00000000002.29.1, 00000000004.32.6, 00000000002.19.1, 00000000004.26.6, 00000000002.25.1, 00000000002.36.1, 00000000004.28.6, 00000000002.24.1, 00000000002.23.1, 00000000002.20.1] 2023-09-29T22:18:14.024 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:14.022+0000 7f29d74d4f40 0 bucket checkpoint complete 2023-09-29T22:18:14.053 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-47 2023-09-29T22:18:14.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:14.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-47', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:14.053 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-47 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:14.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:14.166+0000 7f658107ef40 1 bucket sync caught up with source: 2023-09-29T22:18:14.169 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.99.6, , 00000000002.170.6, , 00000000002.117.6, , , 00000000002.140.6, , , ] 2023-09-29T22:18:14.169 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.99.6, , 00000000002.170.6, , 00000000002.117.6, , , 00000000002.140.6, , , ] 2023-09-29T22:18:14.169 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:14.166+0000 7f658107ef40 0 bucket checkpoint complete 2023-09-29T22:18:14.192 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-47 zones={a2, a1} 2023-09-29T22:18:14.193 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.193 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.193 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.193 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:18:14.193 DEBUG:boto:Data: 2023-09-29T22:18:14.193 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.194 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.194 DEBUG:boto:Params: {} 2023-09-29T22:18:14.194 DEBUG:boto:Token: None 2023-09-29T22:18:14.194 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t8jtxDteNl9BXEgBMrbTvuVBs6A= 2023-09-29T22:18:14.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t8jtxDteNl9BXEgBMrbTvuVBs6A=', 'Content-Length': '0'} 2023-09-29T22:18:14.198 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3c6d766b7b6be5e-0065174d26-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.198 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.198 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.198 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.198 DEBUG:boto:Path: /hcnphz-47/ 2023-09-29T22:18:14.199 DEBUG:boto:Data: 2023-09-29T22:18:14.199 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.199 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.199 DEBUG:boto:Params: {} 2023-09-29T22:18:14.199 DEBUG:boto:Token: None 2023-09-29T22:18:14.199 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t8jtxDteNl9BXEgBMrbTvuVBs6A= 2023-09-29T22:18:14.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t8jtxDteNl9BXEgBMrbTvuVBs6A=', 'Content-Length': '0'} 2023-09-29T22:18:14.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000da8bed5946ead908-0065174d26-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.203 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:18:14.203 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.203 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.203 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.203 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.203 DEBUG:boto:Method: GET 2023-09-29T22:18:14.204 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.204 DEBUG:boto:Data: 2023-09-29T22:18:14.204 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.204 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.204 DEBUG:boto:Params: {} 2023-09-29T22:18:14.204 DEBUG:boto:Token: None 2023-09-29T22:18:14.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/?versions 2023-09-29T22:18:14.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM= 2023-09-29T22:18:14.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM=', 'Content-Length': '0'} 2023-09-29T22:18:14.208 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ca6f69ed8934bb-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.209 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:18:14.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:18:14.211 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.211 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.211 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.211 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.211 DEBUG:boto:Method: GET 2023-09-29T22:18:14.211 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.211 DEBUG:boto:Data: 2023-09-29T22:18:14.211 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.212 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.212 DEBUG:boto:Params: {} 2023-09-29T22:18:14.212 DEBUG:boto:Token: None 2023-09-29T22:18:14.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/?versions 2023-09-29T22:18:14.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM= 2023-09-29T22:18:14.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM=', 'Content-Length': '0'} 2023-09-29T22:18:14.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be8461014fea9f8b-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.258 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:18:14.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:18:14.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:18:14.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:18:14.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:18:14.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:18:14.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:18:14.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:18:14.261 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.261 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.262 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.262 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.262 DEBUG:boto:Method: GET 2023-09-29T22:18:14.262 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.262 DEBUG:boto:Data: 2023-09-29T22:18:14.263 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.263 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.263 DEBUG:boto:Params: {} 2023-09-29T22:18:14.263 DEBUG:boto:Token: None 2023-09-29T22:18:14.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/?versions 2023-09-29T22:18:14.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM= 2023-09-29T22:18:14.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM=', 'Content-Length': '0'} 2023-09-29T22:18:14.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb52f8151af28aa5-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.270 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.271 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.271 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.271 DEBUG:boto:path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.271 DEBUG:boto:auth_path=/hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.271 DEBUG:boto:Method: GET 2023-09-29T22:18:14.271 DEBUG:boto:Path: /hcnphz-47/?versions&max-keys=999 2023-09-29T22:18:14.271 DEBUG:boto:Data: 2023-09-29T22:18:14.271 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.272 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.272 DEBUG:boto:Params: {} 2023-09-29T22:18:14.272 DEBUG:boto:Token: None 2023-09-29T22:18:14.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/?versions 2023-09-29T22:18:14.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM= 2023-09-29T22:18:14.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qDNKj0LWx3ZdNoqBptZeP3SuXoM=', 'Content-Length': '0'} 2023-09-29T22:18:14.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035dba7fe7876859b-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.277 DEBUG:boto:b'hcnphz-47999falseobj1nulltrue2023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:18:14.278 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.278 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.278 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.278 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.278 DEBUG:boto:Method: GET 2023-09-29T22:18:14.279 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.279 DEBUG:boto:Data: 2023-09-29T22:18:14.279 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.279 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.279 DEBUG:boto:Params: {} 2023-09-29T22:18:14.279 DEBUG:boto:Token: None 2023-09-29T22:18:14.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA= 2023-09-29T22:18:14.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA=', 'Content-Length': '0'} 2023-09-29T22:18:14.282 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e3defd9bfa3fdef7-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.326 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.327 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.327 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.327 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.327 DEBUG:boto:Method: GET 2023-09-29T22:18:14.327 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.327 DEBUG:boto:Data: 2023-09-29T22:18:14.327 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.328 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.328 DEBUG:boto:Params: {} 2023-09-29T22:18:14.328 DEBUG:boto:Token: None 2023-09-29T22:18:14.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA= 2023-09-29T22:18:14.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA=', 'Content-Length': '0'} 2023-09-29T22:18:14.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002c24159c397dc3aa-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.331 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.332 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.332 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.332 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.332 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.332 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.332 DEBUG:boto:Data: 2023-09-29T22:18:14.332 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.332 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.332 DEBUG:boto:Params: {} 2023-09-29T22:18:14.333 DEBUG:boto:Token: None 2023-09-29T22:18:14.333 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Afeb14ljcPa+GEl0nMkIsWRTz6U= 2023-09-29T22:18:14.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Afeb14ljcPa+GEl0nMkIsWRTz6U=', 'Content-Length': '0'} 2023-09-29T22:18:14.335 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e592df8504d96910-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.336 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.336 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.336 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.336 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.336 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.337 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.337 DEBUG:boto:Data: 2023-09-29T22:18:14.337 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.337 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.337 DEBUG:boto:Params: {} 2023-09-29T22:18:14.337 DEBUG:boto:Token: None 2023-09-29T22:18:14.337 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Afeb14ljcPa+GEl0nMkIsWRTz6U= 2023-09-29T22:18:14.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Afeb14ljcPa+GEl0nMkIsWRTz6U=', 'Content-Length': '0'} 2023-09-29T22:18:14.340 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042351d6e516b164d-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:18:14.341 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.341 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.341 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.341 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.341 DEBUG:boto:Method: GET 2023-09-29T22:18:14.341 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.342 DEBUG:boto:Data: 2023-09-29T22:18:14.342 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.342 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.342 DEBUG:boto:Params: {} 2023-09-29T22:18:14.342 DEBUG:boto:Token: None 2023-09-29T22:18:14.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA= 2023-09-29T22:18:14.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA=', 'Content-Length': '0'} 2023-09-29T22:18:14.345 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000041007e698ad5d044-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.346 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.346 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.346 DEBUG:boto:path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.346 DEBUG:boto:auth_path=/hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.346 DEBUG:boto:Method: GET 2023-09-29T22:18:14.346 DEBUG:boto:Path: /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.347 DEBUG:boto:Data: 2023-09-29T22:18:14.347 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.347 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.347 DEBUG:boto:Params: {} 2023-09-29T22:18:14.347 DEBUG:boto:Token: None 2023-09-29T22:18:14.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1?versionId=null 2023-09-29T22:18:14.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA= 2023-09-29T22:18:14.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Osnn2rCQ1LZgAZxgWbuLKlG/3mA=', 'Content-Length': '0'} 2023-09-29T22:18:14.350 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d5c7bc9e0144cb71-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.350 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.351 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.351 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.351 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.351 DEBUG:boto:Method: GET 2023-09-29T22:18:14.351 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.351 DEBUG:boto:Data: 2023-09-29T22:18:14.351 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.351 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.352 DEBUG:boto:Params: {} 2023-09-29T22:18:14.352 DEBUG:boto:Token: None 2023-09-29T22:18:14.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005bf76b67b17ca8fe-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.357 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.357 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.357 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.357 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.358 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.358 DEBUG:boto:Method: GET 2023-09-29T22:18:14.358 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.358 DEBUG:boto:Data: 2023-09-29T22:18:14.358 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.358 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.358 DEBUG:boto:Params: {} 2023-09-29T22:18:14.358 DEBUG:boto:Token: None 2023-09-29T22:18:14.359 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.359 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.363 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f09e7a79267f1aec-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.363 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.363 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.363 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.364 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.364 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:18:14.364 DEBUG:boto:Data: 2023-09-29T22:18:14.364 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.364 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.364 DEBUG:boto:Params: {} 2023-09-29T22:18:14.364 DEBUG:boto:Token: None 2023-09-29T22:18:14.364 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1 2023-09-29T22:18:14.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/E1cFqZmKZGW8zl48QDBBcVvhyU= 2023-09-29T22:18:14.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/E1cFqZmKZGW8zl48QDBBcVvhyU=', 'Content-Length': '0'} 2023-09-29T22:18:14.367 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0285b697274df69-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.367 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.367 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.367 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.368 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:18:14.368 DEBUG:boto:Data: 2023-09-29T22:18:14.368 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.368 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.368 DEBUG:boto:Params: {} 2023-09-29T22:18:14.368 DEBUG:boto:Token: None 2023-09-29T22:18:14.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1 2023-09-29T22:18:14.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/E1cFqZmKZGW8zl48QDBBcVvhyU= 2023-09-29T22:18:14.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/E1cFqZmKZGW8zl48QDBBcVvhyU=', 'Content-Length': '0'} 2023-09-29T22:18:14.371 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000191dd8f988a8c008-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:18:14.371 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.371 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.372 DEBUG:boto:Method: GET 2023-09-29T22:18:14.372 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:18:14.372 DEBUG:boto:Data: 2023-09-29T22:18:14.372 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.372 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.372 DEBUG:boto:Params: {} 2023-09-29T22:18:14.372 DEBUG:boto:Token: None 2023-09-29T22:18:14.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1 2023-09-29T22:18:14.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lew1/efnirKYyVf7LFAM+6sQrdc= 2023-09-29T22:18:14.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lew1/efnirKYyVf7LFAM+6sQrdc=', 'Content-Length': '0'} 2023-09-29T22:18:14.375 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000324dff19741680c4-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.418 DEBUG:boto:path=/hcnphz-47/obj1 2023-09-29T22:18:14.419 DEBUG:boto:auth_path=/hcnphz-47/obj1 2023-09-29T22:18:14.419 DEBUG:boto:Method: GET 2023-09-29T22:18:14.419 DEBUG:boto:Path: /hcnphz-47/obj1 2023-09-29T22:18:14.419 DEBUG:boto:Data: 2023-09-29T22:18:14.419 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.419 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.419 DEBUG:boto:Params: {} 2023-09-29T22:18:14.420 DEBUG:boto:Token: None 2023-09-29T22:18:14.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj1 2023-09-29T22:18:14.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lew1/efnirKYyVf7LFAM+6sQrdc= 2023-09-29T22:18:14.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lew1/efnirKYyVf7LFAM+6sQrdc=', 'Content-Length': '0'} 2023-09-29T22:18:14.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000109688f2d39841e8-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.423 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.423 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.423 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.423 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.424 DEBUG:boto:Method: GET 2023-09-29T22:18:14.424 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.424 DEBUG:boto:Data: 2023-09-29T22:18:14.424 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.424 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.424 DEBUG:boto:Params: {} 2023-09-29T22:18:14.424 DEBUG:boto:Token: None 2023-09-29T22:18:14.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004136e596f15baa7d-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.430 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.430 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.431 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.431 DEBUG:boto:path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.431 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.431 DEBUG:boto:Method: GET 2023-09-29T22:18:14.431 DEBUG:boto:Path: /hcnphz-47/?prefix=obj1 2023-09-29T22:18:14.431 DEBUG:boto:Data: 2023-09-29T22:18:14.431 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.431 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.431 DEBUG:boto:Params: {} 2023-09-29T22:18:14.432 DEBUG:boto:Token: None 2023-09-29T22:18:14.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a29afdc650d14e9-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.437 DEBUG:boto:b'hcnphz-47obj11000falseobj12023-09-29T22:15:12.465Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:18:14.437 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.437 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.437 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.438 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.438 DEBUG:boto:Method: GET 2023-09-29T22:18:14.438 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.438 DEBUG:boto:Data: 2023-09-29T22:18:14.438 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.438 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.438 DEBUG:boto:Params: {} 2023-09-29T22:18:14.438 DEBUG:boto:Token: None 2023-09-29T22:18:14.439 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.439 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I= 2023-09-29T22:18:14.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I=', 'Content-Length': '0'} 2023-09-29T22:18:14.441 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004c2de7f302f7b9fa-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.442 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.442 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.442 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.442 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.442 DEBUG:boto:Method: GET 2023-09-29T22:18:14.442 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.443 DEBUG:boto:Data: 2023-09-29T22:18:14.443 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.443 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.443 DEBUG:boto:Params: {} 2023-09-29T22:18:14.443 DEBUG:boto:Token: None 2023-09-29T22:18:14.443 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I= 2023-09-29T22:18:14.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I=', 'Content-Length': '0'} 2023-09-29T22:18:14.446 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a967e3b321a5337-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.446 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.446 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.447 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.447 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.447 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.447 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.447 DEBUG:boto:Data: 2023-09-29T22:18:14.447 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.447 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.447 DEBUG:boto:Params: {} 2023-09-29T22:18:14.447 DEBUG:boto:Token: None 2023-09-29T22:18:14.448 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Gw6dSZzVlcWqw7OQkeipbASMD4k= 2023-09-29T22:18:14.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Gw6dSZzVlcWqw7OQkeipbASMD4k=', 'Content-Length': '0'} 2023-09-29T22:18:14.450 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b415eb0545b14b8-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.450 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.451 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.451 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.451 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.451 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.451 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.451 DEBUG:boto:Data: 2023-09-29T22:18:14.452 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.452 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.452 DEBUG:boto:Params: {} 2023-09-29T22:18:14.452 DEBUG:boto:Token: None 2023-09-29T22:18:14.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Gw6dSZzVlcWqw7OQkeipbASMD4k= 2023-09-29T22:18:14.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Gw6dSZzVlcWqw7OQkeipbASMD4k=', 'Content-Length': '0'} 2023-09-29T22:18:14.456 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a2303b14102f0355-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:18:14.456 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.456 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.456 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.456 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.457 DEBUG:boto:Method: GET 2023-09-29T22:18:14.457 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.457 DEBUG:boto:Data: 2023-09-29T22:18:14.457 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.457 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.457 DEBUG:boto:Params: {} 2023-09-29T22:18:14.457 DEBUG:boto:Token: None 2023-09-29T22:18:14.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.458 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I= 2023-09-29T22:18:14.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I=', 'Content-Length': '0'} 2023-09-29T22:18:14.460 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069a9ad665ec947c1-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.460 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.461 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.461 DEBUG:boto:path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.461 DEBUG:boto:auth_path=/hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.461 DEBUG:boto:Method: GET 2023-09-29T22:18:14.461 DEBUG:boto:Path: /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.461 DEBUG:boto:Data: 2023-09-29T22:18:14.461 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.461 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.462 DEBUG:boto:Params: {} 2023-09-29T22:18:14.462 DEBUG:boto:Token: None 2023-09-29T22:18:14.462 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2?versionId=null 2023-09-29T22:18:14.462 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I= 2023-09-29T22:18:14.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DzBNh9sWGDTKaPtAEuJpjhQYn8I=', 'Content-Length': '0'} 2023-09-29T22:18:14.465 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000844ffd542dc3898a-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.506 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.507 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.508 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.508 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.508 DEBUG:boto:Method: GET 2023-09-29T22:18:14.508 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.509 DEBUG:boto:Data: 2023-09-29T22:18:14.509 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.509 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.510 DEBUG:boto:Params: {} 2023-09-29T22:18:14.510 DEBUG:boto:Token: None 2023-09-29T22:18:14.510 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.511 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.516 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d0636f2d1eafd22-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.517 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.517 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.518 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.518 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.518 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.518 DEBUG:boto:Method: GET 2023-09-29T22:18:14.518 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.519 DEBUG:boto:Data: 2023-09-29T22:18:14.519 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.519 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.520 DEBUG:boto:Params: {} 2023-09-29T22:18:14.520 DEBUG:boto:Token: None 2023-09-29T22:18:14.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.525 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000858aa9b68eefc6c6-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.525 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.526 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.526 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.526 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.527 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:18:14.527 DEBUG:boto:Data: 2023-09-29T22:18:14.527 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.527 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.527 DEBUG:boto:Params: {} 2023-09-29T22:18:14.528 DEBUG:boto:Token: None 2023-09-29T22:18:14.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2 2023-09-29T22:18:14.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i+i3eMvFURiCWKXxvJYFztiiOQ8= 2023-09-29T22:18:14.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i+i3eMvFURiCWKXxvJYFztiiOQ8=', 'Content-Length': '0'} 2023-09-29T22:18:14.531 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c679e9adc1ee13e-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.531 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.532 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.532 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.532 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:18:14.532 DEBUG:boto:Data: 2023-09-29T22:18:14.532 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.533 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.533 DEBUG:boto:Params: {} 2023-09-29T22:18:14.533 DEBUG:boto:Token: None 2023-09-29T22:18:14.533 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2 2023-09-29T22:18:14.534 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i+i3eMvFURiCWKXxvJYFztiiOQ8= 2023-09-29T22:18:14.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i+i3eMvFURiCWKXxvJYFztiiOQ8=', 'Content-Length': '0'} 2023-09-29T22:18:14.536 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa37ee83d33af84c-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:18:14.537 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.537 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.538 DEBUG:boto:Method: GET 2023-09-29T22:18:14.538 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:18:14.538 DEBUG:boto:Data: 2023-09-29T22:18:14.538 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.539 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.539 DEBUG:boto:Params: {} 2023-09-29T22:18:14.539 DEBUG:boto:Token: None 2023-09-29T22:18:14.539 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2 2023-09-29T22:18:14.539 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Iqi5Dc6TicrnMV+S9ml8F+nNohE= 2023-09-29T22:18:14.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Iqi5Dc6TicrnMV+S9ml8F+nNohE=', 'Content-Length': '0'} 2023-09-29T22:18:14.542 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006b5b4abebfa8640-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.586 DEBUG:boto:path=/hcnphz-47/obj2 2023-09-29T22:18:14.587 DEBUG:boto:auth_path=/hcnphz-47/obj2 2023-09-29T22:18:14.587 DEBUG:boto:Method: GET 2023-09-29T22:18:14.587 DEBUG:boto:Path: /hcnphz-47/obj2 2023-09-29T22:18:14.587 DEBUG:boto:Data: 2023-09-29T22:18:14.587 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.587 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.587 DEBUG:boto:Params: {} 2023-09-29T22:18:14.588 DEBUG:boto:Token: None 2023-09-29T22:18:14.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj2 2023-09-29T22:18:14.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Iqi5Dc6TicrnMV+S9ml8F+nNohE= 2023-09-29T22:18:14.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Iqi5Dc6TicrnMV+S9ml8F+nNohE=', 'Content-Length': '0'} 2023-09-29T22:18:14.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e655efe50e9d4598-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.591 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.591 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.592 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.592 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.592 DEBUG:boto:Method: GET 2023-09-29T22:18:14.592 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.592 DEBUG:boto:Data: 2023-09-29T22:18:14.592 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.592 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.592 DEBUG:boto:Params: {} 2023-09-29T22:18:14.593 DEBUG:boto:Token: None 2023-09-29T22:18:14.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.598 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097ecd69f9c4c0ffb-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.598 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.598 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.598 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.599 DEBUG:boto:path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.599 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.599 DEBUG:boto:Method: GET 2023-09-29T22:18:14.599 DEBUG:boto:Path: /hcnphz-47/?prefix=obj2 2023-09-29T22:18:14.600 DEBUG:boto:Data: 2023-09-29T22:18:14.600 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.600 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.600 DEBUG:boto:Params: {} 2023-09-29T22:18:14.601 DEBUG:boto:Token: None 2023-09-29T22:18:14.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005391cd9c445de10e-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.646 DEBUG:boto:b'hcnphz-47obj21000falseobj22023-09-29T22:15:12.484Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:18:14.647 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.647 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.647 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.647 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.647 DEBUG:boto:Method: GET 2023-09-29T22:18:14.648 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.648 DEBUG:boto:Data: 2023-09-29T22:18:14.648 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.648 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.648 DEBUG:boto:Params: {} 2023-09-29T22:18:14.648 DEBUG:boto:Token: None 2023-09-29T22:18:14.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM= 2023-09-29T22:18:14.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM=', 'Content-Length': '0'} 2023-09-29T22:18:14.651 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047fea64e1eb74cfd-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.652 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.652 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.652 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.652 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.652 DEBUG:boto:Method: GET 2023-09-29T22:18:14.653 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.653 DEBUG:boto:Data: 2023-09-29T22:18:14.653 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.653 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.653 DEBUG:boto:Params: {} 2023-09-29T22:18:14.653 DEBUG:boto:Token: None 2023-09-29T22:18:14.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM= 2023-09-29T22:18:14.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM=', 'Content-Length': '0'} 2023-09-29T22:18:14.657 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a9b1dcd1916bc44f-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.657 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.657 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.657 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.658 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.658 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.658 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.658 DEBUG:boto:Data: 2023-09-29T22:18:14.658 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.658 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.658 DEBUG:boto:Params: {} 2023-09-29T22:18:14.659 DEBUG:boto:Token: None 2023-09-29T22:18:14.659 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DW6e1GihOh44yVmV4ZPvY8KPIJg= 2023-09-29T22:18:14.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DW6e1GihOh44yVmV4ZPvY8KPIJg=', 'Content-Length': '0'} 2023-09-29T22:18:14.661 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000559d7e385721966-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.661 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.661 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.662 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.662 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.662 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.662 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.662 DEBUG:boto:Data: 2023-09-29T22:18:14.662 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.662 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.663 DEBUG:boto:Params: {} 2023-09-29T22:18:14.663 DEBUG:boto:Token: None 2023-09-29T22:18:14.663 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DW6e1GihOh44yVmV4ZPvY8KPIJg= 2023-09-29T22:18:14.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DW6e1GihOh44yVmV4ZPvY8KPIJg=', 'Content-Length': '0'} 2023-09-29T22:18:14.665 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c1c247b76cdeb721-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:18:14.665 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.666 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.666 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.666 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.666 DEBUG:boto:Method: GET 2023-09-29T22:18:14.666 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.666 DEBUG:boto:Data: 2023-09-29T22:18:14.666 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.667 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.667 DEBUG:boto:Params: {} 2023-09-29T22:18:14.667 DEBUG:boto:Token: None 2023-09-29T22:18:14.667 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.667 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM= 2023-09-29T22:18:14.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM=', 'Content-Length': '0'} 2023-09-29T22:18:14.669 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f707e398f1c2e4c0-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.669 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.670 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.670 DEBUG:boto:path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.670 DEBUG:boto:auth_path=/hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.670 DEBUG:boto:Method: GET 2023-09-29T22:18:14.670 DEBUG:boto:Path: /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.670 DEBUG:boto:Data: 2023-09-29T22:18:14.670 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.670 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.671 DEBUG:boto:Params: {} 2023-09-29T22:18:14.671 DEBUG:boto:Token: None 2023-09-29T22:18:14.671 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3?versionId=null 2023-09-29T22:18:14.671 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM= 2023-09-29T22:18:14.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IT7jiS4bfVKguj1Ul9MvUJXXktM=', 'Content-Length': '0'} 2023-09-29T22:18:14.673 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c3584476a3eed7-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.673 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.673 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.674 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.674 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.674 DEBUG:boto:Method: GET 2023-09-29T22:18:14.674 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.674 DEBUG:boto:Data: 2023-09-29T22:18:14.674 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.674 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.674 DEBUG:boto:Params: {} 2023-09-29T22:18:14.675 DEBUG:boto:Token: None 2023-09-29T22:18:14.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.679 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a30c350d60afb130-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.679 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.679 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.679 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.679 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.680 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.680 DEBUG:boto:Method: GET 2023-09-29T22:18:14.680 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.680 DEBUG:boto:Data: 2023-09-29T22:18:14.680 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.680 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.680 DEBUG:boto:Params: {} 2023-09-29T22:18:14.680 DEBUG:boto:Token: None 2023-09-29T22:18:14.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.684 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dfd4a6fc04eef092-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.684 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.685 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.685 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.685 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.685 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:18:14.685 DEBUG:boto:Data: 2023-09-29T22:18:14.685 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.686 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.686 DEBUG:boto:Params: {} 2023-09-29T22:18:14.686 DEBUG:boto:Token: None 2023-09-29T22:18:14.686 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3 2023-09-29T22:18:14.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DG4tkQbo7RgHpq39ALOtx/rQdZk= 2023-09-29T22:18:14.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DG4tkQbo7RgHpq39ALOtx/rQdZk=', 'Content-Length': '0'} 2023-09-29T22:18:14.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ba1eb1f781f3e424-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.689 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.689 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.689 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.689 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:18:14.689 DEBUG:boto:Data: 2023-09-29T22:18:14.689 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.689 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.689 DEBUG:boto:Params: {} 2023-09-29T22:18:14.690 DEBUG:boto:Token: None 2023-09-29T22:18:14.690 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3 2023-09-29T22:18:14.690 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DG4tkQbo7RgHpq39ALOtx/rQdZk= 2023-09-29T22:18:14.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DG4tkQbo7RgHpq39ALOtx/rQdZk=', 'Content-Length': '0'} 2023-09-29T22:18:14.692 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5a8f59ad888cbdd-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:18:14.693 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.693 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.693 DEBUG:boto:Method: GET 2023-09-29T22:18:14.693 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:18:14.693 DEBUG:boto:Data: 2023-09-29T22:18:14.694 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.694 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.694 DEBUG:boto:Params: {} 2023-09-29T22:18:14.694 DEBUG:boto:Token: None 2023-09-29T22:18:14.694 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3 2023-09-29T22:18:14.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ih4/e5glXBBC6r+vk86wAAOq+ZA= 2023-09-29T22:18:14.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ih4/e5glXBBC6r+vk86wAAOq+ZA=', 'Content-Length': '0'} 2023-09-29T22:18:14.697 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b8b1550021c669d-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.738 DEBUG:boto:path=/hcnphz-47/obj3 2023-09-29T22:18:14.738 DEBUG:boto:auth_path=/hcnphz-47/obj3 2023-09-29T22:18:14.739 DEBUG:boto:Method: GET 2023-09-29T22:18:14.739 DEBUG:boto:Path: /hcnphz-47/obj3 2023-09-29T22:18:14.739 DEBUG:boto:Data: 2023-09-29T22:18:14.739 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.739 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.740 DEBUG:boto:Params: {} 2023-09-29T22:18:14.740 DEBUG:boto:Token: None 2023-09-29T22:18:14.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj3 2023-09-29T22:18:14.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ih4/e5glXBBC6r+vk86wAAOq+ZA= 2023-09-29T22:18:14.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ih4/e5glXBBC6r+vk86wAAOq+ZA=', 'Content-Length': '0'} 2023-09-29T22:18:14.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000058ab73d8ddd8e946-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.743 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.743 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.743 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.744 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.744 DEBUG:boto:Method: GET 2023-09-29T22:18:14.744 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.744 DEBUG:boto:Data: 2023-09-29T22:18:14.744 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.744 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.744 DEBUG:boto:Params: {} 2023-09-29T22:18:14.744 DEBUG:boto:Token: None 2023-09-29T22:18:14.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.750 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a45a46ff346fb87b-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.750 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.751 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.751 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.751 DEBUG:boto:path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.751 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.751 DEBUG:boto:Method: GET 2023-09-29T22:18:14.752 DEBUG:boto:Path: /hcnphz-47/?prefix=obj3 2023-09-29T22:18:14.752 DEBUG:boto:Data: 2023-09-29T22:18:14.752 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.752 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.752 DEBUG:boto:Params: {} 2023-09-29T22:18:14.752 DEBUG:boto:Token: None 2023-09-29T22:18:14.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011e92cbf7f4d41e0-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.757 DEBUG:boto:b'hcnphz-47obj31000falseobj32023-09-29T22:15:12.502Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:18:14.758 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.758 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.758 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.758 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.758 DEBUG:boto:Method: GET 2023-09-29T22:18:14.759 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.759 DEBUG:boto:Data: 2023-09-29T22:18:14.759 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.759 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.759 DEBUG:boto:Params: {} 2023-09-29T22:18:14.759 DEBUG:boto:Token: None 2023-09-29T22:18:14.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4= 2023-09-29T22:18:14.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4=', 'Content-Length': '0'} 2023-09-29T22:18:14.762 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001d0bdb02f680374f-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.762 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.763 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.763 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.763 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.763 DEBUG:boto:Method: GET 2023-09-29T22:18:14.763 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.763 DEBUG:boto:Data: 2023-09-29T22:18:14.763 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.763 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.764 DEBUG:boto:Params: {} 2023-09-29T22:18:14.764 DEBUG:boto:Token: None 2023-09-29T22:18:14.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4= 2023-09-29T22:18:14.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4=', 'Content-Length': '0'} 2023-09-29T22:18:14.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005288f57171316290-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.767 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.767 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.767 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.767 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.768 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.768 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.768 DEBUG:boto:Data: 2023-09-29T22:18:14.768 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.768 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.768 DEBUG:boto:Params: {} 2023-09-29T22:18:14.768 DEBUG:boto:Token: None 2023-09-29T22:18:14.769 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.769 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qS0yh0LWppiecOJLMf8MjyM5lzg= 2023-09-29T22:18:14.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qS0yh0LWppiecOJLMf8MjyM5lzg=', 'Content-Length': '0'} 2023-09-29T22:18:14.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000023225430b901ceda-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.771 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.772 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.772 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.772 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.772 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.772 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.772 DEBUG:boto:Data: 2023-09-29T22:18:14.772 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.772 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.773 DEBUG:boto:Params: {} 2023-09-29T22:18:14.773 DEBUG:boto:Token: None 2023-09-29T22:18:14.773 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qS0yh0LWppiecOJLMf8MjyM5lzg= 2023-09-29T22:18:14.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qS0yh0LWppiecOJLMf8MjyM5lzg=', 'Content-Length': '0'} 2023-09-29T22:18:14.775 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c0e669beaaba62eb-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:18:14.776 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.776 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.776 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.776 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.776 DEBUG:boto:Method: GET 2023-09-29T22:18:14.776 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.776 DEBUG:boto:Data: 2023-09-29T22:18:14.777 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.777 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.777 DEBUG:boto:Params: {} 2023-09-29T22:18:14.777 DEBUG:boto:Token: None 2023-09-29T22:18:14.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4= 2023-09-29T22:18:14.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4=', 'Content-Length': '0'} 2023-09-29T22:18:14.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dfc20166f4cf92c7-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.822 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.823 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.823 DEBUG:boto:path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.823 DEBUG:boto:auth_path=/hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.823 DEBUG:boto:Method: GET 2023-09-29T22:18:14.823 DEBUG:boto:Path: /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.824 DEBUG:boto:Data: 2023-09-29T22:18:14.824 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.824 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.824 DEBUG:boto:Params: {} 2023-09-29T22:18:14.824 DEBUG:boto:Token: None 2023-09-29T22:18:14.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4?versionId=null 2023-09-29T22:18:14.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4= 2023-09-29T22:18:14.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2vqJzW4tIEi9F1xo1HQ/kYq8XC4=', 'Content-Length': '0'} 2023-09-29T22:18:14.828 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd56335d973766b2-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.829 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.829 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.829 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.829 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.829 DEBUG:boto:Method: GET 2023-09-29T22:18:14.829 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.829 DEBUG:boto:Data: 2023-09-29T22:18:14.829 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.829 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.830 DEBUG:boto:Params: {} 2023-09-29T22:18:14.830 DEBUG:boto:Token: None 2023-09-29T22:18:14.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bde42eac27da9355-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.835 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.835 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.835 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.836 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.836 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.836 DEBUG:boto:Method: GET 2023-09-29T22:18:14.836 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.836 DEBUG:boto:Data: 2023-09-29T22:18:14.836 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.836 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.836 DEBUG:boto:Params: {} 2023-09-29T22:18:14.836 DEBUG:boto:Token: None 2023-09-29T22:18:14.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050c98af87c60aecb-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.841 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.842 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.842 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.842 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.842 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:18:14.842 DEBUG:boto:Data: 2023-09-29T22:18:14.842 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.843 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.843 DEBUG:boto:Params: {} 2023-09-29T22:18:14.843 DEBUG:boto:Token: None 2023-09-29T22:18:14.843 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4 2023-09-29T22:18:14.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oX7y1zHV/2fVK187PTRy4HE2oc8= 2023-09-29T22:18:14.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oX7y1zHV/2fVK187PTRy4HE2oc8=', 'Content-Length': '0'} 2023-09-29T22:18:14.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001c831ab454907492-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.846 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.846 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.846 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.846 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:18:14.846 DEBUG:boto:Data: 2023-09-29T22:18:14.846 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.846 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.846 DEBUG:boto:Params: {} 2023-09-29T22:18:14.847 DEBUG:boto:Token: None 2023-09-29T22:18:14.847 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4 2023-09-29T22:18:14.847 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oX7y1zHV/2fVK187PTRy4HE2oc8= 2023-09-29T22:18:14.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oX7y1zHV/2fVK187PTRy4HE2oc8=', 'Content-Length': '0'} 2023-09-29T22:18:14.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce4138ee7effe47e-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:18:14.850 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.850 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.850 DEBUG:boto:Method: GET 2023-09-29T22:18:14.850 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:18:14.850 DEBUG:boto:Data: 2023-09-29T22:18:14.850 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.850 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.850 DEBUG:boto:Params: {} 2023-09-29T22:18:14.851 DEBUG:boto:Token: None 2023-09-29T22:18:14.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4 2023-09-29T22:18:14.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GmX5SQvXbSFzrLRwDof/iUrVJx4= 2023-09-29T22:18:14.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GmX5SQvXbSFzrLRwDof/iUrVJx4=', 'Content-Length': '0'} 2023-09-29T22:18:14.853 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026151db6d04fa49c-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.853 DEBUG:boto:path=/hcnphz-47/obj4 2023-09-29T22:18:14.853 DEBUG:boto:auth_path=/hcnphz-47/obj4 2023-09-29T22:18:14.854 DEBUG:boto:Method: GET 2023-09-29T22:18:14.854 DEBUG:boto:Path: /hcnphz-47/obj4 2023-09-29T22:18:14.854 DEBUG:boto:Data: 2023-09-29T22:18:14.854 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.854 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.854 DEBUG:boto:Params: {} 2023-09-29T22:18:14.854 DEBUG:boto:Token: None 2023-09-29T22:18:14.854 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj4 2023-09-29T22:18:14.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GmX5SQvXbSFzrLRwDof/iUrVJx4= 2023-09-29T22:18:14.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GmX5SQvXbSFzrLRwDof/iUrVJx4=', 'Content-Length': '0'} 2023-09-29T22:18:14.857 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:15:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003395746ae3c72023-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.857 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.858 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.858 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.858 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.858 DEBUG:boto:Method: GET 2023-09-29T22:18:14.858 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.858 DEBUG:boto:Data: 2023-09-29T22:18:14.858 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.858 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.858 DEBUG:boto:Params: {} 2023-09-29T22:18:14.859 DEBUG:boto:Token: None 2023-09-29T22:18:14.859 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.859 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062186c73da9020bf-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.863 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.864 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.864 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.864 DEBUG:boto:path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.864 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.864 DEBUG:boto:Method: GET 2023-09-29T22:18:14.864 DEBUG:boto:Path: /hcnphz-47/?prefix=obj4 2023-09-29T22:18:14.864 DEBUG:boto:Data: 2023-09-29T22:18:14.864 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.865 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.865 DEBUG:boto:Params: {} 2023-09-29T22:18:14.865 DEBUG:boto:Token: None 2023-09-29T22:18:14.865 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe526f57b2d20e71-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.869 DEBUG:boto:b'hcnphz-47obj41000falseobj42023-09-29T22:15:12.553Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:18:14.870 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.870 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.870 DEBUG:boto:path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.870 DEBUG:boto:auth_path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.871 DEBUG:boto:Method: GET 2023-09-29T22:18:14.871 DEBUG:boto:Path: /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.871 DEBUG:boto:Data: 2023-09-29T22:18:14.871 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.871 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.871 DEBUG:boto:Params: {} 2023-09-29T22:18:14.871 DEBUG:boto:Token: None 2023-09-29T22:18:14.871 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c= 2023-09-29T22:18:14.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c=', 'Content-Length': '0'} 2023-09-29T22:18:14.874 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c33d1aaa34520be4-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.918 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.918 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.919 DEBUG:boto:path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.919 DEBUG:boto:auth_path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.919 DEBUG:boto:Method: GET 2023-09-29T22:18:14.919 DEBUG:boto:Path: /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.919 DEBUG:boto:Data: 2023-09-29T22:18:14.919 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.919 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.919 DEBUG:boto:Params: {} 2023-09-29T22:18:14.920 DEBUG:boto:Token: None 2023-09-29T22:18:14.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c= 2023-09-29T22:18:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c=', 'Content-Length': '0'} 2023-09-29T22:18:14.923 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c6d08548ac7dfce0-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.923 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.924 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.924 DEBUG:boto:path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.924 DEBUG:boto:auth_path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.924 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.924 DEBUG:boto:Path: /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.924 DEBUG:boto:Data: 2023-09-29T22:18:14.924 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.924 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.924 DEBUG:boto:Params: {} 2023-09-29T22:18:14.925 DEBUG:boto:Token: None 2023-09-29T22:18:14.925 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.925 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CKGFWskrasaMNfk2+mVhKsbBGso= 2023-09-29T22:18:14.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CKGFWskrasaMNfk2+mVhKsbBGso=', 'Content-Length': '0'} 2023-09-29T22:18:14.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd6abc23ff243199-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.928 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.928 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.928 DEBUG:boto:path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.928 DEBUG:boto:auth_path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.928 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.928 DEBUG:boto:Path: /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.928 DEBUG:boto:Data: 2023-09-29T22:18:14.929 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.929 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.929 DEBUG:boto:Params: {} 2023-09-29T22:18:14.929 DEBUG:boto:Token: None 2023-09-29T22:18:14.929 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.929 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CKGFWskrasaMNfk2+mVhKsbBGso= 2023-09-29T22:18:14.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CKGFWskrasaMNfk2+mVhKsbBGso=', 'Content-Length': '0'} 2023-09-29T22:18:14.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000246697378509c52d-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:18:14.932 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.932 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.932 DEBUG:boto:path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.933 DEBUG:boto:auth_path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.933 DEBUG:boto:Method: GET 2023-09-29T22:18:14.933 DEBUG:boto:Path: /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.933 DEBUG:boto:Data: 2023-09-29T22:18:14.933 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.933 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.933 DEBUG:boto:Params: {} 2023-09-29T22:18:14.933 DEBUG:boto:Token: None 2023-09-29T22:18:14.934 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c= 2023-09-29T22:18:14.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c=', 'Content-Length': '0'} 2023-09-29T22:18:14.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021f0f0d71ae44942-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.937 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.937 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.937 DEBUG:boto:path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.937 DEBUG:boto:auth_path=/hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.937 DEBUG:boto:Method: GET 2023-09-29T22:18:14.937 DEBUG:boto:Path: /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.937 DEBUG:boto:Data: 2023-09-29T22:18:14.937 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.938 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.938 DEBUG:boto:Params: {} 2023-09-29T22:18:14.938 DEBUG:boto:Token: None 2023-09-29T22:18:14.938 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5?versionId=null 2023-09-29T22:18:14.938 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c= 2023-09-29T22:18:14.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l31nwhrMeETxaMWvRvhayczqg9c=', 'Content-Length': '0'} 2023-09-29T22:18:14.941 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001040e9096a038b91-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.982 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.982 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.983 DEBUG:boto:path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:14.983 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:14.983 DEBUG:boto:Method: GET 2023-09-29T22:18:14.983 DEBUG:boto:Path: /hcnphz-47/?prefix=obj5 2023-09-29T22:18:14.983 DEBUG:boto:Data: 2023-09-29T22:18:14.983 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.983 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.983 DEBUG:boto:Params: {} 2023-09-29T22:18:14.984 DEBUG:boto:Token: None 2023-09-29T22:18:14.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.988 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c71a7dcd1c0ba0f9-0065174d26-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.988 DEBUG:boto:b'hcnphz-47obj51000falseobj52023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.989 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:14.989 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:14.989 DEBUG:boto:path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:14.989 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:14.989 DEBUG:boto:Method: GET 2023-09-29T22:18:14.989 DEBUG:boto:Path: /hcnphz-47/?prefix=obj5 2023-09-29T22:18:14.990 DEBUG:boto:Data: 2023-09-29T22:18:14.990 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:14.990 DEBUG:boto:Port: 8000 2023-09-29T22:18:14.990 DEBUG:boto:Params: {} 2023-09-29T22:18:14.990 DEBUG:boto:Token: None 2023-09-29T22:18:14.990 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/ 2023-09-29T22:18:14.990 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M= 2023-09-29T22:18:14.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7cK4eWBJr5GqJYq/FGSo4gLN3M=', 'Content-Length': '0'} 2023-09-29T22:18:14.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed772ce5a070308d-0065174d26-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:14.995 DEBUG:boto:b'hcnphz-47obj51000falseobj52023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:14.996 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:14.996 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:14.996 DEBUG:boto:Method: HEAD 2023-09-29T22:18:14.996 DEBUG:boto:Path: /hcnphz-47/obj5 2023-09-29T22:18:14.996 DEBUG:boto:Data: 2023-09-29T22:18:14.996 DEBUG:boto:Headers: {} 2023-09-29T22:18:14.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:14.997 DEBUG:boto:Port: 8001 2023-09-29T22:18:14.997 DEBUG:boto:Params: {} 2023-09-29T22:18:14.997 DEBUG:boto:Token: None 2023-09-29T22:18:14.997 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:14 GMT /hcnphz-47/obj5 2023-09-29T22:18:14.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aBIBhEnOSGI4uV6zkqiMbsK4+YU= 2023-09-29T22:18:14.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aBIBhEnOSGI4uV6zkqiMbsK4+YU=', 'Content-Length': '0'} 2023-09-29T22:18:15.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bbe38a3d3f863c31-0065174d26-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.000 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:15.000 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:15.000 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.000 DEBUG:boto:Path: /hcnphz-47/obj5 2023-09-29T22:18:15.001 DEBUG:boto:Data: 2023-09-29T22:18:15.001 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.001 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.001 DEBUG:boto:Params: {} 2023-09-29T22:18:15.001 DEBUG:boto:Token: None 2023-09-29T22:18:15.001 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj5 2023-09-29T22:18:15.001 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UWC1tYwZeTNlWJxYaWEcgOYKbFo= 2023-09-29T22:18:15.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UWC1tYwZeTNlWJxYaWEcgOYKbFo=', 'Content-Length': '0'} 2023-09-29T22:18:15.003 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3ef25af636d7f94-0065174d26-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:18:15.004 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:15.004 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:15.004 DEBUG:boto:Method: GET 2023-09-29T22:18:15.004 DEBUG:boto:Path: /hcnphz-47/obj5 2023-09-29T22:18:15.004 DEBUG:boto:Data: 2023-09-29T22:18:15.004 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.004 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.004 DEBUG:boto:Params: {} 2023-09-29T22:18:15.005 DEBUG:boto:Token: None 2023-09-29T22:18:15.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj5 2023-09-29T22:18:15.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wNBDtsfFKEO0UionHKOdM0kS32o= 2023-09-29T22:18:15.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wNBDtsfFKEO0UionHKOdM0kS32o=', 'Content-Length': '0'} 2023-09-29T22:18:15.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084e7c9d4874a68cb-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.054 DEBUG:boto:path=/hcnphz-47/obj5 2023-09-29T22:18:15.054 DEBUG:boto:auth_path=/hcnphz-47/obj5 2023-09-29T22:18:15.055 DEBUG:boto:Method: GET 2023-09-29T22:18:15.055 DEBUG:boto:Path: /hcnphz-47/obj5 2023-09-29T22:18:15.055 DEBUG:boto:Data: 2023-09-29T22:18:15.055 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.055 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.055 DEBUG:boto:Params: {} 2023-09-29T22:18:15.055 DEBUG:boto:Token: None 2023-09-29T22:18:15.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj5 2023-09-29T22:18:15.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wNBDtsfFKEO0UionHKOdM0kS32o= 2023-09-29T22:18:15.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wNBDtsfFKEO0UionHKOdM0kS32o=', 'Content-Length': '0'} 2023-09-29T22:18:15.058 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006ac7f931b2541294-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.059 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.059 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.059 DEBUG:boto:path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:15.059 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:15.059 DEBUG:boto:Method: GET 2023-09-29T22:18:15.059 DEBUG:boto:Path: /hcnphz-47/?prefix=obj5 2023-09-29T22:18:15.059 DEBUG:boto:Data: 2023-09-29T22:18:15.060 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.060 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.060 DEBUG:boto:Params: {} 2023-09-29T22:18:15.060 DEBUG:boto:Token: None 2023-09-29T22:18:15.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e000c617259abf8d-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.065 DEBUG:boto:b'hcnphz-47obj51000falseobj52023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.066 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.066 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.066 DEBUG:boto:path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:15.066 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj5 2023-09-29T22:18:15.066 DEBUG:boto:Method: GET 2023-09-29T22:18:15.067 DEBUG:boto:Path: /hcnphz-47/?prefix=obj5 2023-09-29T22:18:15.067 DEBUG:boto:Data: 2023-09-29T22:18:15.067 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.067 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.067 DEBUG:boto:Params: {} 2023-09-29T22:18:15.067 DEBUG:boto:Token: None 2023-09-29T22:18:15.067 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.067 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed892ab7daaf905-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.118 DEBUG:boto:b'hcnphz-47obj51000falseobj52023-09-29T22:17:46.046Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:18:15.119 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.119 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.119 DEBUG:boto:path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.120 DEBUG:boto:auth_path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.120 DEBUG:boto:Method: GET 2023-09-29T22:18:15.120 DEBUG:boto:Path: /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.120 DEBUG:boto:Data: 2023-09-29T22:18:15.120 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.120 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.120 DEBUG:boto:Params: {} 2023-09-29T22:18:15.121 DEBUG:boto:Token: None 2023-09-29T22:18:15.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc= 2023-09-29T22:18:15.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc=', 'Content-Length': '0'} 2023-09-29T22:18:15.124 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000011bdbf6e39cf354a-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.124 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.124 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.125 DEBUG:boto:path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.125 DEBUG:boto:auth_path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.125 DEBUG:boto:Method: GET 2023-09-29T22:18:15.125 DEBUG:boto:Path: /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.125 DEBUG:boto:Data: 2023-09-29T22:18:15.125 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.125 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.126 DEBUG:boto:Params: {} 2023-09-29T22:18:15.126 DEBUG:boto:Token: None 2023-09-29T22:18:15.126 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc= 2023-09-29T22:18:15.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc=', 'Content-Length': '0'} 2023-09-29T22:18:15.129 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000aa491af121edd885-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.129 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.129 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.130 DEBUG:boto:path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.130 DEBUG:boto:auth_path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.130 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.130 DEBUG:boto:Path: /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.130 DEBUG:boto:Data: 2023-09-29T22:18:15.130 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.130 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.131 DEBUG:boto:Params: {} 2023-09-29T22:18:15.131 DEBUG:boto:Token: None 2023-09-29T22:18:15.131 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.131 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DHWIVjSZ31d4XWh3kU8lSMwkKKM= 2023-09-29T22:18:15.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DHWIVjSZ31d4XWh3kU8lSMwkKKM=', 'Content-Length': '0'} 2023-09-29T22:18:15.134 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d51e9c4a44d6b821-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.134 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.134 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.134 DEBUG:boto:path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.135 DEBUG:boto:auth_path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.135 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.135 DEBUG:boto:Path: /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.135 DEBUG:boto:Data: 2023-09-29T22:18:15.135 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.135 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.135 DEBUG:boto:Params: {} 2023-09-29T22:18:15.136 DEBUG:boto:Token: None 2023-09-29T22:18:15.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DHWIVjSZ31d4XWh3kU8lSMwkKKM= 2023-09-29T22:18:15.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DHWIVjSZ31d4XWh3kU8lSMwkKKM=', 'Content-Length': '0'} 2023-09-29T22:18:15.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046f41a506c081c25-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:18:15.139 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.139 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.139 DEBUG:boto:path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.139 DEBUG:boto:auth_path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.139 DEBUG:boto:Method: GET 2023-09-29T22:18:15.140 DEBUG:boto:Path: /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.140 DEBUG:boto:Data: 2023-09-29T22:18:15.140 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.140 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.140 DEBUG:boto:Params: {} 2023-09-29T22:18:15.140 DEBUG:boto:Token: None 2023-09-29T22:18:15.141 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc= 2023-09-29T22:18:15.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc=', 'Content-Length': '0'} 2023-09-29T22:18:15.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025104cad8df39bf7-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.144 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.144 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.144 DEBUG:boto:path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.144 DEBUG:boto:auth_path=/hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.144 DEBUG:boto:Method: GET 2023-09-29T22:18:15.145 DEBUG:boto:Path: /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.145 DEBUG:boto:Data: 2023-09-29T22:18:15.145 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.145 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.145 DEBUG:boto:Params: {} 2023-09-29T22:18:15.145 DEBUG:boto:Token: None 2023-09-29T22:18:15.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6?versionId=null 2023-09-29T22:18:15.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc= 2023-09-29T22:18:15.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XdG/fxdeY54q/9oKYob2FnyKXLc=', 'Content-Length': '0'} 2023-09-29T22:18:15.148 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002a8ca6b045f4edd0-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.149 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.149 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.149 DEBUG:boto:path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.149 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.150 DEBUG:boto:Method: GET 2023-09-29T22:18:15.150 DEBUG:boto:Path: /hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.150 DEBUG:boto:Data: 2023-09-29T22:18:15.150 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.150 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.150 DEBUG:boto:Params: {} 2023-09-29T22:18:15.151 DEBUG:boto:Token: None 2023-09-29T22:18:15.151 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.151 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.155 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000498380338aa003a-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.202 DEBUG:boto:b'hcnphz-47obj61000falseobj62023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.203 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.203 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.203 DEBUG:boto:path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.203 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.204 DEBUG:boto:Method: GET 2023-09-29T22:18:15.204 DEBUG:boto:Path: /hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.204 DEBUG:boto:Data: 2023-09-29T22:18:15.204 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.204 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.204 DEBUG:boto:Params: {} 2023-09-29T22:18:15.204 DEBUG:boto:Token: None 2023-09-29T22:18:15.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7d7f9e9a954cc00-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.209 DEBUG:boto:b'hcnphz-47obj61000falseobj62023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.210 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.210 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.210 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.210 DEBUG:boto:Path: /hcnphz-47/obj6 2023-09-29T22:18:15.210 DEBUG:boto:Data: 2023-09-29T22:18:15.210 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.211 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.211 DEBUG:boto:Params: {} 2023-09-29T22:18:15.211 DEBUG:boto:Token: None 2023-09-29T22:18:15.211 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6 2023-09-29T22:18:15.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U7Q/6D1ghuBGSM0ZBQehSciO6OA= 2023-09-29T22:18:15.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U7Q/6D1ghuBGSM0ZBQehSciO6OA=', 'Content-Length': '0'} 2023-09-29T22:18:15.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000465c7b08a9b238b9-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.214 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.214 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.215 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.215 DEBUG:boto:Path: /hcnphz-47/obj6 2023-09-29T22:18:15.215 DEBUG:boto:Data: 2023-09-29T22:18:15.215 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.215 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.215 DEBUG:boto:Params: {} 2023-09-29T22:18:15.215 DEBUG:boto:Token: None 2023-09-29T22:18:15.216 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6 2023-09-29T22:18:15.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U7Q/6D1ghuBGSM0ZBQehSciO6OA= 2023-09-29T22:18:15.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U7Q/6D1ghuBGSM0ZBQehSciO6OA=', 'Content-Length': '0'} 2023-09-29T22:18:15.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008822ca39a78c384d-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:18:15.219 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.219 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.219 DEBUG:boto:Method: GET 2023-09-29T22:18:15.219 DEBUG:boto:Path: /hcnphz-47/obj6 2023-09-29T22:18:15.219 DEBUG:boto:Data: 2023-09-29T22:18:15.220 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.220 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.220 DEBUG:boto:Params: {} 2023-09-29T22:18:15.220 DEBUG:boto:Token: None 2023-09-29T22:18:15.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6 2023-09-29T22:18:15.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7/AGMBgKXNYBXyfFbxBUAU5KwbM= 2023-09-29T22:18:15.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7/AGMBgKXNYBXyfFbxBUAU5KwbM=', 'Content-Length': '0'} 2023-09-29T22:18:15.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ea8dc469588085b9-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.224 DEBUG:boto:path=/hcnphz-47/obj6 2023-09-29T22:18:15.224 DEBUG:boto:auth_path=/hcnphz-47/obj6 2023-09-29T22:18:15.224 DEBUG:boto:Method: GET 2023-09-29T22:18:15.224 DEBUG:boto:Path: /hcnphz-47/obj6 2023-09-29T22:18:15.224 DEBUG:boto:Data: 2023-09-29T22:18:15.224 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.225 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.225 DEBUG:boto:Params: {} 2023-09-29T22:18:15.225 DEBUG:boto:Token: None 2023-09-29T22:18:15.225 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj6 2023-09-29T22:18:15.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7/AGMBgKXNYBXyfFbxBUAU5KwbM= 2023-09-29T22:18:15.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7/AGMBgKXNYBXyfFbxBUAU5KwbM=', 'Content-Length': '0'} 2023-09-29T22:18:15.228 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6aabe4c0cbfa4b5-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.270 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.271 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.271 DEBUG:boto:path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.271 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.271 DEBUG:boto:Method: GET 2023-09-29T22:18:15.271 DEBUG:boto:Path: /hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.271 DEBUG:boto:Data: 2023-09-29T22:18:15.271 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.272 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.272 DEBUG:boto:Params: {} 2023-09-29T22:18:15.272 DEBUG:boto:Token: None 2023-09-29T22:18:15.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059f9f8c96c9269a2-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.277 DEBUG:boto:b'hcnphz-47obj61000falseobj62023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.278 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.278 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.278 DEBUG:boto:path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.278 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.278 DEBUG:boto:Method: GET 2023-09-29T22:18:15.279 DEBUG:boto:Path: /hcnphz-47/?prefix=obj6 2023-09-29T22:18:15.279 DEBUG:boto:Data: 2023-09-29T22:18:15.279 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.279 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.279 DEBUG:boto:Params: {} 2023-09-29T22:18:15.279 DEBUG:boto:Token: None 2023-09-29T22:18:15.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.284 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e03e038c8ddb491-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.284 DEBUG:boto:b'hcnphz-47obj61000falseobj62023-09-29T22:17:46.069Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:18:15.285 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.285 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.285 DEBUG:boto:path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.285 DEBUG:boto:auth_path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.285 DEBUG:boto:Method: GET 2023-09-29T22:18:15.285 DEBUG:boto:Path: /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.285 DEBUG:boto:Data: 2023-09-29T22:18:15.285 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.286 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.286 DEBUG:boto:Params: {} 2023-09-29T22:18:15.286 DEBUG:boto:Token: None 2023-09-29T22:18:15.286 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.286 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE= 2023-09-29T22:18:15.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE=', 'Content-Length': '0'} 2023-09-29T22:18:15.288 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075a806e47e1aac7a-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.289 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.289 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.289 DEBUG:boto:path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.289 DEBUG:boto:auth_path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.289 DEBUG:boto:Method: GET 2023-09-29T22:18:15.289 DEBUG:boto:Path: /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.289 DEBUG:boto:Data: 2023-09-29T22:18:15.289 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.290 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.290 DEBUG:boto:Params: {} 2023-09-29T22:18:15.290 DEBUG:boto:Token: None 2023-09-29T22:18:15.290 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.290 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE= 2023-09-29T22:18:15.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE=', 'Content-Length': '0'} 2023-09-29T22:18:15.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd1d1088c9e055e5-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.293 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.293 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.293 DEBUG:boto:path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.293 DEBUG:boto:auth_path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.294 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.294 DEBUG:boto:Path: /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.294 DEBUG:boto:Data: 2023-09-29T22:18:15.294 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.294 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.294 DEBUG:boto:Params: {} 2023-09-29T22:18:15.294 DEBUG:boto:Token: None 2023-09-29T22:18:15.294 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a70DR7j3TeuRzV0CejI+zYQDK2s= 2023-09-29T22:18:15.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a70DR7j3TeuRzV0CejI+zYQDK2s=', 'Content-Length': '0'} 2023-09-29T22:18:15.297 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000007faef17b6e26d20-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.297 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.297 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.297 DEBUG:boto:path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.297 DEBUG:boto:auth_path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.297 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.297 DEBUG:boto:Path: /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.298 DEBUG:boto:Data: 2023-09-29T22:18:15.298 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.298 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.298 DEBUG:boto:Params: {} 2023-09-29T22:18:15.298 DEBUG:boto:Token: None 2023-09-29T22:18:15.298 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a70DR7j3TeuRzV0CejI+zYQDK2s= 2023-09-29T22:18:15.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a70DR7j3TeuRzV0CejI+zYQDK2s=', 'Content-Length': '0'} 2023-09-29T22:18:15.301 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000293acded026181b-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:18:15.301 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.301 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.301 DEBUG:boto:path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.301 DEBUG:boto:auth_path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.301 DEBUG:boto:Method: GET 2023-09-29T22:18:15.302 DEBUG:boto:Path: /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.302 DEBUG:boto:Data: 2023-09-29T22:18:15.302 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.302 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.302 DEBUG:boto:Params: {} 2023-09-29T22:18:15.302 DEBUG:boto:Token: None 2023-09-29T22:18:15.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE= 2023-09-29T22:18:15.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE=', 'Content-Length': '0'} 2023-09-29T22:18:15.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000000e5e16ee4cef802-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.305 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.306 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.306 DEBUG:boto:path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.306 DEBUG:boto:auth_path=/hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.306 DEBUG:boto:Method: GET 2023-09-29T22:18:15.306 DEBUG:boto:Path: /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.306 DEBUG:boto:Data: 2023-09-29T22:18:15.306 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.306 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.306 DEBUG:boto:Params: {} 2023-09-29T22:18:15.307 DEBUG:boto:Token: None 2023-09-29T22:18:15.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7?versionId=null 2023-09-29T22:18:15.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE= 2023-09-29T22:18:15.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L4aQ7kAbNmiRTYPohkONhKwR1zE=', 'Content-Length': '0'} 2023-09-29T22:18:15.309 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f5d696c3df30ea0-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.310 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.310 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.310 DEBUG:boto:path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.310 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.310 DEBUG:boto:Method: GET 2023-09-29T22:18:15.310 DEBUG:boto:Path: /hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.311 DEBUG:boto:Data: 2023-09-29T22:18:15.311 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.311 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.311 DEBUG:boto:Params: {} 2023-09-29T22:18:15.311 DEBUG:boto:Token: None 2023-09-29T22:18:15.311 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.316 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c56de92c5030537-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.316 DEBUG:boto:b'hcnphz-47obj71000falseobj72023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.316 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.317 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.317 DEBUG:boto:path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.317 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.317 DEBUG:boto:Method: GET 2023-09-29T22:18:15.317 DEBUG:boto:Path: /hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.317 DEBUG:boto:Data: 2023-09-29T22:18:15.317 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.318 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.318 DEBUG:boto:Params: {} 2023-09-29T22:18:15.318 DEBUG:boto:Token: None 2023-09-29T22:18:15.318 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.318 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb59af8c8cc47179-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.322 DEBUG:boto:b'hcnphz-47obj71000falseobj72023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.323 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.323 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.323 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.323 DEBUG:boto:Path: /hcnphz-47/obj7 2023-09-29T22:18:15.323 DEBUG:boto:Data: 2023-09-29T22:18:15.324 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.324 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.324 DEBUG:boto:Params: {} 2023-09-29T22:18:15.324 DEBUG:boto:Token: None 2023-09-29T22:18:15.324 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7 2023-09-29T22:18:15.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M3uFZJ5wf2SaYVTASl6MXsYIzTE= 2023-09-29T22:18:15.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M3uFZJ5wf2SaYVTASl6MXsYIzTE=', 'Content-Length': '0'} 2023-09-29T22:18:15.327 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000066c265832e265ac-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.327 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.328 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.328 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.328 DEBUG:boto:Path: /hcnphz-47/obj7 2023-09-29T22:18:15.328 DEBUG:boto:Data: 2023-09-29T22:18:15.328 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.328 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.329 DEBUG:boto:Params: {} 2023-09-29T22:18:15.329 DEBUG:boto:Token: None 2023-09-29T22:18:15.329 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7 2023-09-29T22:18:15.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M3uFZJ5wf2SaYVTASl6MXsYIzTE= 2023-09-29T22:18:15.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M3uFZJ5wf2SaYVTASl6MXsYIzTE=', 'Content-Length': '0'} 2023-09-29T22:18:15.331 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ee6b33b846782386-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:18:15.332 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.332 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.332 DEBUG:boto:Method: GET 2023-09-29T22:18:15.332 DEBUG:boto:Path: /hcnphz-47/obj7 2023-09-29T22:18:15.332 DEBUG:boto:Data: 2023-09-29T22:18:15.332 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.333 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.333 DEBUG:boto:Params: {} 2023-09-29T22:18:15.333 DEBUG:boto:Token: None 2023-09-29T22:18:15.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7 2023-09-29T22:18:15.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tDSd/mHbERAmgQ3hMPYK5uGbkYM= 2023-09-29T22:18:15.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tDSd/mHbERAmgQ3hMPYK5uGbkYM=', 'Content-Length': '0'} 2023-09-29T22:18:15.336 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000069d7432c3f9aa386-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.378 DEBUG:boto:path=/hcnphz-47/obj7 2023-09-29T22:18:15.379 DEBUG:boto:auth_path=/hcnphz-47/obj7 2023-09-29T22:18:15.379 DEBUG:boto:Method: GET 2023-09-29T22:18:15.379 DEBUG:boto:Path: /hcnphz-47/obj7 2023-09-29T22:18:15.379 DEBUG:boto:Data: 2023-09-29T22:18:15.379 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.379 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.379 DEBUG:boto:Params: {} 2023-09-29T22:18:15.379 DEBUG:boto:Token: None 2023-09-29T22:18:15.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj7 2023-09-29T22:18:15.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tDSd/mHbERAmgQ3hMPYK5uGbkYM= 2023-09-29T22:18:15.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tDSd/mHbERAmgQ3hMPYK5uGbkYM=', 'Content-Length': '0'} 2023-09-29T22:18:15.382 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007fc60975c617e96a-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.383 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.383 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.383 DEBUG:boto:path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.383 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.383 DEBUG:boto:Method: GET 2023-09-29T22:18:15.383 DEBUG:boto:Path: /hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.383 DEBUG:boto:Data: 2023-09-29T22:18:15.383 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.384 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.384 DEBUG:boto:Params: {} 2023-09-29T22:18:15.384 DEBUG:boto:Token: None 2023-09-29T22:18:15.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003039e49710b60d69-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.388 DEBUG:boto:b'hcnphz-47obj71000falseobj72023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.389 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.389 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.389 DEBUG:boto:path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.389 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.389 DEBUG:boto:Method: GET 2023-09-29T22:18:15.389 DEBUG:boto:Path: /hcnphz-47/?prefix=obj7 2023-09-29T22:18:15.390 DEBUG:boto:Data: 2023-09-29T22:18:15.390 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.390 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.390 DEBUG:boto:Params: {} 2023-09-29T22:18:15.390 DEBUG:boto:Token: None 2023-09-29T22:18:15.390 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000782e135e513cb855-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.395 DEBUG:boto:b'hcnphz-47obj71000falseobj72023-09-29T22:17:46.126Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:18:15.395 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.395 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.395 DEBUG:boto:path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.396 DEBUG:boto:auth_path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.396 DEBUG:boto:Method: GET 2023-09-29T22:18:15.396 DEBUG:boto:Path: /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.396 DEBUG:boto:Data: 2023-09-29T22:18:15.396 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.396 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.396 DEBUG:boto:Params: {} 2023-09-29T22:18:15.396 DEBUG:boto:Token: None 2023-09-29T22:18:15.397 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A= 2023-09-29T22:18:15.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A=', 'Content-Length': '0'} 2023-09-29T22:18:15.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000085d6f6713041647d-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.399 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.400 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.400 DEBUG:boto:path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.400 DEBUG:boto:auth_path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.400 DEBUG:boto:Method: GET 2023-09-29T22:18:15.400 DEBUG:boto:Path: /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.400 DEBUG:boto:Data: 2023-09-29T22:18:15.400 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.400 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.401 DEBUG:boto:Params: {} 2023-09-29T22:18:15.401 DEBUG:boto:Token: None 2023-09-29T22:18:15.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A= 2023-09-29T22:18:15.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A=', 'Content-Length': '0'} 2023-09-29T22:18:15.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000af084fe1594a3c69-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.404 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.404 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.404 DEBUG:boto:path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.404 DEBUG:boto:auth_path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.404 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.405 DEBUG:boto:Path: /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.405 DEBUG:boto:Data: 2023-09-29T22:18:15.405 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.405 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.405 DEBUG:boto:Params: {} 2023-09-29T22:18:15.405 DEBUG:boto:Token: None 2023-09-29T22:18:15.405 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nhwaTg9+k4kNnmz8lemj4J6JoKU= 2023-09-29T22:18:15.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nhwaTg9+k4kNnmz8lemj4J6JoKU=', 'Content-Length': '0'} 2023-09-29T22:18:15.408 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000640fead911c89c9b-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.408 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.408 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.408 DEBUG:boto:path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.408 DEBUG:boto:auth_path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.408 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.409 DEBUG:boto:Path: /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.409 DEBUG:boto:Data: 2023-09-29T22:18:15.409 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.409 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.409 DEBUG:boto:Params: {} 2023-09-29T22:18:15.409 DEBUG:boto:Token: None 2023-09-29T22:18:15.409 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nhwaTg9+k4kNnmz8lemj4J6JoKU= 2023-09-29T22:18:15.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nhwaTg9+k4kNnmz8lemj4J6JoKU=', 'Content-Length': '0'} 2023-09-29T22:18:15.412 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000006c057246030e4fa-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:18:15.412 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.412 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.413 DEBUG:boto:path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.413 DEBUG:boto:auth_path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.413 DEBUG:boto:Method: GET 2023-09-29T22:18:15.413 DEBUG:boto:Path: /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.413 DEBUG:boto:Data: 2023-09-29T22:18:15.413 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.413 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.413 DEBUG:boto:Params: {} 2023-09-29T22:18:15.414 DEBUG:boto:Token: None 2023-09-29T22:18:15.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A= 2023-09-29T22:18:15.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A=', 'Content-Length': '0'} 2023-09-29T22:18:15.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000334492dd68fa8469-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.458 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.459 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.459 DEBUG:boto:path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.459 DEBUG:boto:auth_path=/hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.459 DEBUG:boto:Method: GET 2023-09-29T22:18:15.459 DEBUG:boto:Path: /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.460 DEBUG:boto:Data: 2023-09-29T22:18:15.460 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.460 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.460 DEBUG:boto:Params: {} 2023-09-29T22:18:15.461 DEBUG:boto:Token: None 2023-09-29T22:18:15.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8?versionId=null 2023-09-29T22:18:15.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A= 2023-09-29T22:18:15.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bVTZqWZGkZBQkUi4fh++8z3/h8A=', 'Content-Length': '0'} 2023-09-29T22:18:15.464 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b45635913e57b6ba-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.465 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.465 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.465 DEBUG:boto:path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.465 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.465 DEBUG:boto:Method: GET 2023-09-29T22:18:15.466 DEBUG:boto:Path: /hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.466 DEBUG:boto:Data: 2023-09-29T22:18:15.466 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.466 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.466 DEBUG:boto:Params: {} 2023-09-29T22:18:15.467 DEBUG:boto:Token: None 2023-09-29T22:18:15.467 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.467 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db3ef7d8990b211b-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.472 DEBUG:boto:b'hcnphz-47obj81000falseobj82023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.473 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.473 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.473 DEBUG:boto:path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.474 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.474 DEBUG:boto:Method: GET 2023-09-29T22:18:15.474 DEBUG:boto:Path: /hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.474 DEBUG:boto:Data: 2023-09-29T22:18:15.474 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.475 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.475 DEBUG:boto:Params: {} 2023-09-29T22:18:15.475 DEBUG:boto:Token: None 2023-09-29T22:18:15.475 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.475 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a020869eee4cb6c0-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.522 DEBUG:boto:b'hcnphz-47obj81000falseobj82023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.523 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.523 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.523 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.523 DEBUG:boto:Path: /hcnphz-47/obj8 2023-09-29T22:18:15.523 DEBUG:boto:Data: 2023-09-29T22:18:15.524 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.524 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.524 DEBUG:boto:Params: {} 2023-09-29T22:18:15.524 DEBUG:boto:Token: None 2023-09-29T22:18:15.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8 2023-09-29T22:18:15.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tjvOK2y3SNBL21LcqZaU0SlGgIM= 2023-09-29T22:18:15.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tjvOK2y3SNBL21LcqZaU0SlGgIM=', 'Content-Length': '0'} 2023-09-29T22:18:15.528 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009474f920c54e4a5c-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.528 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.528 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.528 DEBUG:boto:Method: HEAD 2023-09-29T22:18:15.528 DEBUG:boto:Path: /hcnphz-47/obj8 2023-09-29T22:18:15.529 DEBUG:boto:Data: 2023-09-29T22:18:15.529 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.529 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.529 DEBUG:boto:Params: {} 2023-09-29T22:18:15.529 DEBUG:boto:Token: None 2023-09-29T22:18:15.529 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8 2023-09-29T22:18:15.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tjvOK2y3SNBL21LcqZaU0SlGgIM= 2023-09-29T22:18:15.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tjvOK2y3SNBL21LcqZaU0SlGgIM=', 'Content-Length': '0'} 2023-09-29T22:18:15.533 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cc67985198e71e36-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:18:15.533 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.533 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.534 DEBUG:boto:Method: GET 2023-09-29T22:18:15.534 DEBUG:boto:Path: /hcnphz-47/obj8 2023-09-29T22:18:15.534 DEBUG:boto:Data: 2023-09-29T22:18:15.534 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.534 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.534 DEBUG:boto:Params: {} 2023-09-29T22:18:15.535 DEBUG:boto:Token: None 2023-09-29T22:18:15.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8 2023-09-29T22:18:15.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BOPFnzyNQWO89VHtGLonRkocItw= 2023-09-29T22:18:15.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BOPFnzyNQWO89VHtGLonRkocItw=', 'Content-Length': '0'} 2023-09-29T22:18:15.538 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f37c1f2b65d810f6-0065174d27-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.539 DEBUG:boto:path=/hcnphz-47/obj8 2023-09-29T22:18:15.539 DEBUG:boto:auth_path=/hcnphz-47/obj8 2023-09-29T22:18:15.539 DEBUG:boto:Method: GET 2023-09-29T22:18:15.539 DEBUG:boto:Path: /hcnphz-47/obj8 2023-09-29T22:18:15.539 DEBUG:boto:Data: 2023-09-29T22:18:15.540 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.540 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.540 DEBUG:boto:Params: {} 2023-09-29T22:18:15.540 DEBUG:boto:Token: None 2023-09-29T22:18:15.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/obj8 2023-09-29T22:18:15.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BOPFnzyNQWO89VHtGLonRkocItw= 2023-09-29T22:18:15.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BOPFnzyNQWO89VHtGLonRkocItw=', 'Content-Length': '0'} 2023-09-29T22:18:15.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:17:46 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-47:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.12'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026bad8384ea7d335-0065174d27-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.544 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.544 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.545 DEBUG:boto:path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.545 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.545 DEBUG:boto:Method: GET 2023-09-29T22:18:15.545 DEBUG:boto:Path: /hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.545 DEBUG:boto:Data: 2023-09-29T22:18:15.545 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:15.546 DEBUG:boto:Port: 8001 2023-09-29T22:18:15.546 DEBUG:boto:Params: {} 2023-09-29T22:18:15.546 DEBUG:boto:Token: None 2023-09-29T22:18:15.546 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.546 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4654a2cfa3f3e3c-0065174d27-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.551 DEBUG:boto:b'hcnphz-47obj81000falseobj82023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.552 DEBUG:boto:path=/hcnphz-47/ 2023-09-29T22:18:15.552 DEBUG:boto:auth_path=/hcnphz-47/ 2023-09-29T22:18:15.552 DEBUG:boto:path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.552 DEBUG:boto:auth_path=/hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.552 DEBUG:boto:Method: GET 2023-09-29T22:18:15.552 DEBUG:boto:Path: /hcnphz-47/?prefix=obj8 2023-09-29T22:18:15.553 DEBUG:boto:Data: 2023-09-29T22:18:15.553 DEBUG:boto:Headers: {} 2023-09-29T22:18:15.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:15.553 DEBUG:boto:Port: 8000 2023-09-29T22:18:15.553 DEBUG:boto:Params: {} 2023-09-29T22:18:15.553 DEBUG:boto:Token: None 2023-09-29T22:18:15.554 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:18:15 GMT /hcnphz-47/ 2023-09-29T22:18:15.554 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M= 2023-09-29T22:18:15.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTtGcru5h5dLWFdGT59IpWmsS/M=', 'Content-Length': '0'} 2023-09-29T22:18:15.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005be2a45a899f3681-0065174d27-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:18:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:15.558 DEBUG:boto:b'hcnphz-47obj81000falseobj82023-09-29T22:17:46.141Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:18:15.559 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-47 zones={a2, a1} 2023-09-29T22:18:15.559 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:18:15.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.559 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:18:15.601 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:15.602 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:15.697 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:58.794118Z" 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005437", 2023-09-29T22:18:15.698 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:45.856163Z" 2023-09-29T22:18:15.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.699 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-09-29T22:18:15.699 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:58.746792Z" 2023-09-29T22:18:15.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.700 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-09-29T22:18:15.700 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:58.740894Z" 2023-09-29T22:18:15.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.700 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:18:15.701 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005437', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-09-29T22:18:15.701 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:18:15.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.702 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:18:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:15.748 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:15.828 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:18:15.828 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:18:15.828 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:18:15.828 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000007269", 2023-09-29T22:18:15.829 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:58.794118Z" 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:18:15.830 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005437", 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:45.856163Z" 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.831 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004316", 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:15.832 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:58.746792Z" 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002625", 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:15.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:58.740894Z" 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:18:15.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.835 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:18:15Z" 2023-09-29T22:18:15.835 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:18:15.835 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007269", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:58.794118Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005437", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:45.856163Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004316", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:58.746792Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002625", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:58.740894Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:18:15Z" } 2023-09-29T22:18:15.836 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007269', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:58.794118Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005437', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:45.856163Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004316', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:58.746792Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002625', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:58.740894Z'}}] 2023-09-29T22:18:15.836 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005437', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-09-29T22:18:15.836 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000007269', 1: '00000000000000000000:00000000000000005437', 2: '00000000000000000000:00000000000000004316', 3: '00000000000000000000:00000000000000002625'} 2023-09-29T22:18:15.836 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:18:15.836 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:18:15.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.837 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:18:15.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:15.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:15.976 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:18:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.976 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006705", 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:48.817033Z" 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002902", 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:15:12.407263Z" 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-09-29T22:18:15.977 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:48.814812Z" 2023-09-29T22:18:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001497", 2023-09-29T22:18:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:17:48.816514Z" 2023-09-29T22:18:15.978 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:15.978 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:18:15.979 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000006705', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002620', 3: '00000000000000000000:00000000000000001497'} 2023-09-29T22:18:15.979 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:18:15.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:15.980 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:18:16.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:16.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:16.157 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:18:16.157 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:18:16.158 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006705", 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:48.817033Z" 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.159 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002902", 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:15:12.407263Z" 2023-09-29T22:18:16.160 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002620", 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.161 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:48.814812Z" 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000001497", 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.162 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:17:48.816514Z" 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:18:16.163 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:18:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.164 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:18:16Z" 2023-09-29T22:18:16.164 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:18:16.165 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006705", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:48.817033Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002902", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:15:12.407263Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002620", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:48.814812Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000001497", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:17:48.816514Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:18:16Z" } 2023-09-29T22:18:16.165 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006705', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:48.817033Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002902', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:15:12.407263Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002620', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:48.814812Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000001497', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:17:48.816514Z'}}] 2023-09-29T22:18:16.165 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000006705', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002620', 3: '00000000000000000000:00000000000000001497'} 2023-09-29T22:18:16.165 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000006705', 1: '00000000000000000000:00000000000000002902', 2: '00000000000000000000:00000000000000002620', 3: '00000000000000000000:00000000000000001497'} 2023-09-29T22:18:16.165 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:18:16.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2023-09-29T22:18:16.167 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-48 2023-09-29T22:18:16.167 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:18:16.167 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:18:16.167 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.167 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:18:16.167 DEBUG:boto:Data: 2023-09-29T22:18:16.167 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.168 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.168 DEBUG:boto:Params: {} 2023-09-29T22:18:16.168 DEBUG:boto:Token: None 2023-09-29T22:18:16.168 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/ 2023-09-29T22:18:16.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LDbo/20gGAckK/gcTY87oFHsqaQ= 2023-09-29T22:18:16.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LDbo/20gGAckK/gcTY87oFHsqaQ=', 'Content-Length': '0'} 2023-09-29T22:18:16.197 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005a7ff7d5a4d18949-0065174d28-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.197 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-49 2023-09-29T22:18:16.197 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:18:16.198 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:18:16.198 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.198 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:18:16.198 DEBUG:boto:Data: 2023-09-29T22:18:16.198 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.198 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.198 DEBUG:boto:Params: {} 2023-09-29T22:18:16.198 DEBUG:boto:Token: None 2023-09-29T22:18:16.199 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/ 2023-09-29T22:18:16.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mr61HH6XDBTmlU4XCq4Edg7ioDc= 2023-09-29T22:18:16.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mr61HH6XDBTmlU4XCq4Edg7ioDc=', 'Content-Length': '0'} 2023-09-29T22:18:16.242 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005293d59a050ae7d0-0065174d28-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.242 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:18:16.243 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:18:16.243 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.243 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:18:16.243 DEBUG:boto:Data: 2023-09-29T22:18:16.243 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.243 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.243 DEBUG:boto:Params: {} 2023-09-29T22:18:16.243 DEBUG:boto:Token: None 2023-09-29T22:18:16.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/ 2023-09-29T22:18:16.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M= 2023-09-29T22:18:16.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M=', 'Content-Length': '0'} 2023-09-29T22:18:16.246 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068cad8341746cec3-0065174d28-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.247 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:18:16.247 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:18:16.247 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.247 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:18:16.248 DEBUG:boto:Data: 2023-09-29T22:18:16.248 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.248 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.248 DEBUG:boto:Params: {} 2023-09-29T22:18:16.249 DEBUG:boto:Token: None 2023-09-29T22:18:16.249 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/obj1 2023-09-29T22:18:16.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JaaXi5ZKpgXiAJMXOdgYbbiCecs= 2023-09-29T22:18:16.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JaaXi5ZKpgXiAJMXOdgYbbiCecs='} 2023-09-29T22:18:16.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d0d3efa0d0f77e5-0065174d28-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.295 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:18:16.295 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:18:16.295 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.295 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:18:16.295 DEBUG:boto:Data: 2023-09-29T22:18:16.295 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.295 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.295 DEBUG:boto:Params: {} 2023-09-29T22:18:16.296 DEBUG:boto:Token: None 2023-09-29T22:18:16.296 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/ 2023-09-29T22:18:16.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M= 2023-09-29T22:18:16.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M=', 'Content-Length': '0'} 2023-09-29T22:18:16.299 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ddf04b6c1e36f60-0065174d28-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.299 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:18:16.299 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:18:16.299 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.299 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:18:16.299 DEBUG:boto:Data: 2023-09-29T22:18:16.300 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.300 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.300 DEBUG:boto:Params: {} 2023-09-29T22:18:16.300 DEBUG:boto:Token: None 2023-09-29T22:18:16.300 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/obj2 2023-09-29T22:18:16.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SC8pFA8Na+GA+M/aWAV/fpoiQX4= 2023-09-29T22:18:16.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SC8pFA8Na+GA+M/aWAV/fpoiQX4='} 2023-09-29T22:18:16.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000077b1f57aaa11f1d-0065174d28-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.309 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:18:16.309 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:18:16.310 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.310 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:18:16.310 DEBUG:boto:Data: 2023-09-29T22:18:16.310 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.310 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.310 DEBUG:boto:Params: {} 2023-09-29T22:18:16.310 DEBUG:boto:Token: None 2023-09-29T22:18:16.310 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/ 2023-09-29T22:18:16.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M= 2023-09-29T22:18:16.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M=', 'Content-Length': '0'} 2023-09-29T22:18:16.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055883062ea2203f2-0065174d28-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.314 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:18:16.314 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:18:16.314 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.314 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:18:16.314 DEBUG:boto:Data: 2023-09-29T22:18:16.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.314 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.315 DEBUG:boto:Params: {} 2023-09-29T22:18:16.315 DEBUG:boto:Token: None 2023-09-29T22:18:16.315 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/obj3 2023-09-29T22:18:16.315 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4PpwpoNKQyU2jqwu4Qq4VE6lWCo= 2023-09-29T22:18:16.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4PpwpoNKQyU2jqwu4Qq4VE6lWCo='} 2023-09-29T22:18:16.362 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eae6f94c1859b70d-0065174d28-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.363 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:18:16.363 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:18:16.363 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.363 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:18:16.363 DEBUG:boto:Data: 2023-09-29T22:18:16.363 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.364 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.364 DEBUG:boto:Params: {} 2023-09-29T22:18:16.364 DEBUG:boto:Token: None 2023-09-29T22:18:16.364 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/ 2023-09-29T22:18:16.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M= 2023-09-29T22:18:16.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9YQxDPl3GiOLaDsyXY2HTu1Tp4M=', 'Content-Length': '0'} 2023-09-29T22:18:16.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf9ed2e2c9d68412-0065174d28-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.369 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:18:16.369 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:18:16.369 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.369 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:18:16.369 DEBUG:boto:Data: 2023-09-29T22:18:16.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:18:16.370 DEBUG:boto:Port: 8000 2023-09-29T22:18:16.371 DEBUG:boto:Params: {} 2023-09-29T22:18:16.371 DEBUG:boto:Token: None 2023-09-29T22:18:16.371 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-48/obj4 2023-09-29T22:18:16.371 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7jc3uXbynxyij4xQg74BteSttKo= 2023-09-29T22:18:16.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7jc3uXbynxyij4xQg74BteSttKo='} 2023-09-29T22:18:16.383 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000826a18d67d6342d1-0065174d28-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.383 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:18:16.383 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:18:16.383 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.383 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:18:16.384 DEBUG:boto:Data: 2023-09-29T22:18:16.384 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.384 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.384 DEBUG:boto:Params: {} 2023-09-29T22:18:16.384 DEBUG:boto:Token: None 2023-09-29T22:18:16.384 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/ 2023-09-29T22:18:16.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0= 2023-09-29T22:18:16.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0=', 'Content-Length': '0'} 2023-09-29T22:18:16.388 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099fee6b573bffa77-0065174d28-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.388 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:18:16.388 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:18:16.389 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.389 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:18:16.389 DEBUG:boto:Data: 2023-09-29T22:18:16.389 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.389 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.389 DEBUG:boto:Params: {} 2023-09-29T22:18:16.389 DEBUG:boto:Token: None 2023-09-29T22:18:16.390 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/obj1 2023-09-29T22:18:16.390 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wEgskVGMRPe7H0iakccf9Qbu4Io= 2023-09-29T22:18:16.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wEgskVGMRPe7H0iakccf9Qbu4Io='} 2023-09-29T22:18:16.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000764eb16777890003-0065174d28-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.435 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:18:16.435 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:18:16.435 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.435 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:18:16.435 DEBUG:boto:Data: 2023-09-29T22:18:16.435 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.436 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.436 DEBUG:boto:Params: {} 2023-09-29T22:18:16.436 DEBUG:boto:Token: None 2023-09-29T22:18:16.436 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/ 2023-09-29T22:18:16.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0= 2023-09-29T22:18:16.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0=', 'Content-Length': '0'} 2023-09-29T22:18:16.440 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4a732472711939a-0065174d28-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.440 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:18:16.440 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:18:16.441 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.441 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:18:16.441 DEBUG:boto:Data: 2023-09-29T22:18:16.441 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.441 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.441 DEBUG:boto:Params: {} 2023-09-29T22:18:16.441 DEBUG:boto:Token: None 2023-09-29T22:18:16.442 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/obj2 2023-09-29T22:18:16.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+JT0gvJ95cjzcWWIUlSwOZoqijg= 2023-09-29T22:18:16.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+JT0gvJ95cjzcWWIUlSwOZoqijg='} 2023-09-29T22:18:16.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008771ca756180bb4a-0065174d28-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.454 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:18:16.454 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:18:16.454 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.454 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:18:16.454 DEBUG:boto:Data: 2023-09-29T22:18:16.454 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.455 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.455 DEBUG:boto:Params: {} 2023-09-29T22:18:16.455 DEBUG:boto:Token: None 2023-09-29T22:18:16.455 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/ 2023-09-29T22:18:16.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0= 2023-09-29T22:18:16.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0=', 'Content-Length': '0'} 2023-09-29T22:18:16.458 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000392ec54d767c6eda-0065174d28-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.459 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:18:16.459 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:18:16.459 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.459 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:18:16.459 DEBUG:boto:Data: 2023-09-29T22:18:16.459 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.460 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.460 DEBUG:boto:Params: {} 2023-09-29T22:18:16.460 DEBUG:boto:Token: None 2023-09-29T22:18:16.460 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/obj3 2023-09-29T22:18:16.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UVymxBlmVtTPTnCJUncyNTxfyMY= 2023-09-29T22:18:16.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UVymxBlmVtTPTnCJUncyNTxfyMY='} 2023-09-29T22:18:16.502 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058f78ab5e38a9c76-0065174d28-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.503 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:18:16.503 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:18:16.503 DEBUG:boto:Method: HEAD 2023-09-29T22:18:16.503 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:18:16.503 DEBUG:boto:Data: 2023-09-29T22:18:16.503 DEBUG:boto:Headers: {} 2023-09-29T22:18:16.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.503 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.503 DEBUG:boto:Params: {} 2023-09-29T22:18:16.504 DEBUG:boto:Token: None 2023-09-29T22:18:16.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/ 2023-09-29T22:18:16.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0= 2023-09-29T22:18:16.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uODOjcqSgBJ/VhSxeAPwDZKNzE0=', 'Content-Length': '0'} 2023-09-29T22:18:16.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000522ecc593419f79e-0065174d28-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.508 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:18:16.508 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:18:16.508 DEBUG:boto:Method: PUT 2023-09-29T22:18:16.508 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:18:16.508 DEBUG:boto:Data: 2023-09-29T22:18:16.509 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:18:16.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:18:16.509 DEBUG:boto:Port: 8001 2023-09-29T22:18:16.509 DEBUG:boto:Params: {} 2023-09-29T22:18:16.509 DEBUG:boto:Token: None 2023-09-29T22:18:16.509 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:18:16 GMT /hcnphz-49/obj4 2023-09-29T22:18:16.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BqCYxA7sZ0NwG+CQldsDA9XK4pk= 2023-09-29T22:18:16.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:18:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BqCYxA7sZ0NwG+CQldsDA9XK4pk='} 2023-09-29T22:18:16.521 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003fa8f2e7db23b3a8-0065174d28-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:18:16.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:16.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:16.522 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:18:16.564 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:16.565 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:16.622 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:18:16.650 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:18:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:18:16.650 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025896.227910_382.1", 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:18:16.227910Z" 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:18:16.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:18:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:18:16.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.652 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:18:16.653 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696025896.227910_382.1', 2: '', 3: ''} 2023-09-29T22:18:16.653 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:18:16.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:16.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:18:16.654 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:18:16.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:16.741 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:16.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:16.790+0000 7f2fe81c6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:18:16.796 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:16.794+0000 7f2fe81c6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:18:16.821 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:18:16.821 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:18:16.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:18:16.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025896.227910_382.1", 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.824 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:18:16.227910Z", 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:18:16.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:18:16.826 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:18:16.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.828 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:18:16.828 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:18:16.828 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:18:16.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:18:16.828 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:18:16Z" 2023-09-29T22:18:16.828 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:18:16.829 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025896.227910_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:18:16.227910Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:18:16Z" } 2023-09-29T22:18:16.829 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025896.227910_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:18:16.227910Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:18:16.829 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696025896.227910_382.1', 2: '', 3: ''} 2023-09-29T22:18:16.830 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025896.227910_382.1', 2: '', 3: ''} 2023-09-29T22:18:16.830 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:18:16.830 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-48 2023-09-29T22:18:16.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:16.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:16.830 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:16.881 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:16.881 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:16.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:16.950+0000 7fdee4cd4f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:21.952 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:21.950+0000 7fdee4cd4f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:26.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:26.950+0000 7fdee4cd4f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:31.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:31.953+0000 7fdee4cd4f40 1 bucket sync caught up with source: 2023-09-29T22:18:31.957 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.35.6, , 00000000001.27.6, , 00000000001.29.6, , , 00000000001.17.6, , , ] 2023-09-29T22:18:31.957 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.35.6, , 00000000001.27.6, , 00000000001.29.6, , , 00000000001.17.6, , , ] 2023-09-29T22:18:31.957 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:31.953+0000 7fdee4cd4f40 0 bucket checkpoint complete 2023-09-29T22:18:31.984 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-48 2023-09-29T22:18:31.985 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:31.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:18:31.985 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:18:32.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:18:32.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:18:32.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:32.089+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:37.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:37.089+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:42.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:42.089+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:47.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:47.097+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:52.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:52.097+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:18:57.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:18:57.097+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:02.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:02.097+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:07.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:07.100+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:12.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:12.100+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:17.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:17.100+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:22.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:22.100+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:27.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:27.104+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:32.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:32.104+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:37.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:37.104+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:42.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:42.104+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:47.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:47.107+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:52.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:52.107+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:19:57.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:19:57.107+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:02.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:02.107+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:07.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:07.111+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:12.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:12.111+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:17.111+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:22.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:22.110+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:27.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:27.114+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:32.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:32.114+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:37.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:37.114+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:42.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:42.114+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:47.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:47.118+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:52.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:52.118+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:20:57.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:20:57.117+0000 7fe699f56f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:02.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:02.121+0000 7fe699f56f40 1 bucket sync caught up with source: 2023-09-29T22:21:02.126 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.185.6, , 00000000001.101.6, , 00000000001.126.6, , , 00000000001.158.6, , , ] 2023-09-29T22:21:02.126 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.185.6, , 00000000001.101.6, , 00000000001.126.6, , , 00000000001.158.6, , , ] 2023-09-29T22:21:02.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:02.121+0000 7fe699f56f40 0 bucket checkpoint complete 2023-09-29T22:21:02.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-48 zones={a2, a1} 2023-09-29T22:21:02.155 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.155 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.155 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.155 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:21:02.155 DEBUG:boto:Data: 2023-09-29T22:21:02.155 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.156 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.156 DEBUG:boto:Params: {} 2023-09-29T22:21:02.156 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:21:02.156 DEBUG:boto:Token: None 2023-09-29T22:21:02.156 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UXyo38SUiX3LHc/EkGKpvCK47hk= 2023-09-29T22:21:02.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UXyo38SUiX3LHc/EkGKpvCK47hk=', 'Content-Length': '0'} 2023-09-29T22:21:02.161 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef24cfc076f75694-0065174dce-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.162 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.162 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.162 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.162 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:21:02.162 DEBUG:boto:Data: 2023-09-29T22:21:02.162 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.163 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.163 DEBUG:boto:Params: {} 2023-09-29T22:21:02.163 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:21:02.163 DEBUG:boto:Token: None 2023-09-29T22:21:02.163 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.163 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UXyo38SUiX3LHc/EkGKpvCK47hk= 2023-09-29T22:21:02.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UXyo38SUiX3LHc/EkGKpvCK47hk=', 'Content-Length': '0'} 2023-09-29T22:21:02.168 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000959fe662931f64fd-0065174dce-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.168 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:21:02.168 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.169 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.169 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.169 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.169 DEBUG:boto:Method: GET 2023-09-29T22:21:02.169 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.169 DEBUG:boto:Data: 2023-09-29T22:21:02.169 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.169 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.170 DEBUG:boto:Params: {} 2023-09-29T22:21:02.170 DEBUG:boto:Token: None 2023-09-29T22:21:02.170 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/?versions 2023-09-29T22:21:02.170 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko= 2023-09-29T22:21:02.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko=', 'Content-Length': '0'} 2023-09-29T22:21:02.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071c14defccc4144d-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.174 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:21:02.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:21:02.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:21:02.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:21:02.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:21:02.175 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.175 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.176 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.176 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.176 DEBUG:boto:Method: GET 2023-09-29T22:21:02.176 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.176 DEBUG:boto:Data: 2023-09-29T22:21:02.176 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.176 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.176 DEBUG:boto:Params: {} 2023-09-29T22:21:02.176 DEBUG:boto:Token: None 2023-09-29T22:21:02.177 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/?versions 2023-09-29T22:21:02.177 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko= 2023-09-29T22:21:02.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko=', 'Content-Length': '0'} 2023-09-29T22:21:02.180 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042efd0fa1143a185-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.181 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:21:02.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:21:02.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:21:02.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:21:02.182 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.182 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.182 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.182 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.182 DEBUG:boto:Method: GET 2023-09-29T22:21:02.182 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.182 DEBUG:boto:Data: 2023-09-29T22:21:02.182 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.183 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.183 DEBUG:boto:Params: {} 2023-09-29T22:21:02.183 DEBUG:boto:Token: None 2023-09-29T22:21:02.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/?versions 2023-09-29T22:21:02.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko= 2023-09-29T22:21:02.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko=', 'Content-Length': '0'} 2023-09-29T22:21:02.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000670ffbf5f88f6911-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.186 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.187 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.187 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.187 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.187 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.187 DEBUG:boto:Method: GET 2023-09-29T22:21:02.187 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:21:02.188 DEBUG:boto:Data: 2023-09-29T22:21:02.188 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.188 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.188 DEBUG:boto:Params: {} 2023-09-29T22:21:02.188 DEBUG:boto:Token: None 2023-09-29T22:21:02.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/?versions 2023-09-29T22:21:02.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko= 2023-09-29T22:21:02.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1o1upRwYVDfJ9AYHSUpfGfOOTko=', 'Content-Length': '0'} 2023-09-29T22:21:02.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005df6587d387f33c1-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.192 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:21:02.193 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.193 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.193 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.193 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.193 DEBUG:boto:Method: GET 2023-09-29T22:21:02.193 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.194 DEBUG:boto:Data: 2023-09-29T22:21:02.194 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.194 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.194 DEBUG:boto:Params: {} 2023-09-29T22:21:02.194 DEBUG:boto:Token: None 2023-09-29T22:21:02.194 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk= 2023-09-29T22:21:02.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk=', 'Content-Length': '0'} 2023-09-29T22:21:02.197 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000081499852083a4ddc-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.197 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.197 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.197 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.198 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.198 DEBUG:boto:Method: GET 2023-09-29T22:21:02.198 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.198 DEBUG:boto:Data: 2023-09-29T22:21:02.198 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.198 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.198 DEBUG:boto:Params: {} 2023-09-29T22:21:02.199 DEBUG:boto:Token: None 2023-09-29T22:21:02.199 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk= 2023-09-29T22:21:02.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk=', 'Content-Length': '0'} 2023-09-29T22:21:02.201 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce4ddcdafa894c68-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.242 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.243 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.243 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.243 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.243 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.243 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.243 DEBUG:boto:Data: 2023-09-29T22:21:02.243 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.244 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.244 DEBUG:boto:Params: {} 2023-09-29T22:21:02.244 DEBUG:boto:Token: None 2023-09-29T22:21:02.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8PIEYxqdcN74DXGnHObl+H2o9hs= 2023-09-29T22:21:02.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8PIEYxqdcN74DXGnHObl+H2o9hs=', 'Content-Length': '0'} 2023-09-29T22:21:02.247 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000021a546054c9b7240-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.247 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.247 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.247 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.247 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.248 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.248 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.248 DEBUG:boto:Data: 2023-09-29T22:21:02.248 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.248 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.248 DEBUG:boto:Params: {} 2023-09-29T22:21:02.248 DEBUG:boto:Token: None 2023-09-29T22:21:02.248 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8PIEYxqdcN74DXGnHObl+H2o9hs= 2023-09-29T22:21:02.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8PIEYxqdcN74DXGnHObl+H2o9hs=', 'Content-Length': '0'} 2023-09-29T22:21:02.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e5072126da0d197-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:21:02.252 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.252 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.252 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.252 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.252 DEBUG:boto:Method: GET 2023-09-29T22:21:02.252 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.252 DEBUG:boto:Data: 2023-09-29T22:21:02.252 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.253 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.253 DEBUG:boto:Params: {} 2023-09-29T22:21:02.253 DEBUG:boto:Token: None 2023-09-29T22:21:02.253 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.253 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk= 2023-09-29T22:21:02.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk=', 'Content-Length': '0'} 2023-09-29T22:21:02.256 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000052292fdd53678966-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.256 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.256 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.256 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.256 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.257 DEBUG:boto:Method: GET 2023-09-29T22:21:02.257 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.257 DEBUG:boto:Data: 2023-09-29T22:21:02.257 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.257 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.257 DEBUG:boto:Params: {} 2023-09-29T22:21:02.257 DEBUG:boto:Token: None 2023-09-29T22:21:02.257 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:21:02.258 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk= 2023-09-29T22:21:02.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mxhr3MJ5rhDgEq6yhpCC4AoNWfk=', 'Content-Length': '0'} 2023-09-29T22:21:02.261 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000534c016f78b5d4c4-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.261 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.261 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.261 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.261 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.261 DEBUG:boto:Method: GET 2023-09-29T22:21:02.262 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.262 DEBUG:boto:Data: 2023-09-29T22:21:02.262 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.262 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.262 DEBUG:boto:Params: {} 2023-09-29T22:21:02.262 DEBUG:boto:Token: None 2023-09-29T22:21:02.262 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1e36e124d140c81-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.314 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.315 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.315 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.315 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.315 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.316 DEBUG:boto:Method: GET 2023-09-29T22:21:02.316 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.316 DEBUG:boto:Data: 2023-09-29T22:21:02.316 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.316 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.316 DEBUG:boto:Params: {} 2023-09-29T22:21:02.316 DEBUG:boto:Token: None 2023-09-29T22:21:02.317 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000361d37f898171222-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.321 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.322 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.322 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.322 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.322 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:21:02.322 DEBUG:boto:Data: 2023-09-29T22:21:02.322 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.323 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.323 DEBUG:boto:Params: {} 2023-09-29T22:21:02.323 DEBUG:boto:Token: None 2023-09-29T22:21:02.323 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1 2023-09-29T22:21:02.323 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wdn7E/7qmwM8mYJd6Fy+3Hwsva0= 2023-09-29T22:21:02.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wdn7E/7qmwM8mYJd6Fy+3Hwsva0=', 'Content-Length': '0'} 2023-09-29T22:21:02.326 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000048f917a10f1988fc-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.326 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.326 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.326 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.326 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:21:02.327 DEBUG:boto:Data: 2023-09-29T22:21:02.327 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.327 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.327 DEBUG:boto:Params: {} 2023-09-29T22:21:02.327 DEBUG:boto:Token: None 2023-09-29T22:21:02.327 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1 2023-09-29T22:21:02.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wdn7E/7qmwM8mYJd6Fy+3Hwsva0= 2023-09-29T22:21:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wdn7E/7qmwM8mYJd6Fy+3Hwsva0=', 'Content-Length': '0'} 2023-09-29T22:21:02.330 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026de59fa8451dd13-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:21:02.330 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.330 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.331 DEBUG:boto:Method: GET 2023-09-29T22:21:02.331 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:21:02.331 DEBUG:boto:Data: 2023-09-29T22:21:02.331 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.331 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.331 DEBUG:boto:Params: {} 2023-09-29T22:21:02.331 DEBUG:boto:Token: None 2023-09-29T22:21:02.331 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1 2023-09-29T22:21:02.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yCpIXf3OGD10S6WXaSLHohXenS0= 2023-09-29T22:21:02.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCpIXf3OGD10S6WXaSLHohXenS0=', 'Content-Length': '0'} 2023-09-29T22:21:02.334 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000335a86d62b2b33d4-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.334 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:21:02.335 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:21:02.335 DEBUG:boto:Method: GET 2023-09-29T22:21:02.335 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:21:02.335 DEBUG:boto:Data: 2023-09-29T22:21:02.335 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.335 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.335 DEBUG:boto:Params: {} 2023-09-29T22:21:02.335 DEBUG:boto:Token: None 2023-09-29T22:21:02.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj1 2023-09-29T22:21:02.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yCpIXf3OGD10S6WXaSLHohXenS0= 2023-09-29T22:21:02.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCpIXf3OGD10S6WXaSLHohXenS0=', 'Content-Length': '0'} 2023-09-29T22:21:02.339 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006badd02b11c5e41f-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.382 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.383 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.383 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.383 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.383 DEBUG:boto:Method: GET 2023-09-29T22:21:02.383 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.383 DEBUG:boto:Data: 2023-09-29T22:21:02.383 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.383 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.383 DEBUG:boto:Params: {} 2023-09-29T22:21:02.384 DEBUG:boto:Token: None 2023-09-29T22:21:02.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b3813c290529b4a-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.389 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.389 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.389 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.389 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.390 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.390 DEBUG:boto:Method: GET 2023-09-29T22:21:02.390 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:21:02.390 DEBUG:boto:Data: 2023-09-29T22:21:02.390 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.390 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.390 DEBUG:boto:Params: {} 2023-09-29T22:21:02.390 DEBUG:boto:Token: None 2023-09-29T22:21:02.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000108e26110692068b-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.395 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:21:02.395 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.395 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.396 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.396 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.396 DEBUG:boto:Method: GET 2023-09-29T22:21:02.396 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.396 DEBUG:boto:Data: 2023-09-29T22:21:02.396 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.396 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.396 DEBUG:boto:Params: {} 2023-09-29T22:21:02.396 DEBUG:boto:Token: None 2023-09-29T22:21:02.397 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE= 2023-09-29T22:21:02.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE=', 'Content-Length': '0'} 2023-09-29T22:21:02.399 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a1d9c864ba820dc1-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.399 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.400 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.400 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.400 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.400 DEBUG:boto:Method: GET 2023-09-29T22:21:02.400 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.400 DEBUG:boto:Data: 2023-09-29T22:21:02.400 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.400 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.400 DEBUG:boto:Params: {} 2023-09-29T22:21:02.401 DEBUG:boto:Token: None 2023-09-29T22:21:02.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE= 2023-09-29T22:21:02.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE=', 'Content-Length': '0'} 2023-09-29T22:21:02.403 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000842d81227e3e4e98-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.404 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.404 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.404 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.404 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.404 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.404 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.404 DEBUG:boto:Data: 2023-09-29T22:21:02.404 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.404 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.405 DEBUG:boto:Params: {} 2023-09-29T22:21:02.405 DEBUG:boto:Token: None 2023-09-29T22:21:02.405 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G8Su/e7FnXUDRJg92yXI7XHEt+Y= 2023-09-29T22:21:02.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G8Su/e7FnXUDRJg92yXI7XHEt+Y=', 'Content-Length': '0'} 2023-09-29T22:21:02.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003f0f81d0a9ea3d65-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.407 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.407 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.408 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.408 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.408 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.408 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.408 DEBUG:boto:Data: 2023-09-29T22:21:02.408 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.408 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.408 DEBUG:boto:Params: {} 2023-09-29T22:21:02.408 DEBUG:boto:Token: None 2023-09-29T22:21:02.409 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G8Su/e7FnXUDRJg92yXI7XHEt+Y= 2023-09-29T22:21:02.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G8Su/e7FnXUDRJg92yXI7XHEt+Y=', 'Content-Length': '0'} 2023-09-29T22:21:02.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3ca2e5cee2dc37d-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:21:02.411 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.412 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.412 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.412 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.412 DEBUG:boto:Method: GET 2023-09-29T22:21:02.412 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.412 DEBUG:boto:Data: 2023-09-29T22:21:02.412 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.412 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.412 DEBUG:boto:Params: {} 2023-09-29T22:21:02.413 DEBUG:boto:Token: None 2023-09-29T22:21:02.413 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE= 2023-09-29T22:21:02.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE=', 'Content-Length': '0'} 2023-09-29T22:21:02.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000074c163f1360d3850-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.458 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.458 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.459 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.459 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.459 DEBUG:boto:Method: GET 2023-09-29T22:21:02.459 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.459 DEBUG:boto:Data: 2023-09-29T22:21:02.459 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.459 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.459 DEBUG:boto:Params: {} 2023-09-29T22:21:02.460 DEBUG:boto:Token: None 2023-09-29T22:21:02.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:21:02.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE= 2023-09-29T22:21:02.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CY429RYvXbp139cIHR4QlqbguoE=', 'Content-Length': '0'} 2023-09-29T22:21:02.463 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008e98c6a965d4874e-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.464 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.464 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.464 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.464 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.464 DEBUG:boto:Method: GET 2023-09-29T22:21:02.464 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.464 DEBUG:boto:Data: 2023-09-29T22:21:02.464 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.465 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.465 DEBUG:boto:Params: {} 2023-09-29T22:21:02.465 DEBUG:boto:Token: None 2023-09-29T22:21:02.465 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca6144ed46669395-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.470 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.470 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.470 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.470 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.470 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.470 DEBUG:boto:Method: GET 2023-09-29T22:21:02.471 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.471 DEBUG:boto:Data: 2023-09-29T22:21:02.471 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.471 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.471 DEBUG:boto:Params: {} 2023-09-29T22:21:02.471 DEBUG:boto:Token: None 2023-09-29T22:21:02.471 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6e13b60fe818bff-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.476 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.476 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.477 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.477 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.477 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:21:02.477 DEBUG:boto:Data: 2023-09-29T22:21:02.477 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.477 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.477 DEBUG:boto:Params: {} 2023-09-29T22:21:02.477 DEBUG:boto:Token: None 2023-09-29T22:21:02.478 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2 2023-09-29T22:21:02.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mv1Qg2TRxfvRUmfbHjOy0jFianE= 2023-09-29T22:21:02.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mv1Qg2TRxfvRUmfbHjOy0jFianE=', 'Content-Length': '0'} 2023-09-29T22:21:02.480 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3387e4636f558e7-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.480 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.480 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.481 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.481 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:21:02.481 DEBUG:boto:Data: 2023-09-29T22:21:02.481 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.481 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.481 DEBUG:boto:Params: {} 2023-09-29T22:21:02.481 DEBUG:boto:Token: None 2023-09-29T22:21:02.482 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2 2023-09-29T22:21:02.482 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mv1Qg2TRxfvRUmfbHjOy0jFianE= 2023-09-29T22:21:02.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mv1Qg2TRxfvRUmfbHjOy0jFianE=', 'Content-Length': '0'} 2023-09-29T22:21:02.484 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000056413d79515250b1-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:21:02.484 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.484 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.484 DEBUG:boto:Method: GET 2023-09-29T22:21:02.485 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:21:02.485 DEBUG:boto:Data: 2023-09-29T22:21:02.485 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.485 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.485 DEBUG:boto:Params: {} 2023-09-29T22:21:02.485 DEBUG:boto:Token: None 2023-09-29T22:21:02.485 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2 2023-09-29T22:21:02.486 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PTpKlozdbCk/I/NHwdpE5Va3myw= 2023-09-29T22:21:02.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PTpKlozdbCk/I/NHwdpE5Va3myw=', 'Content-Length': '0'} 2023-09-29T22:21:02.488 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009f539034d0f53d4a-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.488 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:21:02.488 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:21:02.489 DEBUG:boto:Method: GET 2023-09-29T22:21:02.489 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:21:02.489 DEBUG:boto:Data: 2023-09-29T22:21:02.489 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.489 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.489 DEBUG:boto:Params: {} 2023-09-29T22:21:02.489 DEBUG:boto:Token: None 2023-09-29T22:21:02.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj2 2023-09-29T22:21:02.490 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PTpKlozdbCk/I/NHwdpE5Va3myw= 2023-09-29T22:21:02.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PTpKlozdbCk/I/NHwdpE5Va3myw=', 'Content-Length': '0'} 2023-09-29T22:21:02.492 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000096d20971138fbf7-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.492 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.493 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.493 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.493 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.493 DEBUG:boto:Method: GET 2023-09-29T22:21:02.493 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.493 DEBUG:boto:Data: 2023-09-29T22:21:02.493 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.493 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.494 DEBUG:boto:Params: {} 2023-09-29T22:21:02.494 DEBUG:boto:Token: None 2023-09-29T22:21:02.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.494 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc94f1aa7abb8d88-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.498 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.498 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.498 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.499 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.499 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.499 DEBUG:boto:Method: GET 2023-09-29T22:21:02.499 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:21:02.499 DEBUG:boto:Data: 2023-09-29T22:21:02.499 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.499 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.499 DEBUG:boto:Params: {} 2023-09-29T22:21:02.500 DEBUG:boto:Token: None 2023-09-29T22:21:02.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069b89045012eb7a1-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.504 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:21:02.504 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.504 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.504 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.504 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.505 DEBUG:boto:Method: GET 2023-09-29T22:21:02.505 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.505 DEBUG:boto:Data: 2023-09-29T22:21:02.505 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.505 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.505 DEBUG:boto:Params: {} 2023-09-29T22:21:02.505 DEBUG:boto:Token: None 2023-09-29T22:21:02.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA= 2023-09-29T22:21:02.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA=', 'Content-Length': '0'} 2023-09-29T22:21:02.508 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c881849c3d9b5c-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.508 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.508 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.508 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.508 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.509 DEBUG:boto:Method: GET 2023-09-29T22:21:02.509 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.509 DEBUG:boto:Data: 2023-09-29T22:21:02.509 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.509 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.509 DEBUG:boto:Params: {} 2023-09-29T22:21:02.509 DEBUG:boto:Token: None 2023-09-29T22:21:02.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.510 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA= 2023-09-29T22:21:02.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA=', 'Content-Length': '0'} 2023-09-29T22:21:02.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e5c6c8dd2442e4b5-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.554 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.554 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.555 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.555 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.555 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.555 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.555 DEBUG:boto:Data: 2023-09-29T22:21:02.555 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.555 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.555 DEBUG:boto:Params: {} 2023-09-29T22:21:02.556 DEBUG:boto:Token: None 2023-09-29T22:21:02.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bcBVuMQLsuAte6mqm+5V5C2xsmw= 2023-09-29T22:21:02.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bcBVuMQLsuAte6mqm+5V5C2xsmw=', 'Content-Length': '0'} 2023-09-29T22:21:02.558 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c4c8ab4591a96ced-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.559 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.559 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.559 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.559 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.559 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.559 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.559 DEBUG:boto:Data: 2023-09-29T22:21:02.559 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.559 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.560 DEBUG:boto:Params: {} 2023-09-29T22:21:02.560 DEBUG:boto:Token: None 2023-09-29T22:21:02.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bcBVuMQLsuAte6mqm+5V5C2xsmw= 2023-09-29T22:21:02.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bcBVuMQLsuAte6mqm+5V5C2xsmw=', 'Content-Length': '0'} 2023-09-29T22:21:02.562 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006fbb3b191b5e9b13-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:21:02.563 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.563 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.563 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.563 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.563 DEBUG:boto:Method: GET 2023-09-29T22:21:02.563 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.564 DEBUG:boto:Data: 2023-09-29T22:21:02.564 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.564 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.564 DEBUG:boto:Params: {} 2023-09-29T22:21:02.564 DEBUG:boto:Token: None 2023-09-29T22:21:02.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA= 2023-09-29T22:21:02.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA=', 'Content-Length': '0'} 2023-09-29T22:21:02.567 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002dadf73f39cf675c-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.610 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.610 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.611 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.611 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.611 DEBUG:boto:Method: GET 2023-09-29T22:21:02.611 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.611 DEBUG:boto:Data: 2023-09-29T22:21:02.611 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.611 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.611 DEBUG:boto:Params: {} 2023-09-29T22:21:02.611 DEBUG:boto:Token: None 2023-09-29T22:21:02.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:21:02.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA= 2023-09-29T22:21:02.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H3i5lO46lcuY4GRDsg+oPVDQJWA=', 'Content-Length': '0'} 2023-09-29T22:21:02.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063b275222387f055-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.615 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.615 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.615 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.615 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.616 DEBUG:boto:Method: GET 2023-09-29T22:21:02.616 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.616 DEBUG:boto:Data: 2023-09-29T22:21:02.616 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.616 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.616 DEBUG:boto:Params: {} 2023-09-29T22:21:02.616 DEBUG:boto:Token: None 2023-09-29T22:21:02.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bb7ad3f20c00159-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.621 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.622 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.622 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.622 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.622 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.622 DEBUG:boto:Method: GET 2023-09-29T22:21:02.623 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.623 DEBUG:boto:Data: 2023-09-29T22:21:02.623 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.623 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.623 DEBUG:boto:Params: {} 2023-09-29T22:21:02.623 DEBUG:boto:Token: None 2023-09-29T22:21:02.623 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e3b31d907a229d-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.628 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.628 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.629 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.629 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.629 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:21:02.629 DEBUG:boto:Data: 2023-09-29T22:21:02.629 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.629 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.629 DEBUG:boto:Params: {} 2023-09-29T22:21:02.629 DEBUG:boto:Token: None 2023-09-29T22:21:02.630 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3 2023-09-29T22:21:02.630 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wKDPAXMOk2LS1rfJtMgkkY/Fz00= 2023-09-29T22:21:02.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wKDPAXMOk2LS1rfJtMgkkY/Fz00=', 'Content-Length': '0'} 2023-09-29T22:21:02.632 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c80af510ee95e798-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.632 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.633 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.633 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.633 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:21:02.633 DEBUG:boto:Data: 2023-09-29T22:21:02.633 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.633 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.633 DEBUG:boto:Params: {} 2023-09-29T22:21:02.633 DEBUG:boto:Token: None 2023-09-29T22:21:02.634 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3 2023-09-29T22:21:02.634 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wKDPAXMOk2LS1rfJtMgkkY/Fz00= 2023-09-29T22:21:02.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wKDPAXMOk2LS1rfJtMgkkY/Fz00=', 'Content-Length': '0'} 2023-09-29T22:21:02.636 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d985506a18b2ec3f-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:21:02.637 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.637 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.637 DEBUG:boto:Method: GET 2023-09-29T22:21:02.637 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:21:02.637 DEBUG:boto:Data: 2023-09-29T22:21:02.637 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.637 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.637 DEBUG:boto:Params: {} 2023-09-29T22:21:02.638 DEBUG:boto:Token: None 2023-09-29T22:21:02.638 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3 2023-09-29T22:21:02.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eidzqBjWsYYf1bpZ+TRr2irCf1o= 2023-09-29T22:21:02.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eidzqBjWsYYf1bpZ+TRr2irCf1o=', 'Content-Length': '0'} 2023-09-29T22:21:02.640 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000053d7a9b3982f44fb-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.641 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:21:02.641 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:21:02.641 DEBUG:boto:Method: GET 2023-09-29T22:21:02.641 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:21:02.641 DEBUG:boto:Data: 2023-09-29T22:21:02.641 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.641 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.642 DEBUG:boto:Params: {} 2023-09-29T22:21:02.642 DEBUG:boto:Token: None 2023-09-29T22:21:02.642 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj3 2023-09-29T22:21:02.642 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eidzqBjWsYYf1bpZ+TRr2irCf1o= 2023-09-29T22:21:02.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eidzqBjWsYYf1bpZ+TRr2irCf1o=', 'Content-Length': '0'} 2023-09-29T22:21:02.645 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000469e84e187d31cb7-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.686 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.687 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.687 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.687 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.687 DEBUG:boto:Method: GET 2023-09-29T22:21:02.687 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.687 DEBUG:boto:Data: 2023-09-29T22:21:02.687 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.687 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.687 DEBUG:boto:Params: {} 2023-09-29T22:21:02.688 DEBUG:boto:Token: None 2023-09-29T22:21:02.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.694 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006348692840bf8ba4-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.694 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.694 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.695 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.695 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.695 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.695 DEBUG:boto:Method: GET 2023-09-29T22:21:02.695 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:21:02.695 DEBUG:boto:Data: 2023-09-29T22:21:02.695 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.695 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.695 DEBUG:boto:Params: {} 2023-09-29T22:21:02.696 DEBUG:boto:Token: None 2023-09-29T22:21:02.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005352f331086dbb3a-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.700 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:21:02.701 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.701 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.701 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.701 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.701 DEBUG:boto:Method: GET 2023-09-29T22:21:02.701 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.701 DEBUG:boto:Data: 2023-09-29T22:21:02.702 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.702 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.702 DEBUG:boto:Params: {} 2023-09-29T22:21:02.702 DEBUG:boto:Token: None 2023-09-29T22:21:02.702 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc= 2023-09-29T22:21:02.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc=', 'Content-Length': '0'} 2023-09-29T22:21:02.704 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d06283a525dae2ee-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.746 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.746 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.747 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.747 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.747 DEBUG:boto:Method: GET 2023-09-29T22:21:02.747 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.747 DEBUG:boto:Data: 2023-09-29T22:21:02.747 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.747 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.747 DEBUG:boto:Params: {} 2023-09-29T22:21:02.747 DEBUG:boto:Token: None 2023-09-29T22:21:02.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc= 2023-09-29T22:21:02.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc=', 'Content-Length': '0'} 2023-09-29T22:21:02.751 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003edf727b842d0dfc-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.751 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.751 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.751 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.751 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.752 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.752 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.752 DEBUG:boto:Data: 2023-09-29T22:21:02.752 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.752 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.752 DEBUG:boto:Params: {} 2023-09-29T22:21:02.752 DEBUG:boto:Token: None 2023-09-29T22:21:02.752 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3kzDncEIsebsC6NaG9JzkGxOU1k= 2023-09-29T22:21:02.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3kzDncEIsebsC6NaG9JzkGxOU1k=', 'Content-Length': '0'} 2023-09-29T22:21:02.755 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c6fdc89b6311f2e-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.755 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.756 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.756 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.756 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.756 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.756 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.756 DEBUG:boto:Data: 2023-09-29T22:21:02.756 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.756 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.757 DEBUG:boto:Params: {} 2023-09-29T22:21:02.757 DEBUG:boto:Token: None 2023-09-29T22:21:02.757 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.757 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3kzDncEIsebsC6NaG9JzkGxOU1k= 2023-09-29T22:21:02.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3kzDncEIsebsC6NaG9JzkGxOU1k=', 'Content-Length': '0'} 2023-09-29T22:21:02.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000524df5e9af60ed03-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:21:02.760 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.761 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.761 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.761 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.761 DEBUG:boto:Method: GET 2023-09-29T22:21:02.761 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.761 DEBUG:boto:Data: 2023-09-29T22:21:02.761 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.761 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.761 DEBUG:boto:Params: {} 2023-09-29T22:21:02.762 DEBUG:boto:Token: None 2023-09-29T22:21:02.762 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.762 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc= 2023-09-29T22:21:02.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc=', 'Content-Length': '0'} 2023-09-29T22:21:02.765 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c207dadb1b46abd1-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.765 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.765 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.765 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.765 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.765 DEBUG:boto:Method: GET 2023-09-29T22:21:02.765 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.766 DEBUG:boto:Data: 2023-09-29T22:21:02.766 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.766 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.766 DEBUG:boto:Params: {} 2023-09-29T22:21:02.766 DEBUG:boto:Token: None 2023-09-29T22:21:02.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:21:02.766 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc= 2023-09-29T22:21:02.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5YPELzAtG31dg9GpXpQUXl+DAzc=', 'Content-Length': '0'} 2023-09-29T22:21:02.769 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ec2e2e626719945e-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.810 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.811 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.811 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.811 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.811 DEBUG:boto:Method: GET 2023-09-29T22:21:02.811 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.811 DEBUG:boto:Data: 2023-09-29T22:21:02.811 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.812 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.812 DEBUG:boto:Params: {} 2023-09-29T22:21:02.812 DEBUG:boto:Token: None 2023-09-29T22:21:02.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.816 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cef022c6b352ec1-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.817 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.817 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.817 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.818 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.818 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.818 DEBUG:boto:Method: GET 2023-09-29T22:21:02.818 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.818 DEBUG:boto:Data: 2023-09-29T22:21:02.818 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.818 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.818 DEBUG:boto:Params: {} 2023-09-29T22:21:02.818 DEBUG:boto:Token: None 2023-09-29T22:21:02.819 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.819 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b98f20f661407647-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.823 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.823 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.823 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.824 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.824 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:21:02.824 DEBUG:boto:Data: 2023-09-29T22:21:02.824 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.824 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.824 DEBUG:boto:Params: {} 2023-09-29T22:21:02.824 DEBUG:boto:Token: None 2023-09-29T22:21:02.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4 2023-09-29T22:21:02.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hjlwopcjG2mTwisSiX69iLAVNDw= 2023-09-29T22:21:02.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hjlwopcjG2mTwisSiX69iLAVNDw=', 'Content-Length': '0'} 2023-09-29T22:21:02.827 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b227ab054bfe0405-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.827 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.827 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.827 DEBUG:boto:Method: HEAD 2023-09-29T22:21:02.827 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:21:02.827 DEBUG:boto:Data: 2023-09-29T22:21:02.827 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.828 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.828 DEBUG:boto:Params: {} 2023-09-29T22:21:02.828 DEBUG:boto:Token: None 2023-09-29T22:21:02.828 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4 2023-09-29T22:21:02.828 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hjlwopcjG2mTwisSiX69iLAVNDw= 2023-09-29T22:21:02.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hjlwopcjG2mTwisSiX69iLAVNDw=', 'Content-Length': '0'} 2023-09-29T22:21:02.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002ca74b7683fa24da-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:21:02.831 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.831 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.831 DEBUG:boto:Method: GET 2023-09-29T22:21:02.831 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:21:02.831 DEBUG:boto:Data: 2023-09-29T22:21:02.831 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.832 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.832 DEBUG:boto:Params: {} 2023-09-29T22:21:02.832 DEBUG:boto:Token: None 2023-09-29T22:21:02.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4 2023-09-29T22:21:02.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XH7klzv3SESHP+GGps72QEsYI0M= 2023-09-29T22:21:02.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XH7klzv3SESHP+GGps72QEsYI0M=', 'Content-Length': '0'} 2023-09-29T22:21:02.834 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2f79555ccffa628-0065174dce-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.878 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:21:02.879 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:21:02.879 DEBUG:boto:Method: GET 2023-09-29T22:21:02.879 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:21:02.879 DEBUG:boto:Data: 2023-09-29T22:21:02.879 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.879 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.879 DEBUG:boto:Params: {} 2023-09-29T22:21:02.879 DEBUG:boto:Token: None 2023-09-29T22:21:02.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/obj4 2023-09-29T22:21:02.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XH7klzv3SESHP+GGps72QEsYI0M= 2023-09-29T22:21:02.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XH7klzv3SESHP+GGps72QEsYI0M=', 'Content-Length': '0'} 2023-09-29T22:21:02.884 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f069e2d1adccc31-0065174dce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.884 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.884 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.884 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.885 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.885 DEBUG:boto:Method: GET 2023-09-29T22:21:02.885 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.885 DEBUG:boto:Data: 2023-09-29T22:21:02.885 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:02.885 DEBUG:boto:Port: 8001 2023-09-29T22:21:02.885 DEBUG:boto:Params: {} 2023-09-29T22:21:02.885 DEBUG:boto:Token: None 2023-09-29T22:21:02.886 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e812da1dbb13e535-0065174dce-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.891 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.891 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:02.891 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:02.891 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.891 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.891 DEBUG:boto:Method: GET 2023-09-29T22:21:02.892 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:21:02.892 DEBUG:boto:Data: 2023-09-29T22:21:02.892 DEBUG:boto:Headers: {} 2023-09-29T22:21:02.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:02.892 DEBUG:boto:Port: 8000 2023-09-29T22:21:02.892 DEBUG:boto:Params: {} 2023-09-29T22:21:02.892 DEBUG:boto:Token: None 2023-09-29T22:21:02.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:02 GMT /hcnphz-48/ 2023-09-29T22:21:02.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo= 2023-09-29T22:21:02.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zSPd7kVLsltWdKneuRkK1eh1xPo=', 'Content-Length': '0'} 2023-09-29T22:21:02.896 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e5dcfe988839ae9-0065174dce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:02.938 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:02.939 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-48 zones={a2, a1} 2023-09-29T22:21:02.939 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-49 2023-09-29T22:21:02.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:21:02.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:21:02.940 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:21:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:02.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:03.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:03.057+0000 7f8aa6433f40 1 bucket sync caught up with source: 2023-09-29T22:21:03.062 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.27.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-09-29T22:21:03.062 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.27.6, , 00000000001.22.6, , 00000000001.14.6, , , 00000000001.23.6, , , ] 2023-09-29T22:21:03.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:03.057+0000 7f8aa6433f40 0 bucket checkpoint complete 2023-09-29T22:21:03.088 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-49 2023-09-29T22:21:03.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:21:03.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:21:03.089 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:21:03.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:03.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:03.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:03.201+0000 7fd834433f40 1 bucket sync caught up with source: 2023-09-29T22:21:03.205 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.229.6, , 00000000001.176.6, , 00000000001.29.6, , , 00000000001.56.6, , , ] 2023-09-29T22:21:03.205 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.229.6, , 00000000001.176.6, , 00000000001.29.6, , , 00000000001.56.6, , , ] 2023-09-29T22:21:03.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:03.201+0000 7fd834433f40 0 bucket checkpoint complete 2023-09-29T22:21:03.231 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-49 zones={a2, a1} 2023-09-29T22:21:03.231 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.231 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.232 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.232 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:21:03.232 DEBUG:boto:Data: 2023-09-29T22:21:03.232 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.232 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.232 DEBUG:boto:Params: {} 2023-09-29T22:21:03.232 DEBUG:boto:Token: None 2023-09-29T22:21:03.232 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4bciWjwEN3ZMJEdwnaj4PXFEKPg= 2023-09-29T22:21:03.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4bciWjwEN3ZMJEdwnaj4PXFEKPg=', 'Content-Length': '0'} 2023-09-29T22:21:03.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d38542735589b957-0065174dcf-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.236 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.237 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.237 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.237 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:21:03.237 DEBUG:boto:Data: 2023-09-29T22:21:03.237 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.237 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.237 DEBUG:boto:Params: {} 2023-09-29T22:21:03.237 DEBUG:boto:Token: None 2023-09-29T22:21:03.238 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4bciWjwEN3ZMJEdwnaj4PXFEKPg= 2023-09-29T22:21:03.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4bciWjwEN3ZMJEdwnaj4PXFEKPg=', 'Content-Length': '0'} 2023-09-29T22:21:03.242 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000054a52fd2ff3a2969-0065174dcf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.242 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:21:03.242 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.242 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.242 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.242 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.242 DEBUG:boto:Method: GET 2023-09-29T22:21:03.243 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.243 DEBUG:boto:Data: 2023-09-29T22:21:03.243 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.243 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.243 DEBUG:boto:Params: {} 2023-09-29T22:21:03.243 DEBUG:boto:Token: None 2023-09-29T22:21:03.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/?versions 2023-09-29T22:21:03.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw= 2023-09-29T22:21:03.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw=', 'Content-Length': '0'} 2023-09-29T22:21:03.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000971e2ade1eaa64b8-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.247 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:21:03.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:21:03.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:21:03.248 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:21:03.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:21:03.248 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.248 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.249 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.249 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.249 DEBUG:boto:Method: GET 2023-09-29T22:21:03.249 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.249 DEBUG:boto:Data: 2023-09-29T22:21:03.249 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.249 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.249 DEBUG:boto:Params: {} 2023-09-29T22:21:03.249 DEBUG:boto:Token: None 2023-09-29T22:21:03.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/?versions 2023-09-29T22:21:03.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw= 2023-09-29T22:21:03.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw=', 'Content-Length': '0'} 2023-09-29T22:21:03.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2077f1b4c399ad8-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.254 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:21:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:21:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:21:03.255 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:21:03.255 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.255 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.255 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.256 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.256 DEBUG:boto:Method: GET 2023-09-29T22:21:03.256 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.256 DEBUG:boto:Data: 2023-09-29T22:21:03.256 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.256 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.256 DEBUG:boto:Params: {} 2023-09-29T22:21:03.256 DEBUG:boto:Token: None 2023-09-29T22:21:03.257 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/?versions 2023-09-29T22:21:03.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw= 2023-09-29T22:21:03.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw=', 'Content-Length': '0'} 2023-09-29T22:21:03.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d19c7b9faac119-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.260 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.261 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.261 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.261 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.261 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.261 DEBUG:boto:Method: GET 2023-09-29T22:21:03.261 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:21:03.261 DEBUG:boto:Data: 2023-09-29T22:21:03.262 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.262 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.262 DEBUG:boto:Params: {} 2023-09-29T22:21:03.262 DEBUG:boto:Token: None 2023-09-29T22:21:03.262 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/?versions 2023-09-29T22:21:03.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw= 2023-09-29T22:21:03.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Z2eXrU+oD882mfWK6u49/25Yhw=', 'Content-Length': '0'} 2023-09-29T22:21:03.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2373fa9a98ed713-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.266 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:21:03.267 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.267 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.267 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.267 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.267 DEBUG:boto:Method: GET 2023-09-29T22:21:03.267 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.267 DEBUG:boto:Data: 2023-09-29T22:21:03.267 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.267 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.268 DEBUG:boto:Params: {} 2023-09-29T22:21:03.268 DEBUG:boto:Token: None 2023-09-29T22:21:03.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY= 2023-09-29T22:21:03.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY=', 'Content-Length': '0'} 2023-09-29T22:21:03.271 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d012f91ddaf9c297-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.314 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.315 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.315 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.315 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.315 DEBUG:boto:Method: GET 2023-09-29T22:21:03.315 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.315 DEBUG:boto:Data: 2023-09-29T22:21:03.315 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.315 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.316 DEBUG:boto:Params: {} 2023-09-29T22:21:03.316 DEBUG:boto:Token: None 2023-09-29T22:21:03.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY= 2023-09-29T22:21:03.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY=', 'Content-Length': '0'} 2023-09-29T22:21:03.319 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f7e88d6b299ab400-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.319 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.320 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.320 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.320 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.320 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.320 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.320 DEBUG:boto:Data: 2023-09-29T22:21:03.320 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.320 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.321 DEBUG:boto:Params: {} 2023-09-29T22:21:03.321 DEBUG:boto:Token: None 2023-09-29T22:21:03.321 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UP5ZqlE0PbTCWr3A2ot5oE6gCZQ= 2023-09-29T22:21:03.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UP5ZqlE0PbTCWr3A2ot5oE6gCZQ=', 'Content-Length': '0'} 2023-09-29T22:21:03.324 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003014f3531e92839b-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.324 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.324 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.324 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.324 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.324 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.324 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.325 DEBUG:boto:Data: 2023-09-29T22:21:03.325 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.325 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.325 DEBUG:boto:Params: {} 2023-09-29T22:21:03.325 DEBUG:boto:Token: None 2023-09-29T22:21:03.325 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UP5ZqlE0PbTCWr3A2ot5oE6gCZQ= 2023-09-29T22:21:03.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UP5ZqlE0PbTCWr3A2ot5oE6gCZQ=', 'Content-Length': '0'} 2023-09-29T22:21:03.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e901ef6d5821bde-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:21:03.328 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.328 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.329 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.329 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.329 DEBUG:boto:Method: GET 2023-09-29T22:21:03.329 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.329 DEBUG:boto:Data: 2023-09-29T22:21:03.329 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.329 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.329 DEBUG:boto:Params: {} 2023-09-29T22:21:03.329 DEBUG:boto:Token: None 2023-09-29T22:21:03.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY= 2023-09-29T22:21:03.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY=', 'Content-Length': '0'} 2023-09-29T22:21:03.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037dfa89619ec39f7-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.333 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.333 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.333 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.333 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.334 DEBUG:boto:Method: GET 2023-09-29T22:21:03.334 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.334 DEBUG:boto:Data: 2023-09-29T22:21:03.334 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.334 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.334 DEBUG:boto:Params: {} 2023-09-29T22:21:03.334 DEBUG:boto:Token: None 2023-09-29T22:21:03.335 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:21:03.335 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY= 2023-09-29T22:21:03.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C8fvOHRHb/o3Dx/KLk+WDIMhfCY=', 'Content-Length': '0'} 2023-09-29T22:21:03.338 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007b066b5458c90a5c-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.378 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.378 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.379 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.379 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.379 DEBUG:boto:Method: GET 2023-09-29T22:21:03.379 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.379 DEBUG:boto:Data: 2023-09-29T22:21:03.379 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.379 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.379 DEBUG:boto:Params: {} 2023-09-29T22:21:03.380 DEBUG:boto:Token: None 2023-09-29T22:21:03.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be9ec775d9f9dad3-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.385 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.386 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.386 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.386 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.386 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.386 DEBUG:boto:Method: GET 2023-09-29T22:21:03.386 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.386 DEBUG:boto:Data: 2023-09-29T22:21:03.387 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.387 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.387 DEBUG:boto:Params: {} 2023-09-29T22:21:03.387 DEBUG:boto:Token: None 2023-09-29T22:21:03.387 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.391 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000840325c745401379-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.392 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.392 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.392 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.392 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.392 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:21:03.392 DEBUG:boto:Data: 2023-09-29T22:21:03.393 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.393 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.393 DEBUG:boto:Params: {} 2023-09-29T22:21:03.393 DEBUG:boto:Token: None 2023-09-29T22:21:03.393 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1 2023-09-29T22:21:03.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n54wkPB4WNWBkrlOI89xl6hye38= 2023-09-29T22:21:03.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n54wkPB4WNWBkrlOI89xl6hye38=', 'Content-Length': '0'} 2023-09-29T22:21:03.396 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1a00a73bda02b83-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.396 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.396 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.396 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.396 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:21:03.397 DEBUG:boto:Data: 2023-09-29T22:21:03.397 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.397 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.397 DEBUG:boto:Params: {} 2023-09-29T22:21:03.397 DEBUG:boto:Token: None 2023-09-29T22:21:03.397 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1 2023-09-29T22:21:03.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n54wkPB4WNWBkrlOI89xl6hye38= 2023-09-29T22:21:03.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n54wkPB4WNWBkrlOI89xl6hye38=', 'Content-Length': '0'} 2023-09-29T22:21:03.400 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1debccfe504ce5b-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:21:03.400 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.400 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.401 DEBUG:boto:Method: GET 2023-09-29T22:21:03.401 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:21:03.401 DEBUG:boto:Data: 2023-09-29T22:21:03.401 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.401 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.401 DEBUG:boto:Params: {} 2023-09-29T22:21:03.401 DEBUG:boto:Token: None 2023-09-29T22:21:03.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1 2023-09-29T22:21:03.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eipsLUSl94nfXBoRuaB8uyPLcdM= 2023-09-29T22:21:03.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eipsLUSl94nfXBoRuaB8uyPLcdM=', 'Content-Length': '0'} 2023-09-29T22:21:03.404 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000095012b9f2bab5d57-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.446 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:21:03.447 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:21:03.447 DEBUG:boto:Method: GET 2023-09-29T22:21:03.447 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:21:03.447 DEBUG:boto:Data: 2023-09-29T22:21:03.447 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.447 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.447 DEBUG:boto:Params: {} 2023-09-29T22:21:03.448 DEBUG:boto:Token: None 2023-09-29T22:21:03.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj1 2023-09-29T22:21:03.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eipsLUSl94nfXBoRuaB8uyPLcdM= 2023-09-29T22:21:03.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eipsLUSl94nfXBoRuaB8uyPLcdM=', 'Content-Length': '0'} 2023-09-29T22:21:03.451 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000effd7cbf961fd628-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.451 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.451 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.451 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.451 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.451 DEBUG:boto:Method: GET 2023-09-29T22:21:03.452 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.452 DEBUG:boto:Data: 2023-09-29T22:21:03.452 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.452 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.452 DEBUG:boto:Params: {} 2023-09-29T22:21:03.452 DEBUG:boto:Token: None 2023-09-29T22:21:03.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006335f5a5ca4e6058-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.457 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.457 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.458 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.458 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.458 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.458 DEBUG:boto:Method: GET 2023-09-29T22:21:03.458 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:21:03.458 DEBUG:boto:Data: 2023-09-29T22:21:03.458 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.458 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.459 DEBUG:boto:Params: {} 2023-09-29T22:21:03.459 DEBUG:boto:Token: None 2023-09-29T22:21:03.459 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.463 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aed3d4bcef3103e-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.506 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:21:03.507 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.507 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.507 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.507 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.507 DEBUG:boto:Method: GET 2023-09-29T22:21:03.507 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.507 DEBUG:boto:Data: 2023-09-29T22:21:03.508 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.508 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.508 DEBUG:boto:Params: {} 2023-09-29T22:21:03.508 DEBUG:boto:Token: None 2023-09-29T22:21:03.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ= 2023-09-29T22:21:03.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ=', 'Content-Length': '0'} 2023-09-29T22:21:03.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a586176076bbda5e-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.511 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.511 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.512 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.512 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.512 DEBUG:boto:Method: GET 2023-09-29T22:21:03.512 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.512 DEBUG:boto:Data: 2023-09-29T22:21:03.512 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.512 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.512 DEBUG:boto:Params: {} 2023-09-29T22:21:03.513 DEBUG:boto:Token: None 2023-09-29T22:21:03.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ= 2023-09-29T22:21:03.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ=', 'Content-Length': '0'} 2023-09-29T22:21:03.516 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000932f944e64a76cb5-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.516 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.516 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.516 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.516 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.516 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.517 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.517 DEBUG:boto:Data: 2023-09-29T22:21:03.517 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.517 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.517 DEBUG:boto:Params: {} 2023-09-29T22:21:03.517 DEBUG:boto:Token: None 2023-09-29T22:21:03.517 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4meZ+Ouc0Nre+znGSI2XOC9gxxA= 2023-09-29T22:21:03.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4meZ+Ouc0Nre+znGSI2XOC9gxxA=', 'Content-Length': '0'} 2023-09-29T22:21:03.520 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003b3adf173f2e4358-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.520 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.520 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.520 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.521 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.521 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.521 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.521 DEBUG:boto:Data: 2023-09-29T22:21:03.521 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.521 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.521 DEBUG:boto:Params: {} 2023-09-29T22:21:03.521 DEBUG:boto:Token: None 2023-09-29T22:21:03.522 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.522 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4meZ+Ouc0Nre+znGSI2XOC9gxxA= 2023-09-29T22:21:03.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4meZ+Ouc0Nre+znGSI2XOC9gxxA=', 'Content-Length': '0'} 2023-09-29T22:21:03.524 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000063b2cac0cf70cba6-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:21:03.525 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.525 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.525 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.525 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.525 DEBUG:boto:Method: GET 2023-09-29T22:21:03.525 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.525 DEBUG:boto:Data: 2023-09-29T22:21:03.526 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.526 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.526 DEBUG:boto:Params: {} 2023-09-29T22:21:03.526 DEBUG:boto:Token: None 2023-09-29T22:21:03.526 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.526 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ= 2023-09-29T22:21:03.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ=', 'Content-Length': '0'} 2023-09-29T22:21:03.529 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000905ffcd8da24815c-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.570 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.570 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.571 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.571 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.571 DEBUG:boto:Method: GET 2023-09-29T22:21:03.571 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.571 DEBUG:boto:Data: 2023-09-29T22:21:03.571 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.571 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.571 DEBUG:boto:Params: {} 2023-09-29T22:21:03.572 DEBUG:boto:Token: None 2023-09-29T22:21:03.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:21:03.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ= 2023-09-29T22:21:03.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kCmVQoHsNQmspsGl9cftDQWYsVQ=', 'Content-Length': '0'} 2023-09-29T22:21:03.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000732cea38c6d89ddc-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.575 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.575 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.576 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.576 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.576 DEBUG:boto:Method: GET 2023-09-29T22:21:03.576 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.576 DEBUG:boto:Data: 2023-09-29T22:21:03.576 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.576 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.576 DEBUG:boto:Params: {} 2023-09-29T22:21:03.576 DEBUG:boto:Token: None 2023-09-29T22:21:03.577 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033c2bdfe1d849dcf-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.582 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.582 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.582 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.582 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.582 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.583 DEBUG:boto:Method: GET 2023-09-29T22:21:03.583 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.583 DEBUG:boto:Data: 2023-09-29T22:21:03.583 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.583 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.583 DEBUG:boto:Params: {} 2023-09-29T22:21:03.583 DEBUG:boto:Token: None 2023-09-29T22:21:03.583 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000868c91b253328901-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.630 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.631 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.631 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.631 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.631 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:21:03.631 DEBUG:boto:Data: 2023-09-29T22:21:03.631 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.631 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.632 DEBUG:boto:Params: {} 2023-09-29T22:21:03.632 DEBUG:boto:Token: None 2023-09-29T22:21:03.632 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2 2023-09-29T22:21:03.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1l2WrK3/Y3attT8ZC/P4I1tyJuY= 2023-09-29T22:21:03.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1l2WrK3/Y3attT8ZC/P4I1tyJuY=', 'Content-Length': '0'} 2023-09-29T22:21:03.634 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed09ec3498c0e9f3-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.635 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.635 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.635 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.635 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:21:03.635 DEBUG:boto:Data: 2023-09-29T22:21:03.635 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.636 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.636 DEBUG:boto:Params: {} 2023-09-29T22:21:03.636 DEBUG:boto:Token: None 2023-09-29T22:21:03.636 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2 2023-09-29T22:21:03.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1l2WrK3/Y3attT8ZC/P4I1tyJuY= 2023-09-29T22:21:03.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1l2WrK3/Y3attT8ZC/P4I1tyJuY=', 'Content-Length': '0'} 2023-09-29T22:21:03.639 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d11fae3fa96878e-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:21:03.639 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.639 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.639 DEBUG:boto:Method: GET 2023-09-29T22:21:03.639 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:21:03.639 DEBUG:boto:Data: 2023-09-29T22:21:03.639 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.640 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.640 DEBUG:boto:Params: {} 2023-09-29T22:21:03.640 DEBUG:boto:Token: None 2023-09-29T22:21:03.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2 2023-09-29T22:21:03.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Jy5c0zyI+S+o3EUoI/vG+1x40LM= 2023-09-29T22:21:03.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Jy5c0zyI+S+o3EUoI/vG+1x40LM=', 'Content-Length': '0'} 2023-09-29T22:21:03.643 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e12884e3d1ba8d59-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.643 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:21:03.643 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:21:03.643 DEBUG:boto:Method: GET 2023-09-29T22:21:03.644 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:21:03.644 DEBUG:boto:Data: 2023-09-29T22:21:03.644 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.644 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.644 DEBUG:boto:Params: {} 2023-09-29T22:21:03.644 DEBUG:boto:Token: None 2023-09-29T22:21:03.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj2 2023-09-29T22:21:03.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Jy5c0zyI+S+o3EUoI/vG+1x40LM= 2023-09-29T22:21:03.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Jy5c0zyI+S+o3EUoI/vG+1x40LM=', 'Content-Length': '0'} 2023-09-29T22:21:03.647 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000030fb7458d539e244-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.647 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.648 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.648 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.648 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.648 DEBUG:boto:Method: GET 2023-09-29T22:21:03.648 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.648 DEBUG:boto:Data: 2023-09-29T22:21:03.648 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.648 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.649 DEBUG:boto:Params: {} 2023-09-29T22:21:03.649 DEBUG:boto:Token: None 2023-09-29T22:21:03.649 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043be404d6fe74956-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.698 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.699 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.699 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.699 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.699 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.699 DEBUG:boto:Method: GET 2023-09-29T22:21:03.699 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:21:03.699 DEBUG:boto:Data: 2023-09-29T22:21:03.699 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.700 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.700 DEBUG:boto:Params: {} 2023-09-29T22:21:03.700 DEBUG:boto:Token: None 2023-09-29T22:21:03.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c9695a4121c3717-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.706 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:21:03.707 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.707 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.707 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.707 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.707 DEBUG:boto:Method: GET 2023-09-29T22:21:03.707 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.707 DEBUG:boto:Data: 2023-09-29T22:21:03.707 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.708 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.708 DEBUG:boto:Params: {} 2023-09-29T22:21:03.708 DEBUG:boto:Token: None 2023-09-29T22:21:03.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA= 2023-09-29T22:21:03.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA=', 'Content-Length': '0'} 2023-09-29T22:21:03.713 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a08aca836dc244ba-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.713 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.713 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.713 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.713 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.714 DEBUG:boto:Method: GET 2023-09-29T22:21:03.714 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.714 DEBUG:boto:Data: 2023-09-29T22:21:03.714 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.714 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.714 DEBUG:boto:Params: {} 2023-09-29T22:21:03.714 DEBUG:boto:Token: None 2023-09-29T22:21:03.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA= 2023-09-29T22:21:03.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA=', 'Content-Length': '0'} 2023-09-29T22:21:03.718 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbfe552bfd1df755-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.762 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.762 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.763 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.763 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.763 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.763 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.763 DEBUG:boto:Data: 2023-09-29T22:21:03.763 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.763 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.763 DEBUG:boto:Params: {} 2023-09-29T22:21:03.764 DEBUG:boto:Token: None 2023-09-29T22:21:03.764 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2/Fm6/lkcv3OWLB4rMHNREGOImE= 2023-09-29T22:21:03.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2/Fm6/lkcv3OWLB4rMHNREGOImE=', 'Content-Length': '0'} 2023-09-29T22:21:03.766 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2e98a5d3953a2d4-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.767 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.767 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.767 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.767 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.767 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.767 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.768 DEBUG:boto:Data: 2023-09-29T22:21:03.768 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.768 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.768 DEBUG:boto:Params: {} 2023-09-29T22:21:03.768 DEBUG:boto:Token: None 2023-09-29T22:21:03.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2/Fm6/lkcv3OWLB4rMHNREGOImE= 2023-09-29T22:21:03.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2/Fm6/lkcv3OWLB4rMHNREGOImE=', 'Content-Length': '0'} 2023-09-29T22:21:03.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000221df25cafae0a87-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:21:03.772 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.772 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.772 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.772 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.772 DEBUG:boto:Method: GET 2023-09-29T22:21:03.772 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.772 DEBUG:boto:Data: 2023-09-29T22:21:03.772 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.773 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.773 DEBUG:boto:Params: {} 2023-09-29T22:21:03.773 DEBUG:boto:Token: None 2023-09-29T22:21:03.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA= 2023-09-29T22:21:03.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA=', 'Content-Length': '0'} 2023-09-29T22:21:03.776 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006962792dda7b45a3-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.776 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.776 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.776 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.776 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.777 DEBUG:boto:Method: GET 2023-09-29T22:21:03.777 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.777 DEBUG:boto:Data: 2023-09-29T22:21:03.777 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.777 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.777 DEBUG:boto:Params: {} 2023-09-29T22:21:03.777 DEBUG:boto:Token: None 2023-09-29T22:21:03.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:21:03.778 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA= 2023-09-29T22:21:03.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0SaHbCatgM6LW09rvl7ZLbhQLHA=', 'Content-Length': '0'} 2023-09-29T22:21:03.780 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002b622fc448d3ca73-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.781 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.781 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.781 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.781 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.781 DEBUG:boto:Method: GET 2023-09-29T22:21:03.781 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.781 DEBUG:boto:Data: 2023-09-29T22:21:03.781 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.782 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.782 DEBUG:boto:Params: {} 2023-09-29T22:21:03.782 DEBUG:boto:Token: None 2023-09-29T22:21:03.782 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.782 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000003ec502b0f57c1f-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.834 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.835 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.835 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.835 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.835 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.835 DEBUG:boto:Method: GET 2023-09-29T22:21:03.835 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.836 DEBUG:boto:Data: 2023-09-29T22:21:03.836 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.836 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.836 DEBUG:boto:Params: {} 2023-09-29T22:21:03.836 DEBUG:boto:Token: None 2023-09-29T22:21:03.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062f35cb822cca2ed-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.841 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.841 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.841 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.842 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.842 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:21:03.842 DEBUG:boto:Data: 2023-09-29T22:21:03.842 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.842 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.842 DEBUG:boto:Params: {} 2023-09-29T22:21:03.842 DEBUG:boto:Token: None 2023-09-29T22:21:03.842 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3 2023-09-29T22:21:03.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+EQGe1hsg/bLCBu8etcoAH/yEWs= 2023-09-29T22:21:03.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+EQGe1hsg/bLCBu8etcoAH/yEWs=', 'Content-Length': '0'} 2023-09-29T22:21:03.845 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8781e724a513599-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.845 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.845 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.846 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.846 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:21:03.846 DEBUG:boto:Data: 2023-09-29T22:21:03.846 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.846 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.846 DEBUG:boto:Params: {} 2023-09-29T22:21:03.846 DEBUG:boto:Token: None 2023-09-29T22:21:03.846 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3 2023-09-29T22:21:03.847 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+EQGe1hsg/bLCBu8etcoAH/yEWs= 2023-09-29T22:21:03.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+EQGe1hsg/bLCBu8etcoAH/yEWs=', 'Content-Length': '0'} 2023-09-29T22:21:03.849 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db4e9d191f20e750-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:21:03.850 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.850 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.850 DEBUG:boto:Method: GET 2023-09-29T22:21:03.850 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:21:03.850 DEBUG:boto:Data: 2023-09-29T22:21:03.850 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.850 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.851 DEBUG:boto:Params: {} 2023-09-29T22:21:03.851 DEBUG:boto:Token: None 2023-09-29T22:21:03.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3 2023-09-29T22:21:03.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OuOHYOpOllyb2ylMBvd+Qv1+keo= 2023-09-29T22:21:03.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OuOHYOpOllyb2ylMBvd+Qv1+keo=', 'Content-Length': '0'} 2023-09-29T22:21:03.854 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000be19f3823b1efb68-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.854 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:21:03.854 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:21:03.854 DEBUG:boto:Method: GET 2023-09-29T22:21:03.854 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:21:03.854 DEBUG:boto:Data: 2023-09-29T22:21:03.855 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.855 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.855 DEBUG:boto:Params: {} 2023-09-29T22:21:03.855 DEBUG:boto:Token: None 2023-09-29T22:21:03.855 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj3 2023-09-29T22:21:03.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OuOHYOpOllyb2ylMBvd+Qv1+keo= 2023-09-29T22:21:03.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OuOHYOpOllyb2ylMBvd+Qv1+keo=', 'Content-Length': '0'} 2023-09-29T22:21:03.858 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006bcb9d4ada5a8f7e-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.902 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.903 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.903 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.903 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.903 DEBUG:boto:Method: GET 2023-09-29T22:21:03.903 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.903 DEBUG:boto:Data: 2023-09-29T22:21:03.903 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.903 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.903 DEBUG:boto:Params: {} 2023-09-29T22:21:03.904 DEBUG:boto:Token: None 2023-09-29T22:21:03.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000046c0f0912ee39c9-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.909 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.909 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.909 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.909 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.909 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.909 DEBUG:boto:Method: GET 2023-09-29T22:21:03.910 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:21:03.910 DEBUG:boto:Data: 2023-09-29T22:21:03.910 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.910 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.910 DEBUG:boto:Params: {} 2023-09-29T22:21:03.910 DEBUG:boto:Token: None 2023-09-29T22:21:03.910 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.911 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.915 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000680af781afd482e5-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.915 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:21:03.916 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.916 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.916 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.916 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.916 DEBUG:boto:Method: GET 2023-09-29T22:21:03.916 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.916 DEBUG:boto:Data: 2023-09-29T22:21:03.916 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.917 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.917 DEBUG:boto:Params: {} 2023-09-29T22:21:03.917 DEBUG:boto:Token: None 2023-09-29T22:21:03.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s= 2023-09-29T22:21:03.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s=', 'Content-Length': '0'} 2023-09-29T22:21:03.919 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000621f7999f854a168-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.920 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.920 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.920 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.920 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.920 DEBUG:boto:Method: GET 2023-09-29T22:21:03.920 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.921 DEBUG:boto:Data: 2023-09-29T22:21:03.921 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.921 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.921 DEBUG:boto:Params: {} 2023-09-29T22:21:03.921 DEBUG:boto:Token: None 2023-09-29T22:21:03.921 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.921 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s= 2023-09-29T22:21:03.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s=', 'Content-Length': '0'} 2023-09-29T22:21:03.924 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005b5cdbe7ed60ad5f-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.924 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.924 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.924 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.924 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.925 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.925 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.925 DEBUG:boto:Data: 2023-09-29T22:21:03.925 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.925 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.925 DEBUG:boto:Params: {} 2023-09-29T22:21:03.925 DEBUG:boto:Token: None 2023-09-29T22:21:03.925 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.926 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l43belNOB2Wyj4yHGnfBe2EsjJ4= 2023-09-29T22:21:03.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l43belNOB2Wyj4yHGnfBe2EsjJ4=', 'Content-Length': '0'} 2023-09-29T22:21:03.928 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ba61263d96bd119-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.928 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.928 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.928 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.928 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.928 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.929 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.929 DEBUG:boto:Data: 2023-09-29T22:21:03.929 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.929 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.929 DEBUG:boto:Params: {} 2023-09-29T22:21:03.929 DEBUG:boto:Token: None 2023-09-29T22:21:03.929 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.929 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l43belNOB2Wyj4yHGnfBe2EsjJ4= 2023-09-29T22:21:03.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l43belNOB2Wyj4yHGnfBe2EsjJ4=', 'Content-Length': '0'} 2023-09-29T22:21:03.932 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ac855b781c5d96d6-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:21:03.932 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.932 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.933 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.933 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.933 DEBUG:boto:Method: GET 2023-09-29T22:21:03.933 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.933 DEBUG:boto:Data: 2023-09-29T22:21:03.933 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.933 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.933 DEBUG:boto:Params: {} 2023-09-29T22:21:03.934 DEBUG:boto:Token: None 2023-09-29T22:21:03.934 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s= 2023-09-29T22:21:03.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s=', 'Content-Length': '0'} 2023-09-29T22:21:03.936 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000517b73d3b9f60fcf-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.978 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.978 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.979 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.979 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.979 DEBUG:boto:Method: GET 2023-09-29T22:21:03.979 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.979 DEBUG:boto:Data: 2023-09-29T22:21:03.979 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.979 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.979 DEBUG:boto:Params: {} 2023-09-29T22:21:03.980 DEBUG:boto:Token: None 2023-09-29T22:21:03.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:21:03.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s= 2023-09-29T22:21:03.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+cARmKDkO6K70xA+7ZIFM0Hle8s=', 'Content-Length': '0'} 2023-09-29T22:21:03.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000213e1716e1c3d565-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.983 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.983 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.983 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:03.984 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:03.984 DEBUG:boto:Method: GET 2023-09-29T22:21:03.984 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:21:03.984 DEBUG:boto:Data: 2023-09-29T22:21:03.984 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.984 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.984 DEBUG:boto:Params: {} 2023-09-29T22:21:03.984 DEBUG:boto:Token: None 2023-09-29T22:21:03.985 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.985 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a57149444b3fb08b-0065174dcf-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.989 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.990 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:03.990 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:03.990 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:03.990 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:03.990 DEBUG:boto:Method: GET 2023-09-29T22:21:03.990 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:21:03.990 DEBUG:boto:Data: 2023-09-29T22:21:03.991 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:03.991 DEBUG:boto:Port: 8000 2023-09-29T22:21:03.991 DEBUG:boto:Params: {} 2023-09-29T22:21:03.991 DEBUG:boto:Token: None 2023-09-29T22:21:03.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/ 2023-09-29T22:21:03.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE= 2023-09-29T22:21:03.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xjzmkUGMMFL9IUTERo8X2kHoNTE=', 'Content-Length': '0'} 2023-09-29T22:21:03.996 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056510f7c6e5c8e9f-0065174dcf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:03.996 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:03.996 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:03.996 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:03.997 DEBUG:boto:Method: HEAD 2023-09-29T22:21:03.997 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:21:03.997 DEBUG:boto:Data: 2023-09-29T22:21:03.997 DEBUG:boto:Headers: {} 2023-09-29T22:21:03.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:03.997 DEBUG:boto:Port: 8001 2023-09-29T22:21:03.997 DEBUG:boto:Params: {} 2023-09-29T22:21:03.997 DEBUG:boto:Token: None 2023-09-29T22:21:03.998 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:03 GMT /hcnphz-49/obj4 2023-09-29T22:21:03.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+VhuQCR9T3rzhRud29z5Ak6v/nk= 2023-09-29T22:21:03.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+VhuQCR9T3rzhRud29z5Ak6v/nk=', 'Content-Length': '0'} 2023-09-29T22:21:04.000 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000027d59d1cd325abe4-0065174dcf-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:04.000 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:04.000 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:04.000 DEBUG:boto:Method: HEAD 2023-09-29T22:21:04.001 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:21:04.001 DEBUG:boto:Data: 2023-09-29T22:21:04.001 DEBUG:boto:Headers: {} 2023-09-29T22:21:04.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:04.001 DEBUG:boto:Port: 8000 2023-09-29T22:21:04.001 DEBUG:boto:Params: {} 2023-09-29T22:21:04.001 DEBUG:boto:Token: None 2023-09-29T22:21:04.001 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:04 GMT /hcnphz-49/obj4 2023-09-29T22:21:04.001 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+U/PUi960S2xT9rqE/q+Fxxkrjw= 2023-09-29T22:21:04.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+U/PUi960S2xT9rqE/q+Fxxkrjw=', 'Content-Length': '0'} 2023-09-29T22:21:04.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006c8d350ea2f97947-0065174dcf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:04.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:21:04.004 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:04.004 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:04.004 DEBUG:boto:Method: GET 2023-09-29T22:21:04.005 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:21:04.005 DEBUG:boto:Data: 2023-09-29T22:21:04.005 DEBUG:boto:Headers: {} 2023-09-29T22:21:04.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:04.005 DEBUG:boto:Port: 8001 2023-09-29T22:21:04.005 DEBUG:boto:Params: {} 2023-09-29T22:21:04.005 DEBUG:boto:Token: None 2023-09-29T22:21:04.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:04 GMT /hcnphz-49/obj4 2023-09-29T22:21:04.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BMSMw28xNAPoN7V2nQXeawpOm58= 2023-09-29T22:21:04.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BMSMw28xNAPoN7V2nQXeawpOm58=', 'Content-Length': '0'} 2023-09-29T22:21:04.008 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000235dea0c47ea730a-0065174dd0-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:04.008 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:21:04.008 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:21:04.009 DEBUG:boto:Method: GET 2023-09-29T22:21:04.009 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:21:04.009 DEBUG:boto:Data: 2023-09-29T22:21:04.009 DEBUG:boto:Headers: {} 2023-09-29T22:21:04.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:04.009 DEBUG:boto:Port: 8000 2023-09-29T22:21:04.009 DEBUG:boto:Params: {} 2023-09-29T22:21:04.009 DEBUG:boto:Token: None 2023-09-29T22:21:04.009 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:04 GMT /hcnphz-49/obj4 2023-09-29T22:21:04.010 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BMSMw28xNAPoN7V2nQXeawpOm58= 2023-09-29T22:21:04.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BMSMw28xNAPoN7V2nQXeawpOm58=', 'Content-Length': '0'} 2023-09-29T22:21:04.012 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000090f147865d5ae098-0065174dd0-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:04.054 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:04.054 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:04.054 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:04.055 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:04.055 DEBUG:boto:Method: GET 2023-09-29T22:21:04.055 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:21:04.055 DEBUG:boto:Data: 2023-09-29T22:21:04.055 DEBUG:boto:Headers: {} 2023-09-29T22:21:04.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:04.055 DEBUG:boto:Port: 8001 2023-09-29T22:21:04.055 DEBUG:boto:Params: {} 2023-09-29T22:21:04.055 DEBUG:boto:Token: None 2023-09-29T22:21:04.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:04 GMT /hcnphz-49/ 2023-09-29T22:21:04.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a/h6eNBWicYMZfEhnr83QY98+08= 2023-09-29T22:21:04.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a/h6eNBWicYMZfEhnr83QY98+08=', 'Content-Length': '0'} 2023-09-29T22:21:04.060 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042a084d1e65ee3fe-0065174dd0-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:04.060 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:04.060 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:04.060 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:04.061 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:04.061 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:21:04.061 DEBUG:boto:Method: GET 2023-09-29T22:21:04.061 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:21:04.061 DEBUG:boto:Data: 2023-09-29T22:21:04.061 DEBUG:boto:Headers: {} 2023-09-29T22:21:04.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:04.061 DEBUG:boto:Port: 8000 2023-09-29T22:21:04.061 DEBUG:boto:Params: {} 2023-09-29T22:21:04.062 DEBUG:boto:Token: None 2023-09-29T22:21:04.062 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:21:04 GMT /hcnphz-49/ 2023-09-29T22:21:04.062 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a/h6eNBWicYMZfEhnr83QY98+08= 2023-09-29T22:21:04.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a/h6eNBWicYMZfEhnr83QY98+08=', 'Content-Length': '0'} 2023-09-29T22:21:04.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336d17fb30cf1799-0065174dd0-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:21:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:04.066 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:21:04.067 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-49 zones={a2, a1} 2023-09-29T22:21:04.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hcnphz-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:04.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hcnphz-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:04.067 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hcnphz-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:21:04.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:04.110 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:04.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'hcnphz-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:04.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'hcnphz-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:04.236 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket hcnphz-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:21:04.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:04.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:04.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:04.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:04.410 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:21:04.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:04.454 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:04.511 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:21:04.538 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:21:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:21:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:21:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026064.382525_386.1", 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:04.382525Z" 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:21:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:04.540 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:21:04.541 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026064.382525_386.1', 2: '', 3: ''} 2023-09-29T22:21:04.541 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:21:04.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:04.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:04.541 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:21:04.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:04.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:04.644 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:04.641+0000 7f640f258f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:04.645 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:04.641+0000 7f640f258f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:04.671 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:21:04.671 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:21:04.671 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:21:04.671 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:04.672 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.673 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025896.227910_382.1", 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:18:16.227910Z", 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:04.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:04.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:04.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:21:04.677 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:21:04.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:04.678 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:21:04Z" 2023-09-29T22:21:04.678 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:21:04.679 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025896.227910_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:18:16.227910Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:21:04Z" } 2023-09-29T22:21:04.679 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025896.227910_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:18:16.227910Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:21:04.679 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026064.382525_386.1', 2: '', 3: ''} 2023-09-29T22:21:04.679 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025896.227910_382.1', 2: '', 3: ''} 2023-09-29T22:21:04.679 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026064.382525_386.1 target=1_1696025896.227910_382.1 2023-09-29T22:21:09.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:09.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:09.681 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:21:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:09.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:09.782 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:09.781+0000 7f2cf831bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:09.783 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:09.785+0000 7f2cf831bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:09.817 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:21:09.817 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:21:09.817 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:21:09.817 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:21:09.818 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:09.819 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696025896.227910_382.1", 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:18:16.227910Z", 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:09.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:09.821 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:09.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:21:09.823 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:21:09.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:09.824 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:21:09Z" 2023-09-29T22:21:09.824 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:21:09.825 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696025896.227910_382.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:18:16.227910Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:21:09Z" } 2023-09-29T22:21:09.825 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696025896.227910_382.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:18:16.227910Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:21:09.825 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026064.382525_386.1', 2: '', 3: ''} 2023-09-29T22:21:09.825 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696025896.227910_382.1', 2: '', 3: ''} 2023-09-29T22:21:09.825 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026064.382525_386.1 target=1_1696025896.227910_382.1 2023-09-29T22:21:14.826 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:14.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:14.826 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:21:14.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:14.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:14.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:14.929+0000 7f3a29332f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:14.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:14.929+0000 7f3a29332f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:14.956 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:21:14.956 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:21:14.956 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:21:14.956 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:21:14.956 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:21:14.956 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:14.957 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:14.958 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026064.204636_383.1", 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:04.204636Z", 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:14.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:14.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:14.961 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:21:14.962 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:21:14.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:14.963 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:21:14Z" 2023-09-29T22:21:14.963 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:21:14.964 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026064.204636_383.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:04.204636Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:21:14Z" } 2023-09-29T22:21:14.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026064.204636_383.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:04.204636Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:21:14.964 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026064.382525_386.1', 2: '', 3: ''} 2023-09-29T22:21:14.964 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026064.204636_383.1', 2: '', 3: ''} 2023-09-29T22:21:14.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026064.382525_386.1 target=1_1696026064.204636_383.1 2023-09-29T22:21:19.011 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:21:19.008+0000 7efd48c31700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:21:19.012 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:21:19.008+0000 7efd4bc37700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:21:19.012 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:21:19.008+0000 7efd4cc39700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:21:19.012 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:21:19.008+0000 7efd4fc3f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:21:19.012 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:21:19.008+0000 7efd4d43a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:21:19.012 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T22:21:19.008+0000 7efd51c43700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:21:19.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:19.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:21:19.966 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:21:20.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:20.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:20.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:20.065+0000 7f65481aef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:20.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:20.069+0000 7f65481aef40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:21:20.091 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:21:20.091 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:21:20.091 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:21:20.091 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:21:20.091 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:21:20.092 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:20.093 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026064.382525_386.1", 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:04.382525Z", 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:20.094 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:20.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:21:20.096 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:21:20Z" 2023-09-29T22:21:20.097 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:21:20.098 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026064.382525_386.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:04.382525Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:21:20Z" } 2023-09-29T22:21:20.099 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026064.382525_386.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:04.382525Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:21:20.099 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026064.382525_386.1', 2: '', 3: ''} 2023-09-29T22:21:20.099 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026064.382525_386.1', 2: '', 3: ''} 2023-09-29T22:21:20.099 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:21:20.099 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:20.099 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:20.099 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.099 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:21:20.100 DEBUG:boto:Data: 2023-09-29T22:21:20.100 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.100 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.100 DEBUG:boto:Params: {} 2023-09-29T22:21:20.100 DEBUG:boto:Token: None 2023-09-29T22:21:20.100 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/ 2023-09-29T22:21:20.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg= 2023-09-29T22:21:20.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg=', 'Content-Length': '0'} 2023-09-29T22:21:20.104 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a4edbec1d558decb-0065174de0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.104 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:21:20.104 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:21:20.104 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.105 DEBUG:boto:Path: /hcnphz-48/obj5 2023-09-29T22:21:20.105 DEBUG:boto:Data: 2023-09-29T22:21:20.105 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.105 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.105 DEBUG:boto:Params: {} 2023-09-29T22:21:20.105 DEBUG:boto:Token: None 2023-09-29T22:21:20.105 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/obj5 2023-09-29T22:21:20.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQEm6X2n6Pwa7K+BhQCgzoyx3uI= 2023-09-29T22:21:20.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQEm6X2n6Pwa7K+BhQCgzoyx3uI='} 2023-09-29T22:21:20.114 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a02c5a9aff3dff1-0065174de0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.114 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:20.114 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:20.115 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.115 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:21:20.115 DEBUG:boto:Data: 2023-09-29T22:21:20.115 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.115 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.115 DEBUG:boto:Params: {} 2023-09-29T22:21:20.115 DEBUG:boto:Token: None 2023-09-29T22:21:20.115 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/ 2023-09-29T22:21:20.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg= 2023-09-29T22:21:20.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg=', 'Content-Length': '0'} 2023-09-29T22:21:20.118 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc052b4b3ea9987f-0065174de0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.119 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:21:20.119 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:21:20.119 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.119 DEBUG:boto:Path: /hcnphz-48/obj6 2023-09-29T22:21:20.119 DEBUG:boto:Data: 2023-09-29T22:21:20.119 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.119 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.119 DEBUG:boto:Params: {} 2023-09-29T22:21:20.120 DEBUG:boto:Token: None 2023-09-29T22:21:20.120 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/obj6 2023-09-29T22:21:20.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S+5zzLBM27FaWbObkauO3z6L/x4= 2023-09-29T22:21:20.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S+5zzLBM27FaWbObkauO3z6L/x4='} 2023-09-29T22:21:20.162 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f0251dd96904b5d5-0065174de0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.163 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:20.163 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:20.163 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.163 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:21:20.164 DEBUG:boto:Data: 2023-09-29T22:21:20.164 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.164 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.164 DEBUG:boto:Params: {} 2023-09-29T22:21:20.164 DEBUG:boto:Token: None 2023-09-29T22:21:20.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/ 2023-09-29T22:21:20.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg= 2023-09-29T22:21:20.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg=', 'Content-Length': '0'} 2023-09-29T22:21:20.169 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba1b6f44d5e14508-0065174de0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.169 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:21:20.169 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:21:20.169 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.169 DEBUG:boto:Path: /hcnphz-48/obj7 2023-09-29T22:21:20.169 DEBUG:boto:Data: 2023-09-29T22:21:20.170 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.170 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.170 DEBUG:boto:Params: {} 2023-09-29T22:21:20.170 DEBUG:boto:Token: None 2023-09-29T22:21:20.170 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/obj7 2023-09-29T22:21:20.170 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j0BNj6DBMcP6c1Za59HK6n5aPXo= 2023-09-29T22:21:20.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j0BNj6DBMcP6c1Za59HK6n5aPXo='} 2023-09-29T22:21:20.182 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d9558ad798ed7d5-0065174de0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.182 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:21:20.182 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:21:20.182 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.182 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:21:20.183 DEBUG:boto:Data: 2023-09-29T22:21:20.183 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.183 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.183 DEBUG:boto:Params: {} 2023-09-29T22:21:20.183 DEBUG:boto:Token: None 2023-09-29T22:21:20.183 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/ 2023-09-29T22:21:20.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg= 2023-09-29T22:21:20.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NLfvFQ8EWVw9U4GDUHcOWITpeZg=', 'Content-Length': '0'} 2023-09-29T22:21:20.187 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000325daa6c7d497fd2-0065174de0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.187 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:21:20.187 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:21:20.187 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.187 DEBUG:boto:Path: /hcnphz-48/obj8 2023-09-29T22:21:20.187 DEBUG:boto:Data: 2023-09-29T22:21:20.187 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:21:20.188 DEBUG:boto:Port: 8000 2023-09-29T22:21:20.188 DEBUG:boto:Params: {} 2023-09-29T22:21:20.188 DEBUG:boto:Token: None 2023-09-29T22:21:20.188 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-48/obj8 2023-09-29T22:21:20.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nCoMOTEM6KBj92yrsoDNLZYsufc= 2023-09-29T22:21:20.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nCoMOTEM6KBj92yrsoDNLZYsufc='} 2023-09-29T22:21:20.230 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000121de39508195c4d-0065174de0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.231 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:20.231 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:20.231 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.231 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:21:20.231 DEBUG:boto:Data: 2023-09-29T22:21:20.231 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.232 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.232 DEBUG:boto:Params: {} 2023-09-29T22:21:20.232 DEBUG:boto:Token: None 2023-09-29T22:21:20.232 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/ 2023-09-29T22:21:20.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE= 2023-09-29T22:21:20.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE=', 'Content-Length': '0'} 2023-09-29T22:21:20.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6a9cdd8b16e5448-0065174de0-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.237 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:21:20.237 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:21:20.237 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.237 DEBUG:boto:Path: /hcnphz-49/obj5 2023-09-29T22:21:20.237 DEBUG:boto:Data: 2023-09-29T22:21:20.237 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.237 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.238 DEBUG:boto:Params: {} 2023-09-29T22:21:20.238 DEBUG:boto:Token: None 2023-09-29T22:21:20.238 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/obj5 2023-09-29T22:21:20.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C6pUYaXJXu/1M4k2mU5Mqp4HzHY= 2023-09-29T22:21:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C6pUYaXJXu/1M4k2mU5Mqp4HzHY='} 2023-09-29T22:21:20.283 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002acada972c7f23d6-0065174de0-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.283 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:20.283 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:20.283 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.283 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:21:20.283 DEBUG:boto:Data: 2023-09-29T22:21:20.284 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.284 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.284 DEBUG:boto:Params: {} 2023-09-29T22:21:20.284 DEBUG:boto:Token: None 2023-09-29T22:21:20.284 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/ 2023-09-29T22:21:20.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE= 2023-09-29T22:21:20.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE=', 'Content-Length': '0'} 2023-09-29T22:21:20.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000562ffab277292e19-0065174de0-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.289 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:21:20.289 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:21:20.289 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.289 DEBUG:boto:Path: /hcnphz-49/obj6 2023-09-29T22:21:20.289 DEBUG:boto:Data: 2023-09-29T22:21:20.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.290 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.290 DEBUG:boto:Params: {} 2023-09-29T22:21:20.290 DEBUG:boto:Token: None 2023-09-29T22:21:20.290 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/obj6 2023-09-29T22:21:20.290 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZB7XgnLUIvJQabKs7NugQccg3+Q= 2023-09-29T22:21:20.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZB7XgnLUIvJQabKs7NugQccg3+Q='} 2023-09-29T22:21:20.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000024233690f71aa147-0065174de0-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.299 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:20.299 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:20.299 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.299 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:21:20.299 DEBUG:boto:Data: 2023-09-29T22:21:20.300 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.300 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.300 DEBUG:boto:Params: {} 2023-09-29T22:21:20.300 DEBUG:boto:Token: None 2023-09-29T22:21:20.300 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/ 2023-09-29T22:21:20.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE= 2023-09-29T22:21:20.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE=', 'Content-Length': '0'} 2023-09-29T22:21:20.304 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082881f3ce3574e6b-0065174de0-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.304 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:21:20.305 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:21:20.305 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.305 DEBUG:boto:Path: /hcnphz-49/obj7 2023-09-29T22:21:20.305 DEBUG:boto:Data: 2023-09-29T22:21:20.305 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.305 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.305 DEBUG:boto:Params: {} 2023-09-29T22:21:20.306 DEBUG:boto:Token: None 2023-09-29T22:21:20.306 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/obj7 2023-09-29T22:21:20.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6dA7XFlsJr7CyoKEr70ZthF7zsA= 2023-09-29T22:21:20.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6dA7XFlsJr7CyoKEr70ZthF7zsA='} 2023-09-29T22:21:20.351 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6620ce064f162a9-0065174de0-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.351 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:21:20.351 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:21:20.351 DEBUG:boto:Method: HEAD 2023-09-29T22:21:20.351 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:21:20.351 DEBUG:boto:Data: 2023-09-29T22:21:20.352 DEBUG:boto:Headers: {} 2023-09-29T22:21:20.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.352 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.352 DEBUG:boto:Params: {} 2023-09-29T22:21:20.352 DEBUG:boto:Token: None 2023-09-29T22:21:20.352 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/ 2023-09-29T22:21:20.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE= 2023-09-29T22:21:20.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Knk0AmFbOgkH36Hf2H4NU5L+CE=', 'Content-Length': '0'} 2023-09-29T22:21:20.357 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b98dcb0c31e6e2a2-0065174de0-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.357 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:21:20.357 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:21:20.357 DEBUG:boto:Method: PUT 2023-09-29T22:21:20.357 DEBUG:boto:Path: /hcnphz-49/obj8 2023-09-29T22:21:20.358 DEBUG:boto:Data: 2023-09-29T22:21:20.358 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T22:21:20.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:21:20.358 DEBUG:boto:Port: 8001 2023-09-29T22:21:20.358 DEBUG:boto:Params: {} 2023-09-29T22:21:20.358 DEBUG:boto:Token: None 2023-09-29T22:21:20.358 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 22:21:20 GMT /hcnphz-49/obj8 2023-09-29T22:21:20.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o0EbV7L2hYUqMXRTFPP2+yTS3e4= 2023-09-29T22:21:20.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:21:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o0EbV7L2hYUqMXRTFPP2+yTS3e4='} 2023-09-29T22:21:20.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004150db9e41c29f43-0065174de0-1642-a2'), ('Date', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:21:20.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hcnphz-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:20.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hcnphz-48', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:20.368 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hcnphz-48 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:21:20.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:20.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:20.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'hcnphz-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:20.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'hcnphz-49', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:21:20.545 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket hcnphz-49 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:21:20.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:20.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:20.720 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-48 2023-09-29T22:21:20.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:21:20.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:21:20.720 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-48 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:21:20.768 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:21:20.769 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:21:20.832 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:20.833+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:25.833 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:25.833+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:30.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:30.833+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:35.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:35.836+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:40.837 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:40.836+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:45.838 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:45.836+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:49.025 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:21:49.024+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:21:50.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:50.836+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:21:55.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:21:55.840+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:00.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:00.840+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:05.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:05.840+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:10.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:10.839+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:15.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:15.843+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:20.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:20.843+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:25.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:25.843+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:30.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:30.847+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:35.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:35.847+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:40.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:40.847+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:45.846+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:50.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:50.850+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:22:55.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:22:55.850+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:00.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:00.850+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:05.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:05.854+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:10.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:10.854+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:15.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:15.858+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:20.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:20.854+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:25.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:25.857+0000 7f6330263f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:30.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:30.861+0000 7f6330263f40 1 bucket sync caught up with source: 2023-09-29T22:23:30.864 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.39.1, 00000000002.120.1, 00000000004.31.1, 00000000002.42.1, 00000000004.33.1, 00000000002.15.1, 00000000002.44.1, 00000000004.21.1, 00000000002.30.1, 00000000002.28.1, 00000000002.19.1] 2023-09-29T22:23:30.864 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.39.1, 00000000002.120.1, 00000000004.31.1, 00000000002.42.1, 00000000004.33.1, 00000000002.15.1, 00000000002.44.1, 00000000004.21.1, 00000000002.30.1, 00000000002.28.1, 00000000002.19.1] 2023-09-29T22:23:30.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:30.861+0000 7f6330263f40 0 bucket checkpoint complete 2023-09-29T22:23:30.891 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-48 2023-09-29T22:23:30.892 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:30.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-48', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:30.892 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-48 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:30.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:30.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:31.005 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:31.001+0000 7fbbb09b5f40 1 bucket sync caught up with source: 2023-09-29T22:23:31.005 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.192.1, 00000000002.178.1, 00000000004.109.6, 00000000002.107.1, 00000000004.133.1, 00000000002.111.1, 00000000002.114.1, 00000000004.165.1, 00000000002.241.1, 00000000002.74.1, 00000000002.170.1] 2023-09-29T22:23:31.005 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.192.1, 00000000002.178.1, 00000000004.109.6, 00000000002.107.1, 00000000004.133.1, 00000000002.111.1, 00000000002.114.1, 00000000004.165.1, 00000000002.241.1, 00000000002.74.1, 00000000002.170.1] 2023-09-29T22:23:31.006 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:31.001+0000 7fbbb09b5f40 0 bucket checkpoint complete 2023-09-29T22:23:31.030 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-48 zones={a2, a1} 2023-09-29T22:23:31.030 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.030 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.030 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.030 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:23:31.031 DEBUG:boto:Data: 2023-09-29T22:23:31.031 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.031 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.031 DEBUG:boto:Params: {} 2023-09-29T22:23:31.031 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:23:31.032 DEBUG:boto:Token: None 2023-09-29T22:23:31.032 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jZ3Ka2ZVUQDLFmDNRL0fNZNFfEo= 2023-09-29T22:23:31.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jZ3Ka2ZVUQDLFmDNRL0fNZNFfEo=', 'Content-Length': '0'} 2023-09-29T22:23:31.037 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002ec95d5b10fb4097-0065174e63-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.037 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.038 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.038 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.038 DEBUG:boto:Path: /hcnphz-48/ 2023-09-29T22:23:31.038 DEBUG:boto:Data: 2023-09-29T22:23:31.038 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.038 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.038 DEBUG:boto:Params: {} 2023-09-29T22:23:31.039 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:23:31.039 DEBUG:boto:Token: None 2023-09-29T22:23:31.039 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.039 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jZ3Ka2ZVUQDLFmDNRL0fNZNFfEo= 2023-09-29T22:23:31.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jZ3Ka2ZVUQDLFmDNRL0fNZNFfEo=', 'Content-Length': '0'} 2023-09-29T22:23:31.044 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000612771aac492c6ae-0065174e63-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.044 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:23:31.044 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.044 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.045 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.045 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.045 DEBUG:boto:Method: GET 2023-09-29T22:23:31.045 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.045 DEBUG:boto:Data: 2023-09-29T22:23:31.045 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.045 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.045 DEBUG:boto:Params: {} 2023-09-29T22:23:31.046 DEBUG:boto:Token: None 2023-09-29T22:23:31.046 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/?versions 2023-09-29T22:23:31.046 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ= 2023-09-29T22:23:31.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ=', 'Content-Length': '0'} 2023-09-29T22:23:31.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024b4144fc273366c-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.052 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:23:31.053 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:23:31.054 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:23:31.054 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.054 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.054 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.054 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.054 DEBUG:boto:Method: GET 2023-09-29T22:23:31.054 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.054 DEBUG:boto:Data: 2023-09-29T22:23:31.054 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.055 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.055 DEBUG:boto:Params: {} 2023-09-29T22:23:31.055 DEBUG:boto:Token: None 2023-09-29T22:23:31.055 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/?versions 2023-09-29T22:23:31.055 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ= 2023-09-29T22:23:31.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ=', 'Content-Length': '0'} 2023-09-29T22:23:31.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf44fb2c167b31bf-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.059 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:23:31.060 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:23:31.061 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.061 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.061 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.061 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.061 DEBUG:boto:Method: GET 2023-09-29T22:23:31.061 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.061 DEBUG:boto:Data: 2023-09-29T22:23:31.061 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.062 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.062 DEBUG:boto:Params: {} 2023-09-29T22:23:31.062 DEBUG:boto:Token: None 2023-09-29T22:23:31.062 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/?versions 2023-09-29T22:23:31.062 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ= 2023-09-29T22:23:31.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ=', 'Content-Length': '0'} 2023-09-29T22:23:31.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bc6735574636b89-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.066 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.067 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.067 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.067 DEBUG:boto:path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.067 DEBUG:boto:auth_path=/hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.067 DEBUG:boto:Method: GET 2023-09-29T22:23:31.067 DEBUG:boto:Path: /hcnphz-48/?versions&max-keys=999 2023-09-29T22:23:31.068 DEBUG:boto:Data: 2023-09-29T22:23:31.068 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.068 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.068 DEBUG:boto:Params: {} 2023-09-29T22:23:31.068 DEBUG:boto:Token: None 2023-09-29T22:23:31.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/?versions 2023-09-29T22:23:31.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ= 2023-09-29T22:23:31.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9XXo5YyqysPJnYh4i9GPrFajZgQ=', 'Content-Length': '0'} 2023-09-29T22:23:31.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c7677d79993f459-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.073 DEBUG:boto:b'hcnphz-48999falseobj1nulltrue2023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:23:31.073 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.074 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.074 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.074 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.074 DEBUG:boto:Method: GET 2023-09-29T22:23:31.074 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.074 DEBUG:boto:Data: 2023-09-29T22:23:31.074 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.074 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.075 DEBUG:boto:Params: {} 2023-09-29T22:23:31.075 DEBUG:boto:Token: None 2023-09-29T22:23:31.075 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.075 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4= 2023-09-29T22:23:31.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4=', 'Content-Length': '0'} 2023-09-29T22:23:31.078 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000b87b6fec7df6e8a-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.078 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.078 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.078 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.078 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.078 DEBUG:boto:Method: GET 2023-09-29T22:23:31.079 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.079 DEBUG:boto:Data: 2023-09-29T22:23:31.079 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.079 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.079 DEBUG:boto:Params: {} 2023-09-29T22:23:31.079 DEBUG:boto:Token: None 2023-09-29T22:23:31.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4= 2023-09-29T22:23:31.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4=', 'Content-Length': '0'} 2023-09-29T22:23:31.082 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d22ee5ad411994bc-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.126 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.126 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.127 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.127 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.127 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.127 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.127 DEBUG:boto:Data: 2023-09-29T22:23:31.127 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.127 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.127 DEBUG:boto:Params: {} 2023-09-29T22:23:31.128 DEBUG:boto:Token: None 2023-09-29T22:23:31.128 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8OesLK/Qsk1sOu4cMdqQBqePlj4= 2023-09-29T22:23:31.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8OesLK/Qsk1sOu4cMdqQBqePlj4=', 'Content-Length': '0'} 2023-09-29T22:23:31.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000519fc7d4db731ce4-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.131 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.131 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.131 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.131 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.131 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.131 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.131 DEBUG:boto:Data: 2023-09-29T22:23:31.132 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.132 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.132 DEBUG:boto:Params: {} 2023-09-29T22:23:31.132 DEBUG:boto:Token: None 2023-09-29T22:23:31.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8OesLK/Qsk1sOu4cMdqQBqePlj4= 2023-09-29T22:23:31.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8OesLK/Qsk1sOu4cMdqQBqePlj4=', 'Content-Length': '0'} 2023-09-29T22:23:31.135 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000084893026034db079-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:23:31.135 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.136 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.136 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.136 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.136 DEBUG:boto:Method: GET 2023-09-29T22:23:31.136 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.136 DEBUG:boto:Data: 2023-09-29T22:23:31.136 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.136 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.137 DEBUG:boto:Params: {} 2023-09-29T22:23:31.137 DEBUG:boto:Token: None 2023-09-29T22:23:31.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4= 2023-09-29T22:23:31.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4=', 'Content-Length': '0'} 2023-09-29T22:23:31.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000035d60a2c971470e1-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.186 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.186 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.187 DEBUG:boto:path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.187 DEBUG:boto:auth_path=/hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.187 DEBUG:boto:Method: GET 2023-09-29T22:23:31.187 DEBUG:boto:Path: /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.187 DEBUG:boto:Data: 2023-09-29T22:23:31.187 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.187 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.187 DEBUG:boto:Params: {} 2023-09-29T22:23:31.188 DEBUG:boto:Token: None 2023-09-29T22:23:31.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1?versionId=null 2023-09-29T22:23:31.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4= 2023-09-29T22:23:31.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g+Euj5gQcyGML9bHygElZi9jJs4=', 'Content-Length': '0'} 2023-09-29T22:23:31.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d38fa258279b5eee-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.191 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.191 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.191 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.191 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.191 DEBUG:boto:Method: GET 2023-09-29T22:23:31.191 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.192 DEBUG:boto:Data: 2023-09-29T22:23:31.192 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.192 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.192 DEBUG:boto:Params: {} 2023-09-29T22:23:31.192 DEBUG:boto:Token: None 2023-09-29T22:23:31.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.196 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000877b09d91f9d3a3c-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.197 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.197 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.197 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.197 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.197 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.198 DEBUG:boto:Method: GET 2023-09-29T22:23:31.198 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.198 DEBUG:boto:Data: 2023-09-29T22:23:31.198 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.198 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.198 DEBUG:boto:Params: {} 2023-09-29T22:23:31.198 DEBUG:boto:Token: None 2023-09-29T22:23:31.199 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000533bcbce2894af84-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.203 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.203 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.204 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.204 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.204 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:23:31.204 DEBUG:boto:Data: 2023-09-29T22:23:31.204 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.204 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.204 DEBUG:boto:Params: {} 2023-09-29T22:23:31.205 DEBUG:boto:Token: None 2023-09-29T22:23:31.205 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1 2023-09-29T22:23:31.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sE4V7FQGtxxxsWenquEw+Mxqs54= 2023-09-29T22:23:31.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sE4V7FQGtxxxsWenquEw+Mxqs54=', 'Content-Length': '0'} 2023-09-29T22:23:31.207 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cd9ced663cdb34e0-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.208 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.208 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.208 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.208 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:23:31.208 DEBUG:boto:Data: 2023-09-29T22:23:31.208 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.209 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.209 DEBUG:boto:Params: {} 2023-09-29T22:23:31.209 DEBUG:boto:Token: None 2023-09-29T22:23:31.209 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1 2023-09-29T22:23:31.209 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sE4V7FQGtxxxsWenquEw+Mxqs54= 2023-09-29T22:23:31.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sE4V7FQGtxxxsWenquEw+Mxqs54=', 'Content-Length': '0'} 2023-09-29T22:23:31.211 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ef342bf85798a6d4-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:23:31.212 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.212 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.212 DEBUG:boto:Method: GET 2023-09-29T22:23:31.212 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:23:31.212 DEBUG:boto:Data: 2023-09-29T22:23:31.212 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.212 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.213 DEBUG:boto:Params: {} 2023-09-29T22:23:31.213 DEBUG:boto:Token: None 2023-09-29T22:23:31.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1 2023-09-29T22:23:31.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fPD7KjwcAcgJbyiU/xsNxYgiIs= 2023-09-29T22:23:31.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fPD7KjwcAcgJbyiU/xsNxYgiIs=', 'Content-Length': '0'} 2023-09-29T22:23:31.215 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000009d3a3dfa40908d5-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.258 DEBUG:boto:path=/hcnphz-48/obj1 2023-09-29T22:23:31.258 DEBUG:boto:auth_path=/hcnphz-48/obj1 2023-09-29T22:23:31.259 DEBUG:boto:Method: GET 2023-09-29T22:23:31.259 DEBUG:boto:Path: /hcnphz-48/obj1 2023-09-29T22:23:31.259 DEBUG:boto:Data: 2023-09-29T22:23:31.259 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.259 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.259 DEBUG:boto:Params: {} 2023-09-29T22:23:31.259 DEBUG:boto:Token: None 2023-09-29T22:23:31.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj1 2023-09-29T22:23:31.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fPD7KjwcAcgJbyiU/xsNxYgiIs= 2023-09-29T22:23:31.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fPD7KjwcAcgJbyiU/xsNxYgiIs=', 'Content-Length': '0'} 2023-09-29T22:23:31.262 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2de5e521e8e9213-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.263 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.263 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.263 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.263 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.263 DEBUG:boto:Method: GET 2023-09-29T22:23:31.263 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.264 DEBUG:boto:Data: 2023-09-29T22:23:31.264 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.264 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.264 DEBUG:boto:Params: {} 2023-09-29T22:23:31.264 DEBUG:boto:Token: None 2023-09-29T22:23:31.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc523e4905a9310d-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.269 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.269 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.269 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.270 DEBUG:boto:path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.270 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.270 DEBUG:boto:Method: GET 2023-09-29T22:23:31.270 DEBUG:boto:Path: /hcnphz-48/?prefix=obj1 2023-09-29T22:23:31.270 DEBUG:boto:Data: 2023-09-29T22:23:31.270 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.270 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.270 DEBUG:boto:Params: {} 2023-09-29T22:23:31.271 DEBUG:boto:Token: None 2023-09-29T22:23:31.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044eef0d7b2513218-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.276 DEBUG:boto:b'hcnphz-48obj11000falseobj12023-09-29T22:18:16.254Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:23:31.276 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.276 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.276 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.276 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.277 DEBUG:boto:Method: GET 2023-09-29T22:23:31.277 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.277 DEBUG:boto:Data: 2023-09-29T22:23:31.277 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.277 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.277 DEBUG:boto:Params: {} 2023-09-29T22:23:31.277 DEBUG:boto:Token: None 2023-09-29T22:23:31.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.278 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8= 2023-09-29T22:23:31.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8=', 'Content-Length': '0'} 2023-09-29T22:23:31.280 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000290850121d610896-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.280 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.280 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.281 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.281 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.281 DEBUG:boto:Method: GET 2023-09-29T22:23:31.281 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.281 DEBUG:boto:Data: 2023-09-29T22:23:31.281 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.281 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.281 DEBUG:boto:Params: {} 2023-09-29T22:23:31.282 DEBUG:boto:Token: None 2023-09-29T22:23:31.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8= 2023-09-29T22:23:31.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8=', 'Content-Length': '0'} 2023-09-29T22:23:31.285 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075a27d61a722e396-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.285 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.285 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.285 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.285 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.286 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.286 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.286 DEBUG:boto:Data: 2023-09-29T22:23:31.286 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.286 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.286 DEBUG:boto:Params: {} 2023-09-29T22:23:31.286 DEBUG:boto:Token: None 2023-09-29T22:23:31.286 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SoB+ze2Dx7evNdBadWGVfMGKS9M= 2023-09-29T22:23:31.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SoB+ze2Dx7evNdBadWGVfMGKS9M=', 'Content-Length': '0'} 2023-09-29T22:23:31.289 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a362e2ba549d6c8b-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.290 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.290 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.290 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.290 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.290 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.290 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.290 DEBUG:boto:Data: 2023-09-29T22:23:31.291 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.291 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.291 DEBUG:boto:Params: {} 2023-09-29T22:23:31.291 DEBUG:boto:Token: None 2023-09-29T22:23:31.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SoB+ze2Dx7evNdBadWGVfMGKS9M= 2023-09-29T22:23:31.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SoB+ze2Dx7evNdBadWGVfMGKS9M=', 'Content-Length': '0'} 2023-09-29T22:23:31.294 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007e2e530bade4ef24-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:23:31.294 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.294 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.295 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.295 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.295 DEBUG:boto:Method: GET 2023-09-29T22:23:31.295 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.295 DEBUG:boto:Data: 2023-09-29T22:23:31.295 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.295 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.295 DEBUG:boto:Params: {} 2023-09-29T22:23:31.296 DEBUG:boto:Token: None 2023-09-29T22:23:31.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8= 2023-09-29T22:23:31.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8=', 'Content-Length': '0'} 2023-09-29T22:23:31.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bb1d0205ee6e74da-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.342 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.342 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.343 DEBUG:boto:path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.343 DEBUG:boto:auth_path=/hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.343 DEBUG:boto:Method: GET 2023-09-29T22:23:31.343 DEBUG:boto:Path: /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.343 DEBUG:boto:Data: 2023-09-29T22:23:31.343 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.343 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.343 DEBUG:boto:Params: {} 2023-09-29T22:23:31.344 DEBUG:boto:Token: None 2023-09-29T22:23:31.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2?versionId=null 2023-09-29T22:23:31.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8= 2023-09-29T22:23:31.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pb066TyjQk4Yc0iLb0ixQl/MUd8=', 'Content-Length': '0'} 2023-09-29T22:23:31.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005e3e1fdc08447281-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.348 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.348 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.348 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.348 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.348 DEBUG:boto:Method: GET 2023-09-29T22:23:31.349 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.349 DEBUG:boto:Data: 2023-09-29T22:23:31.349 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.349 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.349 DEBUG:boto:Params: {} 2023-09-29T22:23:31.350 DEBUG:boto:Token: None 2023-09-29T22:23:31.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.350 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043124f6ec1f1ccd5-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.354 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.354 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.355 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.355 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.355 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.355 DEBUG:boto:Method: GET 2023-09-29T22:23:31.355 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.355 DEBUG:boto:Data: 2023-09-29T22:23:31.356 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.356 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.356 DEBUG:boto:Params: {} 2023-09-29T22:23:31.356 DEBUG:boto:Token: None 2023-09-29T22:23:31.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009107a202d41ebe2c-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.406 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.407 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.407 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.407 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.407 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:23:31.407 DEBUG:boto:Data: 2023-09-29T22:23:31.408 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.408 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.408 DEBUG:boto:Params: {} 2023-09-29T22:23:31.408 DEBUG:boto:Token: None 2023-09-29T22:23:31.408 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2 2023-09-29T22:23:31.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gn+ENBVJ8XYJYZLNREiyQQ6H9Ok= 2023-09-29T22:23:31.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gn+ENBVJ8XYJYZLNREiyQQ6H9Ok=', 'Content-Length': '0'} 2023-09-29T22:23:31.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1ae5b1d1509f76b-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.412 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.412 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.412 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.412 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:23:31.412 DEBUG:boto:Data: 2023-09-29T22:23:31.412 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.413 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.413 DEBUG:boto:Params: {} 2023-09-29T22:23:31.413 DEBUG:boto:Token: None 2023-09-29T22:23:31.413 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2 2023-09-29T22:23:31.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gn+ENBVJ8XYJYZLNREiyQQ6H9Ok= 2023-09-29T22:23:31.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gn+ENBVJ8XYJYZLNREiyQQ6H9Ok=', 'Content-Length': '0'} 2023-09-29T22:23:31.416 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a650e9797adf6168-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:23:31.416 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.416 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.416 DEBUG:boto:Method: GET 2023-09-29T22:23:31.416 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:23:31.416 DEBUG:boto:Data: 2023-09-29T22:23:31.417 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.417 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.417 DEBUG:boto:Params: {} 2023-09-29T22:23:31.417 DEBUG:boto:Token: None 2023-09-29T22:23:31.417 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2 2023-09-29T22:23:31.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mclbl7e6z9ZFry0BBfn4AccOZbs= 2023-09-29T22:23:31.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mclbl7e6z9ZFry0BBfn4AccOZbs=', 'Content-Length': '0'} 2023-09-29T22:23:31.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dea1fc10bed304c4-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.420 DEBUG:boto:path=/hcnphz-48/obj2 2023-09-29T22:23:31.420 DEBUG:boto:auth_path=/hcnphz-48/obj2 2023-09-29T22:23:31.420 DEBUG:boto:Method: GET 2023-09-29T22:23:31.421 DEBUG:boto:Path: /hcnphz-48/obj2 2023-09-29T22:23:31.421 DEBUG:boto:Data: 2023-09-29T22:23:31.421 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.421 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.421 DEBUG:boto:Params: {} 2023-09-29T22:23:31.421 DEBUG:boto:Token: None 2023-09-29T22:23:31.421 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj2 2023-09-29T22:23:31.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mclbl7e6z9ZFry0BBfn4AccOZbs= 2023-09-29T22:23:31.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mclbl7e6z9ZFry0BBfn4AccOZbs=', 'Content-Length': '0'} 2023-09-29T22:23:31.424 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000020a903066d679687-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.424 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.425 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.425 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.425 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.425 DEBUG:boto:Method: GET 2023-09-29T22:23:31.425 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.425 DEBUG:boto:Data: 2023-09-29T22:23:31.425 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.425 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.425 DEBUG:boto:Params: {} 2023-09-29T22:23:31.426 DEBUG:boto:Token: None 2023-09-29T22:23:31.426 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd485d8ebecf0e46-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.470 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.471 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.471 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.471 DEBUG:boto:path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.471 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.471 DEBUG:boto:Method: GET 2023-09-29T22:23:31.471 DEBUG:boto:Path: /hcnphz-48/?prefix=obj2 2023-09-29T22:23:31.471 DEBUG:boto:Data: 2023-09-29T22:23:31.471 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.472 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.472 DEBUG:boto:Params: {} 2023-09-29T22:23:31.472 DEBUG:boto:Token: None 2023-09-29T22:23:31.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003f9e8610026df6d3-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.477 DEBUG:boto:b'hcnphz-48obj21000falseobj22023-09-29T22:18:16.304Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:23:31.478 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.478 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.478 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.478 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.478 DEBUG:boto:Method: GET 2023-09-29T22:23:31.478 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.478 DEBUG:boto:Data: 2023-09-29T22:23:31.478 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.479 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.479 DEBUG:boto:Params: {} 2023-09-29T22:23:31.479 DEBUG:boto:Token: None 2023-09-29T22:23:31.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg= 2023-09-29T22:23:31.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg=', 'Content-Length': '0'} 2023-09-29T22:23:31.482 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003e2af77f75128c6b-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.482 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.482 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.483 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.483 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.483 DEBUG:boto:Method: GET 2023-09-29T22:23:31.483 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.483 DEBUG:boto:Data: 2023-09-29T22:23:31.483 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.483 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.483 DEBUG:boto:Params: {} 2023-09-29T22:23:31.483 DEBUG:boto:Token: None 2023-09-29T22:23:31.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg= 2023-09-29T22:23:31.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg=', 'Content-Length': '0'} 2023-09-29T22:23:31.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a70349e075c3f4e5-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.530 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.531 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.531 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.531 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.531 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.531 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.531 DEBUG:boto:Data: 2023-09-29T22:23:31.531 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.531 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.531 DEBUG:boto:Params: {} 2023-09-29T22:23:31.532 DEBUG:boto:Token: None 2023-09-29T22:23:31.532 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WljIlfAgDNDGcjKVNxqqyG/Y12o= 2023-09-29T22:23:31.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WljIlfAgDNDGcjKVNxqqyG/Y12o=', 'Content-Length': '0'} 2023-09-29T22:23:31.535 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b889c0a4d568e92c-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.535 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.535 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.535 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.535 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.535 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.536 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.536 DEBUG:boto:Data: 2023-09-29T22:23:31.536 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.536 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.536 DEBUG:boto:Params: {} 2023-09-29T22:23:31.536 DEBUG:boto:Token: None 2023-09-29T22:23:31.536 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WljIlfAgDNDGcjKVNxqqyG/Y12o= 2023-09-29T22:23:31.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WljIlfAgDNDGcjKVNxqqyG/Y12o=', 'Content-Length': '0'} 2023-09-29T22:23:31.539 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ed62280758df2e4c-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:23:31.540 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.540 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.540 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.540 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.540 DEBUG:boto:Method: GET 2023-09-29T22:23:31.540 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.540 DEBUG:boto:Data: 2023-09-29T22:23:31.540 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.540 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.541 DEBUG:boto:Params: {} 2023-09-29T22:23:31.541 DEBUG:boto:Token: None 2023-09-29T22:23:31.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg= 2023-09-29T22:23:31.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg=', 'Content-Length': '0'} 2023-09-29T22:23:31.544 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b9aee5a4ac149c59-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.544 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.544 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.544 DEBUG:boto:path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.544 DEBUG:boto:auth_path=/hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.545 DEBUG:boto:Method: GET 2023-09-29T22:23:31.545 DEBUG:boto:Path: /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.545 DEBUG:boto:Data: 2023-09-29T22:23:31.545 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.545 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.545 DEBUG:boto:Params: {} 2023-09-29T22:23:31.545 DEBUG:boto:Token: None 2023-09-29T22:23:31.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3?versionId=null 2023-09-29T22:23:31.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg= 2023-09-29T22:23:31.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0kzKbpeAXo2HROY1YsoJ88glmg=', 'Content-Length': '0'} 2023-09-29T22:23:31.548 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008b017bb8f0bfc89a-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.549 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.549 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.549 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.549 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.549 DEBUG:boto:Method: GET 2023-09-29T22:23:31.549 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.549 DEBUG:boto:Data: 2023-09-29T22:23:31.549 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.549 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.550 DEBUG:boto:Params: {} 2023-09-29T22:23:31.550 DEBUG:boto:Token: None 2023-09-29T22:23:31.550 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.550 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf818beba60f07d8-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.594 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.595 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.595 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.595 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.595 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.595 DEBUG:boto:Method: GET 2023-09-29T22:23:31.595 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.595 DEBUG:boto:Data: 2023-09-29T22:23:31.596 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.596 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.596 DEBUG:boto:Params: {} 2023-09-29T22:23:31.596 DEBUG:boto:Token: None 2023-09-29T22:23:31.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f26d574f074c74a3-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.601 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.601 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.602 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.602 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.602 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:23:31.602 DEBUG:boto:Data: 2023-09-29T22:23:31.602 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.602 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.602 DEBUG:boto:Params: {} 2023-09-29T22:23:31.603 DEBUG:boto:Token: None 2023-09-29T22:23:31.603 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3 2023-09-29T22:23:31.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d3hpWfaugJPkKLzBGUn7OZz1Ddc= 2023-09-29T22:23:31.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d3hpWfaugJPkKLzBGUn7OZz1Ddc=', 'Content-Length': '0'} 2023-09-29T22:23:31.606 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000094949396fa7b5298-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.606 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.606 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.606 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.607 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:23:31.607 DEBUG:boto:Data: 2023-09-29T22:23:31.607 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.607 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.607 DEBUG:boto:Params: {} 2023-09-29T22:23:31.607 DEBUG:boto:Token: None 2023-09-29T22:23:31.607 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3 2023-09-29T22:23:31.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d3hpWfaugJPkKLzBGUn7OZz1Ddc= 2023-09-29T22:23:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d3hpWfaugJPkKLzBGUn7OZz1Ddc=', 'Content-Length': '0'} 2023-09-29T22:23:31.610 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000623823511e4e5567-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:23:31.611 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.611 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.611 DEBUG:boto:Method: GET 2023-09-29T22:23:31.611 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:23:31.611 DEBUG:boto:Data: 2023-09-29T22:23:31.611 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.612 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.612 DEBUG:boto:Params: {} 2023-09-29T22:23:31.612 DEBUG:boto:Token: None 2023-09-29T22:23:31.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3 2023-09-29T22:23:31.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:twSTV9kuNdH6kkza3QEql16yhRM= 2023-09-29T22:23:31.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:twSTV9kuNdH6kkza3QEql16yhRM=', 'Content-Length': '0'} 2023-09-29T22:23:31.615 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000267d1428a17937fb-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.615 DEBUG:boto:path=/hcnphz-48/obj3 2023-09-29T22:23:31.615 DEBUG:boto:auth_path=/hcnphz-48/obj3 2023-09-29T22:23:31.616 DEBUG:boto:Method: GET 2023-09-29T22:23:31.616 DEBUG:boto:Path: /hcnphz-48/obj3 2023-09-29T22:23:31.616 DEBUG:boto:Data: 2023-09-29T22:23:31.616 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.616 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.616 DEBUG:boto:Params: {} 2023-09-29T22:23:31.616 DEBUG:boto:Token: None 2023-09-29T22:23:31.617 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj3 2023-09-29T22:23:31.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:twSTV9kuNdH6kkza3QEql16yhRM= 2023-09-29T22:23:31.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:twSTV9kuNdH6kkza3QEql16yhRM=', 'Content-Length': '0'} 2023-09-29T22:23:31.619 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff4c62a862949636-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.662 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.662 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.663 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.663 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.663 DEBUG:boto:Method: GET 2023-09-29T22:23:31.663 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.663 DEBUG:boto:Data: 2023-09-29T22:23:31.663 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.663 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.663 DEBUG:boto:Params: {} 2023-09-29T22:23:31.663 DEBUG:boto:Token: None 2023-09-29T22:23:31.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085d0cc8d07099c8f-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.669 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.669 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.669 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.669 DEBUG:boto:path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.669 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.669 DEBUG:boto:Method: GET 2023-09-29T22:23:31.669 DEBUG:boto:Path: /hcnphz-48/?prefix=obj3 2023-09-29T22:23:31.670 DEBUG:boto:Data: 2023-09-29T22:23:31.670 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.670 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.670 DEBUG:boto:Params: {} 2023-09-29T22:23:31.670 DEBUG:boto:Token: None 2023-09-29T22:23:31.670 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.674 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba054f2e7d134d1-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.675 DEBUG:boto:b'hcnphz-48obj31000falseobj32023-09-29T22:18:16.319Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:23:31.675 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.675 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.675 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.676 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.676 DEBUG:boto:Method: GET 2023-09-29T22:23:31.676 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.676 DEBUG:boto:Data: 2023-09-29T22:23:31.676 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.676 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.676 DEBUG:boto:Params: {} 2023-09-29T22:23:31.676 DEBUG:boto:Token: None 2023-09-29T22:23:31.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s= 2023-09-29T22:23:31.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s=', 'Content-Length': '0'} 2023-09-29T22:23:31.679 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000533eabf2221be5ef-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.679 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.679 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.679 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.680 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.680 DEBUG:boto:Method: GET 2023-09-29T22:23:31.680 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.680 DEBUG:boto:Data: 2023-09-29T22:23:31.680 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.680 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.680 DEBUG:boto:Params: {} 2023-09-29T22:23:31.680 DEBUG:boto:Token: None 2023-09-29T22:23:31.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s= 2023-09-29T22:23:31.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s=', 'Content-Length': '0'} 2023-09-29T22:23:31.683 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e6b1facb42cc1e72-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.684 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.684 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.684 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.684 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.684 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.684 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.684 DEBUG:boto:Data: 2023-09-29T22:23:31.684 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.684 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.685 DEBUG:boto:Params: {} 2023-09-29T22:23:31.685 DEBUG:boto:Token: None 2023-09-29T22:23:31.685 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.685 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QyjcKPyB7N6QNC5MMjHBzcVihjQ= 2023-09-29T22:23:31.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QyjcKPyB7N6QNC5MMjHBzcVihjQ=', 'Content-Length': '0'} 2023-09-29T22:23:31.687 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000092645441a8204122-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.687 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.687 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.687 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.687 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.688 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.688 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.688 DEBUG:boto:Data: 2023-09-29T22:23:31.688 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.688 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.688 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.688 DEBUG:boto:Params: {} 2023-09-29T22:23:31.688 DEBUG:boto:Token: None 2023-09-29T22:23:31.688 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.689 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QyjcKPyB7N6QNC5MMjHBzcVihjQ= 2023-09-29T22:23:31.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QyjcKPyB7N6QNC5MMjHBzcVihjQ=', 'Content-Length': '0'} 2023-09-29T22:23:31.691 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bb13e70f431a6c8-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:23:31.691 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.691 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.692 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.692 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.692 DEBUG:boto:Method: GET 2023-09-29T22:23:31.692 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.692 DEBUG:boto:Data: 2023-09-29T22:23:31.692 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.692 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.692 DEBUG:boto:Params: {} 2023-09-29T22:23:31.692 DEBUG:boto:Token: None 2023-09-29T22:23:31.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.693 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s= 2023-09-29T22:23:31.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s=', 'Content-Length': '0'} 2023-09-29T22:23:31.695 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ff304ca1fb30e4b8-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.738 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.738 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.739 DEBUG:boto:path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.739 DEBUG:boto:auth_path=/hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.739 DEBUG:boto:Method: GET 2023-09-29T22:23:31.739 DEBUG:boto:Path: /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.739 DEBUG:boto:Data: 2023-09-29T22:23:31.739 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.739 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.739 DEBUG:boto:Params: {} 2023-09-29T22:23:31.740 DEBUG:boto:Token: None 2023-09-29T22:23:31.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4?versionId=null 2023-09-29T22:23:31.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s= 2023-09-29T22:23:31.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dt+CxAuuACPgaNDzbVIySM7mC8s=', 'Content-Length': '0'} 2023-09-29T22:23:31.743 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ce9e018d947b3fd3-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.743 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.743 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.743 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.743 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.743 DEBUG:boto:Method: GET 2023-09-29T22:23:31.744 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.744 DEBUG:boto:Data: 2023-09-29T22:23:31.744 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.744 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.744 DEBUG:boto:Params: {} 2023-09-29T22:23:31.744 DEBUG:boto:Token: None 2023-09-29T22:23:31.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e7445c98d534dc0-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.749 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.750 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.750 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.750 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.750 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.750 DEBUG:boto:Method: GET 2023-09-29T22:23:31.750 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.750 DEBUG:boto:Data: 2023-09-29T22:23:31.750 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.751 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.751 DEBUG:boto:Params: {} 2023-09-29T22:23:31.751 DEBUG:boto:Token: None 2023-09-29T22:23:31.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.751 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.756 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025a50cb35735eea6-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.756 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.756 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.757 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.757 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.757 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:23:31.757 DEBUG:boto:Data: 2023-09-29T22:23:31.757 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.757 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.757 DEBUG:boto:Params: {} 2023-09-29T22:23:31.757 DEBUG:boto:Token: None 2023-09-29T22:23:31.758 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4 2023-09-29T22:23:31.758 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CXdHTeGVufUShmNvrMW0a62r47U= 2023-09-29T22:23:31.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CXdHTeGVufUShmNvrMW0a62r47U=', 'Content-Length': '0'} 2023-09-29T22:23:31.760 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0b3a1a4dfa69913-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.760 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.760 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.760 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.761 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:23:31.761 DEBUG:boto:Data: 2023-09-29T22:23:31.761 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.761 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.761 DEBUG:boto:Params: {} 2023-09-29T22:23:31.761 DEBUG:boto:Token: None 2023-09-29T22:23:31.761 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4 2023-09-29T22:23:31.762 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CXdHTeGVufUShmNvrMW0a62r47U= 2023-09-29T22:23:31.762 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CXdHTeGVufUShmNvrMW0a62r47U=', 'Content-Length': '0'} 2023-09-29T22:23:31.764 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3a9a9e140fd4bc1-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:23:31.764 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.764 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.764 DEBUG:boto:Method: GET 2023-09-29T22:23:31.764 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:23:31.765 DEBUG:boto:Data: 2023-09-29T22:23:31.765 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.765 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.765 DEBUG:boto:Params: {} 2023-09-29T22:23:31.765 DEBUG:boto:Token: None 2023-09-29T22:23:31.765 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4 2023-09-29T22:23:31.765 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pAPdrFRHHXJ41PywZRRKgfLMGqE= 2023-09-29T22:23:31.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pAPdrFRHHXJ41PywZRRKgfLMGqE=', 'Content-Length': '0'} 2023-09-29T22:23:31.767 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009548540b29ec50d8-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.768 DEBUG:boto:path=/hcnphz-48/obj4 2023-09-29T22:23:31.768 DEBUG:boto:auth_path=/hcnphz-48/obj4 2023-09-29T22:23:31.768 DEBUG:boto:Method: GET 2023-09-29T22:23:31.768 DEBUG:boto:Path: /hcnphz-48/obj4 2023-09-29T22:23:31.768 DEBUG:boto:Data: 2023-09-29T22:23:31.768 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.769 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.769 DEBUG:boto:Params: {} 2023-09-29T22:23:31.769 DEBUG:boto:Token: None 2023-09-29T22:23:31.769 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj4 2023-09-29T22:23:31.769 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pAPdrFRHHXJ41PywZRRKgfLMGqE= 2023-09-29T22:23:31.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pAPdrFRHHXJ41PywZRRKgfLMGqE=', 'Content-Length': '0'} 2023-09-29T22:23:31.771 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000043ee77f22a50ce3a-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.772 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.772 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.772 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.772 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.772 DEBUG:boto:Method: GET 2023-09-29T22:23:31.772 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.773 DEBUG:boto:Data: 2023-09-29T22:23:31.773 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.773 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.773 DEBUG:boto:Params: {} 2023-09-29T22:23:31.773 DEBUG:boto:Token: None 2023-09-29T22:23:31.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038745173d74ea4bc-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.818 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.819 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.819 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.819 DEBUG:boto:path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.819 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.819 DEBUG:boto:Method: GET 2023-09-29T22:23:31.819 DEBUG:boto:Path: /hcnphz-48/?prefix=obj4 2023-09-29T22:23:31.820 DEBUG:boto:Data: 2023-09-29T22:23:31.820 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.820 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.820 DEBUG:boto:Params: {} 2023-09-29T22:23:31.820 DEBUG:boto:Token: None 2023-09-29T22:23:31.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e879a2cc717b7535-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.825 DEBUG:boto:b'hcnphz-48obj41000falseobj42023-09-29T22:18:16.375Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:23:31.826 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.826 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.826 DEBUG:boto:path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.826 DEBUG:boto:auth_path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.826 DEBUG:boto:Method: GET 2023-09-29T22:23:31.826 DEBUG:boto:Path: /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.826 DEBUG:boto:Data: 2023-09-29T22:23:31.826 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.827 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.827 DEBUG:boto:Params: {} 2023-09-29T22:23:31.827 DEBUG:boto:Token: None 2023-09-29T22:23:31.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU= 2023-09-29T22:23:31.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU=', 'Content-Length': '0'} 2023-09-29T22:23:31.830 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007d2f01303d33f22b-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.831 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.831 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.831 DEBUG:boto:path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.831 DEBUG:boto:auth_path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.831 DEBUG:boto:Method: GET 2023-09-29T22:23:31.831 DEBUG:boto:Path: /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.831 DEBUG:boto:Data: 2023-09-29T22:23:31.831 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.831 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.832 DEBUG:boto:Params: {} 2023-09-29T22:23:31.832 DEBUG:boto:Token: None 2023-09-29T22:23:31.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU= 2023-09-29T22:23:31.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU=', 'Content-Length': '0'} 2023-09-29T22:23:31.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fd306685f4780a93-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.878 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.878 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.879 DEBUG:boto:path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.879 DEBUG:boto:auth_path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.879 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.879 DEBUG:boto:Path: /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.879 DEBUG:boto:Data: 2023-09-29T22:23:31.879 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.879 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.879 DEBUG:boto:Params: {} 2023-09-29T22:23:31.880 DEBUG:boto:Token: None 2023-09-29T22:23:31.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hTOfR74h9sSfbeHRcGRNare6jMA= 2023-09-29T22:23:31.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hTOfR74h9sSfbeHRcGRNare6jMA=', 'Content-Length': '0'} 2023-09-29T22:23:31.883 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dbbaec982b1ddf75-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.884 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.884 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.884 DEBUG:boto:path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.884 DEBUG:boto:auth_path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.884 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.884 DEBUG:boto:Path: /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.884 DEBUG:boto:Data: 2023-09-29T22:23:31.884 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.885 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.885 DEBUG:boto:Params: {} 2023-09-29T22:23:31.885 DEBUG:boto:Token: None 2023-09-29T22:23:31.885 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hTOfR74h9sSfbeHRcGRNare6jMA= 2023-09-29T22:23:31.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hTOfR74h9sSfbeHRcGRNare6jMA=', 'Content-Length': '0'} 2023-09-29T22:23:31.888 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a22e4aa4919632f-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:23:31.889 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.889 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.889 DEBUG:boto:path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.889 DEBUG:boto:auth_path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.889 DEBUG:boto:Method: GET 2023-09-29T22:23:31.889 DEBUG:boto:Path: /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.889 DEBUG:boto:Data: 2023-09-29T22:23:31.889 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.890 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.890 DEBUG:boto:Params: {} 2023-09-29T22:23:31.890 DEBUG:boto:Token: None 2023-09-29T22:23:31.890 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.890 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU= 2023-09-29T22:23:31.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU=', 'Content-Length': '0'} 2023-09-29T22:23:31.892 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a15c881dfe696560-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.893 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.893 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.893 DEBUG:boto:path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.893 DEBUG:boto:auth_path=/hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.893 DEBUG:boto:Method: GET 2023-09-29T22:23:31.893 DEBUG:boto:Path: /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.893 DEBUG:boto:Data: 2023-09-29T22:23:31.894 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.894 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.894 DEBUG:boto:Params: {} 2023-09-29T22:23:31.894 DEBUG:boto:Token: None 2023-09-29T22:23:31.894 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5?versionId=null 2023-09-29T22:23:31.894 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU= 2023-09-29T22:23:31.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sHAxUWVmSh8Zl1tFUam2YGh+dUU=', 'Content-Length': '0'} 2023-09-29T22:23:31.896 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cbe78f746f5fd2d-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.897 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.897 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.897 DEBUG:boto:path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.897 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.897 DEBUG:boto:Method: GET 2023-09-29T22:23:31.897 DEBUG:boto:Path: /hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.897 DEBUG:boto:Data: 2023-09-29T22:23:31.897 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.898 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.898 DEBUG:boto:Params: {} 2023-09-29T22:23:31.898 DEBUG:boto:Token: None 2023-09-29T22:23:31.898 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000675ed44be80df43e-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.902 DEBUG:boto:b'hcnphz-48obj51000falseobj52023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.902 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.902 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.903 DEBUG:boto:path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.903 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.903 DEBUG:boto:Method: GET 2023-09-29T22:23:31.903 DEBUG:boto:Path: /hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.903 DEBUG:boto:Data: 2023-09-29T22:23:31.903 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.903 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.903 DEBUG:boto:Params: {} 2023-09-29T22:23:31.904 DEBUG:boto:Token: None 2023-09-29T22:23:31.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2744a1a178c8458-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.908 DEBUG:boto:b'hcnphz-48obj51000falseobj52023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.908 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.908 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.908 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.909 DEBUG:boto:Path: /hcnphz-48/obj5 2023-09-29T22:23:31.909 DEBUG:boto:Data: 2023-09-29T22:23:31.909 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.909 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.909 DEBUG:boto:Params: {} 2023-09-29T22:23:31.909 DEBUG:boto:Token: None 2023-09-29T22:23:31.909 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5 2023-09-29T22:23:31.909 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VI4PGqXTJZ/lFfJAyYGZ/U7YAvI= 2023-09-29T22:23:31.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VI4PGqXTJZ/lFfJAyYGZ/U7YAvI=', 'Content-Length': '0'} 2023-09-29T22:23:31.912 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f309b54b3897d5c1-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.912 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.912 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.912 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.912 DEBUG:boto:Path: /hcnphz-48/obj5 2023-09-29T22:23:31.912 DEBUG:boto:Data: 2023-09-29T22:23:31.913 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.913 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.913 DEBUG:boto:Params: {} 2023-09-29T22:23:31.913 DEBUG:boto:Token: None 2023-09-29T22:23:31.913 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5 2023-09-29T22:23:31.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VI4PGqXTJZ/lFfJAyYGZ/U7YAvI= 2023-09-29T22:23:31.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VI4PGqXTJZ/lFfJAyYGZ/U7YAvI=', 'Content-Length': '0'} 2023-09-29T22:23:31.916 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e29278d3f22ee22c-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:23:31.916 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.916 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.916 DEBUG:boto:Method: GET 2023-09-29T22:23:31.916 DEBUG:boto:Path: /hcnphz-48/obj5 2023-09-29T22:23:31.916 DEBUG:boto:Data: 2023-09-29T22:23:31.917 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.917 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.917 DEBUG:boto:Params: {} 2023-09-29T22:23:31.917 DEBUG:boto:Token: None 2023-09-29T22:23:31.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5 2023-09-29T22:23:31.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ci199fezkrXBTDa8zsIVlL8+3vU= 2023-09-29T22:23:31.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ci199fezkrXBTDa8zsIVlL8+3vU=', 'Content-Length': '0'} 2023-09-29T22:23:31.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006b08e536fddf1cee-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.962 DEBUG:boto:path=/hcnphz-48/obj5 2023-09-29T22:23:31.962 DEBUG:boto:auth_path=/hcnphz-48/obj5 2023-09-29T22:23:31.963 DEBUG:boto:Method: GET 2023-09-29T22:23:31.963 DEBUG:boto:Path: /hcnphz-48/obj5 2023-09-29T22:23:31.963 DEBUG:boto:Data: 2023-09-29T22:23:31.963 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.963 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.963 DEBUG:boto:Params: {} 2023-09-29T22:23:31.963 DEBUG:boto:Token: None 2023-09-29T22:23:31.963 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj5 2023-09-29T22:23:31.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ci199fezkrXBTDa8zsIVlL8+3vU= 2023-09-29T22:23:31.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ci199fezkrXBTDa8zsIVlL8+3vU=', 'Content-Length': '0'} 2023-09-29T22:23:31.966 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bac75fcc83aa2363-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.967 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.967 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.967 DEBUG:boto:path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.967 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.967 DEBUG:boto:Method: GET 2023-09-29T22:23:31.967 DEBUG:boto:Path: /hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.967 DEBUG:boto:Data: 2023-09-29T22:23:31.967 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.968 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.968 DEBUG:boto:Params: {} 2023-09-29T22:23:31.968 DEBUG:boto:Token: None 2023-09-29T22:23:31.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.972 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e213ae64ebf19094-0065174e63-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.973 DEBUG:boto:b'hcnphz-48obj51000falseobj52023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.973 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:31.974 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:31.974 DEBUG:boto:path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.974 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.974 DEBUG:boto:Method: GET 2023-09-29T22:23:31.974 DEBUG:boto:Path: /hcnphz-48/?prefix=obj5 2023-09-29T22:23:31.974 DEBUG:boto:Data: 2023-09-29T22:23:31.975 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.975 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.975 DEBUG:boto:Params: {} 2023-09-29T22:23:31.975 DEBUG:boto:Token: None 2023-09-29T22:23:31.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/ 2023-09-29T22:23:31.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI= 2023-09-29T22:23:31.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HRc2W/1MLvJAQPz7/ejJpmruuVI=', 'Content-Length': '0'} 2023-09-29T22:23:31.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1897624e9347d1d-0065174e63-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.979 DEBUG:boto:b'hcnphz-48obj51000falseobj52023-09-29T22:21:20.109Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:31.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:23:31.980 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:31.980 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:31.980 DEBUG:boto:path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.980 DEBUG:boto:auth_path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.980 DEBUG:boto:Method: GET 2023-09-29T22:23:31.980 DEBUG:boto:Path: /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.980 DEBUG:boto:Data: 2023-09-29T22:23:31.980 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.981 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.981 DEBUG:boto:Params: {} 2023-09-29T22:23:31.981 DEBUG:boto:Token: None 2023-09-29T22:23:31.981 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kzT29EeyL4wE7iNDrqACE77gFac= 2023-09-29T22:23:31.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kzT29EeyL4wE7iNDrqACE77gFac=', 'Content-Length': '0'} 2023-09-29T22:23:31.984 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003129a00c82ef5ed6-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.984 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:31.984 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:31.984 DEBUG:boto:path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.984 DEBUG:boto:auth_path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.985 DEBUG:boto:Method: GET 2023-09-29T22:23:31.985 DEBUG:boto:Path: /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.985 DEBUG:boto:Data: 2023-09-29T22:23:31.985 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.985 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.985 DEBUG:boto:Params: {} 2023-09-29T22:23:31.985 DEBUG:boto:Token: None 2023-09-29T22:23:31.985 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.986 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kzT29EeyL4wE7iNDrqACE77gFac= 2023-09-29T22:23:31.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kzT29EeyL4wE7iNDrqACE77gFac=', 'Content-Length': '0'} 2023-09-29T22:23:31.988 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003863a35e496751f4-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.988 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:31.989 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:31.989 DEBUG:boto:path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.989 DEBUG:boto:auth_path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.989 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.989 DEBUG:boto:Path: /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.989 DEBUG:boto:Data: 2023-09-29T22:23:31.989 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.989 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.990 DEBUG:boto:Params: {} 2023-09-29T22:23:31.990 DEBUG:boto:Token: None 2023-09-29T22:23:31.990 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.990 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TqGZcA4TSYYy0MQy7ZGqbKfGQPc= 2023-09-29T22:23:31.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TqGZcA4TSYYy0MQy7ZGqbKfGQPc=', 'Content-Length': '0'} 2023-09-29T22:23:31.992 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f004479f91e71aef-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.993 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:31.993 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:31.993 DEBUG:boto:path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.993 DEBUG:boto:auth_path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.993 DEBUG:boto:Method: HEAD 2023-09-29T22:23:31.994 DEBUG:boto:Path: /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.994 DEBUG:boto:Data: 2023-09-29T22:23:31.994 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:31.994 DEBUG:boto:Port: 8000 2023-09-29T22:23:31.994 DEBUG:boto:Params: {} 2023-09-29T22:23:31.995 DEBUG:boto:Token: None 2023-09-29T22:23:31.995 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.995 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TqGZcA4TSYYy0MQy7ZGqbKfGQPc= 2023-09-29T22:23:31.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TqGZcA4TSYYy0MQy7ZGqbKfGQPc=', 'Content-Length': '0'} 2023-09-29T22:23:31.998 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077aff9b0a5a5832d-0065174e63-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:31.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:23:31.998 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:31.998 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:31.998 DEBUG:boto:path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.999 DEBUG:boto:auth_path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.999 DEBUG:boto:Method: GET 2023-09-29T22:23:31.999 DEBUG:boto:Path: /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:31.999 DEBUG:boto:Data: 2023-09-29T22:23:31.999 DEBUG:boto:Headers: {} 2023-09-29T22:23:31.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:31.999 DEBUG:boto:Port: 8001 2023-09-29T22:23:31.999 DEBUG:boto:Params: {} 2023-09-29T22:23:31.999 DEBUG:boto:Token: None 2023-09-29T22:23:32.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:31 GMT /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:32.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kzT29EeyL4wE7iNDrqACE77gFac= 2023-09-29T22:23:32.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kzT29EeyL4wE7iNDrqACE77gFac=', 'Content-Length': '0'} 2023-09-29T22:23:32.002 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000433eaf5583d8bac9-0065174e63-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.046 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:32.046 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:32.047 DEBUG:boto:path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:32.047 DEBUG:boto:auth_path=/hcnphz-48/obj6?versionId=null 2023-09-29T22:23:32.047 DEBUG:boto:Method: GET 2023-09-29T22:23:32.047 DEBUG:boto:Path: /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:32.047 DEBUG:boto:Data: 2023-09-29T22:23:32.047 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.047 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.047 DEBUG:boto:Params: {} 2023-09-29T22:23:32.047 DEBUG:boto:Token: None 2023-09-29T22:23:32.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj6?versionId=null 2023-09-29T22:23:32.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FMCE2jGsoamQJwpB3ewEViDXhzg= 2023-09-29T22:23:32.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FMCE2jGsoamQJwpB3ewEViDXhzg=', 'Content-Length': '0'} 2023-09-29T22:23:32.051 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc2d236ebe317d6b-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.051 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.051 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.051 DEBUG:boto:path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.051 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.051 DEBUG:boto:Method: GET 2023-09-29T22:23:32.052 DEBUG:boto:Path: /hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.052 DEBUG:boto:Data: 2023-09-29T22:23:32.052 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.052 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.052 DEBUG:boto:Params: {} 2023-09-29T22:23:32.052 DEBUG:boto:Token: None 2023-09-29T22:23:32.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35d8c4ab7d8ba72-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.057 DEBUG:boto:b'hcnphz-48obj61000falseobj62023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.058 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.058 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.058 DEBUG:boto:path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.058 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.058 DEBUG:boto:Method: GET 2023-09-29T22:23:32.058 DEBUG:boto:Path: /hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.059 DEBUG:boto:Data: 2023-09-29T22:23:32.059 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.059 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.059 DEBUG:boto:Params: {} 2023-09-29T22:23:32.059 DEBUG:boto:Token: None 2023-09-29T22:23:32.059 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.059 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.064 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009aa20fd5a0f58004-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.106 DEBUG:boto:b'hcnphz-48obj61000falseobj62023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.107 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:32.107 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:32.107 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.107 DEBUG:boto:Path: /hcnphz-48/obj6 2023-09-29T22:23:32.107 DEBUG:boto:Data: 2023-09-29T22:23:32.107 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.107 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.107 DEBUG:boto:Params: {} 2023-09-29T22:23:32.108 DEBUG:boto:Token: None 2023-09-29T22:23:32.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj6 2023-09-29T22:23:32.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iCnNByoaRzrTUAb4BUgw20l70s4= 2023-09-29T22:23:32.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iCnNByoaRzrTUAb4BUgw20l70s4=', 'Content-Length': '0'} 2023-09-29T22:23:32.110 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d37c3d9bb5740c83-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.111 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:32.111 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:32.111 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.111 DEBUG:boto:Path: /hcnphz-48/obj6 2023-09-29T22:23:32.111 DEBUG:boto:Data: 2023-09-29T22:23:32.111 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.111 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.112 DEBUG:boto:Params: {} 2023-09-29T22:23:32.112 DEBUG:boto:Token: None 2023-09-29T22:23:32.112 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj6 2023-09-29T22:23:32.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iCnNByoaRzrTUAb4BUgw20l70s4= 2023-09-29T22:23:32.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iCnNByoaRzrTUAb4BUgw20l70s4=', 'Content-Length': '0'} 2023-09-29T22:23:32.114 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006f457a0db78beac7-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:23:32.115 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:32.115 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:32.115 DEBUG:boto:Method: GET 2023-09-29T22:23:32.115 DEBUG:boto:Path: /hcnphz-48/obj6 2023-09-29T22:23:32.115 DEBUG:boto:Data: 2023-09-29T22:23:32.115 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.115 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.115 DEBUG:boto:Params: {} 2023-09-29T22:23:32.116 DEBUG:boto:Token: None 2023-09-29T22:23:32.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj6 2023-09-29T22:23:32.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2zlnlQTSNDke1A3Plv7Htg20eI= 2023-09-29T22:23:32.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2zlnlQTSNDke1A3Plv7Htg20eI=', 'Content-Length': '0'} 2023-09-29T22:23:32.118 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9544babb51ab59a-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.119 DEBUG:boto:path=/hcnphz-48/obj6 2023-09-29T22:23:32.119 DEBUG:boto:auth_path=/hcnphz-48/obj6 2023-09-29T22:23:32.119 DEBUG:boto:Method: GET 2023-09-29T22:23:32.119 DEBUG:boto:Path: /hcnphz-48/obj6 2023-09-29T22:23:32.119 DEBUG:boto:Data: 2023-09-29T22:23:32.119 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.119 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.120 DEBUG:boto:Params: {} 2023-09-29T22:23:32.120 DEBUG:boto:Token: None 2023-09-29T22:23:32.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj6 2023-09-29T22:23:32.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2zlnlQTSNDke1A3Plv7Htg20eI= 2023-09-29T22:23:32.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2zlnlQTSNDke1A3Plv7Htg20eI=', 'Content-Length': '0'} 2023-09-29T22:23:32.122 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000026de8adc9d63a467-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.122 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.122 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.123 DEBUG:boto:path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.123 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.123 DEBUG:boto:Method: GET 2023-09-29T22:23:32.123 DEBUG:boto:Path: /hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.123 DEBUG:boto:Data: 2023-09-29T22:23:32.123 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.123 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.123 DEBUG:boto:Params: {} 2023-09-29T22:23:32.124 DEBUG:boto:Token: None 2023-09-29T22:23:32.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdf826ba658f7175-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.174 DEBUG:boto:b'hcnphz-48obj61000falseobj62023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.175 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.175 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.175 DEBUG:boto:path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.175 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.175 DEBUG:boto:Method: GET 2023-09-29T22:23:32.175 DEBUG:boto:Path: /hcnphz-48/?prefix=obj6 2023-09-29T22:23:32.175 DEBUG:boto:Data: 2023-09-29T22:23:32.176 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.176 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.176 DEBUG:boto:Params: {} 2023-09-29T22:23:32.176 DEBUG:boto:Token: None 2023-09-29T22:23:32.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058e005dfd3c4b482-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.181 DEBUG:boto:b'hcnphz-48obj61000falseobj62023-09-29T22:21:20.123Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:23:32.182 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.182 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.182 DEBUG:boto:path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.182 DEBUG:boto:auth_path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.182 DEBUG:boto:Method: GET 2023-09-29T22:23:32.183 DEBUG:boto:Path: /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.183 DEBUG:boto:Data: 2023-09-29T22:23:32.183 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.183 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.183 DEBUG:boto:Params: {} 2023-09-29T22:23:32.183 DEBUG:boto:Token: None 2023-09-29T22:23:32.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw= 2023-09-29T22:23:32.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw=', 'Content-Length': '0'} 2023-09-29T22:23:32.186 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c3533b070d46fddd-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.187 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.187 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.187 DEBUG:boto:path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.187 DEBUG:boto:auth_path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.187 DEBUG:boto:Method: GET 2023-09-29T22:23:32.187 DEBUG:boto:Path: /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.187 DEBUG:boto:Data: 2023-09-29T22:23:32.187 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.188 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.188 DEBUG:boto:Params: {} 2023-09-29T22:23:32.188 DEBUG:boto:Token: None 2023-09-29T22:23:32.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw= 2023-09-29T22:23:32.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw=', 'Content-Length': '0'} 2023-09-29T22:23:32.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001b89b46c6d220ac2-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.191 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.192 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.192 DEBUG:boto:path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.192 DEBUG:boto:auth_path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.192 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.192 DEBUG:boto:Path: /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.192 DEBUG:boto:Data: 2023-09-29T22:23:32.192 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.192 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.193 DEBUG:boto:Params: {} 2023-09-29T22:23:32.193 DEBUG:boto:Token: None 2023-09-29T22:23:32.193 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DSEAsI7sdUaKEvLHsQqt4JTGV9A= 2023-09-29T22:23:32.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DSEAsI7sdUaKEvLHsQqt4JTGV9A=', 'Content-Length': '0'} 2023-09-29T22:23:32.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e2a4221700e75a42-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.196 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.196 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.196 DEBUG:boto:path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.196 DEBUG:boto:auth_path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.196 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.197 DEBUG:boto:Path: /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.197 DEBUG:boto:Data: 2023-09-29T22:23:32.197 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.197 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.197 DEBUG:boto:Params: {} 2023-09-29T22:23:32.197 DEBUG:boto:Token: None 2023-09-29T22:23:32.197 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DSEAsI7sdUaKEvLHsQqt4JTGV9A= 2023-09-29T22:23:32.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DSEAsI7sdUaKEvLHsQqt4JTGV9A=', 'Content-Length': '0'} 2023-09-29T22:23:32.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b295b3e6ba57fa25-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:23:32.200 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.201 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.201 DEBUG:boto:path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.201 DEBUG:boto:auth_path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.201 DEBUG:boto:Method: GET 2023-09-29T22:23:32.201 DEBUG:boto:Path: /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.201 DEBUG:boto:Data: 2023-09-29T22:23:32.201 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.201 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.201 DEBUG:boto:Params: {} 2023-09-29T22:23:32.202 DEBUG:boto:Token: None 2023-09-29T22:23:32.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.202 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw= 2023-09-29T22:23:32.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw=', 'Content-Length': '0'} 2023-09-29T22:23:32.205 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004d189b8160b18d9c-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.246 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.246 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.247 DEBUG:boto:path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.247 DEBUG:boto:auth_path=/hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.247 DEBUG:boto:Method: GET 2023-09-29T22:23:32.247 DEBUG:boto:Path: /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.247 DEBUG:boto:Data: 2023-09-29T22:23:32.247 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.247 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.248 DEBUG:boto:Params: {} 2023-09-29T22:23:32.248 DEBUG:boto:Token: None 2023-09-29T22:23:32.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7?versionId=null 2023-09-29T22:23:32.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw= 2023-09-29T22:23:32.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LBXcCNqmyuAEjESO3044gHzdOpw=', 'Content-Length': '0'} 2023-09-29T22:23:32.251 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000bcd026cf180a3558-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.251 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.251 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.252 DEBUG:boto:path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.252 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.252 DEBUG:boto:Method: GET 2023-09-29T22:23:32.252 DEBUG:boto:Path: /hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.252 DEBUG:boto:Data: 2023-09-29T22:23:32.252 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.252 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.252 DEBUG:boto:Params: {} 2023-09-29T22:23:32.253 DEBUG:boto:Token: None 2023-09-29T22:23:32.253 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.253 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.257 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014b1f28c0a8e1758-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.258 DEBUG:boto:b'hcnphz-48obj71000falseobj72023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.258 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.258 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.258 DEBUG:boto:path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.258 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.258 DEBUG:boto:Method: GET 2023-09-29T22:23:32.259 DEBUG:boto:Path: /hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.259 DEBUG:boto:Data: 2023-09-29T22:23:32.259 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.259 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.259 DEBUG:boto:Params: {} 2023-09-29T22:23:32.259 DEBUG:boto:Token: None 2023-09-29T22:23:32.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8b8b61b0a97d32e-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.264 DEBUG:boto:b'hcnphz-48obj71000falseobj72023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.264 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.264 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.265 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.265 DEBUG:boto:Path: /hcnphz-48/obj7 2023-09-29T22:23:32.265 DEBUG:boto:Data: 2023-09-29T22:23:32.265 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.265 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.265 DEBUG:boto:Params: {} 2023-09-29T22:23:32.265 DEBUG:boto:Token: None 2023-09-29T22:23:32.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7 2023-09-29T22:23:32.266 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kL/iswa1tkmxnUZ/sofZjx+NDnQ= 2023-09-29T22:23:32.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kL/iswa1tkmxnUZ/sofZjx+NDnQ=', 'Content-Length': '0'} 2023-09-29T22:23:32.268 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cce7e501653b599a-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.268 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.268 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.269 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.269 DEBUG:boto:Path: /hcnphz-48/obj7 2023-09-29T22:23:32.269 DEBUG:boto:Data: 2023-09-29T22:23:32.269 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.269 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.269 DEBUG:boto:Params: {} 2023-09-29T22:23:32.269 DEBUG:boto:Token: None 2023-09-29T22:23:32.270 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7 2023-09-29T22:23:32.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kL/iswa1tkmxnUZ/sofZjx+NDnQ= 2023-09-29T22:23:32.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kL/iswa1tkmxnUZ/sofZjx+NDnQ=', 'Content-Length': '0'} 2023-09-29T22:23:32.272 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ad3c0675caaabd58-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:23:32.273 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.273 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.273 DEBUG:boto:Method: GET 2023-09-29T22:23:32.273 DEBUG:boto:Path: /hcnphz-48/obj7 2023-09-29T22:23:32.273 DEBUG:boto:Data: 2023-09-29T22:23:32.273 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.274 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.274 DEBUG:boto:Params: {} 2023-09-29T22:23:32.274 DEBUG:boto:Token: None 2023-09-29T22:23:32.274 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7 2023-09-29T22:23:32.274 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CClv2Fx0ySTFdO0v70OTCVVp9Fk= 2023-09-29T22:23:32.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CClv2Fx0ySTFdO0v70OTCVVp9Fk=', 'Content-Length': '0'} 2023-09-29T22:23:32.277 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006de931f119f87ba5-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.277 DEBUG:boto:path=/hcnphz-48/obj7 2023-09-29T22:23:32.277 DEBUG:boto:auth_path=/hcnphz-48/obj7 2023-09-29T22:23:32.277 DEBUG:boto:Method: GET 2023-09-29T22:23:32.277 DEBUG:boto:Path: /hcnphz-48/obj7 2023-09-29T22:23:32.277 DEBUG:boto:Data: 2023-09-29T22:23:32.278 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.278 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.278 DEBUG:boto:Params: {} 2023-09-29T22:23:32.278 DEBUG:boto:Token: None 2023-09-29T22:23:32.278 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj7 2023-09-29T22:23:32.278 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CClv2Fx0ySTFdO0v70OTCVVp9Fk= 2023-09-29T22:23:32.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CClv2Fx0ySTFdO0v70OTCVVp9Fk=', 'Content-Length': '0'} 2023-09-29T22:23:32.281 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b4a8bb49109f4499-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.281 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.281 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.281 DEBUG:boto:path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.281 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.282 DEBUG:boto:Method: GET 2023-09-29T22:23:32.282 DEBUG:boto:Path: /hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.282 DEBUG:boto:Data: 2023-09-29T22:23:32.282 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.282 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.282 DEBUG:boto:Params: {} 2023-09-29T22:23:32.282 DEBUG:boto:Token: None 2023-09-29T22:23:32.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.283 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.287 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d70faf71b1537a0a-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.287 DEBUG:boto:b'hcnphz-48obj71000falseobj72023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.287 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.287 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.287 DEBUG:boto:path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.288 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.288 DEBUG:boto:Method: GET 2023-09-29T22:23:32.288 DEBUG:boto:Path: /hcnphz-48/?prefix=obj7 2023-09-29T22:23:32.288 DEBUG:boto:Data: 2023-09-29T22:23:32.288 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.288 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.288 DEBUG:boto:Params: {} 2023-09-29T22:23:32.288 DEBUG:boto:Token: None 2023-09-29T22:23:32.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088244bd26d4cf989-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.293 DEBUG:boto:b'hcnphz-48obj71000falseobj72023-09-29T22:21:20.174Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:23:32.294 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.294 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.294 DEBUG:boto:path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.294 DEBUG:boto:auth_path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.294 DEBUG:boto:Method: GET 2023-09-29T22:23:32.294 DEBUG:boto:Path: /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.294 DEBUG:boto:Data: 2023-09-29T22:23:32.294 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.295 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.295 DEBUG:boto:Params: {} 2023-09-29T22:23:32.295 DEBUG:boto:Token: None 2023-09-29T22:23:32.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ= 2023-09-29T22:23:32.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ=', 'Content-Length': '0'} 2023-09-29T22:23:32.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b6b38cacb0e5f99a-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.338 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.338 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.339 DEBUG:boto:path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.339 DEBUG:boto:auth_path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.339 DEBUG:boto:Method: GET 2023-09-29T22:23:32.339 DEBUG:boto:Path: /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.339 DEBUG:boto:Data: 2023-09-29T22:23:32.339 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.339 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.339 DEBUG:boto:Params: {} 2023-09-29T22:23:32.340 DEBUG:boto:Token: None 2023-09-29T22:23:32.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ= 2023-09-29T22:23:32.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ=', 'Content-Length': '0'} 2023-09-29T22:23:32.343 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b08204360dbf9bd9-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.343 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.344 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.344 DEBUG:boto:path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.344 DEBUG:boto:auth_path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.344 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.344 DEBUG:boto:Path: /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.344 DEBUG:boto:Data: 2023-09-29T22:23:32.344 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.344 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.344 DEBUG:boto:Params: {} 2023-09-29T22:23:32.345 DEBUG:boto:Token: None 2023-09-29T22:23:32.345 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ri1987rrTjlq2F4I/Mhb80KI2sE= 2023-09-29T22:23:32.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ri1987rrTjlq2F4I/Mhb80KI2sE=', 'Content-Length': '0'} 2023-09-29T22:23:32.348 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000091fd2f264252b5eb-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.348 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.348 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.348 DEBUG:boto:path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.348 DEBUG:boto:auth_path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.348 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.348 DEBUG:boto:Path: /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.349 DEBUG:boto:Data: 2023-09-29T22:23:32.349 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.349 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.349 DEBUG:boto:Params: {} 2023-09-29T22:23:32.349 DEBUG:boto:Token: None 2023-09-29T22:23:32.349 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ri1987rrTjlq2F4I/Mhb80KI2sE= 2023-09-29T22:23:32.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ri1987rrTjlq2F4I/Mhb80KI2sE=', 'Content-Length': '0'} 2023-09-29T22:23:32.352 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000422a877de3a1ad3d-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:23:32.353 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.353 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.353 DEBUG:boto:path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.353 DEBUG:boto:auth_path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.353 DEBUG:boto:Method: GET 2023-09-29T22:23:32.353 DEBUG:boto:Path: /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.353 DEBUG:boto:Data: 2023-09-29T22:23:32.353 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.354 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.354 DEBUG:boto:Params: {} 2023-09-29T22:23:32.354 DEBUG:boto:Token: None 2023-09-29T22:23:32.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ= 2023-09-29T22:23:32.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ=', 'Content-Length': '0'} 2023-09-29T22:23:32.357 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000004d356258a2c5df0-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.357 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.358 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.358 DEBUG:boto:path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.358 DEBUG:boto:auth_path=/hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.358 DEBUG:boto:Method: GET 2023-09-29T22:23:32.358 DEBUG:boto:Path: /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.358 DEBUG:boto:Data: 2023-09-29T22:23:32.358 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.358 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.358 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.359 DEBUG:boto:Params: {} 2023-09-29T22:23:32.359 DEBUG:boto:Token: None 2023-09-29T22:23:32.359 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8?versionId=null 2023-09-29T22:23:32.359 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ= 2023-09-29T22:23:32.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C+VyPOlo5j/g0qc9nuknROMp/UQ=', 'Content-Length': '0'} 2023-09-29T22:23:32.362 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001e95ebe40f0c7e93-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.406 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.407 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.407 DEBUG:boto:path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.407 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.407 DEBUG:boto:Method: GET 2023-09-29T22:23:32.407 DEBUG:boto:Path: /hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.407 DEBUG:boto:Data: 2023-09-29T22:23:32.407 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.407 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.407 DEBUG:boto:Params: {} 2023-09-29T22:23:32.408 DEBUG:boto:Token: None 2023-09-29T22:23:32.408 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000441f93525984d82c-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.413 DEBUG:boto:b'hcnphz-48obj81000falseobj82023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.413 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.413 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.413 DEBUG:boto:path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.413 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.414 DEBUG:boto:Method: GET 2023-09-29T22:23:32.414 DEBUG:boto:Path: /hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.414 DEBUG:boto:Data: 2023-09-29T22:23:32.414 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.414 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.414 DEBUG:boto:Params: {} 2023-09-29T22:23:32.414 DEBUG:boto:Token: None 2023-09-29T22:23:32.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c8ff11389fa61bc-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.419 DEBUG:boto:b'hcnphz-48obj81000falseobj82023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.419 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.419 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.420 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.420 DEBUG:boto:Path: /hcnphz-48/obj8 2023-09-29T22:23:32.420 DEBUG:boto:Data: 2023-09-29T22:23:32.420 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.420 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.420 DEBUG:boto:Params: {} 2023-09-29T22:23:32.420 DEBUG:boto:Token: None 2023-09-29T22:23:32.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8 2023-09-29T22:23:32.421 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bpqNQibqHjpUy3Pg5YeJbFBWhK0= 2023-09-29T22:23:32.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bpqNQibqHjpUy3Pg5YeJbFBWhK0=', 'Content-Length': '0'} 2023-09-29T22:23:32.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000da6ec16dd102a457-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.423 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.423 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.423 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.423 DEBUG:boto:Path: /hcnphz-48/obj8 2023-09-29T22:23:32.424 DEBUG:boto:Data: 2023-09-29T22:23:32.424 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.424 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.424 DEBUG:boto:Params: {} 2023-09-29T22:23:32.424 DEBUG:boto:Token: None 2023-09-29T22:23:32.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8 2023-09-29T22:23:32.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bpqNQibqHjpUy3Pg5YeJbFBWhK0= 2023-09-29T22:23:32.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bpqNQibqHjpUy3Pg5YeJbFBWhK0=', 'Content-Length': '0'} 2023-09-29T22:23:32.426 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d2079a2b1a19f95a-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:23:32.427 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.427 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.427 DEBUG:boto:Method: GET 2023-09-29T22:23:32.427 DEBUG:boto:Path: /hcnphz-48/obj8 2023-09-29T22:23:32.427 DEBUG:boto:Data: 2023-09-29T22:23:32.427 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.428 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.428 DEBUG:boto:Params: {} 2023-09-29T22:23:32.428 DEBUG:boto:Token: None 2023-09-29T22:23:32.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8 2023-09-29T22:23:32.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rzvXHRpa7QMTMk5GuvdT0Wk+wkA= 2023-09-29T22:23:32.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rzvXHRpa7QMTMk5GuvdT0Wk+wkA=', 'Content-Length': '0'} 2023-09-29T22:23:32.431 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-48:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.13'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007299499d90fc9dd1-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.478 DEBUG:boto:path=/hcnphz-48/obj8 2023-09-29T22:23:32.479 DEBUG:boto:auth_path=/hcnphz-48/obj8 2023-09-29T22:23:32.479 DEBUG:boto:Method: GET 2023-09-29T22:23:32.479 DEBUG:boto:Path: /hcnphz-48/obj8 2023-09-29T22:23:32.479 DEBUG:boto:Data: 2023-09-29T22:23:32.479 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.479 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.479 DEBUG:boto:Params: {} 2023-09-29T22:23:32.479 DEBUG:boto:Token: None 2023-09-29T22:23:32.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/obj8 2023-09-29T22:23:32.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rzvXHRpa7QMTMk5GuvdT0Wk+wkA= 2023-09-29T22:23:32.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rzvXHRpa7QMTMk5GuvdT0Wk+wkA=', 'Content-Length': '0'} 2023-09-29T22:23:32.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f2a8472fc115eeea-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.483 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.483 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.483 DEBUG:boto:path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.483 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.483 DEBUG:boto:Method: GET 2023-09-29T22:23:32.484 DEBUG:boto:Path: /hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.484 DEBUG:boto:Data: 2023-09-29T22:23:32.484 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.484 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.484 DEBUG:boto:Params: {} 2023-09-29T22:23:32.484 DEBUG:boto:Token: None 2023-09-29T22:23:32.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d757031445bdcecc-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.489 DEBUG:boto:b'hcnphz-48obj81000falseobj82023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.490 DEBUG:boto:path=/hcnphz-48/ 2023-09-29T22:23:32.490 DEBUG:boto:auth_path=/hcnphz-48/ 2023-09-29T22:23:32.490 DEBUG:boto:path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.490 DEBUG:boto:auth_path=/hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.490 DEBUG:boto:Method: GET 2023-09-29T22:23:32.490 DEBUG:boto:Path: /hcnphz-48/?prefix=obj8 2023-09-29T22:23:32.491 DEBUG:boto:Data: 2023-09-29T22:23:32.491 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.491 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.491 DEBUG:boto:Params: {} 2023-09-29T22:23:32.491 DEBUG:boto:Token: None 2023-09-29T22:23:32.491 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-48/ 2023-09-29T22:23:32.491 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4= 2023-09-29T22:23:32.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:64q9u/F7I5cPafZywxZ6gcrlnD4=', 'Content-Length': '0'} 2023-09-29T22:23:32.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d98ced5480324790-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.538 DEBUG:boto:b'hcnphz-48obj81000falseobj82023-09-29T22:21:20.192Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.539 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-48 zones={a2, a1} 2023-09-29T22:23:32.539 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-49 2023-09-29T22:23:32.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:32.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:32.539 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-49 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:32.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:32.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:32.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:32.649+0000 7f99febfaf40 1 bucket sync caught up with source: 2023-09-29T22:23:32.654 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.33.6, 00000000002.121.1, 00000000004.26.6, 00000000002.43.1, 00000000004.20.6, 00000000002.44.1, 00000000002.18.1, 00000000004.27.6, 00000000002.31.1, 00000000002.122.1, 00000000002.29.1] 2023-09-29T22:23:32.654 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.33.6, 00000000002.121.1, 00000000004.26.6, 00000000002.43.1, 00000000004.20.6, 00000000002.44.1, 00000000002.18.1, 00000000004.27.6, 00000000002.31.1, 00000000002.122.1, 00000000002.29.1] 2023-09-29T22:23:32.654 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:32.649+0000 7f99febfaf40 0 bucket checkpoint complete 2023-09-29T22:23:32.678 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-49 2023-09-29T22:23:32.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:32.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-49', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:32.678 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-49 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:32.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:32.726 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:32.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:32.789+0000 7fae84725f40 1 bucket sync caught up with source: 2023-09-29T22:23:32.794 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000004.240.1, 00000000002.169.1, 00000000004.183.1, 00000000002.98.1, 00000000004.40.1, 00000000002.106.1, 00000000002.41.1, 00000000004.63.1, 00000000002.154.1, 00000000002.177.1, 00000000002.68.1] 2023-09-29T22:23:32.794 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000004.240.1, 00000000002.169.1, 00000000004.183.1, 00000000002.98.1, 00000000004.40.1, 00000000002.106.1, 00000000002.41.1, 00000000004.63.1, 00000000002.154.1, 00000000002.177.1, 00000000002.68.1] 2023-09-29T22:23:32.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:32.789+0000 7fae84725f40 0 bucket checkpoint complete 2023-09-29T22:23:32.819 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-49 zones={a2, a1} 2023-09-29T22:23:32.819 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.819 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.819 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.819 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:23:32.819 DEBUG:boto:Data: 2023-09-29T22:23:32.819 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.820 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.820 DEBUG:boto:Params: {} 2023-09-29T22:23:32.820 DEBUG:boto:Token: None 2023-09-29T22:23:32.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/ 2023-09-29T22:23:32.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HxfnWohr3wSy3rvtzcW2Onk/SvU= 2023-09-29T22:23:32.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HxfnWohr3wSy3rvtzcW2Onk/SvU=', 'Content-Length': '0'} 2023-09-29T22:23:32.824 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b669c376d99b2db2-0065174e64-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.824 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.824 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.824 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.824 DEBUG:boto:Path: /hcnphz-49/ 2023-09-29T22:23:32.824 DEBUG:boto:Data: 2023-09-29T22:23:32.824 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.825 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.825 DEBUG:boto:Params: {} 2023-09-29T22:23:32.825 DEBUG:boto:Token: None 2023-09-29T22:23:32.825 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/ 2023-09-29T22:23:32.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HxfnWohr3wSy3rvtzcW2Onk/SvU= 2023-09-29T22:23:32.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HxfnWohr3wSy3rvtzcW2Onk/SvU=', 'Content-Length': '0'} 2023-09-29T22:23:32.828 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d76a9c1fd2d2103-0065174e64-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.829 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:23:32.829 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.829 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.829 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.829 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.829 DEBUG:boto:Method: GET 2023-09-29T22:23:32.829 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.829 DEBUG:boto:Data: 2023-09-29T22:23:32.829 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.830 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.830 DEBUG:boto:Params: {} 2023-09-29T22:23:32.830 DEBUG:boto:Token: None 2023-09-29T22:23:32.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/?versions 2023-09-29T22:23:32.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU= 2023-09-29T22:23:32.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU=', 'Content-Length': '0'} 2023-09-29T22:23:32.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039bddc5cd8a60868-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.834 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:23:32.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:23:32.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:23:32.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:23:32.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:23:32.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:23:32.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:23:32.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:23:32.836 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:23:32.836 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.836 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.836 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.836 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.836 DEBUG:boto:Method: GET 2023-09-29T22:23:32.836 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.837 DEBUG:boto:Data: 2023-09-29T22:23:32.837 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.837 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.837 DEBUG:boto:Params: {} 2023-09-29T22:23:32.837 DEBUG:boto:Token: None 2023-09-29T22:23:32.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/?versions 2023-09-29T22:23:32.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU= 2023-09-29T22:23:32.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU=', 'Content-Length': '0'} 2023-09-29T22:23:32.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006252337e7c1e4b29-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.842 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2023-09-29T22:23:32.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2023-09-29T22:23:32.843 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.843 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.844 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.844 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.844 DEBUG:boto:Method: GET 2023-09-29T22:23:32.844 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.844 DEBUG:boto:Data: 2023-09-29T22:23:32.844 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.844 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.844 DEBUG:boto:Params: {} 2023-09-29T22:23:32.844 DEBUG:boto:Token: None 2023-09-29T22:23:32.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/?versions 2023-09-29T22:23:32.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU= 2023-09-29T22:23:32.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU=', 'Content-Length': '0'} 2023-09-29T22:23:32.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3c83822ec9def4f-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.849 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.849 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.849 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.849 DEBUG:boto:path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.850 DEBUG:boto:auth_path=/hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.850 DEBUG:boto:Method: GET 2023-09-29T22:23:32.850 DEBUG:boto:Path: /hcnphz-49/?versions&max-keys=999 2023-09-29T22:23:32.850 DEBUG:boto:Data: 2023-09-29T22:23:32.850 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.850 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.850 DEBUG:boto:Params: {} 2023-09-29T22:23:32.850 DEBUG:boto:Token: None 2023-09-29T22:23:32.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/?versions 2023-09-29T22:23:32.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU= 2023-09-29T22:23:32.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V2Yn691EYuN5KCxADMmBenoIQkU=', 'Content-Length': '0'} 2023-09-29T22:23:32.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0de97f7881f5d3f-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.855 DEBUG:boto:b'hcnphz-49999falseobj1nulltrue2023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj2nulltrue2023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj3nulltrue2023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj4nulltrue2023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj5nulltrue2023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj6nulltrue2023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj7nulltrue2023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormalobj8nulltrue2023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:23:32.856 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.856 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.856 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.856 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.856 DEBUG:boto:Method: GET 2023-09-29T22:23:32.856 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.857 DEBUG:boto:Data: 2023-09-29T22:23:32.857 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.857 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.857 DEBUG:boto:Params: {} 2023-09-29T22:23:32.857 DEBUG:boto:Token: None 2023-09-29T22:23:32.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I= 2023-09-29T22:23:32.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I=', 'Content-Length': '0'} 2023-09-29T22:23:32.860 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b121c53f7bc7fe1-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.902 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.903 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.903 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.903 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.903 DEBUG:boto:Method: GET 2023-09-29T22:23:32.903 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.903 DEBUG:boto:Data: 2023-09-29T22:23:32.903 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.903 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.903 DEBUG:boto:Params: {} 2023-09-29T22:23:32.904 DEBUG:boto:Token: None 2023-09-29T22:23:32.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I= 2023-09-29T22:23:32.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I=', 'Content-Length': '0'} 2023-09-29T22:23:32.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007a8cfd60926b6557-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.907 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.907 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.907 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.907 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.907 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.908 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.908 DEBUG:boto:Data: 2023-09-29T22:23:32.908 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.908 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.908 DEBUG:boto:Params: {} 2023-09-29T22:23:32.908 DEBUG:boto:Token: None 2023-09-29T22:23:32.908 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VlsqHoVD6Q/N1StOU11daZUSC3g= 2023-09-29T22:23:32.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VlsqHoVD6Q/N1StOU11daZUSC3g=', 'Content-Length': '0'} 2023-09-29T22:23:32.911 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000129df15734279ce4-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.912 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.912 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.912 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.912 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.912 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.912 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.912 DEBUG:boto:Data: 2023-09-29T22:23:32.912 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.912 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.912 DEBUG:boto:Params: {} 2023-09-29T22:23:32.913 DEBUG:boto:Token: None 2023-09-29T22:23:32.913 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VlsqHoVD6Q/N1StOU11daZUSC3g= 2023-09-29T22:23:32.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VlsqHoVD6Q/N1StOU11daZUSC3g=', 'Content-Length': '0'} 2023-09-29T22:23:32.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000040c5029f894ba9a3-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:23:32.916 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.916 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.916 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.916 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.916 DEBUG:boto:Method: GET 2023-09-29T22:23:32.916 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.917 DEBUG:boto:Data: 2023-09-29T22:23:32.917 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.917 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.917 DEBUG:boto:Params: {} 2023-09-29T22:23:32.917 DEBUG:boto:Token: None 2023-09-29T22:23:32.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I= 2023-09-29T22:23:32.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I=', 'Content-Length': '0'} 2023-09-29T22:23:32.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000959c74da4f09024-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.920 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.921 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.921 DEBUG:boto:path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.921 DEBUG:boto:auth_path=/hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.921 DEBUG:boto:Method: GET 2023-09-29T22:23:32.921 DEBUG:boto:Path: /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.921 DEBUG:boto:Data: 2023-09-29T22:23:32.921 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.921 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.921 DEBUG:boto:Params: {} 2023-09-29T22:23:32.922 DEBUG:boto:Token: None 2023-09-29T22:23:32.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1?versionId=null 2023-09-29T22:23:32.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I= 2023-09-29T22:23:32.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+vE5Jh1a/3cEX4xFi1vKsyzAR7I=', 'Content-Length': '0'} 2023-09-29T22:23:32.925 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000046a63097f6e92300-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.966 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.967 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.967 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:32.967 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:32.967 DEBUG:boto:Method: GET 2023-09-29T22:23:32.967 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:23:32.967 DEBUG:boto:Data: 2023-09-29T22:23:32.967 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.967 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.967 DEBUG:boto:Params: {} 2023-09-29T22:23:32.968 DEBUG:boto:Token: None 2023-09-29T22:23:32.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/ 2023-09-29T22:23:32.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YSV0ppC4pfb52TaQBu9/iJuXI+o= 2023-09-29T22:23:32.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YSV0ppC4pfb52TaQBu9/iJuXI+o=', 'Content-Length': '0'} 2023-09-29T22:23:32.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000509cd1398c664067-0065174e64-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.973 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.973 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:32.973 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:32.974 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:32.974 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:32.974 DEBUG:boto:Method: GET 2023-09-29T22:23:32.974 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:23:32.974 DEBUG:boto:Data: 2023-09-29T22:23:32.974 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.974 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.974 DEBUG:boto:Params: {} 2023-09-29T22:23:32.974 DEBUG:boto:Token: None 2023-09-29T22:23:32.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/ 2023-09-29T22:23:32.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YSV0ppC4pfb52TaQBu9/iJuXI+o= 2023-09-29T22:23:32.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YSV0ppC4pfb52TaQBu9/iJuXI+o=', 'Content-Length': '0'} 2023-09-29T22:23:32.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d9515be7739769e-0065174e64-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.979 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:32.980 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.980 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.980 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.980 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:23:32.980 DEBUG:boto:Data: 2023-09-29T22:23:32.980 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.980 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.981 DEBUG:boto:Params: {} 2023-09-29T22:23:32.981 DEBUG:boto:Token: None 2023-09-29T22:23:32.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1 2023-09-29T22:23:32.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MWzcvWZXe90CiPSEYGNcrIB2GbM= 2023-09-29T22:23:32.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MWzcvWZXe90CiPSEYGNcrIB2GbM=', 'Content-Length': '0'} 2023-09-29T22:23:32.983 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000075a6e3f868e7d041-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.984 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.984 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.984 DEBUG:boto:Method: HEAD 2023-09-29T22:23:32.984 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:23:32.984 DEBUG:boto:Data: 2023-09-29T22:23:32.984 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:32.985 DEBUG:boto:Port: 8000 2023-09-29T22:23:32.985 DEBUG:boto:Params: {} 2023-09-29T22:23:32.985 DEBUG:boto:Token: None 2023-09-29T22:23:32.985 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1 2023-09-29T22:23:32.985 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MWzcvWZXe90CiPSEYGNcrIB2GbM= 2023-09-29T22:23:32.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MWzcvWZXe90CiPSEYGNcrIB2GbM=', 'Content-Length': '0'} 2023-09-29T22:23:32.987 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8f78f3525da90e2-0065174e64-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:32.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2023-09-29T22:23:32.987 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:32.987 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:32.988 DEBUG:boto:Method: GET 2023-09-29T22:23:32.988 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:23:32.988 DEBUG:boto:Data: 2023-09-29T22:23:32.988 DEBUG:boto:Headers: {} 2023-09-29T22:23:32.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:32.988 DEBUG:boto:Port: 8001 2023-09-29T22:23:32.988 DEBUG:boto:Params: {} 2023-09-29T22:23:32.988 DEBUG:boto:Token: None 2023-09-29T22:23:32.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:32 GMT /hcnphz-49/obj1 2023-09-29T22:23:32.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:smOmOvz0Wnc153fcDS0sCtAEDZM= 2023-09-29T22:23:32.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:smOmOvz0Wnc153fcDS0sCtAEDZM=', 'Content-Length': '0'} 2023-09-29T22:23:32.991 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001fe673fc25f9a8c-0065174e64-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.034 DEBUG:boto:path=/hcnphz-49/obj1 2023-09-29T22:23:33.035 DEBUG:boto:auth_path=/hcnphz-49/obj1 2023-09-29T22:23:33.035 DEBUG:boto:Method: GET 2023-09-29T22:23:33.035 DEBUG:boto:Path: /hcnphz-49/obj1 2023-09-29T22:23:33.035 DEBUG:boto:Data: 2023-09-29T22:23:33.035 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.035 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.035 DEBUG:boto:Params: {} 2023-09-29T22:23:33.035 DEBUG:boto:Token: None 2023-09-29T22:23:33.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj1 2023-09-29T22:23:33.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H+IfFKsrBtI2TmqycnUY298qpnU= 2023-09-29T22:23:33.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H+IfFKsrBtI2TmqycnUY298qpnU=', 'Content-Length': '0'} 2023-09-29T22:23:33.039 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d9ff2f2498e6ffcb-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.039 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.039 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.039 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:33.039 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:33.040 DEBUG:boto:Method: GET 2023-09-29T22:23:33.040 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:23:33.040 DEBUG:boto:Data: 2023-09-29T22:23:33.040 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.040 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.040 DEBUG:boto:Params: {} 2023-09-29T22:23:33.040 DEBUG:boto:Token: None 2023-09-29T22:23:33.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc9bc2a539a4c996-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.045 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.046 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.046 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.046 DEBUG:boto:path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:33.046 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj1 2023-09-29T22:23:33.046 DEBUG:boto:Method: GET 2023-09-29T22:23:33.046 DEBUG:boto:Path: /hcnphz-49/?prefix=obj1 2023-09-29T22:23:33.046 DEBUG:boto:Data: 2023-09-29T22:23:33.046 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.047 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.047 DEBUG:boto:Params: {} 2023-09-29T22:23:33.047 DEBUG:boto:Token: None 2023-09-29T22:23:33.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.051 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb0487114070811d-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.094 DEBUG:boto:b'hcnphz-49obj11000falseobj12023-09-29T22:18:16.394Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:23:33.095 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.095 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.095 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.095 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.095 DEBUG:boto:Method: GET 2023-09-29T22:23:33.095 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.095 DEBUG:boto:Data: 2023-09-29T22:23:33.095 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.096 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.096 DEBUG:boto:Params: {} 2023-09-29T22:23:33.096 DEBUG:boto:Token: None 2023-09-29T22:23:33.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.096 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4= 2023-09-29T22:23:33.096 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4=', 'Content-Length': '0'} 2023-09-29T22:23:33.099 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0006d1cf7f7d10b-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.099 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.099 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.099 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.099 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.099 DEBUG:boto:Method: GET 2023-09-29T22:23:33.100 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.100 DEBUG:boto:Data: 2023-09-29T22:23:33.100 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.100 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.100 DEBUG:boto:Params: {} 2023-09-29T22:23:33.100 DEBUG:boto:Token: None 2023-09-29T22:23:33.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4= 2023-09-29T22:23:33.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4=', 'Content-Length': '0'} 2023-09-29T22:23:33.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000099219689f3c08a81-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.104 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.104 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.104 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.104 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.104 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.104 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.104 DEBUG:boto:Data: 2023-09-29T22:23:33.105 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.105 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.105 DEBUG:boto:Params: {} 2023-09-29T22:23:33.105 DEBUG:boto:Token: None 2023-09-29T22:23:33.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ak1K+wHrfldJ+pYtcuQ7LcgqPI4= 2023-09-29T22:23:33.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ak1K+wHrfldJ+pYtcuQ7LcgqPI4=', 'Content-Length': '0'} 2023-09-29T22:23:33.108 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ddde8120fe7a1ab-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.108 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.108 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.108 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.108 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.108 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.108 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.109 DEBUG:boto:Data: 2023-09-29T22:23:33.109 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.109 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.109 DEBUG:boto:Params: {} 2023-09-29T22:23:33.109 DEBUG:boto:Token: None 2023-09-29T22:23:33.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ak1K+wHrfldJ+pYtcuQ7LcgqPI4= 2023-09-29T22:23:33.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ak1K+wHrfldJ+pYtcuQ7LcgqPI4=', 'Content-Length': '0'} 2023-09-29T22:23:33.112 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a84234a3041559e5-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:23:33.112 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.112 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.113 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.113 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.113 DEBUG:boto:Method: GET 2023-09-29T22:23:33.113 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.113 DEBUG:boto:Data: 2023-09-29T22:23:33.113 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.113 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.113 DEBUG:boto:Params: {} 2023-09-29T22:23:33.114 DEBUG:boto:Token: None 2023-09-29T22:23:33.114 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4= 2023-09-29T22:23:33.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4=', 'Content-Length': '0'} 2023-09-29T22:23:33.117 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e0da3515a841117d-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.158 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.159 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.159 DEBUG:boto:path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.159 DEBUG:boto:auth_path=/hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.159 DEBUG:boto:Method: GET 2023-09-29T22:23:33.159 DEBUG:boto:Path: /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.159 DEBUG:boto:Data: 2023-09-29T22:23:33.159 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.159 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.159 DEBUG:boto:Params: {} 2023-09-29T22:23:33.160 DEBUG:boto:Token: None 2023-09-29T22:23:33.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2?versionId=null 2023-09-29T22:23:33.160 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4= 2023-09-29T22:23:33.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3gwMshXva215ij4qjxs5PeAdJh4=', 'Content-Length': '0'} 2023-09-29T22:23:33.163 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b82f7e278b4b7808-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.163 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.163 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.163 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.164 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.164 DEBUG:boto:Method: GET 2023-09-29T22:23:33.164 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.164 DEBUG:boto:Data: 2023-09-29T22:23:33.164 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.164 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.164 DEBUG:boto:Params: {} 2023-09-29T22:23:33.164 DEBUG:boto:Token: None 2023-09-29T22:23:33.165 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.165 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006df86202e3f596e9-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.170 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.170 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.170 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.170 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.170 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.171 DEBUG:boto:Method: GET 2023-09-29T22:23:33.171 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.171 DEBUG:boto:Data: 2023-09-29T22:23:33.171 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.171 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.171 DEBUG:boto:Params: {} 2023-09-29T22:23:33.171 DEBUG:boto:Token: None 2023-09-29T22:23:33.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe5035c10eb5656e-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.218 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.219 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.219 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.219 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.219 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:23:33.219 DEBUG:boto:Data: 2023-09-29T22:23:33.219 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.220 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.220 DEBUG:boto:Params: {} 2023-09-29T22:23:33.220 DEBUG:boto:Token: None 2023-09-29T22:23:33.220 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2 2023-09-29T22:23:33.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TsRgPm9oRRg4mQcOhoBcBeXQa68= 2023-09-29T22:23:33.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TsRgPm9oRRg4mQcOhoBcBeXQa68=', 'Content-Length': '0'} 2023-09-29T22:23:33.223 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000905609fbbffd3670-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.223 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.223 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.223 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.223 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:23:33.223 DEBUG:boto:Data: 2023-09-29T22:23:33.224 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.224 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.224 DEBUG:boto:Params: {} 2023-09-29T22:23:33.224 DEBUG:boto:Token: None 2023-09-29T22:23:33.224 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2 2023-09-29T22:23:33.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TsRgPm9oRRg4mQcOhoBcBeXQa68= 2023-09-29T22:23:33.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TsRgPm9oRRg4mQcOhoBcBeXQa68=', 'Content-Length': '0'} 2023-09-29T22:23:33.227 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f88ae054e8f428a5-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2023-09-29T22:23:33.227 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.228 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.228 DEBUG:boto:Method: GET 2023-09-29T22:23:33.228 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:23:33.228 DEBUG:boto:Data: 2023-09-29T22:23:33.228 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.228 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.228 DEBUG:boto:Params: {} 2023-09-29T22:23:33.228 DEBUG:boto:Token: None 2023-09-29T22:23:33.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2 2023-09-29T22:23:33.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DQAsl7KT1L46T1gzW6voAkcMGmk= 2023-09-29T22:23:33.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQAsl7KT1L46T1gzW6voAkcMGmk=', 'Content-Length': '0'} 2023-09-29T22:23:33.231 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f4f11feab10c4850-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.232 DEBUG:boto:path=/hcnphz-49/obj2 2023-09-29T22:23:33.232 DEBUG:boto:auth_path=/hcnphz-49/obj2 2023-09-29T22:23:33.232 DEBUG:boto:Method: GET 2023-09-29T22:23:33.232 DEBUG:boto:Path: /hcnphz-49/obj2 2023-09-29T22:23:33.232 DEBUG:boto:Data: 2023-09-29T22:23:33.232 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.232 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.232 DEBUG:boto:Params: {} 2023-09-29T22:23:33.233 DEBUG:boto:Token: None 2023-09-29T22:23:33.233 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj2 2023-09-29T22:23:33.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DQAsl7KT1L46T1gzW6voAkcMGmk= 2023-09-29T22:23:33.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQAsl7KT1L46T1gzW6voAkcMGmk=', 'Content-Length': '0'} 2023-09-29T22:23:33.235 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f9f9a69a026ec90f-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.236 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.236 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.236 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.236 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.236 DEBUG:boto:Method: GET 2023-09-29T22:23:33.236 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.236 DEBUG:boto:Data: 2023-09-29T22:23:33.236 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.237 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.237 DEBUG:boto:Params: {} 2023-09-29T22:23:33.237 DEBUG:boto:Token: None 2023-09-29T22:23:33.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.237 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000481c485c51354f27-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.282 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.283 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.283 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.283 DEBUG:boto:path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.283 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.283 DEBUG:boto:Method: GET 2023-09-29T22:23:33.283 DEBUG:boto:Path: /hcnphz-49/?prefix=obj2 2023-09-29T22:23:33.283 DEBUG:boto:Data: 2023-09-29T22:23:33.283 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.284 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.284 DEBUG:boto:Params: {} 2023-09-29T22:23:33.284 DEBUG:boto:Token: None 2023-09-29T22:23:33.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de5a24f1e6f9f884-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.288 DEBUG:boto:b'hcnphz-49obj21000falseobj22023-09-29T22:18:16.446Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:23:33.289 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.289 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.289 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.289 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.289 DEBUG:boto:Method: GET 2023-09-29T22:23:33.290 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.290 DEBUG:boto:Data: 2023-09-29T22:23:33.290 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.290 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.290 DEBUG:boto:Params: {} 2023-09-29T22:23:33.290 DEBUG:boto:Token: None 2023-09-29T22:23:33.290 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc= 2023-09-29T22:23:33.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc=', 'Content-Length': '0'} 2023-09-29T22:23:33.293 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000042ec450abfa19fc6-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.293 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.294 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.294 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.294 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.294 DEBUG:boto:Method: GET 2023-09-29T22:23:33.294 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.294 DEBUG:boto:Data: 2023-09-29T22:23:33.294 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.295 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.295 DEBUG:boto:Params: {} 2023-09-29T22:23:33.295 DEBUG:boto:Token: None 2023-09-29T22:23:33.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc= 2023-09-29T22:23:33.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc=', 'Content-Length': '0'} 2023-09-29T22:23:33.298 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e899d420f407f543-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.338 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.338 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.339 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.339 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.339 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.339 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.339 DEBUG:boto:Data: 2023-09-29T22:23:33.339 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.339 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.339 DEBUG:boto:Params: {} 2023-09-29T22:23:33.339 DEBUG:boto:Token: None 2023-09-29T22:23:33.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rBG2BarvMT9eUNnamAMK+hY6Yrg= 2023-09-29T22:23:33.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rBG2BarvMT9eUNnamAMK+hY6Yrg=', 'Content-Length': '0'} 2023-09-29T22:23:33.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b8de1d9c995e6f8d-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.343 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.343 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.343 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.343 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.343 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.343 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.343 DEBUG:boto:Data: 2023-09-29T22:23:33.343 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.344 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.344 DEBUG:boto:Params: {} 2023-09-29T22:23:33.344 DEBUG:boto:Token: None 2023-09-29T22:23:33.344 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rBG2BarvMT9eUNnamAMK+hY6Yrg= 2023-09-29T22:23:33.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rBG2BarvMT9eUNnamAMK+hY6Yrg=', 'Content-Length': '0'} 2023-09-29T22:23:33.347 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003de5fffa1679f0b3-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:23:33.347 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.347 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.347 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.347 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.347 DEBUG:boto:Method: GET 2023-09-29T22:23:33.348 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.348 DEBUG:boto:Data: 2023-09-29T22:23:33.348 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.348 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.348 DEBUG:boto:Params: {} 2023-09-29T22:23:33.348 DEBUG:boto:Token: None 2023-09-29T22:23:33.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc= 2023-09-29T22:23:33.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc=', 'Content-Length': '0'} 2023-09-29T22:23:33.353 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000029fd8f3c63980b5b-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.353 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.354 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.354 DEBUG:boto:path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.354 DEBUG:boto:auth_path=/hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.354 DEBUG:boto:Method: GET 2023-09-29T22:23:33.354 DEBUG:boto:Path: /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.354 DEBUG:boto:Data: 2023-09-29T22:23:33.354 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.354 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.354 DEBUG:boto:Params: {} 2023-09-29T22:23:33.355 DEBUG:boto:Token: None 2023-09-29T22:23:33.355 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3?versionId=null 2023-09-29T22:23:33.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc= 2023-09-29T22:23:33.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tmlRbrIiTMPvOR30o6Ez6MQbzRc=', 'Content-Length': '0'} 2023-09-29T22:23:33.358 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e8303df44da984d4-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.358 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.358 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.358 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.359 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.359 DEBUG:boto:Method: GET 2023-09-29T22:23:33.359 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.359 DEBUG:boto:Data: 2023-09-29T22:23:33.359 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.359 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.359 DEBUG:boto:Params: {} 2023-09-29T22:23:33.359 DEBUG:boto:Token: None 2023-09-29T22:23:33.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080aedb98acecd31e-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.406 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.407 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.407 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.407 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.407 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.407 DEBUG:boto:Method: GET 2023-09-29T22:23:33.407 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.407 DEBUG:boto:Data: 2023-09-29T22:23:33.408 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.408 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.408 DEBUG:boto:Params: {} 2023-09-29T22:23:33.408 DEBUG:boto:Token: None 2023-09-29T22:23:33.408 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.413 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000655b878922d1f8cf-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.413 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.414 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.414 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.414 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.414 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:23:33.414 DEBUG:boto:Data: 2023-09-29T22:23:33.414 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.415 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.415 DEBUG:boto:Params: {} 2023-09-29T22:23:33.415 DEBUG:boto:Token: None 2023-09-29T22:23:33.415 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3 2023-09-29T22:23:33.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7iHCLwnyG0BNYGhoUsc1oTNrJnA= 2023-09-29T22:23:33.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7iHCLwnyG0BNYGhoUsc1oTNrJnA=', 'Content-Length': '0'} 2023-09-29T22:23:33.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000234bc46979696a7f-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.418 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.418 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.418 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.419 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:23:33.419 DEBUG:boto:Data: 2023-09-29T22:23:33.419 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.419 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.419 DEBUG:boto:Params: {} 2023-09-29T22:23:33.420 DEBUG:boto:Token: None 2023-09-29T22:23:33.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3 2023-09-29T22:23:33.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7iHCLwnyG0BNYGhoUsc1oTNrJnA= 2023-09-29T22:23:33.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7iHCLwnyG0BNYGhoUsc1oTNrJnA=', 'Content-Length': '0'} 2023-09-29T22:23:33.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d88e0a03c8325fc6-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2023-09-29T22:23:33.423 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.423 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.424 DEBUG:boto:Method: GET 2023-09-29T22:23:33.424 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:23:33.424 DEBUG:boto:Data: 2023-09-29T22:23:33.424 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.424 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.424 DEBUG:boto:Params: {} 2023-09-29T22:23:33.425 DEBUG:boto:Token: None 2023-09-29T22:23:33.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3 2023-09-29T22:23:33.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OV0erZYdHwQ9vI4MGf8AwzfXhhA= 2023-09-29T22:23:33.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OV0erZYdHwQ9vI4MGf8AwzfXhhA=', 'Content-Length': '0'} 2023-09-29T22:23:33.427 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e9eca3bcb6348c11-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.428 DEBUG:boto:path=/hcnphz-49/obj3 2023-09-29T22:23:33.428 DEBUG:boto:auth_path=/hcnphz-49/obj3 2023-09-29T22:23:33.428 DEBUG:boto:Method: GET 2023-09-29T22:23:33.428 DEBUG:boto:Path: /hcnphz-49/obj3 2023-09-29T22:23:33.428 DEBUG:boto:Data: 2023-09-29T22:23:33.428 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.429 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.429 DEBUG:boto:Params: {} 2023-09-29T22:23:33.429 DEBUG:boto:Token: None 2023-09-29T22:23:33.429 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj3 2023-09-29T22:23:33.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OV0erZYdHwQ9vI4MGf8AwzfXhhA= 2023-09-29T22:23:33.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OV0erZYdHwQ9vI4MGf8AwzfXhhA=', 'Content-Length': '0'} 2023-09-29T22:23:33.432 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f6a866ed795d184b-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.478 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.478 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.479 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.479 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.479 DEBUG:boto:Method: GET 2023-09-29T22:23:33.479 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.479 DEBUG:boto:Data: 2023-09-29T22:23:33.479 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.479 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.479 DEBUG:boto:Params: {} 2023-09-29T22:23:33.480 DEBUG:boto:Token: None 2023-09-29T22:23:33.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061d7c7a6b38b963b-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.485 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.485 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.485 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.485 DEBUG:boto:path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.486 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.486 DEBUG:boto:Method: GET 2023-09-29T22:23:33.486 DEBUG:boto:Path: /hcnphz-49/?prefix=obj3 2023-09-29T22:23:33.486 DEBUG:boto:Data: 2023-09-29T22:23:33.486 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.486 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.486 DEBUG:boto:Params: {} 2023-09-29T22:23:33.486 DEBUG:boto:Token: None 2023-09-29T22:23:33.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bc513159fbd8d8f-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.491 DEBUG:boto:b'hcnphz-49obj31000falseobj32023-09-29T22:18:16.464Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:23:33.491 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.492 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.492 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.492 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.492 DEBUG:boto:Method: GET 2023-09-29T22:23:33.492 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.492 DEBUG:boto:Data: 2023-09-29T22:23:33.492 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.492 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.492 DEBUG:boto:Params: {} 2023-09-29T22:23:33.493 DEBUG:boto:Token: None 2023-09-29T22:23:33.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54= 2023-09-29T22:23:33.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54=', 'Content-Length': '0'} 2023-09-29T22:23:33.496 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000fbca4bdfcd516328-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.496 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.496 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.496 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.496 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.496 DEBUG:boto:Method: GET 2023-09-29T22:23:33.496 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.496 DEBUG:boto:Data: 2023-09-29T22:23:33.497 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.497 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.497 DEBUG:boto:Params: {} 2023-09-29T22:23:33.497 DEBUG:boto:Token: None 2023-09-29T22:23:33.497 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54= 2023-09-29T22:23:33.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54=', 'Content-Length': '0'} 2023-09-29T22:23:33.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5e695b37344d7bf-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.500 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.500 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.500 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.500 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.500 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.500 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.500 DEBUG:boto:Data: 2023-09-29T22:23:33.500 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.500 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.501 DEBUG:boto:Params: {} 2023-09-29T22:23:33.501 DEBUG:boto:Token: None 2023-09-29T22:23:33.501 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NxXroWeV9m+j8X8COb9bKg5PYPE= 2023-09-29T22:23:33.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NxXroWeV9m+j8X8COb9bKg5PYPE=', 'Content-Length': '0'} 2023-09-29T22:23:33.503 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f8bce3ec0f3f777e-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.503 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.503 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.504 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.504 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.504 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.504 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.504 DEBUG:boto:Data: 2023-09-29T22:23:33.504 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.504 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.504 DEBUG:boto:Params: {} 2023-09-29T22:23:33.504 DEBUG:boto:Token: None 2023-09-29T22:23:33.505 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NxXroWeV9m+j8X8COb9bKg5PYPE= 2023-09-29T22:23:33.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NxXroWeV9m+j8X8COb9bKg5PYPE=', 'Content-Length': '0'} 2023-09-29T22:23:33.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000036773675bf54db46-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:23:33.507 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.508 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.508 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.508 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.508 DEBUG:boto:Method: GET 2023-09-29T22:23:33.508 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.508 DEBUG:boto:Data: 2023-09-29T22:23:33.508 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.508 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.508 DEBUG:boto:Params: {} 2023-09-29T22:23:33.509 DEBUG:boto:Token: None 2023-09-29T22:23:33.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54= 2023-09-29T22:23:33.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54=', 'Content-Length': '0'} 2023-09-29T22:23:33.511 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c07eb7a0fc9b8bd4-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.554 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.554 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.555 DEBUG:boto:path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.555 DEBUG:boto:auth_path=/hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.555 DEBUG:boto:Method: GET 2023-09-29T22:23:33.555 DEBUG:boto:Path: /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.555 DEBUG:boto:Data: 2023-09-29T22:23:33.555 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.555 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.555 DEBUG:boto:Params: {} 2023-09-29T22:23:33.556 DEBUG:boto:Token: None 2023-09-29T22:23:33.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4?versionId=null 2023-09-29T22:23:33.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54= 2023-09-29T22:23:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jJ3l7OorVwxtoNKSiDNUQEO7X54=', 'Content-Length': '0'} 2023-09-29T22:23:33.559 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ffef99e0970476b3-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.559 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.559 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.559 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.559 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.559 DEBUG:boto:Method: GET 2023-09-29T22:23:33.559 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.560 DEBUG:boto:Data: 2023-09-29T22:23:33.560 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.560 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.560 DEBUG:boto:Params: {} 2023-09-29T22:23:33.560 DEBUG:boto:Token: None 2023-09-29T22:23:33.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9143dc8db7b88ba-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.566 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.566 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.566 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.566 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.566 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.567 DEBUG:boto:Method: GET 2023-09-29T22:23:33.567 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.567 DEBUG:boto:Data: 2023-09-29T22:23:33.567 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.567 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.567 DEBUG:boto:Params: {} 2023-09-29T22:23:33.567 DEBUG:boto:Token: None 2023-09-29T22:23:33.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000484652bee0acf8f1-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.572 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.572 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.572 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.572 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.572 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:23:33.572 DEBUG:boto:Data: 2023-09-29T22:23:33.573 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.573 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.573 DEBUG:boto:Params: {} 2023-09-29T22:23:33.573 DEBUG:boto:Token: None 2023-09-29T22:23:33.573 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4 2023-09-29T22:23:33.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oWgYvhzTSoDwkWfNCeQY4AcFqfI= 2023-09-29T22:23:33.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oWgYvhzTSoDwkWfNCeQY4AcFqfI=', 'Content-Length': '0'} 2023-09-29T22:23:33.575 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c5c1709ba68d7ac8-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.576 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.576 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.576 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.576 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:23:33.576 DEBUG:boto:Data: 2023-09-29T22:23:33.576 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.576 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.576 DEBUG:boto:Params: {} 2023-09-29T22:23:33.576 DEBUG:boto:Token: None 2023-09-29T22:23:33.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4 2023-09-29T22:23:33.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oWgYvhzTSoDwkWfNCeQY4AcFqfI= 2023-09-29T22:23:33.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oWgYvhzTSoDwkWfNCeQY4AcFqfI=', 'Content-Length': '0'} 2023-09-29T22:23:33.579 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000062dc99d1767c58b5-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2023-09-29T22:23:33.579 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.580 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.580 DEBUG:boto:Method: GET 2023-09-29T22:23:33.580 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:23:33.580 DEBUG:boto:Data: 2023-09-29T22:23:33.580 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.580 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.580 DEBUG:boto:Params: {} 2023-09-29T22:23:33.580 DEBUG:boto:Token: None 2023-09-29T22:23:33.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4 2023-09-29T22:23:33.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:boMNMHNNqLxmpuhsWiYKohRBr3o= 2023-09-29T22:23:33.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:boMNMHNNqLxmpuhsWiYKohRBr3o=', 'Content-Length': '0'} 2023-09-29T22:23:33.583 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a7cff87c6e4842cf-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.626 DEBUG:boto:path=/hcnphz-49/obj4 2023-09-29T22:23:33.626 DEBUG:boto:auth_path=/hcnphz-49/obj4 2023-09-29T22:23:33.627 DEBUG:boto:Method: GET 2023-09-29T22:23:33.627 DEBUG:boto:Path: /hcnphz-49/obj4 2023-09-29T22:23:33.627 DEBUG:boto:Data: 2023-09-29T22:23:33.627 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.627 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.627 DEBUG:boto:Params: {} 2023-09-29T22:23:33.627 DEBUG:boto:Token: None 2023-09-29T22:23:33.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj4 2023-09-29T22:23:33.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:boMNMHNNqLxmpuhsWiYKohRBr3o= 2023-09-29T22:23:33.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:boMNMHNNqLxmpuhsWiYKohRBr3o=', 'Content-Length': '0'} 2023-09-29T22:23:33.630 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:18:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009fc5efeef9011783-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.631 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.631 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.631 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.631 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.631 DEBUG:boto:Method: GET 2023-09-29T22:23:33.631 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.631 DEBUG:boto:Data: 2023-09-29T22:23:33.631 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.632 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.632 DEBUG:boto:Params: {} 2023-09-29T22:23:33.632 DEBUG:boto:Token: None 2023-09-29T22:23:33.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b1555005eab89ba-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.637 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.637 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.637 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.637 DEBUG:boto:path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.638 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.638 DEBUG:boto:Method: GET 2023-09-29T22:23:33.638 DEBUG:boto:Path: /hcnphz-49/?prefix=obj4 2023-09-29T22:23:33.638 DEBUG:boto:Data: 2023-09-29T22:23:33.638 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.638 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.638 DEBUG:boto:Params: {} 2023-09-29T22:23:33.638 DEBUG:boto:Token: None 2023-09-29T22:23:33.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048aed01e77091577-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.682 DEBUG:boto:b'hcnphz-49obj41000falseobj42023-09-29T22:18:16.513Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:23:33.683 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.683 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.683 DEBUG:boto:path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.683 DEBUG:boto:auth_path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.684 DEBUG:boto:Method: GET 2023-09-29T22:23:33.684 DEBUG:boto:Path: /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.684 DEBUG:boto:Data: 2023-09-29T22:23:33.684 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.684 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.685 DEBUG:boto:Params: {} 2023-09-29T22:23:33.685 DEBUG:boto:Token: None 2023-09-29T22:23:33.685 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.685 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8= 2023-09-29T22:23:33.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8=', 'Content-Length': '0'} 2023-09-29T22:23:33.688 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009af54d2251635df6-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.689 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.689 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.689 DEBUG:boto:path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.689 DEBUG:boto:auth_path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.689 DEBUG:boto:Method: GET 2023-09-29T22:23:33.690 DEBUG:boto:Path: /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.690 DEBUG:boto:Data: 2023-09-29T22:23:33.690 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.690 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.690 DEBUG:boto:Params: {} 2023-09-29T22:23:33.691 DEBUG:boto:Token: None 2023-09-29T22:23:33.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8= 2023-09-29T22:23:33.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8=', 'Content-Length': '0'} 2023-09-29T22:23:33.694 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000072a150532ebd6ed0-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.695 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.695 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.695 DEBUG:boto:path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.695 DEBUG:boto:auth_path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.695 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.695 DEBUG:boto:Path: /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.696 DEBUG:boto:Data: 2023-09-29T22:23:33.696 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.696 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.696 DEBUG:boto:Params: {} 2023-09-29T22:23:33.696 DEBUG:boto:Token: None 2023-09-29T22:23:33.697 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.697 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kHQFh2+30iKoyyFB6Mdqgh0iUnI= 2023-09-29T22:23:33.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kHQFh2+30iKoyyFB6Mdqgh0iUnI=', 'Content-Length': '0'} 2023-09-29T22:23:33.699 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000077f964fb4cc5a681-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.700 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.700 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.700 DEBUG:boto:path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.700 DEBUG:boto:auth_path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.700 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.701 DEBUG:boto:Path: /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.701 DEBUG:boto:Data: 2023-09-29T22:23:33.701 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.701 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.701 DEBUG:boto:Params: {} 2023-09-29T22:23:33.702 DEBUG:boto:Token: None 2023-09-29T22:23:33.702 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kHQFh2+30iKoyyFB6Mdqgh0iUnI= 2023-09-29T22:23:33.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kHQFh2+30iKoyyFB6Mdqgh0iUnI=', 'Content-Length': '0'} 2023-09-29T22:23:33.705 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000d55abbcc1a57184d-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:23:33.706 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.706 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.706 DEBUG:boto:path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.706 DEBUG:boto:auth_path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.706 DEBUG:boto:Method: GET 2023-09-29T22:23:33.707 DEBUG:boto:Path: /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.707 DEBUG:boto:Data: 2023-09-29T22:23:33.707 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.707 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.707 DEBUG:boto:Params: {} 2023-09-29T22:23:33.708 DEBUG:boto:Token: None 2023-09-29T22:23:33.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8= 2023-09-29T22:23:33.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8=', 'Content-Length': '0'} 2023-09-29T22:23:33.711 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007bdfa759414af3ad-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.758 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.758 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.759 DEBUG:boto:path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.759 DEBUG:boto:auth_path=/hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.759 DEBUG:boto:Method: GET 2023-09-29T22:23:33.759 DEBUG:boto:Path: /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.759 DEBUG:boto:Data: 2023-09-29T22:23:33.759 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.760 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.760 DEBUG:boto:Params: {} 2023-09-29T22:23:33.760 DEBUG:boto:Token: None 2023-09-29T22:23:33.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5?versionId=null 2023-09-29T22:23:33.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8= 2023-09-29T22:23:33.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/xllme3zPykly5Af9KA2LtH5rW8=', 'Content-Length': '0'} 2023-09-29T22:23:33.763 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009de7f5bc57ceeb80-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.764 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.765 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.765 DEBUG:boto:path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.765 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.766 DEBUG:boto:Method: GET 2023-09-29T22:23:33.766 DEBUG:boto:Path: /hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.766 DEBUG:boto:Data: 2023-09-29T22:23:33.766 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.766 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.766 DEBUG:boto:Params: {} 2023-09-29T22:23:33.766 DEBUG:boto:Token: None 2023-09-29T22:23:33.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.767 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd211859db20991-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.772 DEBUG:boto:b'hcnphz-49obj51000falseobj52023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.772 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.772 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.772 DEBUG:boto:path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.772 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.773 DEBUG:boto:Method: GET 2023-09-29T22:23:33.773 DEBUG:boto:Path: /hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.773 DEBUG:boto:Data: 2023-09-29T22:23:33.773 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.774 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.774 DEBUG:boto:Params: {} 2023-09-29T22:23:33.774 DEBUG:boto:Token: None 2023-09-29T22:23:33.774 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddb490bf7c4749c1-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.826 DEBUG:boto:b'hcnphz-49obj51000falseobj52023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.827 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.827 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.827 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.827 DEBUG:boto:Path: /hcnphz-49/obj5 2023-09-29T22:23:33.827 DEBUG:boto:Data: 2023-09-29T22:23:33.827 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.827 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.828 DEBUG:boto:Params: {} 2023-09-29T22:23:33.828 DEBUG:boto:Token: None 2023-09-29T22:23:33.828 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5 2023-09-29T22:23:33.828 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:umKtFD7eoLyzxCIuPI0umwX8ef4= 2023-09-29T22:23:33.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:umKtFD7eoLyzxCIuPI0umwX8ef4=', 'Content-Length': '0'} 2023-09-29T22:23:33.831 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000070580eac88f83ed7-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.831 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.831 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.831 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.832 DEBUG:boto:Path: /hcnphz-49/obj5 2023-09-29T22:23:33.832 DEBUG:boto:Data: 2023-09-29T22:23:33.832 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.832 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.832 DEBUG:boto:Params: {} 2023-09-29T22:23:33.832 DEBUG:boto:Token: None 2023-09-29T22:23:33.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5 2023-09-29T22:23:33.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:umKtFD7eoLyzxCIuPI0umwX8ef4= 2023-09-29T22:23:33.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:umKtFD7eoLyzxCIuPI0umwX8ef4=', 'Content-Length': '0'} 2023-09-29T22:23:33.835 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005fb1c0479ec7d591-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2023-09-29T22:23:33.836 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.836 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.836 DEBUG:boto:Method: GET 2023-09-29T22:23:33.837 DEBUG:boto:Path: /hcnphz-49/obj5 2023-09-29T22:23:33.837 DEBUG:boto:Data: 2023-09-29T22:23:33.837 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.837 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.837 DEBUG:boto:Params: {} 2023-09-29T22:23:33.838 DEBUG:boto:Token: None 2023-09-29T22:23:33.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5 2023-09-29T22:23:33.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UvpHoARVG0Bmy0UH9OXxOYRK9bI= 2023-09-29T22:23:33.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UvpHoARVG0Bmy0UH9OXxOYRK9bI=', 'Content-Length': '0'} 2023-09-29T22:23:33.841 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003c2db1b4c725be58-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.842 DEBUG:boto:path=/hcnphz-49/obj5 2023-09-29T22:23:33.842 DEBUG:boto:auth_path=/hcnphz-49/obj5 2023-09-29T22:23:33.842 DEBUG:boto:Method: GET 2023-09-29T22:23:33.842 DEBUG:boto:Path: /hcnphz-49/obj5 2023-09-29T22:23:33.842 DEBUG:boto:Data: 2023-09-29T22:23:33.842 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.843 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.843 DEBUG:boto:Params: {} 2023-09-29T22:23:33.843 DEBUG:boto:Token: None 2023-09-29T22:23:33.843 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj5 2023-09-29T22:23:33.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UvpHoARVG0Bmy0UH9OXxOYRK9bI= 2023-09-29T22:23:33.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UvpHoARVG0Bmy0UH9OXxOYRK9bI=', 'Content-Length': '0'} 2023-09-29T22:23:33.846 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3313b0e5d75f0a0-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.847 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.847 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.847 DEBUG:boto:path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.847 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.847 DEBUG:boto:Method: GET 2023-09-29T22:23:33.848 DEBUG:boto:Path: /hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.848 DEBUG:boto:Data: 2023-09-29T22:23:33.848 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.848 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.848 DEBUG:boto:Params: {} 2023-09-29T22:23:33.849 DEBUG:boto:Token: None 2023-09-29T22:23:33.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.853 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3af158a88a4df08-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.854 DEBUG:boto:b'hcnphz-49obj51000falseobj52023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.854 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.854 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.854 DEBUG:boto:path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.854 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.855 DEBUG:boto:Method: GET 2023-09-29T22:23:33.855 DEBUG:boto:Path: /hcnphz-49/?prefix=obj5 2023-09-29T22:23:33.855 DEBUG:boto:Data: 2023-09-29T22:23:33.855 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.856 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.856 DEBUG:boto:Params: {} 2023-09-29T22:23:33.856 DEBUG:boto:Token: None 2023-09-29T22:23:33.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.861 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048077fe828a0018a-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.902 DEBUG:boto:b'hcnphz-49obj51000falseobj52023-09-29T22:21:20.242Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:23:33.903 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.903 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.903 DEBUG:boto:path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.904 DEBUG:boto:auth_path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.904 DEBUG:boto:Method: GET 2023-09-29T22:23:33.904 DEBUG:boto:Path: /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.904 DEBUG:boto:Data: 2023-09-29T22:23:33.904 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.905 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.905 DEBUG:boto:Params: {} 2023-09-29T22:23:33.905 DEBUG:boto:Token: None 2023-09-29T22:23:33.905 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY= 2023-09-29T22:23:33.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY=', 'Content-Length': '0'} 2023-09-29T22:23:33.908 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ab7c4c17ace0c261-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.909 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.909 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.909 DEBUG:boto:path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.909 DEBUG:boto:auth_path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.910 DEBUG:boto:Method: GET 2023-09-29T22:23:33.910 DEBUG:boto:Path: /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.910 DEBUG:boto:Data: 2023-09-29T22:23:33.910 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.910 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.911 DEBUG:boto:Params: {} 2023-09-29T22:23:33.911 DEBUG:boto:Token: None 2023-09-29T22:23:33.911 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.911 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY= 2023-09-29T22:23:33.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY=', 'Content-Length': '0'} 2023-09-29T22:23:33.915 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000047f5c8df12d77dde-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.915 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.915 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.915 DEBUG:boto:path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.915 DEBUG:boto:auth_path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.916 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.916 DEBUG:boto:Path: /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.916 DEBUG:boto:Data: 2023-09-29T22:23:33.916 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.917 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.917 DEBUG:boto:Params: {} 2023-09-29T22:23:33.917 DEBUG:boto:Token: None 2023-09-29T22:23:33.917 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B5ThLjol70TKetmGbJHbMZB7NqI= 2023-09-29T22:23:33.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B5ThLjol70TKetmGbJHbMZB7NqI=', 'Content-Length': '0'} 2023-09-29T22:23:33.920 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004b889f993f2ac14c-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.920 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.921 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.921 DEBUG:boto:path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.921 DEBUG:boto:auth_path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.921 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.921 DEBUG:boto:Path: /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.921 DEBUG:boto:Data: 2023-09-29T22:23:33.922 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.922 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.922 DEBUG:boto:Params: {} 2023-09-29T22:23:33.922 DEBUG:boto:Token: None 2023-09-29T22:23:33.922 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B5ThLjol70TKetmGbJHbMZB7NqI= 2023-09-29T22:23:33.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B5ThLjol70TKetmGbJHbMZB7NqI=', 'Content-Length': '0'} 2023-09-29T22:23:33.926 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a5f3a673269dada7-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:23:33.926 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.926 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.927 DEBUG:boto:path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.927 DEBUG:boto:auth_path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.927 DEBUG:boto:Method: GET 2023-09-29T22:23:33.927 DEBUG:boto:Path: /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.927 DEBUG:boto:Data: 2023-09-29T22:23:33.928 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.928 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.928 DEBUG:boto:Params: {} 2023-09-29T22:23:33.928 DEBUG:boto:Token: None 2023-09-29T22:23:33.929 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.929 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY= 2023-09-29T22:23:33.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY=', 'Content-Length': '0'} 2023-09-29T22:23:33.931 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000054c5e20843df0d82-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.932 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.932 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.932 DEBUG:boto:path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.932 DEBUG:boto:auth_path=/hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.932 DEBUG:boto:Method: GET 2023-09-29T22:23:33.933 DEBUG:boto:Path: /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.933 DEBUG:boto:Data: 2023-09-29T22:23:33.933 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.933 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.933 DEBUG:boto:Params: {} 2023-09-29T22:23:33.934 DEBUG:boto:Token: None 2023-09-29T22:23:33.934 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6?versionId=null 2023-09-29T22:23:33.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY= 2023-09-29T22:23:33.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:62nfjuiiKjY5wJGwyDxhiYi41XY=', 'Content-Length': '0'} 2023-09-29T22:23:33.937 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008edeafc24051722a-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.937 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.938 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.938 DEBUG:boto:path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:33.938 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:33.938 DEBUG:boto:Method: GET 2023-09-29T22:23:33.938 DEBUG:boto:Path: /hcnphz-49/?prefix=obj6 2023-09-29T22:23:33.938 DEBUG:boto:Data: 2023-09-29T22:23:33.938 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.938 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.939 DEBUG:boto:Params: {} 2023-09-29T22:23:33.939 DEBUG:boto:Token: None 2023-09-29T22:23:33.939 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.939 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.944 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005c61e830ce22270-0065174e65-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.986 DEBUG:boto:b'hcnphz-49obj61000falseobj62023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.987 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:33.987 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:33.987 DEBUG:boto:path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:33.987 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:33.987 DEBUG:boto:Method: GET 2023-09-29T22:23:33.987 DEBUG:boto:Path: /hcnphz-49/?prefix=obj6 2023-09-29T22:23:33.988 DEBUG:boto:Data: 2023-09-29T22:23:33.988 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:33.988 DEBUG:boto:Port: 8000 2023-09-29T22:23:33.988 DEBUG:boto:Params: {} 2023-09-29T22:23:33.989 DEBUG:boto:Token: None 2023-09-29T22:23:33.989 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/ 2023-09-29T22:23:33.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY= 2023-09-29T22:23:33.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1UlMGGVoLuoFHayZGXuoS0hvxY=', 'Content-Length': '0'} 2023-09-29T22:23:33.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dbdd1099ad698fd-0065174e65-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.994 DEBUG:boto:b'hcnphz-49obj61000falseobj62023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:33.995 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:33.995 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:33.995 DEBUG:boto:Method: HEAD 2023-09-29T22:23:33.995 DEBUG:boto:Path: /hcnphz-49/obj6 2023-09-29T22:23:33.995 DEBUG:boto:Data: 2023-09-29T22:23:33.995 DEBUG:boto:Headers: {} 2023-09-29T22:23:33.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:33.996 DEBUG:boto:Port: 8001 2023-09-29T22:23:33.996 DEBUG:boto:Params: {} 2023-09-29T22:23:33.996 DEBUG:boto:Token: None 2023-09-29T22:23:33.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:33 GMT /hcnphz-49/obj6 2023-09-29T22:23:33.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZfrQFkIWBPGwBe9Olm98l6S7e0A= 2023-09-29T22:23:33.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZfrQFkIWBPGwBe9Olm98l6S7e0A=', 'Content-Length': '0'} 2023-09-29T22:23:33.999 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a3d3e3dac8db61e4-0065174e65-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:33.999 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:34.000 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:34.000 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.000 DEBUG:boto:Path: /hcnphz-49/obj6 2023-09-29T22:23:34.000 DEBUG:boto:Data: 2023-09-29T22:23:34.000 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.001 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.001 DEBUG:boto:Params: {} 2023-09-29T22:23:34.001 DEBUG:boto:Token: None 2023-09-29T22:23:34.001 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj6 2023-09-29T22:23:34.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4e5PB1moyKThtTf4mYdPdoxVaiU= 2023-09-29T22:23:34.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4e5PB1moyKThtTf4mYdPdoxVaiU=', 'Content-Length': '0'} 2023-09-29T22:23:34.004 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000f53521391160581-0065174e65-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2023-09-29T22:23:34.005 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:34.005 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:34.005 DEBUG:boto:Method: GET 2023-09-29T22:23:34.005 DEBUG:boto:Path: /hcnphz-49/obj6 2023-09-29T22:23:34.005 DEBUG:boto:Data: 2023-09-29T22:23:34.005 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.006 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.006 DEBUG:boto:Params: {} 2023-09-29T22:23:34.006 DEBUG:boto:Token: None 2023-09-29T22:23:34.006 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj6 2023-09-29T22:23:34.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kA9QDkuAawZHcTveW7eKEow33Jo= 2023-09-29T22:23:34.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kA9QDkuAawZHcTveW7eKEow33Jo=', 'Content-Length': '0'} 2023-09-29T22:23:34.009 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c8af652cc37c62b3-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.009 DEBUG:boto:path=/hcnphz-49/obj6 2023-09-29T22:23:34.010 DEBUG:boto:auth_path=/hcnphz-49/obj6 2023-09-29T22:23:34.010 DEBUG:boto:Method: GET 2023-09-29T22:23:34.010 DEBUG:boto:Path: /hcnphz-49/obj6 2023-09-29T22:23:34.010 DEBUG:boto:Data: 2023-09-29T22:23:34.010 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.011 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.011 DEBUG:boto:Params: {} 2023-09-29T22:23:34.011 DEBUG:boto:Token: None 2023-09-29T22:23:34.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj6 2023-09-29T22:23:34.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kA9QDkuAawZHcTveW7eKEow33Jo= 2023-09-29T22:23:34.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kA9QDkuAawZHcTveW7eKEow33Jo=', 'Content-Length': '0'} 2023-09-29T22:23:34.015 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b3e784b4f9d7c10b-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.058 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.059 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.059 DEBUG:boto:path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:34.059 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:34.059 DEBUG:boto:Method: GET 2023-09-29T22:23:34.059 DEBUG:boto:Path: /hcnphz-49/?prefix=obj6 2023-09-29T22:23:34.060 DEBUG:boto:Data: 2023-09-29T22:23:34.060 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.060 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.060 DEBUG:boto:Params: {} 2023-09-29T22:23:34.061 DEBUG:boto:Token: None 2023-09-29T22:23:34.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.066 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a0cf0aa66e58fa0-0065174e66-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.066 DEBUG:boto:b'hcnphz-49obj61000falseobj62023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.067 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.067 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.067 DEBUG:boto:path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:34.067 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj6 2023-09-29T22:23:34.068 DEBUG:boto:Method: GET 2023-09-29T22:23:34.068 DEBUG:boto:Path: /hcnphz-49/?prefix=obj6 2023-09-29T22:23:34.068 DEBUG:boto:Data: 2023-09-29T22:23:34.068 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.068 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.068 DEBUG:boto:Params: {} 2023-09-29T22:23:34.069 DEBUG:boto:Token: None 2023-09-29T22:23:34.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077b4835ff969dcc7-0065174e66-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.078 DEBUG:boto:b'hcnphz-49obj61000falseobj62023-09-29T22:21:20.294Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:23:34.079 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.079 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.079 DEBUG:boto:path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.079 DEBUG:boto:auth_path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.079 DEBUG:boto:Method: GET 2023-09-29T22:23:34.079 DEBUG:boto:Path: /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.080 DEBUG:boto:Data: 2023-09-29T22:23:34.080 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.080 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.080 DEBUG:boto:Params: {} 2023-09-29T22:23:34.080 DEBUG:boto:Token: None 2023-09-29T22:23:34.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk= 2023-09-29T22:23:34.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk=', 'Content-Length': '0'} 2023-09-29T22:23:34.084 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dd39230faf78c608-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.126 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.127 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.127 DEBUG:boto:path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.127 DEBUG:boto:auth_path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.127 DEBUG:boto:Method: GET 2023-09-29T22:23:34.127 DEBUG:boto:Path: /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.127 DEBUG:boto:Data: 2023-09-29T22:23:34.127 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.128 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.128 DEBUG:boto:Params: {} 2023-09-29T22:23:34.128 DEBUG:boto:Token: None 2023-09-29T22:23:34.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk= 2023-09-29T22:23:34.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk=', 'Content-Length': '0'} 2023-09-29T22:23:34.131 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000017686c083377ed9e-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.132 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.132 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.132 DEBUG:boto:path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.132 DEBUG:boto:auth_path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.132 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.132 DEBUG:boto:Path: /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.132 DEBUG:boto:Data: 2023-09-29T22:23:34.132 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.132 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.133 DEBUG:boto:Params: {} 2023-09-29T22:23:34.133 DEBUG:boto:Token: None 2023-09-29T22:23:34.133 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FRSjmckixwBDO7Lr73/BkaOCMlw= 2023-09-29T22:23:34.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FRSjmckixwBDO7Lr73/BkaOCMlw=', 'Content-Length': '0'} 2023-09-29T22:23:34.136 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000738202aaeaf40527-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.136 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.136 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.136 DEBUG:boto:path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.136 DEBUG:boto:auth_path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.136 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.137 DEBUG:boto:Path: /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.137 DEBUG:boto:Data: 2023-09-29T22:23:34.137 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.137 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.137 DEBUG:boto:Params: {} 2023-09-29T22:23:34.137 DEBUG:boto:Token: None 2023-09-29T22:23:34.137 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FRSjmckixwBDO7Lr73/BkaOCMlw= 2023-09-29T22:23:34.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FRSjmckixwBDO7Lr73/BkaOCMlw=', 'Content-Length': '0'} 2023-09-29T22:23:34.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000026dc609e0125f62-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:23:34.140 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.140 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.141 DEBUG:boto:path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.141 DEBUG:boto:auth_path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.141 DEBUG:boto:Method: GET 2023-09-29T22:23:34.141 DEBUG:boto:Path: /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.141 DEBUG:boto:Data: 2023-09-29T22:23:34.141 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.141 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.141 DEBUG:boto:Params: {} 2023-09-29T22:23:34.142 DEBUG:boto:Token: None 2023-09-29T22:23:34.142 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.142 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk= 2023-09-29T22:23:34.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk=', 'Content-Length': '0'} 2023-09-29T22:23:34.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000009e74f9df29b0072f-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.145 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.145 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.145 DEBUG:boto:path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.145 DEBUG:boto:auth_path=/hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.145 DEBUG:boto:Method: GET 2023-09-29T22:23:34.145 DEBUG:boto:Path: /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.145 DEBUG:boto:Data: 2023-09-29T22:23:34.146 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.146 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.146 DEBUG:boto:Params: {} 2023-09-29T22:23:34.146 DEBUG:boto:Token: None 2023-09-29T22:23:34.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7?versionId=null 2023-09-29T22:23:34.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk= 2023-09-29T22:23:34.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1HIbS1wtQkyoek+aO/ykUTg9Mkk=', 'Content-Length': '0'} 2023-09-29T22:23:34.149 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a306053b346d6876-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.191 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.191 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.191 DEBUG:boto:path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.191 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.192 DEBUG:boto:Method: GET 2023-09-29T22:23:34.192 DEBUG:boto:Path: /hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.192 DEBUG:boto:Data: 2023-09-29T22:23:34.192 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.192 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.192 DEBUG:boto:Params: {} 2023-09-29T22:23:34.192 DEBUG:boto:Token: None 2023-09-29T22:23:34.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef730605935855d6-0065174e66-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.198 DEBUG:boto:b'hcnphz-49obj71000falseobj72023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.198 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.199 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.199 DEBUG:boto:path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.199 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.199 DEBUG:boto:Method: GET 2023-09-29T22:23:34.199 DEBUG:boto:Path: /hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.199 DEBUG:boto:Data: 2023-09-29T22:23:34.199 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.199 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.199 DEBUG:boto:Params: {} 2023-09-29T22:23:34.200 DEBUG:boto:Token: None 2023-09-29T22:23:34.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aedd2cde39e752b9-0065174e66-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.205 DEBUG:boto:b'hcnphz-49obj71000falseobj72023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.206 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.206 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.206 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.206 DEBUG:boto:Path: /hcnphz-49/obj7 2023-09-29T22:23:34.206 DEBUG:boto:Data: 2023-09-29T22:23:34.206 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.206 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.206 DEBUG:boto:Params: {} 2023-09-29T22:23:34.207 DEBUG:boto:Token: None 2023-09-29T22:23:34.207 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7 2023-09-29T22:23:34.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6p6CJWZ30kCvAs6BJsAEx2EoAzQ= 2023-09-29T22:23:34.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6p6CJWZ30kCvAs6BJsAEx2EoAzQ=', 'Content-Length': '0'} 2023-09-29T22:23:34.209 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001cf9b3f827468eb4-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.210 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.210 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.210 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.210 DEBUG:boto:Path: /hcnphz-49/obj7 2023-09-29T22:23:34.210 DEBUG:boto:Data: 2023-09-29T22:23:34.210 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.210 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.210 DEBUG:boto:Params: {} 2023-09-29T22:23:34.211 DEBUG:boto:Token: None 2023-09-29T22:23:34.211 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7 2023-09-29T22:23:34.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6p6CJWZ30kCvAs6BJsAEx2EoAzQ= 2023-09-29T22:23:34.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6p6CJWZ30kCvAs6BJsAEx2EoAzQ=', 'Content-Length': '0'} 2023-09-29T22:23:34.214 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004ae7bf383dd8419d-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2023-09-29T22:23:34.214 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.214 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.214 DEBUG:boto:Method: GET 2023-09-29T22:23:34.214 DEBUG:boto:Path: /hcnphz-49/obj7 2023-09-29T22:23:34.214 DEBUG:boto:Data: 2023-09-29T22:23:34.215 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.215 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.215 DEBUG:boto:Params: {} 2023-09-29T22:23:34.215 DEBUG:boto:Token: None 2023-09-29T22:23:34.215 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7 2023-09-29T22:23:34.215 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P2J87gX4pbB/RMj2bZoFp4Pgx4Q= 2023-09-29T22:23:34.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P2J87gX4pbB/RMj2bZoFp4Pgx4Q=', 'Content-Length': '0'} 2023-09-29T22:23:34.218 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007f5d70cb21bc8836-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.258 DEBUG:boto:path=/hcnphz-49/obj7 2023-09-29T22:23:34.259 DEBUG:boto:auth_path=/hcnphz-49/obj7 2023-09-29T22:23:34.259 DEBUG:boto:Method: GET 2023-09-29T22:23:34.259 DEBUG:boto:Path: /hcnphz-49/obj7 2023-09-29T22:23:34.259 DEBUG:boto:Data: 2023-09-29T22:23:34.259 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.259 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.259 DEBUG:boto:Params: {} 2023-09-29T22:23:34.260 DEBUG:boto:Token: None 2023-09-29T22:23:34.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj7 2023-09-29T22:23:34.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P2J87gX4pbB/RMj2bZoFp4Pgx4Q= 2023-09-29T22:23:34.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P2J87gX4pbB/RMj2bZoFp4Pgx4Q=', 'Content-Length': '0'} 2023-09-29T22:23:34.263 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000024bfe0da4f873601-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.263 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.263 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.264 DEBUG:boto:path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.264 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.264 DEBUG:boto:Method: GET 2023-09-29T22:23:34.264 DEBUG:boto:Path: /hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.264 DEBUG:boto:Data: 2023-09-29T22:23:34.264 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.264 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.264 DEBUG:boto:Params: {} 2023-09-29T22:23:34.264 DEBUG:boto:Token: None 2023-09-29T22:23:34.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.270 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009345b5d7be367b0b-0065174e66-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.270 DEBUG:boto:b'hcnphz-49obj71000falseobj72023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.270 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.270 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.270 DEBUG:boto:path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.271 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.271 DEBUG:boto:Method: GET 2023-09-29T22:23:34.271 DEBUG:boto:Path: /hcnphz-49/?prefix=obj7 2023-09-29T22:23:34.271 DEBUG:boto:Data: 2023-09-29T22:23:34.271 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.271 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.271 DEBUG:boto:Params: {} 2023-09-29T22:23:34.271 DEBUG:boto:Token: None 2023-09-29T22:23:34.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a18b2ed5eb357d1a-0065174e66-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.318 DEBUG:boto:b'hcnphz-49obj71000falseobj72023-09-29T22:21:20.310Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:23:34.319 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.319 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.319 DEBUG:boto:path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.319 DEBUG:boto:auth_path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.319 DEBUG:boto:Method: GET 2023-09-29T22:23:34.319 DEBUG:boto:Path: /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.319 DEBUG:boto:Data: 2023-09-29T22:23:34.319 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.320 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.320 DEBUG:boto:Params: {} 2023-09-29T22:23:34.320 DEBUG:boto:Token: None 2023-09-29T22:23:34.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA= 2023-09-29T22:23:34.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA=', 'Content-Length': '0'} 2023-09-29T22:23:34.323 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004dfa510021cfc094-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.323 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.324 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.324 DEBUG:boto:path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.324 DEBUG:boto:auth_path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.324 DEBUG:boto:Method: GET 2023-09-29T22:23:34.324 DEBUG:boto:Path: /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.324 DEBUG:boto:Data: 2023-09-29T22:23:34.324 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.324 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.324 DEBUG:boto:Params: {} 2023-09-29T22:23:34.325 DEBUG:boto:Token: None 2023-09-29T22:23:34.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA= 2023-09-29T22:23:34.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA=', 'Content-Length': '0'} 2023-09-29T22:23:34.328 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f1d3b2463b6f6611-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.329 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.329 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.329 DEBUG:boto:path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.329 DEBUG:boto:auth_path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.329 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.329 DEBUG:boto:Path: /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.329 DEBUG:boto:Data: 2023-09-29T22:23:34.330 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.330 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.330 DEBUG:boto:Params: {} 2023-09-29T22:23:34.330 DEBUG:boto:Token: None 2023-09-29T22:23:34.330 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:svr8FSPgwQ01Rdyq+er3m9z1TFs= 2023-09-29T22:23:34.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:svr8FSPgwQ01Rdyq+er3m9z1TFs=', 'Content-Length': '0'} 2023-09-29T22:23:34.333 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000001448da5f24ce549-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.333 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.333 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.333 DEBUG:boto:path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.333 DEBUG:boto:auth_path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.334 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.334 DEBUG:boto:Path: /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.334 DEBUG:boto:Data: 2023-09-29T22:23:34.334 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.334 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.334 DEBUG:boto:Params: {} 2023-09-29T22:23:34.334 DEBUG:boto:Token: None 2023-09-29T22:23:34.334 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.335 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:svr8FSPgwQ01Rdyq+er3m9z1TFs= 2023-09-29T22:23:34.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:svr8FSPgwQ01Rdyq+er3m9z1TFs=', 'Content-Length': '0'} 2023-09-29T22:23:34.337 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003910df29a113bdf9-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:23:34.338 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.338 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.338 DEBUG:boto:path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.338 DEBUG:boto:auth_path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.338 DEBUG:boto:Method: GET 2023-09-29T22:23:34.338 DEBUG:boto:Path: /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.338 DEBUG:boto:Data: 2023-09-29T22:23:34.338 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.339 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.339 DEBUG:boto:Params: {} 2023-09-29T22:23:34.339 DEBUG:boto:Token: None 2023-09-29T22:23:34.339 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.339 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA= 2023-09-29T22:23:34.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA=', 'Content-Length': '0'} 2023-09-29T22:23:34.342 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cf33bf5f5fee883-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.342 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.342 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.342 DEBUG:boto:path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.342 DEBUG:boto:auth_path=/hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.342 DEBUG:boto:Method: GET 2023-09-29T22:23:34.343 DEBUG:boto:Path: /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.343 DEBUG:boto:Data: 2023-09-29T22:23:34.343 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.343 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.343 DEBUG:boto:Params: {} 2023-09-29T22:23:34.343 DEBUG:boto:Token: None 2023-09-29T22:23:34.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8?versionId=null 2023-09-29T22:23:34.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA= 2023-09-29T22:23:34.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YWoIQl4gfYwyzwOxidDQJFnEsOA=', 'Content-Length': '0'} 2023-09-29T22:23:34.346 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000ca3cf3aba6529201-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.346 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.347 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.347 DEBUG:boto:path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.347 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.347 DEBUG:boto:Method: GET 2023-09-29T22:23:34.347 DEBUG:boto:Path: /hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.347 DEBUG:boto:Data: 2023-09-29T22:23:34.347 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.347 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.347 DEBUG:boto:Params: {} 2023-09-29T22:23:34.348 DEBUG:boto:Token: None 2023-09-29T22:23:34.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095fa26c9dc2af9c5-0065174e66-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.353 DEBUG:boto:b'hcnphz-49obj81000falseobj82023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.353 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.353 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.353 DEBUG:boto:path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.353 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.354 DEBUG:boto:Method: GET 2023-09-29T22:23:34.354 DEBUG:boto:Path: /hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.354 DEBUG:boto:Data: 2023-09-29T22:23:34.354 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.354 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.354 DEBUG:boto:Params: {} 2023-09-29T22:23:34.354 DEBUG:boto:Token: None 2023-09-29T22:23:34.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e06888d05c40034-0065174e66-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.402 DEBUG:boto:b'hcnphz-49obj81000falseobj82023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.403 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.403 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.403 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.403 DEBUG:boto:Path: /hcnphz-49/obj8 2023-09-29T22:23:34.403 DEBUG:boto:Data: 2023-09-29T22:23:34.403 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.403 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.403 DEBUG:boto:Params: {} 2023-09-29T22:23:34.404 DEBUG:boto:Token: None 2023-09-29T22:23:34.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8 2023-09-29T22:23:34.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qnqAS6Y9OKma5/f1a1W/N7BBOG4= 2023-09-29T22:23:34.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qnqAS6Y9OKma5/f1a1W/N7BBOG4=', 'Content-Length': '0'} 2023-09-29T22:23:34.407 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000039956ca01bc84642-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.407 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.407 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.407 DEBUG:boto:Method: HEAD 2023-09-29T22:23:34.407 DEBUG:boto:Path: /hcnphz-49/obj8 2023-09-29T22:23:34.407 DEBUG:boto:Data: 2023-09-29T22:23:34.407 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.408 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.408 DEBUG:boto:Params: {} 2023-09-29T22:23:34.408 DEBUG:boto:Token: None 2023-09-29T22:23:34.408 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8 2023-09-29T22:23:34.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qnqAS6Y9OKma5/f1a1W/N7BBOG4= 2023-09-29T22:23:34.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qnqAS6Y9OKma5/f1a1W/N7BBOG4=', 'Content-Length': '0'} 2023-09-29T22:23:34.411 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001f824243439013fb-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2023-09-29T22:23:34.411 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.411 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.411 DEBUG:boto:Method: GET 2023-09-29T22:23:34.412 DEBUG:boto:Path: /hcnphz-49/obj8 2023-09-29T22:23:34.412 DEBUG:boto:Data: 2023-09-29T22:23:34.412 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.412 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.412 DEBUG:boto:Params: {} 2023-09-29T22:23:34.412 DEBUG:boto:Token: None 2023-09-29T22:23:34.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8 2023-09-29T22:23:34.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ue8TOT4uxfW7tmx8n+DDxRk26zY= 2023-09-29T22:23:34.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ue8TOT4uxfW7tmx8n+DDxRk26zY=', 'Content-Length': '0'} 2023-09-29T22:23:34.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2c15ff740047aa7-0065174e66-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.458 DEBUG:boto:path=/hcnphz-49/obj8 2023-09-29T22:23:34.458 DEBUG:boto:auth_path=/hcnphz-49/obj8 2023-09-29T22:23:34.459 DEBUG:boto:Method: GET 2023-09-29T22:23:34.459 DEBUG:boto:Path: /hcnphz-49/obj8 2023-09-29T22:23:34.459 DEBUG:boto:Data: 2023-09-29T22:23:34.459 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.459 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.459 DEBUG:boto:Params: {} 2023-09-29T22:23:34.459 DEBUG:boto:Token: None 2023-09-29T22:23:34.459 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/obj8 2023-09-29T22:23:34.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ue8TOT4uxfW7tmx8n+DDxRk26zY= 2023-09-29T22:23:34.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ue8TOT4uxfW7tmx8n+DDxRk26zY=', 'Content-Length': '0'} 2023-09-29T22:23:34.462 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:21:20 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-49:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.14'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006e160976993818ea-0065174e66-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.463 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.463 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.463 DEBUG:boto:path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.463 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.463 DEBUG:boto:Method: GET 2023-09-29T22:23:34.463 DEBUG:boto:Path: /hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.463 DEBUG:boto:Data: 2023-09-29T22:23:34.463 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:34.464 DEBUG:boto:Port: 8001 2023-09-29T22:23:34.464 DEBUG:boto:Params: {} 2023-09-29T22:23:34.464 DEBUG:boto:Token: None 2023-09-29T22:23:34.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aef11ef5748d5419-0065174e66-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.469 DEBUG:boto:b'hcnphz-49obj81000falseobj82023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.470 DEBUG:boto:path=/hcnphz-49/ 2023-09-29T22:23:34.470 DEBUG:boto:auth_path=/hcnphz-49/ 2023-09-29T22:23:34.470 DEBUG:boto:path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.470 DEBUG:boto:auth_path=/hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.470 DEBUG:boto:Method: GET 2023-09-29T22:23:34.470 DEBUG:boto:Path: /hcnphz-49/?prefix=obj8 2023-09-29T22:23:34.470 DEBUG:boto:Data: 2023-09-29T22:23:34.470 DEBUG:boto:Headers: {} 2023-09-29T22:23:34.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:34.471 DEBUG:boto:Port: 8000 2023-09-29T22:23:34.471 DEBUG:boto:Params: {} 2023-09-29T22:23:34.471 DEBUG:boto:Token: None 2023-09-29T22:23:34.471 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:34 GMT /hcnphz-49/ 2023-09-29T22:23:34.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s= 2023-09-29T22:23:34.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BcUeeAWLKXdcpfCoUJFZ3Wd7w9s=', 'Content-Length': '0'} 2023-09-29T22:23:34.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e93de7d4998a824e-0065174e66-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:34.476 DEBUG:boto:b'hcnphz-49obj81000falseobj82023-09-29T22:21:20.362Z"a8f5f167f44f4964e6c998dee827110c"6STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:23:34.476 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-49 zones={a2, a1} 2023-09-29T22:23:34.477 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:23:34.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.477 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:34.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:34.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:34.608 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:23:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000010089", 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:39.881752Z" 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000007129", 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:20.682900Z" 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:38.785289Z" 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004317", 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:38.784073Z" 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.610 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:23:34.611 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000010089', 1: '00000000000000000000:00000000000000007129', 2: '00000000000000000000:00000000000000006576', 3: '00000000000000000000:00000000000000004317'} 2023-09-29T22:23:34.611 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:23:34.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.612 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:34.701 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:34.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:34.785 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:23:34.785 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 5605844595349065692 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:23:34.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000010089", 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:39.881752Z" 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.787 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000007129", 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:20.682900Z" 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006576", 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:38.785289Z" 2023-09-29T22:23:34.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004317", 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:34.790 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:38.784073Z" 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:23:34Z" 2023-09-29T22:23:34.791 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:23:34.792 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 5605844595349065692 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010089", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:39.881752Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000007129", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:20.682900Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006576", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:38.785289Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004317", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:38.784073Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:23:34Z" } 2023-09-29T22:23:34.792 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010089', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:39.881752Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000007129', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:20.682900Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006576', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:38.785289Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004317', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:38.784073Z'}}] 2023-09-29T22:23:34.793 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000010089', 1: '00000000000000000000:00000000000000007129', 2: '00000000000000000000:00000000000000006576', 3: '00000000000000000000:00000000000000004317'} 2023-09-29T22:23:34.793 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000010089', 1: '00000000000000000000:00000000000000007129', 2: '00000000000000000000:00000000000000006576', 3: '00000000000000000000:00000000000000004317'} 2023-09-29T22:23:34.793 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:23:34.793 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:23:34.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.793 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:34.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:34.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:34.932 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:23:34.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.932 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-09-29T22:23:34.932 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:37.107883Z" 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:37.111169Z" 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004739", 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:22:09.817243Z" 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:34.933 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:34.934 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002907", 2023-09-29T22:23:34.934 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:21:37.117679Z" 2023-09-29T22:23:34.934 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:34.934 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:23:34.935 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000008961', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004739', 3: '00000000000000000000:00000000000000002907'} 2023-09-29T22:23:34.936 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:23:34.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:34.936 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:34.982 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:34.982 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:35.066 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:23:35.066 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:23:35.066 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:23:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:23:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:23:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:23:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:23:35.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:23:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000008961", 2023-09-29T22:23:35.068 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:37.107883Z" 2023-09-29T22:23:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:35.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:23:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004593", 2023-09-29T22:23:35.070 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:37.111169Z" 2023-09-29T22:23:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:35.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:23:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000004739", 2023-09-29T22:23:35.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:22:09.817243Z" 2023-09-29T22:23:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:35.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:23:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:23:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000002907", 2023-09-29T22:23:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:35.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:35.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:35.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:21:37.117679Z" 2023-09-29T22:23:35.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:35.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:35.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:23:35.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:35.076 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:23:35.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:23:35.076 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:23:35.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:35.076 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:23:35Z" 2023-09-29T22:23:35.077 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:23:35.078 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000008961", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:37.107883Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004593", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:37.111169Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000004739", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:22:09.817243Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000002907", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:21:37.117679Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:23:35Z" } 2023-09-29T22:23:35.078 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000008961', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:37.107883Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004593', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:37.111169Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000004739', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:22:09.817243Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000002907', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:21:37.117679Z'}}] 2023-09-29T22:23:35.078 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000008961', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004739', 3: '00000000000000000000:00000000000000002907'} 2023-09-29T22:23:35.078 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000008961', 1: '00000000000000000000:00000000000000004593', 2: '00000000000000000000:00000000000000004739', 3: '00000000000000000000:00000000000000002907'} 2023-09-29T22:23:35.078 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:23:35.079 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2023-09-29T22:23:35.080 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-50 2023-09-29T22:23:35.080 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:23:35.080 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:23:35.080 DEBUG:boto:Method: PUT 2023-09-29T22:23:35.081 DEBUG:boto:Path: /hcnphz-50/ 2023-09-29T22:23:35.081 DEBUG:boto:Data: 2023-09-29T22:23:35.081 DEBUG:boto:Headers: {} 2023-09-29T22:23:35.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:35.081 DEBUG:boto:Port: 8000 2023-09-29T22:23:35.082 DEBUG:boto:Params: {} 2023-09-29T22:23:35.082 DEBUG:boto:Token: None 2023-09-29T22:23:35.082 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:23:35 GMT /hcnphz-50/ 2023-09-29T22:23:35.082 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y2w3E2/9AO0M3KzS9mk/F0QH97A= 2023-09-29T22:23:35.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y2w3E2/9AO0M3KzS9mk/F0QH97A=', 'Content-Length': '0'} 2023-09-29T22:23:35.113 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd84891b5a7c2828-0065174e67-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:35.113 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-51 2023-09-29T22:23:35.113 DEBUG:boto:path=/hcnphz-51/ 2023-09-29T22:23:35.113 DEBUG:boto:auth_path=/hcnphz-51/ 2023-09-29T22:23:35.114 DEBUG:boto:Method: PUT 2023-09-29T22:23:35.114 DEBUG:boto:Path: /hcnphz-51/ 2023-09-29T22:23:35.114 DEBUG:boto:Data: 2023-09-29T22:23:35.114 DEBUG:boto:Headers: {} 2023-09-29T22:23:35.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:23:35.114 DEBUG:boto:Port: 8001 2023-09-29T22:23:35.115 DEBUG:boto:Params: {} 2023-09-29T22:23:35.115 DEBUG:boto:Token: None 2023-09-29T22:23:35.115 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:23:35 GMT /hcnphz-51/ 2023-09-29T22:23:35.115 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:neSnvVcoodvkO2tff9R6UinHY5w= 2023-09-29T22:23:35.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:neSnvVcoodvkO2tff9R6UinHY5w=', 'Content-Length': '0'} 2023-09-29T22:23:35.160 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000573dfc85f024f0d1-0065174e67-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:23:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:35.161 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:35.161 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:35.161 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploads 2023-09-29T22:23:35.161 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploads 2023-09-29T22:23:35.161 DEBUG:boto:Method: POST 2023-09-29T22:23:35.161 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploads 2023-09-29T22:23:35.161 DEBUG:boto:Data: 2023-09-29T22:23:35.161 DEBUG:boto:Headers: {} 2023-09-29T22:23:35.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:35.162 DEBUG:boto:Port: 8000 2023-09-29T22:23:35.162 DEBUG:boto:Params: {} 2023-09-29T22:23:35.162 DEBUG:boto:Token: None 2023-09-29T22:23:35.162 DEBUG:boto:StringToSign: POST Fri, 29 Sep 2023 22:23:35 GMT /hcnphz-50/MULTIPART?uploads 2023-09-29T22:23:35.162 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xn/q1i0oYgYzDznbbQjpzT4krWQ= 2023-09-29T22:23:35.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xn/q1i0oYgYzDznbbQjpzT4krWQ=', 'Content-Length': '0'} 2023-09-29T22:23:36.321 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000af6762111b13cd05-0065174e67-1648-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '249'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.322 DEBUG:boto:b'hcnphz-50MULTIPART2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0' 2023-09-29T22:23:36.353 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.353 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.353 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=1 2023-09-29T22:23:36.353 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=1 2023-09-29T22:23:36.353 DEBUG:boto:Method: PUT 2023-09-29T22:23:36.353 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=1 2023-09-29T22:23:36.353 DEBUG:boto:Data: 2023-09-29T22:23:36.354 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T22:23:36.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:36.354 DEBUG:boto:Port: 8000 2023-09-29T22:23:36.354 DEBUG:boto:Params: {} 2023-09-29T22:23:36.354 DEBUG:boto:Token: None 2023-09-29T22:23:36.354 DEBUG:boto:StringToSign: PUT ebKBBg0ze5srhMzzkK3PdA== application/octet-stream Fri, 29 Sep 2023 22:23:36 GMT /hcnphz-50/MULTIPART?partNumber=1&uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4PXZ+8W3yKQMCx45/XSVEtaK2pI= 2023-09-29T22:23:36.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'ebKBBg0ze5srhMzzkK3PdA==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:23:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4PXZ+8W3yKQMCx45/XSVEtaK2pI='} 2023-09-29T22:23:36.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"79b281060d337b9b2b84ccf390adcf74"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007749b21fb1759fda-0065174e68-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.431 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.431 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.431 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=2 2023-09-29T22:23:36.431 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=2 2023-09-29T22:23:36.431 DEBUG:boto:Method: PUT 2023-09-29T22:23:36.431 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=2 2023-09-29T22:23:36.432 DEBUG:boto:Data: 2023-09-29T22:23:36.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T22:23:36.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:36.432 DEBUG:boto:Port: 8000 2023-09-29T22:23:36.432 DEBUG:boto:Params: {} 2023-09-29T22:23:36.432 DEBUG:boto:Token: None 2023-09-29T22:23:36.432 DEBUG:boto:StringToSign: PUT dIQ6OrGTo4m87YmUAtmdXw== application/octet-stream Fri, 29 Sep 2023 22:23:36 GMT /hcnphz-50/MULTIPART?partNumber=2&uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HZRSkxu+VoFgJ/gfb3DBBkumR7Y= 2023-09-29T22:23:36.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'dIQ6OrGTo4m87YmUAtmdXw==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:23:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HZRSkxu+VoFgJ/gfb3DBBkumR7Y='} 2023-09-29T22:23:36.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"74843a3ab193a389bced899402d99d5f"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007aa7e033dc433116-0065174e68-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.502 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.503 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.503 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=3 2023-09-29T22:23:36.503 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=3 2023-09-29T22:23:36.503 DEBUG:boto:Method: PUT 2023-09-29T22:23:36.503 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=3 2023-09-29T22:23:36.503 DEBUG:boto:Data: 2023-09-29T22:23:36.503 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T22:23:36.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:36.504 DEBUG:boto:Port: 8000 2023-09-29T22:23:36.504 DEBUG:boto:Params: {} 2023-09-29T22:23:36.504 DEBUG:boto:Token: None 2023-09-29T22:23:36.504 DEBUG:boto:StringToSign: PUT e4RW4edMN49Fhh9TYZ51tg== application/octet-stream Fri, 29 Sep 2023 22:23:36 GMT /hcnphz-50/MULTIPART?partNumber=3&uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QK2tVjU+YqfZpDrD2waWEWtKV7Q= 2023-09-29T22:23:36.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'e4RW4edMN49Fhh9TYZ51tg==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:23:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QK2tVjU+YqfZpDrD2waWEWtKV7Q='} 2023-09-29T22:23:36.540 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"7b8456e1e74c378f45861f53619e75b6"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8838d14473b61f1-0065174e68-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.560 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.560 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.560 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=4 2023-09-29T22:23:36.560 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=4 2023-09-29T22:23:36.560 DEBUG:boto:Method: PUT 2023-09-29T22:23:36.560 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0&partNumber=4 2023-09-29T22:23:36.560 DEBUG:boto:Data: 2023-09-29T22:23:36.561 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue'} 2023-09-29T22:23:36.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:36.561 DEBUG:boto:Port: 8000 2023-09-29T22:23:36.561 DEBUG:boto:Params: {} 2023-09-29T22:23:36.561 DEBUG:boto:Token: None 2023-09-29T22:23:36.561 DEBUG:boto:StringToSign: PUT l+jDi4yPcIlVvEiQwDQZTQ== application/octet-stream Fri, 29 Sep 2023 22:23:36 GMT /hcnphz-50/MULTIPART?partNumber=4&uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0wkLQuKP3S3gchUX1yksGOvFmI= 2023-09-29T22:23:36.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'l+jDi4yPcIlVvEiQwDQZTQ==', 'Content-Length': '5242880', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:23:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0wkLQuKP3S3gchUX1yksGOvFmI='} 2023-09-29T22:23:36.604 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"97e8c38b8c8f708955bc4890c034194d"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c59eccc362b2548f-0065174e68-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.604 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.604 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.604 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.604 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.605 DEBUG:boto:Method: GET 2023-09-29T22:23:36.605 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.605 DEBUG:boto:Data: 2023-09-29T22:23:36.605 DEBUG:boto:Headers: {} 2023-09-29T22:23:36.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:36.605 DEBUG:boto:Port: 8000 2023-09-29T22:23:36.605 DEBUG:boto:Params: {} 2023-09-29T22:23:36.605 DEBUG:boto:Token: None 2023-09-29T22:23:36.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:23:36 GMT /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.606 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7b8ySd6QcXlfF7SFrIZKH6d2ETY= 2023-09-29T22:23:36.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7b8ySd6QcXlfF7SFrIZKH6d2ETY=', 'Content-Length': '0'} 2023-09-29T22:23:36.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000623d88e2e2d725fb-0065174e68-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.651 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.651 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:23:36.651 DEBUG:boto:path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.651 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.651 DEBUG:boto:Method: POST 2023-09-29T22:23:36.652 DEBUG:boto:Path: /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.652 DEBUG:boto:Data: 1 "79b281060d337b9b2b84ccf390adcf74" 2 "74843a3ab193a389bced899402d99d5f" 3 "7b8456e1e74c378f45861f53619e75b6" 4 "97e8c38b8c8f708955bc4890c034194d" 2023-09-29T22:23:36.652 DEBUG:boto:Headers: {'Content-Type': 'text/xml'} 2023-09-29T22:23:36.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:23:36.652 DEBUG:boto:Port: 8000 2023-09-29T22:23:36.652 DEBUG:boto:Params: {} 2023-09-29T22:23:36.652 DEBUG:boto:Token: None 2023-09-29T22:23:36.652 DEBUG:boto:StringToSign: POST text/xml Fri, 29 Sep 2023 22:23:36 GMT /hcnphz-50/MULTIPART?uploadId=2~x-0pagV4KnW6rZGgaxT9PMU_ycowQs0 2023-09-29T22:23:36.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MaSZN2aeXFfleMSl870FPQYHJQM= 2023-09-29T22:23:36.653 DEBUG:boto:Final headers: {'Content-Type': 'text/xml', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:23:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MaSZN2aeXFfleMSl870FPQYHJQM=', 'Content-Length': '460'} 2023-09-29T22:23:36.673 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000007157dccd396476c2-0065174e68-1648-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '291'), ('Date', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:23:36.673 DEBUG:boto:http://smithi112.front.sepia.ceph.com:8000/hcnphz-50/MULTIPARThcnphz-50MULTIPART 2023-09-29T22:23:36.674 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:36.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:36.674 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:36.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:36.715 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:36.773 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:23:36.799 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:23:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:23:36.799 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026215.145303_406.1", 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:23:35.145303Z" 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.800 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:23:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:23:36.801 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:36.801 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:23:36.802 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026215.145303_406.1', 2: '', 3: ''} 2023-09-29T22:23:36.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:23:36.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:36.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:23:36.802 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:23:36.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:36.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:36.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:36.901+0000 7f1075cabf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:36.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:36.905+0000 7f1075cabf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:23:36.930 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:23:36.930 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:23:36.931 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.932 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026215.145303_406.1", 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:23:35.145303Z", 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:23:36.933 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:36.934 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:23:36.935 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:23:36.936 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:23:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:23:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:23:36.937 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:23:36Z" 2023-09-29T22:23:36.937 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:23:36.938 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026215.145303_406.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:23:35.145303Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:23:36Z" } 2023-09-29T22:23:36.938 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026215.145303_406.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:23:35.145303Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:23:36.938 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026215.145303_406.1', 2: '', 3: ''} 2023-09-29T22:23:36.938 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026215.145303_406.1', 2: '', 3: ''} 2023-09-29T22:23:36.938 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:23:36.938 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-50 2023-09-29T22:23:36.939 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:36.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-50', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:36.939 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-50 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:37.031 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:37.031 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:37.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:37.093+0000 7f2278fb4f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:42.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:42.093+0000 7f2278fb4f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:47.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:47.093+0000 7f2278fb4f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:52.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:52.093+0000 7f2278fb4f40 1 bucket sync caught up with source: 2023-09-29T22:23:52.096 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000011.38.6, , , , , ] 2023-09-29T22:23:52.096 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000011.38.6, , , , , ] 2023-09-29T22:23:52.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:52.093+0000 7f2278fb4f40 0 bucket checkpoint complete 2023-09-29T22:23:52.121 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-50 2023-09-29T22:23:52.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:52.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-50', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:23:52.122 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-50 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:23:52.173 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:23:52.174 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:23:52.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:52.241+0000 7fe59fc67f40 1 waiting to reach incremental sync.. 2023-09-29T22:23:57.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:23:57.241+0000 7fe59fc67f40 1 waiting to reach incremental sync.. 2023-09-29T22:24:02.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:02.244+0000 7fe59fc67f40 1 bucket sync caught up with source: 2023-09-29T22:24:02.244 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.147.6, , , , , ] 2023-09-29T22:24:02.245 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.147.6, , , , , ] 2023-09-29T22:24:02.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:02.244+0000 7fe59fc67f40 0 bucket checkpoint complete 2023-09-29T22:24:02.273 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-50 zones={a2, a1} 2023-09-29T22:24:02.273 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.273 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.274 DEBUG:boto:Method: HEAD 2023-09-29T22:24:02.274 DEBUG:boto:Path: /hcnphz-50/ 2023-09-29T22:24:02.274 DEBUG:boto:Data: 2023-09-29T22:24:02.274 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.274 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.274 DEBUG:boto:Params: {} 2023-09-29T22:24:02.275 DEBUG:boto:Token: None 2023-09-29T22:24:02.275 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/ 2023-09-29T22:24:02.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3rCFIfoqtQPoSQS/Y6Q6khMbCTE= 2023-09-29T22:24:02.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3rCFIfoqtQPoSQS/Y6Q6khMbCTE=', 'Content-Length': '0'} 2023-09-29T22:24:02.279 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c663d230ed6d3126-0065174e82-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.279 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.279 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.279 DEBUG:boto:Method: HEAD 2023-09-29T22:24:02.280 DEBUG:boto:Path: /hcnphz-50/ 2023-09-29T22:24:02.280 DEBUG:boto:Data: 2023-09-29T22:24:02.280 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.280 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.280 DEBUG:boto:Params: {} 2023-09-29T22:24:02.280 DEBUG:boto:Token: None 2023-09-29T22:24:02.280 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/ 2023-09-29T22:24:02.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3rCFIfoqtQPoSQS/Y6Q6khMbCTE= 2023-09-29T22:24:02.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3rCFIfoqtQPoSQS/Y6Q6khMbCTE=', 'Content-Length': '0'} 2023-09-29T22:24:02.285 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '20971520'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099046fc1807abaee-0065174e82-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.285 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:24:02.285 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.285 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.285 DEBUG:boto:path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.286 DEBUG:boto:auth_path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.286 DEBUG:boto:Method: GET 2023-09-29T22:24:02.286 DEBUG:boto:Path: /hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.286 DEBUG:boto:Data: 2023-09-29T22:24:02.286 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.287 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.287 DEBUG:boto:Params: {} 2023-09-29T22:24:02.287 DEBUG:boto:Token: None 2023-09-29T22:24:02.287 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/?versions 2023-09-29T22:24:02.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8= 2023-09-29T22:24:02.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8=', 'Content-Length': '0'} 2023-09-29T22:24:02.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000219d5a923cd589fa-0065174e82-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.291 DEBUG:boto:b'hcnphz-50999falseMULTIPARTnulltrue2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.291 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-09-29T22:24:02.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:24:02.292 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.292 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.292 DEBUG:boto:path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.292 DEBUG:boto:auth_path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.292 DEBUG:boto:Method: GET 2023-09-29T22:24:02.292 DEBUG:boto:Path: /hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.292 DEBUG:boto:Data: 2023-09-29T22:24:02.292 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.292 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.293 DEBUG:boto:Params: {} 2023-09-29T22:24:02.293 DEBUG:boto:Token: None 2023-09-29T22:24:02.293 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/?versions 2023-09-29T22:24:02.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8= 2023-09-29T22:24:02.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8=', 'Content-Length': '0'} 2023-09-29T22:24:02.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057e9ffd4278f11b3-0065174e82-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.297 DEBUG:boto:b'hcnphz-50999falseMULTIPARTnulltrue2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.297 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2023-09-29T22:24:02.297 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.298 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.298 DEBUG:boto:path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.298 DEBUG:boto:auth_path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.298 DEBUG:boto:Method: GET 2023-09-29T22:24:02.298 DEBUG:boto:Path: /hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.298 DEBUG:boto:Data: 2023-09-29T22:24:02.298 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.298 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.299 DEBUG:boto:Params: {} 2023-09-29T22:24:02.299 DEBUG:boto:Token: None 2023-09-29T22:24:02.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/?versions 2023-09-29T22:24:02.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8= 2023-09-29T22:24:02.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8=', 'Content-Length': '0'} 2023-09-29T22:24:02.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a9c0cc6cba645f0-0065174e82-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.346 DEBUG:boto:b'hcnphz-50999falseMULTIPARTnulltrue2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.347 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.347 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.347 DEBUG:boto:path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.347 DEBUG:boto:auth_path=/hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.347 DEBUG:boto:Method: GET 2023-09-29T22:24:02.348 DEBUG:boto:Path: /hcnphz-50/?versions&max-keys=999 2023-09-29T22:24:02.348 DEBUG:boto:Data: 2023-09-29T22:24:02.348 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.348 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.348 DEBUG:boto:Params: {} 2023-09-29T22:24:02.348 DEBUG:boto:Token: None 2023-09-29T22:24:02.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/?versions 2023-09-29T22:24:02.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8= 2023-09-29T22:24:02.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXtuzynzVipyhMIh5fXTIrZCtQ8=', 'Content-Length': '0'} 2023-09-29T22:24:02.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d704da35a337862-0065174e82-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.353 DEBUG:boto:b'hcnphz-50999falseMULTIPARTnulltrue2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-09-29T22:24:02.353 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.354 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.354 DEBUG:boto:path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.354 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.354 DEBUG:boto:Method: GET 2023-09-29T22:24:02.354 DEBUG:boto:Path: /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.354 DEBUG:boto:Data: 2023-09-29T22:24:02.354 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.354 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.354 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.354 DEBUG:boto:Params: {} 2023-09-29T22:24:02.355 DEBUG:boto:Token: None 2023-09-29T22:24:02.355 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw= 2023-09-29T22:24:02.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw=', 'Content-Length': '0'} 2023-09-29T22:24:02.369 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-50:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000af2cd09191b17f6e-0065174e82-1642-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.465 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.466 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.466 DEBUG:boto:path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.466 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.466 DEBUG:boto:Method: GET 2023-09-29T22:24:02.466 DEBUG:boto:Path: /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.466 DEBUG:boto:Data: 2023-09-29T22:24:02.466 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.467 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.467 DEBUG:boto:Params: {} 2023-09-29T22:24:02.467 DEBUG:boto:Token: None 2023-09-29T22:24:02.467 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.467 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw= 2023-09-29T22:24:02.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw=', 'Content-Length': '0'} 2023-09-29T22:24:02.473 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d0b2fd326b516ea8-0065174e82-1648-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.559 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.560 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.560 DEBUG:boto:path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.560 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.560 DEBUG:boto:Method: HEAD 2023-09-29T22:24:02.560 DEBUG:boto:Path: /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.560 DEBUG:boto:Data: 2023-09-29T22:24:02.560 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.561 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.561 DEBUG:boto:Params: {} 2023-09-29T22:24:02.561 DEBUG:boto:Token: None 2023-09-29T22:24:02.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MOr3NjARGevwax8/SbLVHCmqmBc= 2023-09-29T22:24:02.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MOr3NjARGevwax8/SbLVHCmqmBc=', 'Content-Length': '0'} 2023-09-29T22:24:02.564 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-50:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx0000055418b9d2f655213-0065174e82-1642-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.565 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.565 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.565 DEBUG:boto:path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.565 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.565 DEBUG:boto:Method: HEAD 2023-09-29T22:24:02.565 DEBUG:boto:Path: /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.565 DEBUG:boto:Data: 2023-09-29T22:24:02.566 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.566 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.566 DEBUG:boto:Params: {} 2023-09-29T22:24:02.566 DEBUG:boto:Token: None 2023-09-29T22:24:02.566 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.566 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MOr3NjARGevwax8/SbLVHCmqmBc= 2023-09-29T22:24:02.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MOr3NjARGevwax8/SbLVHCmqmBc=', 'Content-Length': '0'} 2023-09-29T22:24:02.569 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000d2a3e352c0b6978b-0065174e82-1648-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-09-29T22:24:02.570 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.570 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.570 DEBUG:boto:path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.570 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.570 DEBUG:boto:Method: GET 2023-09-29T22:24:02.570 DEBUG:boto:Path: /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.570 DEBUG:boto:Data: 2023-09-29T22:24:02.571 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.571 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.571 DEBUG:boto:Params: {} 2023-09-29T22:24:02.571 DEBUG:boto:Token: None 2023-09-29T22:24:02.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw= 2023-09-29T22:24:02.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw=', 'Content-Length': '0'} 2023-09-29T22:24:02.586 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-50:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000002f8f32ff067deeca-0065174e82-1642-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.667 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.667 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.668 DEBUG:boto:path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.668 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.668 DEBUG:boto:Method: GET 2023-09-29T22:24:02.668 DEBUG:boto:Path: /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.668 DEBUG:boto:Data: 2023-09-29T22:24:02.668 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.668 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.668 DEBUG:boto:Params: {} 2023-09-29T22:24:02.669 DEBUG:boto:Token: None 2023-09-29T22:24:02.669 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART?versionId=null 2023-09-29T22:24:02.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw= 2023-09-29T22:24:02.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XiILFYM/zKgTRnT9YnX0g3ol9Mw=', 'Content-Length': '0'} 2023-09-29T22:24:02.675 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000000e58261d40fc2bbe-0065174e82-1648-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.763 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.763 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.763 DEBUG:boto:path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.763 DEBUG:boto:auth_path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.763 DEBUG:boto:Method: GET 2023-09-29T22:24:02.764 DEBUG:boto:Path: /hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.764 DEBUG:boto:Data: 2023-09-29T22:24:02.764 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.764 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.764 DEBUG:boto:Params: {} 2023-09-29T22:24:02.764 DEBUG:boto:Token: None 2023-09-29T22:24:02.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/ 2023-09-29T22:24:02.765 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ= 2023-09-29T22:24:02.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ=', 'Content-Length': '0'} 2023-09-29T22:24:02.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b0470e2d50d8a73-0065174e82-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.770 DEBUG:boto:b'hcnphz-50MULTIPART1000falseMULTIPART2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.770 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.770 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.770 DEBUG:boto:path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.771 DEBUG:boto:auth_path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.771 DEBUG:boto:Method: GET 2023-09-29T22:24:02.771 DEBUG:boto:Path: /hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.771 DEBUG:boto:Data: 2023-09-29T22:24:02.771 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.771 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.771 DEBUG:boto:Params: {} 2023-09-29T22:24:02.771 DEBUG:boto:Token: None 2023-09-29T22:24:02.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/ 2023-09-29T22:24:02.772 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ= 2023-09-29T22:24:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ=', 'Content-Length': '0'} 2023-09-29T22:24:02.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070263f5c7d7c6184-0065174e82-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.776 DEBUG:boto:b'hcnphz-50MULTIPART1000falseMULTIPART2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.777 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.777 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.777 DEBUG:boto:Method: HEAD 2023-09-29T22:24:02.777 DEBUG:boto:Path: /hcnphz-50/MULTIPART 2023-09-29T22:24:02.777 DEBUG:boto:Data: 2023-09-29T22:24:02.777 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.777 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.777 DEBUG:boto:Params: {} 2023-09-29T22:24:02.778 DEBUG:boto:Token: None 2023-09-29T22:24:02.778 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART 2023-09-29T22:24:02.778 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uQc48Oyw4/rsCMbuj7bqh0nIV/g= 2023-09-29T22:24:02.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uQc48Oyw4/rsCMbuj7bqh0nIV/g=', 'Content-Length': '0'} 2023-09-29T22:24:02.780 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-50:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000000e1e6449a21ae9e0-0065174e82-1642-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.781 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.781 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.781 DEBUG:boto:Method: HEAD 2023-09-29T22:24:02.781 DEBUG:boto:Path: /hcnphz-50/MULTIPART 2023-09-29T22:24:02.781 DEBUG:boto:Data: 2023-09-29T22:24:02.781 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.781 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.782 DEBUG:boto:Params: {} 2023-09-29T22:24:02.782 DEBUG:boto:Token: None 2023-09-29T22:24:02.782 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART 2023-09-29T22:24:02.782 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uQc48Oyw4/rsCMbuj7bqh0nIV/g= 2023-09-29T22:24:02.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uQc48Oyw4/rsCMbuj7bqh0nIV/g=', 'Content-Length': '0'} 2023-09-29T22:24:02.784 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000004eef04c420890437-0065174e82-1648-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2023-09-29T22:24:02.785 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.785 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.785 DEBUG:boto:Method: GET 2023-09-29T22:24:02.785 DEBUG:boto:Path: /hcnphz-50/MULTIPART 2023-09-29T22:24:02.785 DEBUG:boto:Data: 2023-09-29T22:24:02.785 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.785 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.786 DEBUG:boto:Params: {} 2023-09-29T22:24:02.786 DEBUG:boto:Token: None 2023-09-29T22:24:02.786 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART 2023-09-29T22:24:02.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2N5NlsHtfT3IRKyvf6X0Tb70bWk= 2023-09-29T22:24:02.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2N5NlsHtfT3IRKyvf6X0Tb70bWk=', 'Content-Length': '0'} 2023-09-29T22:24:02.799 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-50:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.15'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx000005c2241b02985b39f-0065174e82-1642-a2'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.879 DEBUG:boto:path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.879 DEBUG:boto:auth_path=/hcnphz-50/MULTIPART 2023-09-29T22:24:02.879 DEBUG:boto:Method: GET 2023-09-29T22:24:02.879 DEBUG:boto:Path: /hcnphz-50/MULTIPART 2023-09-29T22:24:02.879 DEBUG:boto:Data: 2023-09-29T22:24:02.879 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.880 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.880 DEBUG:boto:Params: {} 2023-09-29T22:24:02.880 DEBUG:boto:Token: None 2023-09-29T22:24:02.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/MULTIPART 2023-09-29T22:24:02.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2N5NlsHtfT3IRKyvf6X0Tb70bWk= 2023-09-29T22:24:02.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2N5NlsHtfT3IRKyvf6X0Tb70bWk=', 'Content-Length': '0'} 2023-09-29T22:24:02.887 DEBUG:boto:Response headers: [('Content-Length', '20971520'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:23:36 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"d07d411772772538cd4b657da660bfb5-4"'), ('x-amz-request-id', 'tx00000b34520e831d50022-0065174e82-1648-a1'), ('Content-Type', 'binary/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.975 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.975 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.976 DEBUG:boto:path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.976 DEBUG:boto:auth_path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.976 DEBUG:boto:Method: GET 2023-09-29T22:24:02.976 DEBUG:boto:Path: /hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.976 DEBUG:boto:Data: 2023-09-29T22:24:02.976 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:24:02.976 DEBUG:boto:Port: 8001 2023-09-29T22:24:02.976 DEBUG:boto:Params: {} 2023-09-29T22:24:02.977 DEBUG:boto:Token: None 2023-09-29T22:24:02.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/ 2023-09-29T22:24:02.977 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ= 2023-09-29T22:24:02.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ=', 'Content-Length': '0'} 2023-09-29T22:24:02.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d632aa53d7d1a47b-0065174e82-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.982 DEBUG:boto:b'hcnphz-50MULTIPART1000falseMULTIPART2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.982 DEBUG:boto:path=/hcnphz-50/ 2023-09-29T22:24:02.982 DEBUG:boto:auth_path=/hcnphz-50/ 2023-09-29T22:24:02.983 DEBUG:boto:path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.983 DEBUG:boto:auth_path=/hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.983 DEBUG:boto:Method: GET 2023-09-29T22:24:02.983 DEBUG:boto:Path: /hcnphz-50/?prefix=MULTIPART 2023-09-29T22:24:02.983 DEBUG:boto:Data: 2023-09-29T22:24:02.983 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.983 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.983 DEBUG:boto:Params: {} 2023-09-29T22:24:02.984 DEBUG:boto:Token: None 2023-09-29T22:24:02.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-50/ 2023-09-29T22:24:02.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ= 2023-09-29T22:24:02.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v6NEs9YukgNRJXNGS3ctXTr0NrQ=', 'Content-Length': '0'} 2023-09-29T22:24:02.989 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a61d80a29d5d0e0d-0065174e82-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:24:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:02.989 DEBUG:boto:b'hcnphz-50MULTIPART1000falseMULTIPART2023-09-29T22:23:36.663Z"d07d411772772538cd4b657da660bfb5-4"20971520STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:24:02.989 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-50 zones={a2, a1} 2023-09-29T22:24:02.990 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2023-09-29T22:24:02.990 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-52 2023-09-29T22:24:02.990 DEBUG:boto:path=/hcnphz-52/ 2023-09-29T22:24:02.990 DEBUG:boto:auth_path=/hcnphz-52/ 2023-09-29T22:24:02.991 DEBUG:boto:Method: PUT 2023-09-29T22:24:02.991 DEBUG:boto:Path: /hcnphz-52/ 2023-09-29T22:24:02.991 DEBUG:boto:Data: 2023-09-29T22:24:02.991 DEBUG:boto:Headers: {} 2023-09-29T22:24:02.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:02.991 DEBUG:boto:Port: 8000 2023-09-29T22:24:02.991 DEBUG:boto:Params: {} 2023-09-29T22:24:02.991 DEBUG:boto:Token: None 2023-09-29T22:24:02.992 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:24:02 GMT /hcnphz-52/ 2023-09-29T22:24:02.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YPYp2ipQ6K6sK836/qeUrOH1zMI= 2023-09-29T22:24:02.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:24:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YPYp2ipQ6K6sK836/qeUrOH1zMI=', 'Content-Length': '0'} 2023-09-29T22:24:03.027 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dd6be019b4df8920-0065174e82-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:03.027 DEBUG:boto:path=/hcnphz-52/testobj-sse-c 2023-09-29T22:24:03.027 DEBUG:boto:auth_path=/hcnphz-52/testobj-sse-c 2023-09-29T22:24:03.027 DEBUG:boto:Method: PUT 2023-09-29T22:24:03.028 DEBUG:boto:Path: /hcnphz-52/testobj-sse-c 2023-09-29T22:24:03.028 DEBUG:boto:Data: 2023-09-29T22:24:03.028 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:24:03.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:03.028 DEBUG:boto:Port: 8000 2023-09-29T22:24:03.028 DEBUG:boto:Params: {} 2023-09-29T22:24:03.028 DEBUG:boto:Token: None 2023-09-29T22:24:03.028 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 29 Sep 2023 22:24:03 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hcnphz-52/testobj-sse-c 2023-09-29T22:24:03.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SHEfANk8QcYgl5yejCrWG7omtl4= 2023-09-29T22:24:03.029 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:24:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SHEfANk8QcYgl5yejCrWG7omtl4='} 2023-09-29T22:24:03.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('x-amz-request-id', 'tx000002ae07d3a44f9900a-0065174e83-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:03.071 DEBUG:boto:path=/hcnphz-52/testobj-sse-kms 2023-09-29T22:24:03.071 DEBUG:boto:auth_path=/hcnphz-52/testobj-sse-kms 2023-09-29T22:24:03.071 DEBUG:boto:Method: PUT 2023-09-29T22:24:03.071 DEBUG:boto:Path: /hcnphz-52/testobj-sse-kms 2023-09-29T22:24:03.071 DEBUG:boto:Data: 2023-09-29T22:24:03.071 DEBUG:boto:Headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue'} 2023-09-29T22:24:03.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:24:03.072 DEBUG:boto:Port: 8000 2023-09-29T22:24:03.072 DEBUG:boto:Params: {} 2023-09-29T22:24:03.072 DEBUG:boto:Token: None 2023-09-29T22:24:03.072 DEBUG:boto:StringToSign: PUT 3FCGuEcom6i4veFJuDiBdQ== application/octet-stream Fri, 29 Sep 2023 22:24:03 GMT x-amz-server-side-encryption:aws:kms x-amz-server-side-encryption-aws-kms-key-id:testkey-1 /hcnphz-52/testobj-sse-kms 2023-09-29T22:24:03.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:trliZW31SoCKDK0mGAJS10s3ncU= 2023-09-29T22:24:03.072 DEBUG:boto:Final headers: {'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'testkey-1', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': '3FCGuEcom6i4veFJuDiBdQ==', 'Content-Length': '512', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:24:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:trliZW31SoCKDK0mGAJS10s3ncU='} 2023-09-29T22:24:03.093 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('Accept-Ranges', 'bytes'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('x-amz-server-side-encryption-context', 'eyJhd3M6czM6YXJuIjoiYXJuOmF3czpzMzo6cmd3LW11bHRpc2l0ZS10ZXN0LXVzZXI6aGNucGh6LTUyL3Rlc3RvYmotc3NlLWttcyJ9'), ('x-amz-request-id', 'tx000003d49cab17a0e2df8-0065174e83-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:24:03.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:03.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:03.094 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:24:03.139 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:03.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:03.197 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.225 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026243.024962_410.1", 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:24:03.024962Z" 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:24:03.226 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:24:03.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:24:03.227 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:24:03.228 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026243.024962_410.1', 2: '', 3: ''} 2023-09-29T22:24:03.228 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:24:03.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:03.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:24:03.229 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:24:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:03.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:03.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:03.368+0000 7f2827ac6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:24:03.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:03.372+0000 7f2827ac6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:24:03.399 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:24:03.400 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026243.024962_410.1", 2023-09-29T22:24:03.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:24:03.024962Z", 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:24:03.402 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:24:03.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:24:03Z" 2023-09-29T22:24:03.405 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:24:03.406 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026243.024962_410.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:24:03.024962Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:24:03Z" } 2023-09-29T22:24:03.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026243.024962_410.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:24:03.024962Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:24:03.407 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026243.024962_410.1', 2: '', 3: ''} 2023-09-29T22:24:03.407 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026243.024962_410.1', 2: '', 3: ''} 2023-09-29T22:24:03.407 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:24:03.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:03.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-52', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:24:03.408 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-52 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:24:03.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:24:03.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:24:03.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:03.524+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:08.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:08.524+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:13.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:13.524+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:18.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:18.528+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:23.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:23.528+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:28.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:28.528+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:33.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:33.532+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:38.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:38.531+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:43.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:43.531+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:48.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:48.531+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:53.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:53.531+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:24:58.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:24:58.535+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:25:03.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:03.535+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:25:08.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:08.535+0000 7f8365dbaf40 1 waiting to reach incremental sync.. 2023-09-29T22:25:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:13.538+0000 7f8365dbaf40 1 bucket sync caught up with source: 2023-09-29T22:25:13.542 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.56.6, , , , 00000000001.15.6, , , , , ] 2023-09-29T22:25:13.542 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.56.6, , , , 00000000001.15.6, , , , , ] 2023-09-29T22:25:13.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:13.538+0000 7f8365dbaf40 0 bucket checkpoint complete 2023-09-29T22:25:13.573 DEBUG:boto:path=/hcnphz-52/ 2023-09-29T22:25:13.573 DEBUG:boto:auth_path=/hcnphz-52/ 2023-09-29T22:25:13.573 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.573 DEBUG:boto:Path: /hcnphz-52/ 2023-09-29T22:25:13.573 DEBUG:boto:Data: 2023-09-29T22:25:13.573 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:13.574 DEBUG:boto:Port: 8001 2023-09-29T22:25:13.574 DEBUG:boto:Params: {} 2023-09-29T22:25:13.574 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:25:13.574 DEBUG:boto:Token: None 2023-09-29T22:25:13.575 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-52/ 2023-09-29T22:25:13.575 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J1ZtBabVJ5ZW0+dK11rZoZI3VdM= 2023-09-29T22:25:13.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J1ZtBabVJ5ZW0+dK11rZoZI3VdM=', 'Content-Length': '0'} 2023-09-29T22:25:13.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '1024'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbc31e973f117e24-0065174ec9-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.580 DEBUG:boto:path=/hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.581 DEBUG:boto:auth_path=/hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.581 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.581 DEBUG:boto:Path: /hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.581 DEBUG:boto:Data: 2023-09-29T22:25:13.581 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-09-29T22:25:13.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:13.581 DEBUG:boto:Port: 8001 2023-09-29T22:25:13.581 DEBUG:boto:Params: {} 2023-09-29T22:25:13.582 DEBUG:boto:Token: None 2023-09-29T22:25:13.582 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KaSyJeznRjokQelZtuj0Afv0fcQ= 2023-09-29T22:25:13.582 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KaSyJeznRjokQelZtuj0Afv0fcQ=', 'Content-Length': '0'} 2023-09-29T22:25:13.585 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-52:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000070376d804d1023fe-0065174ec9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.585 DEBUG:boto:path=/hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.585 DEBUG:boto:auth_path=/hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.585 DEBUG:boto:Method: GET 2023-09-29T22:25:13.586 DEBUG:boto:Path: /hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.586 DEBUG:boto:Data: 2023-09-29T22:25:13.586 DEBUG:boto:Headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw=='} 2023-09-29T22:25:13.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:13.586 DEBUG:boto:Port: 8001 2023-09-29T22:25:13.586 DEBUG:boto:Params: {} 2023-09-29T22:25:13.586 DEBUG:boto:Token: None 2023-09-29T22:25:13.586 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:13 GMT x-amz-server-side-encryption-customer-algorithm:AES256 x-amz-server-side-encryption-customer-key:pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= x-amz-server-side-encryption-customer-key-md5:DWygnHRtgiJ77HCm+1rvHw== /hcnphz-52/testobj-sse-c 2023-09-29T22:25:13.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KcF9OlkYaXYUPqUytXPB8Mkhaac= 2023-09-29T22:25:13.587 DEBUG:boto:Final headers: {'x-amz-server-side-encryption-customer-algorithm': 'AES256', 'x-amz-server-side-encryption-customer-key': 'pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs=', 'x-amz-server-side-encryption-customer-key-md5': 'DWygnHRtgiJ77HCm+1rvHw==', 'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KcF9OlkYaXYUPqUytXPB8Mkhaac=', 'Content-Length': '0'} 2023-09-29T22:25:13.592 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption-customer-algorithm', 'AES256'), ('x-amz-server-side-encryption-customer-key-MD5', 'DWygnHRtgiJ77HCm+1rvHw=='), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-52:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx0000083bd43ac6e58e663-0065174ec9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.592 DEBUG:boto:path=/hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.593 DEBUG:boto:auth_path=/hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.593 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.593 DEBUG:boto:Path: /hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.593 DEBUG:boto:Data: 2023-09-29T22:25:13.593 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:13.593 DEBUG:boto:Port: 8001 2023-09-29T22:25:13.593 DEBUG:boto:Params: {} 2023-09-29T22:25:13.593 DEBUG:boto:Token: None 2023-09-29T22:25:13.594 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.594 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IBn+3jnf5gkYSlx6ooAZeZFf+uo= 2023-09-29T22:25:13.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBn+3jnf5gkYSlx6ooAZeZFf+uo=', 'Content-Length': '0'} 2023-09-29T22:25:13.596 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-52:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000008c02577eab756138-0065174ec9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.597 DEBUG:boto:path=/hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.597 DEBUG:boto:auth_path=/hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.597 DEBUG:boto:Method: GET 2023-09-29T22:25:13.597 DEBUG:boto:Path: /hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.597 DEBUG:boto:Data: 2023-09-29T22:25:13.597 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:13.597 DEBUG:boto:Port: 8001 2023-09-29T22:25:13.598 DEBUG:boto:Params: {} 2023-09-29T22:25:13.598 DEBUG:boto:Token: None 2023-09-29T22:25:13.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-52/testobj-sse-kms 2023-09-29T22:25:13.598 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UP2dPXwZvYpInh/KV2+WB80rh4U= 2023-09-29T22:25:13.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UP2dPXwZvYpInh/KV2+WB80rh4U=', 'Content-Length': '0'} 2023-09-29T22:25:13.601 DEBUG:boto:Response headers: [('Content-Length', '512'), ('x-amz-server-side-encryption', 'aws:kms'), ('x-amz-server-side-encryption-aws-kms-key-id', 'testkey-1'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:24:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-52:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.17'), ('ETag', '"dc5086b847289ba8b8bde149b8388175"'), ('x-amz-request-id', 'tx000007b98145a8c61f8ad-0065174ec9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.601 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2023-09-29T22:25:13.602 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-53 2023-09-29T22:25:13.602 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:13.603 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:13.603 DEBUG:boto:Method: PUT 2023-09-29T22:25:13.603 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:13.603 DEBUG:boto:Data: 2023-09-29T22:25:13.603 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.603 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.603 DEBUG:boto:Params: {} 2023-09-29T22:25:13.604 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:25:13.604 DEBUG:boto:Token: None 2023-09-29T22:25:13.604 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/ 2023-09-29T22:25:13.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:133eAPDyTrMFnfbD5v2m8qWGzH0= 2023-09-29T22:25:13.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:133eAPDyTrMFnfbD5v2m8qWGzH0=', 'Content-Length': '0'} 2023-09-29T22:25:13.635 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f05aa0a15a68dbdc-0065174ec9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.635 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:13.635 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:13.635 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.635 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:13.635 DEBUG:boto:Data: 2023-09-29T22:25:13.636 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.636 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.636 DEBUG:boto:Params: {} 2023-09-29T22:25:13.636 DEBUG:boto:Token: None 2023-09-29T22:25:13.636 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/ 2023-09-29T22:25:13.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q= 2023-09-29T22:25:13.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q=', 'Content-Length': '0'} 2023-09-29T22:25:13.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000521abc966a3ce1d1-0065174ec9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.640 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:13.640 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:13.640 DEBUG:boto:Method: PUT 2023-09-29T22:25:13.640 DEBUG:boto:Path: /hcnphz-53/a 2023-09-29T22:25:13.640 DEBUG:boto:Data: 2023-09-29T22:25:13.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:13.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.640 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.641 DEBUG:boto:Params: {} 2023-09-29T22:25:13.641 DEBUG:boto:Token: None 2023-09-29T22:25:13.641 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/a 2023-09-29T22:25:13.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kNn7WDcs5dXG0kzwxrHedJD91OU= 2023-09-29T22:25:13.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kNn7WDcs5dXG0kzwxrHedJD91OU='} 2023-09-29T22:25:13.651 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f22dc29febd5eccc-0065174ec9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.652 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:13.652 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:13.652 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.652 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:13.652 DEBUG:boto:Data: 2023-09-29T22:25:13.652 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.652 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.652 DEBUG:boto:Params: {} 2023-09-29T22:25:13.653 DEBUG:boto:Token: None 2023-09-29T22:25:13.653 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/ 2023-09-29T22:25:13.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q= 2023-09-29T22:25:13.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q=', 'Content-Length': '0'} 2023-09-29T22:25:13.656 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000878ce12045eec9d4-0065174ec9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.656 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:13.656 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:13.656 DEBUG:boto:Method: PUT 2023-09-29T22:25:13.656 DEBUG:boto:Path: /hcnphz-53/b 2023-09-29T22:25:13.656 DEBUG:boto:Data: 2023-09-29T22:25:13.657 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:13.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.657 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.657 DEBUG:boto:Params: {} 2023-09-29T22:25:13.657 DEBUG:boto:Token: None 2023-09-29T22:25:13.657 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/b 2023-09-29T22:25:13.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DBWAhxmpF6h7SvHYj4e1P8RwKz4= 2023-09-29T22:25:13.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DBWAhxmpF6h7SvHYj4e1P8RwKz4='} 2023-09-29T22:25:13.702 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052e493f86dcc4af4-0065174ec9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.703 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:13.703 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:13.703 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.703 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:13.703 DEBUG:boto:Data: 2023-09-29T22:25:13.703 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.703 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.704 DEBUG:boto:Params: {} 2023-09-29T22:25:13.704 DEBUG:boto:Token: None 2023-09-29T22:25:13.704 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/ 2023-09-29T22:25:13.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q= 2023-09-29T22:25:13.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q=', 'Content-Length': '0'} 2023-09-29T22:25:13.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005c353ead66784bd4-0065174ec9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.707 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:13.707 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:13.707 DEBUG:boto:Method: PUT 2023-09-29T22:25:13.707 DEBUG:boto:Path: /hcnphz-53/c 2023-09-29T22:25:13.708 DEBUG:boto:Data: 2023-09-29T22:25:13.708 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:13.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.708 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.708 DEBUG:boto:Params: {} 2023-09-29T22:25:13.708 DEBUG:boto:Token: None 2023-09-29T22:25:13.708 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/c 2023-09-29T22:25:13.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Iz90ymnJVkAZ4WA+jJuGpjEkYDo= 2023-09-29T22:25:13.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Iz90ymnJVkAZ4WA+jJuGpjEkYDo='} 2023-09-29T22:25:13.716 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000004f9291ff0118a00-0065174ec9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.716 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:13.716 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:13.716 DEBUG:boto:Method: HEAD 2023-09-29T22:25:13.716 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:13.716 DEBUG:boto:Data: 2023-09-29T22:25:13.716 DEBUG:boto:Headers: {} 2023-09-29T22:25:13.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.717 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.717 DEBUG:boto:Params: {} 2023-09-29T22:25:13.717 DEBUG:boto:Token: None 2023-09-29T22:25:13.717 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/ 2023-09-29T22:25:13.717 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q= 2023-09-29T22:25:13.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5OFVglxXFyZOBGgqV1uGGg8y+/Q=', 'Content-Length': '0'} 2023-09-29T22:25:13.720 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f8ba2069246cd08-0065174ec9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.720 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:13.720 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:13.721 DEBUG:boto:Method: PUT 2023-09-29T22:25:13.721 DEBUG:boto:Path: /hcnphz-53/d 2023-09-29T22:25:13.721 DEBUG:boto:Data: 2023-09-29T22:25:13.721 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:13.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:13.721 DEBUG:boto:Port: 8000 2023-09-29T22:25:13.721 DEBUG:boto:Params: {} 2023-09-29T22:25:13.721 DEBUG:boto:Token: None 2023-09-29T22:25:13.722 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:13 GMT /hcnphz-53/d 2023-09-29T22:25:13.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qiGx/Ijnatffeu0e2Ny4hRlN62Y= 2023-09-29T22:25:13.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qiGx/Ijnatffeu0e2Ny4hRlN62Y='} 2023-09-29T22:25:13.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007c7113a155bd1912-0065174ec9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:13.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:13.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:13.767 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:13.811 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:13.812 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:13.869 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:25:13.894 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:25:13.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:13.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:25:13.894 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:25:13.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:13.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026313.634007_414.1", 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:25:13.634007Z" 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:13.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:13.896 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:13.896 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:25:13.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:13.896 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:25:13.897 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026313.634007_414.1', 2: '', 3: ''} 2023-09-29T22:25:13.897 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:25:13.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:13.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:13.898 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:13.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:13.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:13.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:13.990+0000 7fd6d4dc8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:13.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:13.994+0000 7fd6d4dc8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026313.624459_411.1", 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:13.624459Z", 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:25:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:25:13Z" 2023-09-29T22:25:14.028 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:25:14.029 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026313.624459_411.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:13.624459Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:13Z" } 2023-09-29T22:25:14.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026313.624459_411.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:13.624459Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:25:14.030 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026313.634007_414.1', 2: '', 3: ''} 2023-09-29T22:25:14.030 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026313.624459_411.1', 2: '', 3: ''} 2023-09-29T22:25:14.030 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026313.634007_414.1 target=1_1696026313.624459_411.1 2023-09-29T22:25:19.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:19.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:19.032 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:19.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:19.079 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:19.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:19.134+0000 7f838a731f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:19.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:19.134+0000 7f838a731f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:19.165 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:25:19.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:19.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026313.634007_414.1", 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:19.168 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:13.634007Z", 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:25:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:25:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:25:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:25:19Z" 2023-09-29T22:25:19.172 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:25:19.173 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026313.634007_414.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:13.634007Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:19Z" } 2023-09-29T22:25:19.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026313.634007_414.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:13.634007Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:25:19.173 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026313.634007_414.1', 2: '', 3: ''} 2023-09-29T22:25:19.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026313.634007_414.1', 2: '', 3: ''} 2023-09-29T22:25:19.174 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:25:19.174 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-53 2023-09-29T22:25:19.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:19.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-53', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:19.174 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-53 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:19.226 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:19.226 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:19.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:19.286+0000 7fb7157cef40 1 waiting to reach incremental sync.. 2023-09-29T22:25:24.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:24.290+0000 7fb7157cef40 1 waiting to reach incremental sync.. 2023-09-29T22:25:29.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:29.290+0000 7fb7157cef40 1 waiting to reach incremental sync.. 2023-09-29T22:25:34.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:34.294+0000 7fb7157cef40 1 bucket sync caught up with source: 2023-09-29T22:25:34.297 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.18.6, , , , 00000000001.37.6, , , 00000000001.31.6, , 00000000001.48.6] 2023-09-29T22:25:34.297 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.18.6, , , , 00000000001.37.6, , , 00000000001.31.6, , 00000000001.48.6] 2023-09-29T22:25:34.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:34.294+0000 7fb7157cef40 0 bucket checkpoint complete 2023-09-29T22:25:34.325 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-53 2023-09-29T22:25:34.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:34.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-53', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:34.326 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-53 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:34.377 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:34.377 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:34.443 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:34.438+0000 7f4ec6c6af40 1 waiting to reach incremental sync.. 2023-09-29T22:25:39.444 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:39.442+0000 7f4ec6c6af40 1 waiting to reach incremental sync.. 2023-09-29T22:25:44.447 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:44.442+0000 7f4ec6c6af40 1 bucket sync caught up with source: 2023-09-29T22:25:44.447 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.150.6, , , , 00000000001.253.6, , , 00000000001.284.6, , 00000000001.119.6] 2023-09-29T22:25:44.448 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.150.6, , , , 00000000001.253.6, , , 00000000001.284.6, , 00000000001.119.6] 2023-09-29T22:25:44.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:44.442+0000 7f4ec6c6af40 0 bucket checkpoint complete 2023-09-29T22:25:44.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-53 zones={a2, a1} 2023-09-29T22:25:44.478 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.478 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.479 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.479 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:44.479 DEBUG:boto:Data: 2023-09-29T22:25:44.479 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.480 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.480 DEBUG:boto:Params: {} 2023-09-29T22:25:44.480 DEBUG:boto:Token: None 2023-09-29T22:25:44.481 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i2Rz0WnDADnrIG/50ozC6MeLBMc= 2023-09-29T22:25:44.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i2Rz0WnDADnrIG/50ozC6MeLBMc=', 'Content-Length': '0'} 2023-09-29T22:25:44.485 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef8c4dcb8a61672c-0065174ee8-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.485 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.485 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.486 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.486 DEBUG:boto:Path: /hcnphz-53/ 2023-09-29T22:25:44.486 DEBUG:boto:Data: 2023-09-29T22:25:44.486 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.486 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.487 DEBUG:boto:Params: {} 2023-09-29T22:25:44.487 DEBUG:boto:Token: None 2023-09-29T22:25:44.487 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i2Rz0WnDADnrIG/50ozC6MeLBMc= 2023-09-29T22:25:44.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i2Rz0WnDADnrIG/50ozC6MeLBMc=', 'Content-Length': '0'} 2023-09-29T22:25:44.491 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed93df5d580f2f5b-0065174ee8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.491 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:25:44.491 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.492 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.492 DEBUG:boto:path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.492 DEBUG:boto:auth_path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.492 DEBUG:boto:Method: GET 2023-09-29T22:25:44.492 DEBUG:boto:Path: /hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.492 DEBUG:boto:Data: 2023-09-29T22:25:44.492 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.492 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.492 DEBUG:boto:Params: {} 2023-09-29T22:25:44.493 DEBUG:boto:Token: None 2023-09-29T22:25:44.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/?versions 2023-09-29T22:25:44.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA= 2023-09-29T22:25:44.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA=', 'Content-Length': '0'} 2023-09-29T22:25:44.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b84666a74ae42d20-0065174ee8-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.497 DEBUG:boto:b'hcnphz-53999falseanulltrue2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.497 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:25:44.498 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:25:44.498 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:25:44.498 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:25:44.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:25:44.498 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.498 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.499 DEBUG:boto:path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.499 DEBUG:boto:auth_path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.499 DEBUG:boto:Method: GET 2023-09-29T22:25:44.499 DEBUG:boto:Path: /hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.499 DEBUG:boto:Data: 2023-09-29T22:25:44.499 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.499 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.499 DEBUG:boto:Params: {} 2023-09-29T22:25:44.499 DEBUG:boto:Token: None 2023-09-29T22:25:44.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/?versions 2023-09-29T22:25:44.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA= 2023-09-29T22:25:44.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA=', 'Content-Length': '0'} 2023-09-29T22:25:44.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000227b9c16b89b48ab-0065174ee8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.504 DEBUG:boto:b'hcnphz-53999falseanulltrue2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:25:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:25:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:25:44.504 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:25:44.504 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.505 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.505 DEBUG:boto:path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.505 DEBUG:boto:auth_path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.505 DEBUG:boto:Method: GET 2023-09-29T22:25:44.505 DEBUG:boto:Path: /hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.505 DEBUG:boto:Data: 2023-09-29T22:25:44.505 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.505 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.505 DEBUG:boto:Params: {} 2023-09-29T22:25:44.506 DEBUG:boto:Token: None 2023-09-29T22:25:44.506 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/?versions 2023-09-29T22:25:44.506 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA= 2023-09-29T22:25:44.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA=', 'Content-Length': '0'} 2023-09-29T22:25:44.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008306c399d60e68cf-0065174ee8-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.510 DEBUG:boto:b'hcnphz-53999falseanulltrue2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.511 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.511 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.511 DEBUG:boto:path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.511 DEBUG:boto:auth_path=/hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.512 DEBUG:boto:Method: GET 2023-09-29T22:25:44.512 DEBUG:boto:Path: /hcnphz-53/?versions&max-keys=999 2023-09-29T22:25:44.512 DEBUG:boto:Data: 2023-09-29T22:25:44.512 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.512 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.512 DEBUG:boto:Params: {} 2023-09-29T22:25:44.513 DEBUG:boto:Token: None 2023-09-29T22:25:44.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/?versions 2023-09-29T22:25:44.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA= 2023-09-29T22:25:44.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66YZpSwXNWVbhv0B9aDfg8C3lFA=', 'Content-Length': '0'} 2023-09-29T22:25:44.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3123912567c5fd7-0065174ee8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.517 DEBUG:boto:b'hcnphz-53999falseanulltrue2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:25:44.518 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.518 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.519 DEBUG:boto:path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.519 DEBUG:boto:auth_path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.519 DEBUG:boto:Method: GET 2023-09-29T22:25:44.519 DEBUG:boto:Path: /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.519 DEBUG:boto:Data: 2023-09-29T22:25:44.520 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.520 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.520 DEBUG:boto:Params: {} 2023-09-29T22:25:44.520 DEBUG:boto:Token: None 2023-09-29T22:25:44.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY= 2023-09-29T22:25:44.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY=', 'Content-Length': '0'} 2023-09-29T22:25:44.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2ece5ecf225823a-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.566 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.566 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.567 DEBUG:boto:path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.567 DEBUG:boto:auth_path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.567 DEBUG:boto:Method: GET 2023-09-29T22:25:44.567 DEBUG:boto:Path: /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.567 DEBUG:boto:Data: 2023-09-29T22:25:44.567 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.567 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.567 DEBUG:boto:Params: {} 2023-09-29T22:25:44.567 DEBUG:boto:Token: None 2023-09-29T22:25:44.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY= 2023-09-29T22:25:44.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY=', 'Content-Length': '0'} 2023-09-29T22:25:44.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b4c004fa836ded-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.571 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.571 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.571 DEBUG:boto:path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.571 DEBUG:boto:auth_path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.571 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.572 DEBUG:boto:Path: /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.572 DEBUG:boto:Data: 2023-09-29T22:25:44.572 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.572 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.572 DEBUG:boto:Params: {} 2023-09-29T22:25:44.572 DEBUG:boto:Token: None 2023-09-29T22:25:44.572 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nxA4Pkj+2KcvK/xdA3ef16x93NQ= 2023-09-29T22:25:44.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nxA4Pkj+2KcvK/xdA3ef16x93NQ=', 'Content-Length': '0'} 2023-09-29T22:25:44.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095a7fa71b7b92001-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.575 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.575 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.576 DEBUG:boto:path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.576 DEBUG:boto:auth_path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.576 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.576 DEBUG:boto:Path: /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.576 DEBUG:boto:Data: 2023-09-29T22:25:44.576 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.576 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.576 DEBUG:boto:Params: {} 2023-09-29T22:25:44.576 DEBUG:boto:Token: None 2023-09-29T22:25:44.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nxA4Pkj+2KcvK/xdA3ef16x93NQ= 2023-09-29T22:25:44.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nxA4Pkj+2KcvK/xdA3ef16x93NQ=', 'Content-Length': '0'} 2023-09-29T22:25:44.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eaeca70d01eb8c9-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:25:44.579 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.580 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.580 DEBUG:boto:path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.580 DEBUG:boto:auth_path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.580 DEBUG:boto:Method: GET 2023-09-29T22:25:44.580 DEBUG:boto:Path: /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.580 DEBUG:boto:Data: 2023-09-29T22:25:44.580 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.580 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.580 DEBUG:boto:Params: {} 2023-09-29T22:25:44.581 DEBUG:boto:Token: None 2023-09-29T22:25:44.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY= 2023-09-29T22:25:44.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY=', 'Content-Length': '0'} 2023-09-29T22:25:44.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c4bd89c3df7f8a-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.584 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.584 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.584 DEBUG:boto:path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.584 DEBUG:boto:auth_path=/hcnphz-53/a?versionId=null 2023-09-29T22:25:44.584 DEBUG:boto:Method: GET 2023-09-29T22:25:44.585 DEBUG:boto:Path: /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.585 DEBUG:boto:Data: 2023-09-29T22:25:44.585 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.585 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.585 DEBUG:boto:Params: {} 2023-09-29T22:25:44.585 DEBUG:boto:Token: None 2023-09-29T22:25:44.585 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a?versionId=null 2023-09-29T22:25:44.585 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY= 2023-09-29T22:25:44.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tifrkeN9ZSE0B4CyrEkKOBYemFY=', 'Content-Length': '0'} 2023-09-29T22:25:44.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046f757a7955b5c4e-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.630 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.631 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.631 DEBUG:boto:path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.631 DEBUG:boto:auth_path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.631 DEBUG:boto:Method: GET 2023-09-29T22:25:44.631 DEBUG:boto:Path: /hcnphz-53/?prefix=a 2023-09-29T22:25:44.631 DEBUG:boto:Data: 2023-09-29T22:25:44.631 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.631 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.632 DEBUG:boto:Params: {} 2023-09-29T22:25:44.632 DEBUG:boto:Token: None 2023-09-29T22:25:44.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e35eb91932397e8-0065174ee8-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.637 DEBUG:boto:b'hcnphz-53a1000falsea2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.637 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.637 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.638 DEBUG:boto:path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.638 DEBUG:boto:auth_path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.638 DEBUG:boto:Method: GET 2023-09-29T22:25:44.638 DEBUG:boto:Path: /hcnphz-53/?prefix=a 2023-09-29T22:25:44.638 DEBUG:boto:Data: 2023-09-29T22:25:44.638 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.638 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.638 DEBUG:boto:Params: {} 2023-09-29T22:25:44.639 DEBUG:boto:Token: None 2023-09-29T22:25:44.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d52351782ee9a172-0065174ee8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.643 DEBUG:boto:b'hcnphz-53a1000falsea2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.644 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.644 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.644 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.644 DEBUG:boto:Path: /hcnphz-53/a 2023-09-29T22:25:44.644 DEBUG:boto:Data: 2023-09-29T22:25:44.644 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.644 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.644 DEBUG:boto:Params: {} 2023-09-29T22:25:44.645 DEBUG:boto:Token: None 2023-09-29T22:25:44.645 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a 2023-09-29T22:25:44.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8XOFz4honPliRZEQlF994DZzx30= 2023-09-29T22:25:44.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8XOFz4honPliRZEQlF994DZzx30=', 'Content-Length': '0'} 2023-09-29T22:25:44.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002773c27ba3fde18-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.648 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.648 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.648 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.648 DEBUG:boto:Path: /hcnphz-53/a 2023-09-29T22:25:44.648 DEBUG:boto:Data: 2023-09-29T22:25:44.648 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.649 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.649 DEBUG:boto:Params: {} 2023-09-29T22:25:44.649 DEBUG:boto:Token: None 2023-09-29T22:25:44.649 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a 2023-09-29T22:25:44.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8XOFz4honPliRZEQlF994DZzx30= 2023-09-29T22:25:44.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8XOFz4honPliRZEQlF994DZzx30=', 'Content-Length': '0'} 2023-09-29T22:25:44.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b50cfd64fb6eb518-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:25:44.651 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.652 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.652 DEBUG:boto:Method: GET 2023-09-29T22:25:44.652 DEBUG:boto:Path: /hcnphz-53/a 2023-09-29T22:25:44.652 DEBUG:boto:Data: 2023-09-29T22:25:44.652 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.652 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.652 DEBUG:boto:Params: {} 2023-09-29T22:25:44.652 DEBUG:boto:Token: None 2023-09-29T22:25:44.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a 2023-09-29T22:25:44.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Iff1VOIdBsNSyIqOlc7OImhZ3Eg= 2023-09-29T22:25:44.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Iff1VOIdBsNSyIqOlc7OImhZ3Eg=', 'Content-Length': '0'} 2023-09-29T22:25:44.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008149624c6c859e1f-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.698 DEBUG:boto:path=/hcnphz-53/a 2023-09-29T22:25:44.699 DEBUG:boto:auth_path=/hcnphz-53/a 2023-09-29T22:25:44.699 DEBUG:boto:Method: GET 2023-09-29T22:25:44.699 DEBUG:boto:Path: /hcnphz-53/a 2023-09-29T22:25:44.699 DEBUG:boto:Data: 2023-09-29T22:25:44.699 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.699 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.699 DEBUG:boto:Params: {} 2023-09-29T22:25:44.700 DEBUG:boto:Token: None 2023-09-29T22:25:44.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/a 2023-09-29T22:25:44.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Iff1VOIdBsNSyIqOlc7OImhZ3Eg= 2023-09-29T22:25:44.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Iff1VOIdBsNSyIqOlc7OImhZ3Eg=', 'Content-Length': '0'} 2023-09-29T22:25:44.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000925b78e19d4eb759-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.703 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.703 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.704 DEBUG:boto:path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.704 DEBUG:boto:auth_path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.704 DEBUG:boto:Method: GET 2023-09-29T22:25:44.704 DEBUG:boto:Path: /hcnphz-53/?prefix=a 2023-09-29T22:25:44.704 DEBUG:boto:Data: 2023-09-29T22:25:44.704 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.704 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.704 DEBUG:boto:Params: {} 2023-09-29T22:25:44.705 DEBUG:boto:Token: None 2023-09-29T22:25:44.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a6091fc0aa0d26-0065174ee8-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.710 DEBUG:boto:b'hcnphz-53a1000falsea2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.711 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.711 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.711 DEBUG:boto:path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.711 DEBUG:boto:auth_path=/hcnphz-53/?prefix=a 2023-09-29T22:25:44.711 DEBUG:boto:Method: GET 2023-09-29T22:25:44.711 DEBUG:boto:Path: /hcnphz-53/?prefix=a 2023-09-29T22:25:44.712 DEBUG:boto:Data: 2023-09-29T22:25:44.712 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.712 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.712 DEBUG:boto:Params: {} 2023-09-29T22:25:44.712 DEBUG:boto:Token: None 2023-09-29T22:25:44.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b020f0fd44da23a1-0065174ee8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.758 DEBUG:boto:b'hcnphz-53a1000falsea2023-09-29T22:25:13.645Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:25:44.759 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.759 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.759 DEBUG:boto:path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.759 DEBUG:boto:auth_path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.760 DEBUG:boto:Method: GET 2023-09-29T22:25:44.760 DEBUG:boto:Path: /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.760 DEBUG:boto:Data: 2023-09-29T22:25:44.760 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.760 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.760 DEBUG:boto:Params: {} 2023-09-29T22:25:44.760 DEBUG:boto:Token: None 2023-09-29T22:25:44.761 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw= 2023-09-29T22:25:44.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw=', 'Content-Length': '0'} 2023-09-29T22:25:44.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc90f877f110ad6b-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.764 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.765 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.765 DEBUG:boto:path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.765 DEBUG:boto:auth_path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.765 DEBUG:boto:Method: GET 2023-09-29T22:25:44.765 DEBUG:boto:Path: /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.765 DEBUG:boto:Data: 2023-09-29T22:25:44.765 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.766 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.766 DEBUG:boto:Params: {} 2023-09-29T22:25:44.766 DEBUG:boto:Token: None 2023-09-29T22:25:44.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.766 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw= 2023-09-29T22:25:44.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw=', 'Content-Length': '0'} 2023-09-29T22:25:44.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f2a50740979a71-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.770 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.770 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.770 DEBUG:boto:path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.770 DEBUG:boto:auth_path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.770 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.770 DEBUG:boto:Path: /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.770 DEBUG:boto:Data: 2023-09-29T22:25:44.771 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.771 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.771 DEBUG:boto:Params: {} 2023-09-29T22:25:44.771 DEBUG:boto:Token: None 2023-09-29T22:25:44.771 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.771 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jzbz1yfBWG4+xkTnEe1KyP5pDDE= 2023-09-29T22:25:44.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jzbz1yfBWG4+xkTnEe1KyP5pDDE=', 'Content-Length': '0'} 2023-09-29T22:25:44.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000990f2dfb82ed65bd-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.775 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.775 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.775 DEBUG:boto:path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.775 DEBUG:boto:auth_path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.775 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.775 DEBUG:boto:Path: /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.775 DEBUG:boto:Data: 2023-09-29T22:25:44.775 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.776 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.776 DEBUG:boto:Params: {} 2023-09-29T22:25:44.776 DEBUG:boto:Token: None 2023-09-29T22:25:44.776 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jzbz1yfBWG4+xkTnEe1KyP5pDDE= 2023-09-29T22:25:44.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jzbz1yfBWG4+xkTnEe1KyP5pDDE=', 'Content-Length': '0'} 2023-09-29T22:25:44.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aaf8b55030af15af-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:25:44.779 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.779 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.780 DEBUG:boto:path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.780 DEBUG:boto:auth_path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.780 DEBUG:boto:Method: GET 2023-09-29T22:25:44.780 DEBUG:boto:Path: /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.780 DEBUG:boto:Data: 2023-09-29T22:25:44.780 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.780 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.780 DEBUG:boto:Params: {} 2023-09-29T22:25:44.781 DEBUG:boto:Token: None 2023-09-29T22:25:44.781 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw= 2023-09-29T22:25:44.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw=', 'Content-Length': '0'} 2023-09-29T22:25:44.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a1a5f3a2a5ae33f-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.830 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.831 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.831 DEBUG:boto:path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.831 DEBUG:boto:auth_path=/hcnphz-53/b?versionId=null 2023-09-29T22:25:44.831 DEBUG:boto:Method: GET 2023-09-29T22:25:44.831 DEBUG:boto:Path: /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.831 DEBUG:boto:Data: 2023-09-29T22:25:44.831 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.831 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.832 DEBUG:boto:Params: {} 2023-09-29T22:25:44.832 DEBUG:boto:Token: None 2023-09-29T22:25:44.832 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b?versionId=null 2023-09-29T22:25:44.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw= 2023-09-29T22:25:44.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mD6+PmlNCtcwUwhL/mNrV79GAhw=', 'Content-Length': '0'} 2023-09-29T22:25:44.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd3f30c94a3eaa2d-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.836 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.836 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.836 DEBUG:boto:path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.836 DEBUG:boto:auth_path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.836 DEBUG:boto:Method: GET 2023-09-29T22:25:44.836 DEBUG:boto:Path: /hcnphz-53/?prefix=b 2023-09-29T22:25:44.836 DEBUG:boto:Data: 2023-09-29T22:25:44.836 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.837 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.837 DEBUG:boto:Params: {} 2023-09-29T22:25:44.837 DEBUG:boto:Token: None 2023-09-29T22:25:44.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecdb3fc629068652-0065174ee8-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.841 DEBUG:boto:b'hcnphz-53b1000falseb2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.842 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.842 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.842 DEBUG:boto:path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.842 DEBUG:boto:auth_path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.842 DEBUG:boto:Method: GET 2023-09-29T22:25:44.842 DEBUG:boto:Path: /hcnphz-53/?prefix=b 2023-09-29T22:25:44.842 DEBUG:boto:Data: 2023-09-29T22:25:44.843 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.843 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.843 DEBUG:boto:Params: {} 2023-09-29T22:25:44.843 DEBUG:boto:Token: None 2023-09-29T22:25:44.843 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087cd283ebce7f467-0065174ee8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.848 DEBUG:boto:b'hcnphz-53b1000falseb2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.848 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.848 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.848 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.849 DEBUG:boto:Path: /hcnphz-53/b 2023-09-29T22:25:44.849 DEBUG:boto:Data: 2023-09-29T22:25:44.849 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.849 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.849 DEBUG:boto:Params: {} 2023-09-29T22:25:44.849 DEBUG:boto:Token: None 2023-09-29T22:25:44.850 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b 2023-09-29T22:25:44.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5WqP4b1JFv3NDOMAmUOcL1pS+w0= 2023-09-29T22:25:44.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5WqP4b1JFv3NDOMAmUOcL1pS+w0=', 'Content-Length': '0'} 2023-09-29T22:25:44.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9dcf51b0b6fedc-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.852 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.853 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.853 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.853 DEBUG:boto:Path: /hcnphz-53/b 2023-09-29T22:25:44.853 DEBUG:boto:Data: 2023-09-29T22:25:44.853 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.853 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.853 DEBUG:boto:Params: {} 2023-09-29T22:25:44.853 DEBUG:boto:Token: None 2023-09-29T22:25:44.854 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b 2023-09-29T22:25:44.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5WqP4b1JFv3NDOMAmUOcL1pS+w0= 2023-09-29T22:25:44.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5WqP4b1JFv3NDOMAmUOcL1pS+w0=', 'Content-Length': '0'} 2023-09-29T22:25:44.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7822173a2bed0f4-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:25:44.856 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.856 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.856 DEBUG:boto:Method: GET 2023-09-29T22:25:44.857 DEBUG:boto:Path: /hcnphz-53/b 2023-09-29T22:25:44.857 DEBUG:boto:Data: 2023-09-29T22:25:44.857 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.857 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.857 DEBUG:boto:Params: {} 2023-09-29T22:25:44.857 DEBUG:boto:Token: None 2023-09-29T22:25:44.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b 2023-09-29T22:25:44.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GWHsoLxAOzD2wbDylqn3tFeyFZE= 2023-09-29T22:25:44.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GWHsoLxAOzD2wbDylqn3tFeyFZE=', 'Content-Length': '0'} 2023-09-29T22:25:44.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055be89cd49995c73-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.860 DEBUG:boto:path=/hcnphz-53/b 2023-09-29T22:25:44.860 DEBUG:boto:auth_path=/hcnphz-53/b 2023-09-29T22:25:44.860 DEBUG:boto:Method: GET 2023-09-29T22:25:44.861 DEBUG:boto:Path: /hcnphz-53/b 2023-09-29T22:25:44.861 DEBUG:boto:Data: 2023-09-29T22:25:44.861 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.861 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.861 DEBUG:boto:Params: {} 2023-09-29T22:25:44.861 DEBUG:boto:Token: None 2023-09-29T22:25:44.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/b 2023-09-29T22:25:44.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GWHsoLxAOzD2wbDylqn3tFeyFZE= 2023-09-29T22:25:44.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GWHsoLxAOzD2wbDylqn3tFeyFZE=', 'Content-Length': '0'} 2023-09-29T22:25:44.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005028e2b997d7cef0-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.906 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.907 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.907 DEBUG:boto:path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.907 DEBUG:boto:auth_path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.907 DEBUG:boto:Method: GET 2023-09-29T22:25:44.907 DEBUG:boto:Path: /hcnphz-53/?prefix=b 2023-09-29T22:25:44.907 DEBUG:boto:Data: 2023-09-29T22:25:44.907 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.907 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.908 DEBUG:boto:Params: {} 2023-09-29T22:25:44.908 DEBUG:boto:Token: None 2023-09-29T22:25:44.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.913 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080231113fbe2391c-0065174ee8-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.913 DEBUG:boto:b'hcnphz-53b1000falseb2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.914 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:44.914 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:44.914 DEBUG:boto:path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.914 DEBUG:boto:auth_path=/hcnphz-53/?prefix=b 2023-09-29T22:25:44.914 DEBUG:boto:Method: GET 2023-09-29T22:25:44.914 DEBUG:boto:Path: /hcnphz-53/?prefix=b 2023-09-29T22:25:44.914 DEBUG:boto:Data: 2023-09-29T22:25:44.915 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.915 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.915 DEBUG:boto:Params: {} 2023-09-29T22:25:44.915 DEBUG:boto:Token: None 2023-09-29T22:25:44.915 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/ 2023-09-29T22:25:44.915 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk= 2023-09-29T22:25:44.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rH0a9mCl/EAOnn09BLZasDHLWbk=', 'Content-Length': '0'} 2023-09-29T22:25:44.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000977eed84be3e1e62-0065174ee8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.920 DEBUG:boto:b'hcnphz-53b1000falseb2023-09-29T22:25:13.661Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:44.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:25:44.921 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:44.921 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:44.921 DEBUG:boto:path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.921 DEBUG:boto:auth_path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.921 DEBUG:boto:Method: GET 2023-09-29T22:25:44.921 DEBUG:boto:Path: /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.921 DEBUG:boto:Data: 2023-09-29T22:25:44.921 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.922 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.922 DEBUG:boto:Params: {} 2023-09-29T22:25:44.922 DEBUG:boto:Token: None 2023-09-29T22:25:44.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag= 2023-09-29T22:25:44.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag=', 'Content-Length': '0'} 2023-09-29T22:25:44.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074d8774f8ba1eb93-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.970 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:44.971 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:44.971 DEBUG:boto:path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.971 DEBUG:boto:auth_path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.971 DEBUG:boto:Method: GET 2023-09-29T22:25:44.971 DEBUG:boto:Path: /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.971 DEBUG:boto:Data: 2023-09-29T22:25:44.971 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.972 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.972 DEBUG:boto:Params: {} 2023-09-29T22:25:44.972 DEBUG:boto:Token: None 2023-09-29T22:25:44.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag= 2023-09-29T22:25:44.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag=', 'Content-Length': '0'} 2023-09-29T22:25:44.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b68cfde6eab3f2b-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.975 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:44.976 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:44.976 DEBUG:boto:path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.976 DEBUG:boto:auth_path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.976 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.976 DEBUG:boto:Path: /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.976 DEBUG:boto:Data: 2023-09-29T22:25:44.976 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.976 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.976 DEBUG:boto:Params: {} 2023-09-29T22:25:44.977 DEBUG:boto:Token: None 2023-09-29T22:25:44.977 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.977 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PvjzXOyQRR7AW1tdjUfKh4wx+Zg= 2023-09-29T22:25:44.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PvjzXOyQRR7AW1tdjUfKh4wx+Zg=', 'Content-Length': '0'} 2023-09-29T22:25:44.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9f2f44dd341f898-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.980 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:44.980 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:44.980 DEBUG:boto:path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.980 DEBUG:boto:auth_path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.980 DEBUG:boto:Method: HEAD 2023-09-29T22:25:44.980 DEBUG:boto:Path: /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.980 DEBUG:boto:Data: 2023-09-29T22:25:44.981 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.981 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.981 DEBUG:boto:Params: {} 2023-09-29T22:25:44.981 DEBUG:boto:Token: None 2023-09-29T22:25:44.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PvjzXOyQRR7AW1tdjUfKh4wx+Zg= 2023-09-29T22:25:44.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PvjzXOyQRR7AW1tdjUfKh4wx+Zg=', 'Content-Length': '0'} 2023-09-29T22:25:44.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040f59e99f729ae18-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:25:44.985 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:44.985 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:44.985 DEBUG:boto:path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.985 DEBUG:boto:auth_path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.985 DEBUG:boto:Method: GET 2023-09-29T22:25:44.985 DEBUG:boto:Path: /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.986 DEBUG:boto:Data: 2023-09-29T22:25:44.986 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:44.986 DEBUG:boto:Port: 8001 2023-09-29T22:25:44.986 DEBUG:boto:Params: {} 2023-09-29T22:25:44.986 DEBUG:boto:Token: None 2023-09-29T22:25:44.986 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.986 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag= 2023-09-29T22:25:44.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag=', 'Content-Length': '0'} 2023-09-29T22:25:44.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007be0a704c4699dc1-0065174ee8-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:44.989 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:44.990 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:44.990 DEBUG:boto:path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.990 DEBUG:boto:auth_path=/hcnphz-53/c?versionId=null 2023-09-29T22:25:44.990 DEBUG:boto:Method: GET 2023-09-29T22:25:44.990 DEBUG:boto:Path: /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.990 DEBUG:boto:Data: 2023-09-29T22:25:44.990 DEBUG:boto:Headers: {} 2023-09-29T22:25:44.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:44.991 DEBUG:boto:Port: 8000 2023-09-29T22:25:44.991 DEBUG:boto:Params: {} 2023-09-29T22:25:44.991 DEBUG:boto:Token: None 2023-09-29T22:25:44.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:44 GMT /hcnphz-53/c?versionId=null 2023-09-29T22:25:44.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag= 2023-09-29T22:25:44.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uBSKudvVSQAFWK68wUCjlpGyXag=', 'Content-Length': '0'} 2023-09-29T22:25:44.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c781148db8d7b687-0065174ee8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.034 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.035 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.035 DEBUG:boto:path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.035 DEBUG:boto:auth_path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.035 DEBUG:boto:Method: GET 2023-09-29T22:25:45.035 DEBUG:boto:Path: /hcnphz-53/?prefix=c 2023-09-29T22:25:45.035 DEBUG:boto:Data: 2023-09-29T22:25:45.035 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.035 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.035 DEBUG:boto:Params: {} 2023-09-29T22:25:45.036 DEBUG:boto:Token: None 2023-09-29T22:25:45.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.040 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000354a78ca835743fe-0065174ee9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.041 DEBUG:boto:b'hcnphz-53c1000falsec2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.041 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.041 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.041 DEBUG:boto:path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.041 DEBUG:boto:auth_path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.042 DEBUG:boto:Method: GET 2023-09-29T22:25:45.042 DEBUG:boto:Path: /hcnphz-53/?prefix=c 2023-09-29T22:25:45.042 DEBUG:boto:Data: 2023-09-29T22:25:45.042 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.042 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.042 DEBUG:boto:Params: {} 2023-09-29T22:25:45.042 DEBUG:boto:Token: None 2023-09-29T22:25:45.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.043 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.047 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f99bffdcfb1e214-0065174ee9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.047 DEBUG:boto:b'hcnphz-53c1000falsec2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.047 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:45.047 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:45.048 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.048 DEBUG:boto:Path: /hcnphz-53/c 2023-09-29T22:25:45.048 DEBUG:boto:Data: 2023-09-29T22:25:45.048 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.048 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.048 DEBUG:boto:Params: {} 2023-09-29T22:25:45.048 DEBUG:boto:Token: None 2023-09-29T22:25:45.048 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/c 2023-09-29T22:25:45.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IANfGSa3GfUFdpMk0AGCQyzs66w= 2023-09-29T22:25:45.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IANfGSa3GfUFdpMk0AGCQyzs66w=', 'Content-Length': '0'} 2023-09-29T22:25:45.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8802f366d3a5a49-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.051 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:45.051 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:45.051 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.051 DEBUG:boto:Path: /hcnphz-53/c 2023-09-29T22:25:45.052 DEBUG:boto:Data: 2023-09-29T22:25:45.052 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.052 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.052 DEBUG:boto:Params: {} 2023-09-29T22:25:45.052 DEBUG:boto:Token: None 2023-09-29T22:25:45.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/c 2023-09-29T22:25:45.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IANfGSa3GfUFdpMk0AGCQyzs66w= 2023-09-29T22:25:45.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IANfGSa3GfUFdpMk0AGCQyzs66w=', 'Content-Length': '0'} 2023-09-29T22:25:45.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4da6cb5ccc7f96e-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:25:45.055 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:45.055 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:45.055 DEBUG:boto:Method: GET 2023-09-29T22:25:45.055 DEBUG:boto:Path: /hcnphz-53/c 2023-09-29T22:25:45.056 DEBUG:boto:Data: 2023-09-29T22:25:45.056 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.056 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.056 DEBUG:boto:Params: {} 2023-09-29T22:25:45.056 DEBUG:boto:Token: None 2023-09-29T22:25:45.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/c 2023-09-29T22:25:45.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5l2JFLdH4KO20rneBNwG0mO3ER8= 2023-09-29T22:25:45.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5l2JFLdH4KO20rneBNwG0mO3ER8=', 'Content-Length': '0'} 2023-09-29T22:25:45.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aba2d098edd15478-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.106 DEBUG:boto:path=/hcnphz-53/c 2023-09-29T22:25:45.107 DEBUG:boto:auth_path=/hcnphz-53/c 2023-09-29T22:25:45.107 DEBUG:boto:Method: GET 2023-09-29T22:25:45.107 DEBUG:boto:Path: /hcnphz-53/c 2023-09-29T22:25:45.107 DEBUG:boto:Data: 2023-09-29T22:25:45.107 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.107 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.107 DEBUG:boto:Params: {} 2023-09-29T22:25:45.107 DEBUG:boto:Token: None 2023-09-29T22:25:45.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/c 2023-09-29T22:25:45.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5l2JFLdH4KO20rneBNwG0mO3ER8= 2023-09-29T22:25:45.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5l2JFLdH4KO20rneBNwG0mO3ER8=', 'Content-Length': '0'} 2023-09-29T22:25:45.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad0f8401f70bed48-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.111 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.111 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.111 DEBUG:boto:path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.111 DEBUG:boto:auth_path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.111 DEBUG:boto:Method: GET 2023-09-29T22:25:45.111 DEBUG:boto:Path: /hcnphz-53/?prefix=c 2023-09-29T22:25:45.112 DEBUG:boto:Data: 2023-09-29T22:25:45.112 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.112 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.112 DEBUG:boto:Params: {} 2023-09-29T22:25:45.112 DEBUG:boto:Token: None 2023-09-29T22:25:45.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c71e087fed0b51bb-0065174ee9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.117 DEBUG:boto:b'hcnphz-53c1000falsec2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.118 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.118 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.118 DEBUG:boto:path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.118 DEBUG:boto:auth_path=/hcnphz-53/?prefix=c 2023-09-29T22:25:45.118 DEBUG:boto:Method: GET 2023-09-29T22:25:45.119 DEBUG:boto:Path: /hcnphz-53/?prefix=c 2023-09-29T22:25:45.119 DEBUG:boto:Data: 2023-09-29T22:25:45.119 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.119 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.119 DEBUG:boto:Params: {} 2023-09-29T22:25:45.119 DEBUG:boto:Token: None 2023-09-29T22:25:45.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058870bb53208a45e-0065174ee9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.166 DEBUG:boto:b'hcnphz-53c1000falsec2023-09-29T22:25:13.712Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:25:45.167 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.167 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.167 DEBUG:boto:path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.168 DEBUG:boto:auth_path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.168 DEBUG:boto:Method: GET 2023-09-29T22:25:45.168 DEBUG:boto:Path: /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.168 DEBUG:boto:Data: 2023-09-29T22:25:45.168 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.169 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.169 DEBUG:boto:Params: {} 2023-09-29T22:25:45.169 DEBUG:boto:Token: None 2023-09-29T22:25:45.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.170 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A= 2023-09-29T22:25:45.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A=', 'Content-Length': '0'} 2023-09-29T22:25:45.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af890f78e08c2538-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.173 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.173 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.174 DEBUG:boto:path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.174 DEBUG:boto:auth_path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.174 DEBUG:boto:Method: GET 2023-09-29T22:25:45.174 DEBUG:boto:Path: /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.175 DEBUG:boto:Data: 2023-09-29T22:25:45.175 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.175 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.175 DEBUG:boto:Params: {} 2023-09-29T22:25:45.175 DEBUG:boto:Token: None 2023-09-29T22:25:45.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A= 2023-09-29T22:25:45.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A=', 'Content-Length': '0'} 2023-09-29T22:25:45.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd56c5d662556707-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.180 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.180 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.180 DEBUG:boto:path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.180 DEBUG:boto:auth_path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.180 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.181 DEBUG:boto:Path: /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.181 DEBUG:boto:Data: 2023-09-29T22:25:45.181 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.181 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.182 DEBUG:boto:Params: {} 2023-09-29T22:25:45.182 DEBUG:boto:Token: None 2023-09-29T22:25:45.182 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.182 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:54C8Ad2ynHyiMnKozsRbEv08YLI= 2023-09-29T22:25:45.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:54C8Ad2ynHyiMnKozsRbEv08YLI=', 'Content-Length': '0'} 2023-09-29T22:25:45.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048dca9c3faa9affd-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.186 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.186 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.186 DEBUG:boto:path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.186 DEBUG:boto:auth_path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.187 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.187 DEBUG:boto:Path: /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.187 DEBUG:boto:Data: 2023-09-29T22:25:45.187 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.187 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.188 DEBUG:boto:Params: {} 2023-09-29T22:25:45.188 DEBUG:boto:Token: None 2023-09-29T22:25:45.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:54C8Ad2ynHyiMnKozsRbEv08YLI= 2023-09-29T22:25:45.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:54C8Ad2ynHyiMnKozsRbEv08YLI=', 'Content-Length': '0'} 2023-09-29T22:25:45.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf5b99572ff4eb44-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:25:45.192 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.192 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.192 DEBUG:boto:path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.192 DEBUG:boto:auth_path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.193 DEBUG:boto:Method: GET 2023-09-29T22:25:45.193 DEBUG:boto:Path: /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.193 DEBUG:boto:Data: 2023-09-29T22:25:45.193 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.193 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.194 DEBUG:boto:Params: {} 2023-09-29T22:25:45.194 DEBUG:boto:Token: None 2023-09-29T22:25:45.194 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A= 2023-09-29T22:25:45.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A=', 'Content-Length': '0'} 2023-09-29T22:25:45.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e751408f7189d763-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.198 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.198 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.198 DEBUG:boto:path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.198 DEBUG:boto:auth_path=/hcnphz-53/d?versionId=null 2023-09-29T22:25:45.198 DEBUG:boto:Method: GET 2023-09-29T22:25:45.199 DEBUG:boto:Path: /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.199 DEBUG:boto:Data: 2023-09-29T22:25:45.199 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.199 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.200 DEBUG:boto:Params: {} 2023-09-29T22:25:45.200 DEBUG:boto:Token: None 2023-09-29T22:25:45.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d?versionId=null 2023-09-29T22:25:45.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A= 2023-09-29T22:25:45.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylw8CnT6kbhQ0oaNhrK+LLAak4A=', 'Content-Length': '0'} 2023-09-29T22:25:45.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c14882b0f9d96f43-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.246 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.247 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.247 DEBUG:boto:path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.247 DEBUG:boto:auth_path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.247 DEBUG:boto:Method: GET 2023-09-29T22:25:45.247 DEBUG:boto:Path: /hcnphz-53/?prefix=d 2023-09-29T22:25:45.247 DEBUG:boto:Data: 2023-09-29T22:25:45.247 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.247 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.248 DEBUG:boto:Params: {} 2023-09-29T22:25:45.248 DEBUG:boto:Token: None 2023-09-29T22:25:45.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003bd7d4963e2d37af-0065174ee9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.253 DEBUG:boto:b'hcnphz-53d1000falsed2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.253 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.254 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.254 DEBUG:boto:path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.254 DEBUG:boto:auth_path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.254 DEBUG:boto:Method: GET 2023-09-29T22:25:45.254 DEBUG:boto:Path: /hcnphz-53/?prefix=d 2023-09-29T22:25:45.254 DEBUG:boto:Data: 2023-09-29T22:25:45.254 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.254 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.254 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.255 DEBUG:boto:Params: {} 2023-09-29T22:25:45.255 DEBUG:boto:Token: None 2023-09-29T22:25:45.255 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.255 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.255 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.259 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503f57da1c454bec-0065174ee9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.259 DEBUG:boto:b'hcnphz-53d1000falsed2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.260 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.260 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.260 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.260 DEBUG:boto:Path: /hcnphz-53/d 2023-09-29T22:25:45.260 DEBUG:boto:Data: 2023-09-29T22:25:45.260 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.260 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.261 DEBUG:boto:Params: {} 2023-09-29T22:25:45.261 DEBUG:boto:Token: None 2023-09-29T22:25:45.261 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d 2023-09-29T22:25:45.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e8w8C0y23xY3kB+rlXuI2i6pqTQ= 2023-09-29T22:25:45.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e8w8C0y23xY3kB+rlXuI2i6pqTQ=', 'Content-Length': '0'} 2023-09-29T22:25:45.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5753a12d09cf7c0-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.264 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.264 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.264 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.264 DEBUG:boto:Path: /hcnphz-53/d 2023-09-29T22:25:45.264 DEBUG:boto:Data: 2023-09-29T22:25:45.264 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.265 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.265 DEBUG:boto:Params: {} 2023-09-29T22:25:45.265 DEBUG:boto:Token: None 2023-09-29T22:25:45.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d 2023-09-29T22:25:45.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e8w8C0y23xY3kB+rlXuI2i6pqTQ= 2023-09-29T22:25:45.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e8w8C0y23xY3kB+rlXuI2i6pqTQ=', 'Content-Length': '0'} 2023-09-29T22:25:45.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024381e6f2fcc4188-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:25:45.268 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.268 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.268 DEBUG:boto:Method: GET 2023-09-29T22:25:45.268 DEBUG:boto:Path: /hcnphz-53/d 2023-09-29T22:25:45.268 DEBUG:boto:Data: 2023-09-29T22:25:45.268 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.269 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.269 DEBUG:boto:Params: {} 2023-09-29T22:25:45.269 DEBUG:boto:Token: None 2023-09-29T22:25:45.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d 2023-09-29T22:25:45.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JC6sscpsdHK++fjylTXAig7VYMQ= 2023-09-29T22:25:45.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JC6sscpsdHK++fjylTXAig7VYMQ=', 'Content-Length': '0'} 2023-09-29T22:25:45.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028c80fb129dcc94a-0065174ee9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.314 DEBUG:boto:path=/hcnphz-53/d 2023-09-29T22:25:45.315 DEBUG:boto:auth_path=/hcnphz-53/d 2023-09-29T22:25:45.315 DEBUG:boto:Method: GET 2023-09-29T22:25:45.315 DEBUG:boto:Path: /hcnphz-53/d 2023-09-29T22:25:45.315 DEBUG:boto:Data: 2023-09-29T22:25:45.315 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.315 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.316 DEBUG:boto:Params: {} 2023-09-29T22:25:45.316 DEBUG:boto:Token: None 2023-09-29T22:25:45.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/d 2023-09-29T22:25:45.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JC6sscpsdHK++fjylTXAig7VYMQ= 2023-09-29T22:25:45.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JC6sscpsdHK++fjylTXAig7VYMQ=', 'Content-Length': '0'} 2023-09-29T22:25:45.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8440515724a8e8-0065174ee9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.320 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.320 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.320 DEBUG:boto:path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.320 DEBUG:boto:auth_path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.320 DEBUG:boto:Method: GET 2023-09-29T22:25:45.320 DEBUG:boto:Path: /hcnphz-53/?prefix=d 2023-09-29T22:25:45.320 DEBUG:boto:Data: 2023-09-29T22:25:45.321 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:25:45.321 DEBUG:boto:Port: 8001 2023-09-29T22:25:45.321 DEBUG:boto:Params: {} 2023-09-29T22:25:45.321 DEBUG:boto:Token: None 2023-09-29T22:25:45.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba562d4ff61bd572-0065174ee9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.327 DEBUG:boto:b'hcnphz-53d1000falsed2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.327 DEBUG:boto:path=/hcnphz-53/ 2023-09-29T22:25:45.327 DEBUG:boto:auth_path=/hcnphz-53/ 2023-09-29T22:25:45.327 DEBUG:boto:path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.327 DEBUG:boto:auth_path=/hcnphz-53/?prefix=d 2023-09-29T22:25:45.327 DEBUG:boto:Method: GET 2023-09-29T22:25:45.328 DEBUG:boto:Path: /hcnphz-53/?prefix=d 2023-09-29T22:25:45.328 DEBUG:boto:Data: 2023-09-29T22:25:45.328 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.328 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.328 DEBUG:boto:Params: {} 2023-09-29T22:25:45.328 DEBUG:boto:Token: None 2023-09-29T22:25:45.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-53/ 2023-09-29T22:25:45.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A= 2023-09-29T22:25:45.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qCr77XpRMgtdaZs4GO2jH77xu5A=', 'Content-Length': '0'} 2023-09-29T22:25:45.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b0112135a909814-0065174ee9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.378 DEBUG:boto:b'hcnphz-53d1000falsed2023-09-29T22:25:13.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:25:45.379 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-53 zones={a2, a1} 2023-09-29T22:25:45.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:25:45.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:25:45.379 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:25:45.421 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:45.422 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:45.509 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-54 2023-09-29T22:25:45.509 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:25:45.509 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:25:45.509 DEBUG:boto:Method: PUT 2023-09-29T22:25:45.510 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:25:45.510 DEBUG:boto:Data: 2023-09-29T22:25:45.510 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.510 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.510 DEBUG:boto:Params: {} 2023-09-29T22:25:45.510 DEBUG:boto:Token: None 2023-09-29T22:25:45.511 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/ 2023-09-29T22:25:45.511 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Jg9SzbWuZgAeyakJYAAOPSNPxGY= 2023-09-29T22:25:45.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Jg9SzbWuZgAeyakJYAAOPSNPxGY=', 'Content-Length': '0'} 2023-09-29T22:25:45.543 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b46016f24f877fce-0065174ee9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.543 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:25:45.543 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:25:45.543 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.544 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:25:45.544 DEBUG:boto:Data: 2023-09-29T22:25:45.544 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.544 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.544 DEBUG:boto:Params: {} 2023-09-29T22:25:45.544 DEBUG:boto:Token: None 2023-09-29T22:25:45.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/ 2023-09-29T22:25:45.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE= 2023-09-29T22:25:45.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE=', 'Content-Length': '0'} 2023-09-29T22:25:45.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1beecb83455ec68-0065174ee9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.548 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:25:45.548 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:25:45.548 DEBUG:boto:Method: PUT 2023-09-29T22:25:45.548 DEBUG:boto:Path: /hcnphz-54/a 2023-09-29T22:25:45.549 DEBUG:boto:Data: 2023-09-29T22:25:45.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:45.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.549 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.549 DEBUG:boto:Params: {} 2023-09-29T22:25:45.549 DEBUG:boto:Token: None 2023-09-29T22:25:45.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/a 2023-09-29T22:25:45.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:va8AP31ydJGlnVXZQA48SciVW9I= 2023-09-29T22:25:45.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:va8AP31ydJGlnVXZQA48SciVW9I='} 2023-09-29T22:25:45.559 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000039500b5c86f5a49-0065174ee9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.560 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:25:45.560 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:25:45.560 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.560 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:25:45.560 DEBUG:boto:Data: 2023-09-29T22:25:45.560 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.560 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.561 DEBUG:boto:Params: {} 2023-09-29T22:25:45.561 DEBUG:boto:Token: None 2023-09-29T22:25:45.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/ 2023-09-29T22:25:45.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE= 2023-09-29T22:25:45.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE=', 'Content-Length': '0'} 2023-09-29T22:25:45.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfa485af55fae91e-0065174ee9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.565 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:25:45.565 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:25:45.565 DEBUG:boto:Method: PUT 2023-09-29T22:25:45.565 DEBUG:boto:Path: /hcnphz-54/b 2023-09-29T22:25:45.565 DEBUG:boto:Data: 2023-09-29T22:25:45.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:45.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.565 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.566 DEBUG:boto:Params: {} 2023-09-29T22:25:45.566 DEBUG:boto:Token: None 2023-09-29T22:25:45.566 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/b 2023-09-29T22:25:45.566 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cotTBAnky1IptM8lxqBxJLjpWXE= 2023-09-29T22:25:45.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cotTBAnky1IptM8lxqBxJLjpWXE='} 2023-09-29T22:25:45.614 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000012d65d87a3d3b4a3-0065174ee9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.615 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:25:45.615 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:25:45.615 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.615 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:25:45.615 DEBUG:boto:Data: 2023-09-29T22:25:45.615 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.616 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.616 DEBUG:boto:Params: {} 2023-09-29T22:25:45.616 DEBUG:boto:Token: None 2023-09-29T22:25:45.616 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/ 2023-09-29T22:25:45.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE= 2023-09-29T22:25:45.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE=', 'Content-Length': '0'} 2023-09-29T22:25:45.620 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dc1edf84ed511db8-0065174ee9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.621 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:25:45.621 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:25:45.621 DEBUG:boto:Method: PUT 2023-09-29T22:25:45.621 DEBUG:boto:Path: /hcnphz-54/c 2023-09-29T22:25:45.621 DEBUG:boto:Data: 2023-09-29T22:25:45.621 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:45.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.621 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.621 DEBUG:boto:Params: {} 2023-09-29T22:25:45.622 DEBUG:boto:Token: None 2023-09-29T22:25:45.622 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/c 2023-09-29T22:25:45.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fgvgBqjN6dRd5dzikosfdpbpx8w= 2023-09-29T22:25:45.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fgvgBqjN6dRd5dzikosfdpbpx8w='} 2023-09-29T22:25:45.632 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003c763c0ffaea4df0-0065174ee9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.633 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:25:45.633 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:25:45.633 DEBUG:boto:Method: HEAD 2023-09-29T22:25:45.633 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:25:45.633 DEBUG:boto:Data: 2023-09-29T22:25:45.633 DEBUG:boto:Headers: {} 2023-09-29T22:25:45.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.633 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.633 DEBUG:boto:Params: {} 2023-09-29T22:25:45.634 DEBUG:boto:Token: None 2023-09-29T22:25:45.634 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/ 2023-09-29T22:25:45.634 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE= 2023-09-29T22:25:45.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bUBYjqnnUCi9wq6Tl25E/LHUtPE=', 'Content-Length': '0'} 2023-09-29T22:25:45.638 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000617adc450c3ca72d-0065174ee9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.638 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:25:45.638 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:25:45.638 DEBUG:boto:Method: PUT 2023-09-29T22:25:45.639 DEBUG:boto:Path: /hcnphz-54/d 2023-09-29T22:25:45.639 DEBUG:boto:Data: 2023-09-29T22:25:45.639 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:25:45.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:25:45.639 DEBUG:boto:Port: 8000 2023-09-29T22:25:45.639 DEBUG:boto:Params: {} 2023-09-29T22:25:45.639 DEBUG:boto:Token: None 2023-09-29T22:25:45.639 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:25:45 GMT /hcnphz-54/d 2023-09-29T22:25:45.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rFOmakoDrI4nBUVA0sdrPwtaH3Q= 2023-09-29T22:25:45.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:25:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rFOmakoDrI4nBUVA0sdrPwtaH3Q='} 2023-09-29T22:25:45.682 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000770506a679dce1f4-0065174ee9-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:25:45.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:45.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:45.683 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:45.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:45.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:45.781 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:25:45.806 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:25:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:25:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:25:45.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026345.541383_419.1", 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:25:45.541383Z" 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:25:45.808 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:45.808 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:25:45.809 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026345.541383_419.1', 2: '', 3: ''} 2023-09-29T22:25:45.809 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:25:45.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:45.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:25:45.810 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:25:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:45.851 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:45.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:45.906+0000 7f361f2c1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:45.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:45.906+0000 7f361f2c1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:25:45.938 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:25:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:25:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:25:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:25:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:25:45.938 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:45.939 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026345.541383_419.1", 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:45.941 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:45.541383Z", 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:45.942 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.943 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:25:45.944 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:25:45.945 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:25:45.946 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:25:45.946 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:25:45Z" 2023-09-29T22:25:45.946 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:25:45.947 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026345.541383_419.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:25:45.541383Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:25:45Z" } 2023-09-29T22:25:45.948 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026345.541383_419.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:25:45.541383Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:25:45.948 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026345.541383_419.1', 2: '', 3: ''} 2023-09-29T22:25:45.948 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026345.541383_419.1', 2: '', 3: ''} 2023-09-29T22:25:45.948 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:25:45.948 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-54 2023-09-29T22:25:45.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:45.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-54', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:45.949 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-54 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:46.000 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:46.000 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:46.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:46.062+0000 7f193bdcaf40 1 waiting to reach incremental sync.. 2023-09-29T22:25:51.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:51.065+0000 7f193bdcaf40 1 bucket sync caught up with source: 2023-09-29T22:25:51.070 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.40.6, , , , 00000000001.36.6, , , 00000000001.36.6, , 00000000001.16.6] 2023-09-29T22:25:51.070 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.40.6, , , , 00000000001.36.6, , , 00000000001.36.6, , 00000000001.16.6] 2023-09-29T22:25:51.070 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:51.065+0000 7f193bdcaf40 0 bucket checkpoint complete 2023-09-29T22:25:51.099 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-54 2023-09-29T22:25:51.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:51.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-54', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:25:51.100 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-54 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:25:51.152 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:25:51.152 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:25:51.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:51.209+0000 7fe00fb6af40 1 waiting to reach incremental sync.. 2023-09-29T22:25:56.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:25:56.209+0000 7fe00fb6af40 1 waiting to reach incremental sync.. 2023-09-29T22:26:01.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:01.213+0000 7fe00fb6af40 1 bucket sync caught up with source: 2023-09-29T22:26:01.219 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.256.6, , , , 00000000001.289.6, , , 00000000001.95.6, , 00000000001.116.6] 2023-09-29T22:26:01.219 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.256.6, , , , 00000000001.289.6, , , 00000000001.95.6, , 00000000001.116.6] 2023-09-29T22:26:01.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:01.213+0000 7fe00fb6af40 0 bucket checkpoint complete 2023-09-29T22:26:01.248 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-54 zones={a2, a1} 2023-09-29T22:26:01.249 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.249 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.249 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.249 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:26:01.249 DEBUG:boto:Data: 2023-09-29T22:26:01.249 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.249 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.250 DEBUG:boto:Params: {} 2023-09-29T22:26:01.250 DEBUG:boto:Token: None 2023-09-29T22:26:01.250 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QEbSmDFBqys9JxxS05FdSQyMNAA= 2023-09-29T22:26:01.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QEbSmDFBqys9JxxS05FdSQyMNAA=', 'Content-Length': '0'} 2023-09-29T22:26:01.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000028a06bf0559bb057-0065174ef9-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.254 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.254 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.254 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.255 DEBUG:boto:Path: /hcnphz-54/ 2023-09-29T22:26:01.255 DEBUG:boto:Data: 2023-09-29T22:26:01.255 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.255 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.255 DEBUG:boto:Params: {} 2023-09-29T22:26:01.255 DEBUG:boto:Token: None 2023-09-29T22:26:01.255 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QEbSmDFBqys9JxxS05FdSQyMNAA= 2023-09-29T22:26:01.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QEbSmDFBqys9JxxS05FdSQyMNAA=', 'Content-Length': '0'} 2023-09-29T22:26:01.259 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d74b1155a595722f-0065174ef9-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.261 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:01.261 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.261 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.262 DEBUG:boto:path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.262 DEBUG:boto:auth_path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.262 DEBUG:boto:Method: GET 2023-09-29T22:26:01.262 DEBUG:boto:Path: /hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.262 DEBUG:boto:Data: 2023-09-29T22:26:01.262 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.262 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.262 DEBUG:boto:Params: {} 2023-09-29T22:26:01.262 DEBUG:boto:Token: None 2023-09-29T22:26:01.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/?versions 2023-09-29T22:26:01.263 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg= 2023-09-29T22:26:01.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg=', 'Content-Length': '0'} 2023-09-29T22:26:01.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d20bcc2cc0f0af4-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.266 DEBUG:boto:b'hcnphz-54999falseanulltrue2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.267 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:26:01.267 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:26:01.267 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:26:01.267 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:26:01.268 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:01.268 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.268 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.268 DEBUG:boto:path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.268 DEBUG:boto:auth_path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.268 DEBUG:boto:Method: GET 2023-09-29T22:26:01.268 DEBUG:boto:Path: /hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.268 DEBUG:boto:Data: 2023-09-29T22:26:01.268 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.269 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.269 DEBUG:boto:Params: {} 2023-09-29T22:26:01.269 DEBUG:boto:Token: None 2023-09-29T22:26:01.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/?versions 2023-09-29T22:26:01.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg= 2023-09-29T22:26:01.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg=', 'Content-Length': '0'} 2023-09-29T22:26:01.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3947b2ac3f6155b-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.274 DEBUG:boto:b'hcnphz-54999falseanulltrue2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.274 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:26:01.274 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:26:01.274 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:26:01.274 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:26:01.275 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.275 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.275 DEBUG:boto:path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.275 DEBUG:boto:auth_path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.275 DEBUG:boto:Method: GET 2023-09-29T22:26:01.275 DEBUG:boto:Path: /hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.275 DEBUG:boto:Data: 2023-09-29T22:26:01.275 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.275 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.276 DEBUG:boto:Params: {} 2023-09-29T22:26:01.276 DEBUG:boto:Token: None 2023-09-29T22:26:01.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/?versions 2023-09-29T22:26:01.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg= 2023-09-29T22:26:01.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg=', 'Content-Length': '0'} 2023-09-29T22:26:01.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb7603c97410f867-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.280 DEBUG:boto:b'hcnphz-54999falseanulltrue2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.281 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.281 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.281 DEBUG:boto:path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.281 DEBUG:boto:auth_path=/hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.281 DEBUG:boto:Method: GET 2023-09-29T22:26:01.281 DEBUG:boto:Path: /hcnphz-54/?versions&max-keys=999 2023-09-29T22:26:01.281 DEBUG:boto:Data: 2023-09-29T22:26:01.281 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.282 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.282 DEBUG:boto:Params: {} 2023-09-29T22:26:01.282 DEBUG:boto:Token: None 2023-09-29T22:26:01.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/?versions 2023-09-29T22:26:01.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg= 2023-09-29T22:26:01.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITPgyCjJSsLpWw3vGJj+ERtvVlg=', 'Content-Length': '0'} 2023-09-29T22:26:01.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f38792b44546919c-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.286 DEBUG:boto:b'hcnphz-54999falseanulltrue2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:26:01.287 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.287 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.287 DEBUG:boto:path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.288 DEBUG:boto:auth_path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.288 DEBUG:boto:Method: GET 2023-09-29T22:26:01.288 DEBUG:boto:Path: /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.288 DEBUG:boto:Data: 2023-09-29T22:26:01.288 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.288 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.288 DEBUG:boto:Params: {} 2023-09-29T22:26:01.288 DEBUG:boto:Token: None 2023-09-29T22:26:01.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA= 2023-09-29T22:26:01.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA=', 'Content-Length': '0'} 2023-09-29T22:26:01.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af9bc72879f40c86-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.334 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.335 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.335 DEBUG:boto:path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.335 DEBUG:boto:auth_path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.335 DEBUG:boto:Method: GET 2023-09-29T22:26:01.335 DEBUG:boto:Path: /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.335 DEBUG:boto:Data: 2023-09-29T22:26:01.335 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.336 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.336 DEBUG:boto:Params: {} 2023-09-29T22:26:01.336 DEBUG:boto:Token: None 2023-09-29T22:26:01.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA= 2023-09-29T22:26:01.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA=', 'Content-Length': '0'} 2023-09-29T22:26:01.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbf05da1736816dc-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.339 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.339 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.339 DEBUG:boto:path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.339 DEBUG:boto:auth_path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.340 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.340 DEBUG:boto:Path: /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.340 DEBUG:boto:Data: 2023-09-29T22:26:01.340 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.340 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.340 DEBUG:boto:Params: {} 2023-09-29T22:26:01.340 DEBUG:boto:Token: None 2023-09-29T22:26:01.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.341 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3k25brhenhO8XTsCJwa2wRkmVBk= 2023-09-29T22:26:01.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3k25brhenhO8XTsCJwa2wRkmVBk=', 'Content-Length': '0'} 2023-09-29T22:26:01.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3f3884ee341dbf-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.344 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.344 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.344 DEBUG:boto:path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.344 DEBUG:boto:auth_path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.344 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.344 DEBUG:boto:Path: /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.344 DEBUG:boto:Data: 2023-09-29T22:26:01.344 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.344 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.345 DEBUG:boto:Params: {} 2023-09-29T22:26:01.345 DEBUG:boto:Token: None 2023-09-29T22:26:01.345 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3k25brhenhO8XTsCJwa2wRkmVBk= 2023-09-29T22:26:01.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3k25brhenhO8XTsCJwa2wRkmVBk=', 'Content-Length': '0'} 2023-09-29T22:26:01.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000075bfe3b33aff740-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:26:01.348 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.348 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.348 DEBUG:boto:path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.348 DEBUG:boto:auth_path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.348 DEBUG:boto:Method: GET 2023-09-29T22:26:01.349 DEBUG:boto:Path: /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.349 DEBUG:boto:Data: 2023-09-29T22:26:01.349 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.349 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.349 DEBUG:boto:Params: {} 2023-09-29T22:26:01.349 DEBUG:boto:Token: None 2023-09-29T22:26:01.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA= 2023-09-29T22:26:01.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA=', 'Content-Length': '0'} 2023-09-29T22:26:01.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae1068a5b64db012-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.352 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.352 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.353 DEBUG:boto:path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.353 DEBUG:boto:auth_path=/hcnphz-54/a?versionId=null 2023-09-29T22:26:01.353 DEBUG:boto:Method: GET 2023-09-29T22:26:01.353 DEBUG:boto:Path: /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.353 DEBUG:boto:Data: 2023-09-29T22:26:01.353 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.353 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.353 DEBUG:boto:Params: {} 2023-09-29T22:26:01.354 DEBUG:boto:Token: None 2023-09-29T22:26:01.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a?versionId=null 2023-09-29T22:26:01.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA= 2023-09-29T22:26:01.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Szamk4YChkTaXjNtDg9iSkcziA=', 'Content-Length': '0'} 2023-09-29T22:26:01.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f0799da1bee0c9c-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.398 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.398 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.399 DEBUG:boto:path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.399 DEBUG:boto:auth_path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.399 DEBUG:boto:Method: GET 2023-09-29T22:26:01.399 DEBUG:boto:Path: /hcnphz-54/?prefix=a 2023-09-29T22:26:01.399 DEBUG:boto:Data: 2023-09-29T22:26:01.399 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.399 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.399 DEBUG:boto:Params: {} 2023-09-29T22:26:01.400 DEBUG:boto:Token: None 2023-09-29T22:26:01.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784dc41511d330b6-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.404 DEBUG:boto:b'hcnphz-54a1000falsea2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.405 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.405 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.405 DEBUG:boto:path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.405 DEBUG:boto:auth_path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.405 DEBUG:boto:Method: GET 2023-09-29T22:26:01.405 DEBUG:boto:Path: /hcnphz-54/?prefix=a 2023-09-29T22:26:01.405 DEBUG:boto:Data: 2023-09-29T22:26:01.405 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.406 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.406 DEBUG:boto:Params: {} 2023-09-29T22:26:01.406 DEBUG:boto:Token: None 2023-09-29T22:26:01.406 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.411 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f720eb99481e215-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.411 DEBUG:boto:b'hcnphz-54a1000falsea2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.411 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.411 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.411 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.412 DEBUG:boto:Path: /hcnphz-54/a 2023-09-29T22:26:01.412 DEBUG:boto:Data: 2023-09-29T22:26:01.412 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.412 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.412 DEBUG:boto:Params: {} 2023-09-29T22:26:01.412 DEBUG:boto:Token: None 2023-09-29T22:26:01.412 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a 2023-09-29T22:26:01.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Gz2oJIU3kX19/8WDl9J+bcKwE1Y= 2023-09-29T22:26:01.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Gz2oJIU3kX19/8WDl9J+bcKwE1Y=', 'Content-Length': '0'} 2023-09-29T22:26:01.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7d5cd3c8bd71956-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.415 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.415 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.415 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.415 DEBUG:boto:Path: /hcnphz-54/a 2023-09-29T22:26:01.415 DEBUG:boto:Data: 2023-09-29T22:26:01.415 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.416 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.416 DEBUG:boto:Params: {} 2023-09-29T22:26:01.416 DEBUG:boto:Token: None 2023-09-29T22:26:01.416 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a 2023-09-29T22:26:01.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Gz2oJIU3kX19/8WDl9J+bcKwE1Y= 2023-09-29T22:26:01.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Gz2oJIU3kX19/8WDl9J+bcKwE1Y=', 'Content-Length': '0'} 2023-09-29T22:26:01.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2a81871491c3bd1-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:26:01.419 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.419 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.419 DEBUG:boto:Method: GET 2023-09-29T22:26:01.419 DEBUG:boto:Path: /hcnphz-54/a 2023-09-29T22:26:01.419 DEBUG:boto:Data: 2023-09-29T22:26:01.419 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.419 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.420 DEBUG:boto:Params: {} 2023-09-29T22:26:01.420 DEBUG:boto:Token: None 2023-09-29T22:26:01.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a 2023-09-29T22:26:01.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+35eng79JnJXM58fhUSvEce4ZU= 2023-09-29T22:26:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+35eng79JnJXM58fhUSvEce4ZU=', 'Content-Length': '0'} 2023-09-29T22:26:01.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015ec56e946caf422-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.470 DEBUG:boto:path=/hcnphz-54/a 2023-09-29T22:26:01.471 DEBUG:boto:auth_path=/hcnphz-54/a 2023-09-29T22:26:01.471 DEBUG:boto:Method: GET 2023-09-29T22:26:01.471 DEBUG:boto:Path: /hcnphz-54/a 2023-09-29T22:26:01.471 DEBUG:boto:Data: 2023-09-29T22:26:01.471 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.471 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.471 DEBUG:boto:Params: {} 2023-09-29T22:26:01.471 DEBUG:boto:Token: None 2023-09-29T22:26:01.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/a 2023-09-29T22:26:01.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+35eng79JnJXM58fhUSvEce4ZU= 2023-09-29T22:26:01.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+35eng79JnJXM58fhUSvEce4ZU=', 'Content-Length': '0'} 2023-09-29T22:26:01.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b5ae3bc8aed35a-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.475 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.475 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.475 DEBUG:boto:path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.475 DEBUG:boto:auth_path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.475 DEBUG:boto:Method: GET 2023-09-29T22:26:01.475 DEBUG:boto:Path: /hcnphz-54/?prefix=a 2023-09-29T22:26:01.475 DEBUG:boto:Data: 2023-09-29T22:26:01.475 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.476 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.476 DEBUG:boto:Params: {} 2023-09-29T22:26:01.476 DEBUG:boto:Token: None 2023-09-29T22:26:01.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.480 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fcef7995a87e39e-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.480 DEBUG:boto:b'hcnphz-54a1000falsea2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.481 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.481 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.481 DEBUG:boto:path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.481 DEBUG:boto:auth_path=/hcnphz-54/?prefix=a 2023-09-29T22:26:01.481 DEBUG:boto:Method: GET 2023-09-29T22:26:01.481 DEBUG:boto:Path: /hcnphz-54/?prefix=a 2023-09-29T22:26:01.481 DEBUG:boto:Data: 2023-09-29T22:26:01.481 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.482 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.482 DEBUG:boto:Params: {} 2023-09-29T22:26:01.482 DEBUG:boto:Token: None 2023-09-29T22:26:01.482 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.482 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d2ce21d26af3713-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.534 DEBUG:boto:b'hcnphz-54a1000falsea2023-09-29T22:25:45.553Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:26:01.535 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.535 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.535 DEBUG:boto:path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.535 DEBUG:boto:auth_path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.535 DEBUG:boto:Method: GET 2023-09-29T22:26:01.536 DEBUG:boto:Path: /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.536 DEBUG:boto:Data: 2023-09-29T22:26:01.536 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.536 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.536 DEBUG:boto:Params: {} 2023-09-29T22:26:01.536 DEBUG:boto:Token: None 2023-09-29T22:26:01.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI= 2023-09-29T22:26:01.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI=', 'Content-Length': '0'} 2023-09-29T22:26:01.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002be7bea0c789ec39-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.540 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.540 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.540 DEBUG:boto:path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.540 DEBUG:boto:auth_path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.540 DEBUG:boto:Method: GET 2023-09-29T22:26:01.540 DEBUG:boto:Path: /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.541 DEBUG:boto:Data: 2023-09-29T22:26:01.541 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.541 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.541 DEBUG:boto:Params: {} 2023-09-29T22:26:01.541 DEBUG:boto:Token: None 2023-09-29T22:26:01.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI= 2023-09-29T22:26:01.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI=', 'Content-Length': '0'} 2023-09-29T22:26:01.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df93750d958975de-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.545 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.545 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.545 DEBUG:boto:path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.545 DEBUG:boto:auth_path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.545 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.545 DEBUG:boto:Path: /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.546 DEBUG:boto:Data: 2023-09-29T22:26:01.546 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.546 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.546 DEBUG:boto:Params: {} 2023-09-29T22:26:01.546 DEBUG:boto:Token: None 2023-09-29T22:26:01.546 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.546 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A3FM4HQfWcoY5yMO6tVJ6zJ4pQs= 2023-09-29T22:26:01.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A3FM4HQfWcoY5yMO6tVJ6zJ4pQs=', 'Content-Length': '0'} 2023-09-29T22:26:01.549 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a76fbe5694d9a69-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.549 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.549 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.550 DEBUG:boto:path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.550 DEBUG:boto:auth_path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.550 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.550 DEBUG:boto:Path: /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.550 DEBUG:boto:Data: 2023-09-29T22:26:01.550 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.550 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.550 DEBUG:boto:Params: {} 2023-09-29T22:26:01.551 DEBUG:boto:Token: None 2023-09-29T22:26:01.551 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.551 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A3FM4HQfWcoY5yMO6tVJ6zJ4pQs= 2023-09-29T22:26:01.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A3FM4HQfWcoY5yMO6tVJ6zJ4pQs=', 'Content-Length': '0'} 2023-09-29T22:26:01.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef539ac276600b58-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:26:01.554 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.554 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.554 DEBUG:boto:path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.554 DEBUG:boto:auth_path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.554 DEBUG:boto:Method: GET 2023-09-29T22:26:01.554 DEBUG:boto:Path: /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.555 DEBUG:boto:Data: 2023-09-29T22:26:01.555 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.555 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.555 DEBUG:boto:Params: {} 2023-09-29T22:26:01.555 DEBUG:boto:Token: None 2023-09-29T22:26:01.555 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI= 2023-09-29T22:26:01.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI=', 'Content-Length': '0'} 2023-09-29T22:26:01.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7517f7391cab14b-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.558 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.559 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.559 DEBUG:boto:path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.559 DEBUG:boto:auth_path=/hcnphz-54/b?versionId=null 2023-09-29T22:26:01.559 DEBUG:boto:Method: GET 2023-09-29T22:26:01.559 DEBUG:boto:Path: /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.559 DEBUG:boto:Data: 2023-09-29T22:26:01.559 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.559 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.560 DEBUG:boto:Params: {} 2023-09-29T22:26:01.560 DEBUG:boto:Token: None 2023-09-29T22:26:01.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b?versionId=null 2023-09-29T22:26:01.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI= 2023-09-29T22:26:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZOlq+rfaRMxrcXGDtTkRmKAw7yI=', 'Content-Length': '0'} 2023-09-29T22:26:01.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f040443455f767-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.563 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.563 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.563 DEBUG:boto:path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.563 DEBUG:boto:auth_path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.563 DEBUG:boto:Method: GET 2023-09-29T22:26:01.564 DEBUG:boto:Path: /hcnphz-54/?prefix=b 2023-09-29T22:26:01.564 DEBUG:boto:Data: 2023-09-29T22:26:01.564 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.564 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.564 DEBUG:boto:Params: {} 2023-09-29T22:26:01.564 DEBUG:boto:Token: None 2023-09-29T22:26:01.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.569 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086482f2140fdd7d1-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.610 DEBUG:boto:b'hcnphz-54b1000falseb2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.611 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.611 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.611 DEBUG:boto:path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.611 DEBUG:boto:auth_path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.611 DEBUG:boto:Method: GET 2023-09-29T22:26:01.611 DEBUG:boto:Path: /hcnphz-54/?prefix=b 2023-09-29T22:26:01.611 DEBUG:boto:Data: 2023-09-29T22:26:01.611 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.612 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.612 DEBUG:boto:Params: {} 2023-09-29T22:26:01.612 DEBUG:boto:Token: None 2023-09-29T22:26:01.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065b4e4a1f74aad65-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.617 DEBUG:boto:b'hcnphz-54b1000falseb2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.617 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.617 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.618 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.618 DEBUG:boto:Path: /hcnphz-54/b 2023-09-29T22:26:01.618 DEBUG:boto:Data: 2023-09-29T22:26:01.618 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.618 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.618 DEBUG:boto:Params: {} 2023-09-29T22:26:01.618 DEBUG:boto:Token: None 2023-09-29T22:26:01.619 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b 2023-09-29T22:26:01.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9i5oRR18Q6XRFUylMOoSHqS1zhg= 2023-09-29T22:26:01.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9i5oRR18Q6XRFUylMOoSHqS1zhg=', 'Content-Length': '0'} 2023-09-29T22:26:01.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e77893f99cae468-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.621 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.622 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.622 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.622 DEBUG:boto:Path: /hcnphz-54/b 2023-09-29T22:26:01.622 DEBUG:boto:Data: 2023-09-29T22:26:01.622 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.622 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.622 DEBUG:boto:Params: {} 2023-09-29T22:26:01.623 DEBUG:boto:Token: None 2023-09-29T22:26:01.623 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b 2023-09-29T22:26:01.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9i5oRR18Q6XRFUylMOoSHqS1zhg= 2023-09-29T22:26:01.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9i5oRR18Q6XRFUylMOoSHqS1zhg=', 'Content-Length': '0'} 2023-09-29T22:26:01.626 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1437ac6a558ca13-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:26:01.626 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.626 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.627 DEBUG:boto:Method: GET 2023-09-29T22:26:01.627 DEBUG:boto:Path: /hcnphz-54/b 2023-09-29T22:26:01.627 DEBUG:boto:Data: 2023-09-29T22:26:01.627 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.627 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.627 DEBUG:boto:Params: {} 2023-09-29T22:26:01.627 DEBUG:boto:Token: None 2023-09-29T22:26:01.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b 2023-09-29T22:26:01.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2GUAWhCwyFs6KpSsJoKRK5rpGxE= 2023-09-29T22:26:01.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2GUAWhCwyFs6KpSsJoKRK5rpGxE=', 'Content-Length': '0'} 2023-09-29T22:26:01.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e14c32bd1e883f-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.631 DEBUG:boto:path=/hcnphz-54/b 2023-09-29T22:26:01.631 DEBUG:boto:auth_path=/hcnphz-54/b 2023-09-29T22:26:01.631 DEBUG:boto:Method: GET 2023-09-29T22:26:01.631 DEBUG:boto:Path: /hcnphz-54/b 2023-09-29T22:26:01.631 DEBUG:boto:Data: 2023-09-29T22:26:01.631 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.631 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.632 DEBUG:boto:Params: {} 2023-09-29T22:26:01.632 DEBUG:boto:Token: None 2023-09-29T22:26:01.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/b 2023-09-29T22:26:01.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2GUAWhCwyFs6KpSsJoKRK5rpGxE= 2023-09-29T22:26:01.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2GUAWhCwyFs6KpSsJoKRK5rpGxE=', 'Content-Length': '0'} 2023-09-29T22:26:01.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003686fc315612f609-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.679 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.679 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.679 DEBUG:boto:path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.679 DEBUG:boto:auth_path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.679 DEBUG:boto:Method: GET 2023-09-29T22:26:01.679 DEBUG:boto:Path: /hcnphz-54/?prefix=b 2023-09-29T22:26:01.679 DEBUG:boto:Data: 2023-09-29T22:26:01.679 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.680 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.680 DEBUG:boto:Params: {} 2023-09-29T22:26:01.680 DEBUG:boto:Token: None 2023-09-29T22:26:01.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000271cd5428f775823-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.686 DEBUG:boto:b'hcnphz-54b1000falseb2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.686 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.686 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.686 DEBUG:boto:path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.687 DEBUG:boto:auth_path=/hcnphz-54/?prefix=b 2023-09-29T22:26:01.687 DEBUG:boto:Method: GET 2023-09-29T22:26:01.687 DEBUG:boto:Path: /hcnphz-54/?prefix=b 2023-09-29T22:26:01.687 DEBUG:boto:Data: 2023-09-29T22:26:01.687 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.687 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.687 DEBUG:boto:Params: {} 2023-09-29T22:26:01.687 DEBUG:boto:Token: None 2023-09-29T22:26:01.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2157e531a14b20-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.693 DEBUG:boto:b'hcnphz-54b1000falseb2023-09-29T22:25:45.570Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:26:01.693 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.694 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.694 DEBUG:boto:path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.694 DEBUG:boto:auth_path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.694 DEBUG:boto:Method: GET 2023-09-29T22:26:01.694 DEBUG:boto:Path: /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.694 DEBUG:boto:Data: 2023-09-29T22:26:01.694 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.694 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.694 DEBUG:boto:Params: {} 2023-09-29T22:26:01.695 DEBUG:boto:Token: None 2023-09-29T22:26:01.695 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.695 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE= 2023-09-29T22:26:01.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE=', 'Content-Length': '0'} 2023-09-29T22:26:01.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000289546259e0c2443-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.697 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.697 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.697 DEBUG:boto:path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.698 DEBUG:boto:auth_path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.698 DEBUG:boto:Method: GET 2023-09-29T22:26:01.698 DEBUG:boto:Path: /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.698 DEBUG:boto:Data: 2023-09-29T22:26:01.698 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.698 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.698 DEBUG:boto:Params: {} 2023-09-29T22:26:01.698 DEBUG:boto:Token: None 2023-09-29T22:26:01.699 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE= 2023-09-29T22:26:01.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE=', 'Content-Length': '0'} 2023-09-29T22:26:01.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003519a1cff736efe3-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.701 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.701 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.701 DEBUG:boto:path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.701 DEBUG:boto:auth_path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.701 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.701 DEBUG:boto:Path: /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.701 DEBUG:boto:Data: 2023-09-29T22:26:01.702 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.702 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.702 DEBUG:boto:Params: {} 2023-09-29T22:26:01.702 DEBUG:boto:Token: None 2023-09-29T22:26:01.702 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BGVklqiAyBnYXadvYz4OLvkLD+U= 2023-09-29T22:26:01.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BGVklqiAyBnYXadvYz4OLvkLD+U=', 'Content-Length': '0'} 2023-09-29T22:26:01.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007985c49a944932f-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.705 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.705 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.705 DEBUG:boto:path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.705 DEBUG:boto:auth_path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.705 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.705 DEBUG:boto:Path: /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.705 DEBUG:boto:Data: 2023-09-29T22:26:01.705 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.706 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.706 DEBUG:boto:Params: {} 2023-09-29T22:26:01.706 DEBUG:boto:Token: None 2023-09-29T22:26:01.706 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.706 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BGVklqiAyBnYXadvYz4OLvkLD+U= 2023-09-29T22:26:01.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BGVklqiAyBnYXadvYz4OLvkLD+U=', 'Content-Length': '0'} 2023-09-29T22:26:01.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c560db359e2f18e9-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:26:01.709 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.709 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.709 DEBUG:boto:path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.709 DEBUG:boto:auth_path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.709 DEBUG:boto:Method: GET 2023-09-29T22:26:01.709 DEBUG:boto:Path: /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.709 DEBUG:boto:Data: 2023-09-29T22:26:01.709 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.709 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.710 DEBUG:boto:Params: {} 2023-09-29T22:26:01.710 DEBUG:boto:Token: None 2023-09-29T22:26:01.710 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE= 2023-09-29T22:26:01.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE=', 'Content-Length': '0'} 2023-09-29T22:26:01.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000199a50775dc8c4f4-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.713 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.713 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.713 DEBUG:boto:path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.713 DEBUG:boto:auth_path=/hcnphz-54/c?versionId=null 2023-09-29T22:26:01.713 DEBUG:boto:Method: GET 2023-09-29T22:26:01.713 DEBUG:boto:Path: /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.713 DEBUG:boto:Data: 2023-09-29T22:26:01.713 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.713 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.714 DEBUG:boto:Params: {} 2023-09-29T22:26:01.714 DEBUG:boto:Token: None 2023-09-29T22:26:01.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c?versionId=null 2023-09-29T22:26:01.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE= 2023-09-29T22:26:01.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jLuFqR1Lj4PVOnLcRAe7yJhkSIE=', 'Content-Length': '0'} 2023-09-29T22:26:01.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdceeca6c0edb3c0-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.716 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.717 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.717 DEBUG:boto:path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.717 DEBUG:boto:auth_path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.717 DEBUG:boto:Method: GET 2023-09-29T22:26:01.717 DEBUG:boto:Path: /hcnphz-54/?prefix=c 2023-09-29T22:26:01.717 DEBUG:boto:Data: 2023-09-29T22:26:01.717 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.717 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.718 DEBUG:boto:Params: {} 2023-09-29T22:26:01.718 DEBUG:boto:Token: None 2023-09-29T22:26:01.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c8d2982fd0e74fd-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.722 DEBUG:boto:b'hcnphz-54c1000falsec2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.723 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.723 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.723 DEBUG:boto:path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.723 DEBUG:boto:auth_path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.723 DEBUG:boto:Method: GET 2023-09-29T22:26:01.723 DEBUG:boto:Path: /hcnphz-54/?prefix=c 2023-09-29T22:26:01.723 DEBUG:boto:Data: 2023-09-29T22:26:01.723 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.724 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.724 DEBUG:boto:Params: {} 2023-09-29T22:26:01.724 DEBUG:boto:Token: None 2023-09-29T22:26:01.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076ac3381ec242b8c-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.728 DEBUG:boto:b'hcnphz-54c1000falsec2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.728 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.729 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.729 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.729 DEBUG:boto:Path: /hcnphz-54/c 2023-09-29T22:26:01.729 DEBUG:boto:Data: 2023-09-29T22:26:01.729 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.729 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.729 DEBUG:boto:Params: {} 2023-09-29T22:26:01.729 DEBUG:boto:Token: None 2023-09-29T22:26:01.730 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c 2023-09-29T22:26:01.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g7QvFFrdXhUi6u/rcMGKmeTMfww= 2023-09-29T22:26:01.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g7QvFFrdXhUi6u/rcMGKmeTMfww=', 'Content-Length': '0'} 2023-09-29T22:26:01.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d32c1c391d34bc92-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.732 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.732 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.732 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.733 DEBUG:boto:Path: /hcnphz-54/c 2023-09-29T22:26:01.733 DEBUG:boto:Data: 2023-09-29T22:26:01.733 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.733 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.733 DEBUG:boto:Params: {} 2023-09-29T22:26:01.733 DEBUG:boto:Token: None 2023-09-29T22:26:01.733 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c 2023-09-29T22:26:01.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g7QvFFrdXhUi6u/rcMGKmeTMfww= 2023-09-29T22:26:01.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g7QvFFrdXhUi6u/rcMGKmeTMfww=', 'Content-Length': '0'} 2023-09-29T22:26:01.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f9ff2497d3a1505-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:26:01.736 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.736 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.736 DEBUG:boto:Method: GET 2023-09-29T22:26:01.736 DEBUG:boto:Path: /hcnphz-54/c 2023-09-29T22:26:01.737 DEBUG:boto:Data: 2023-09-29T22:26:01.737 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.737 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.737 DEBUG:boto:Params: {} 2023-09-29T22:26:01.737 DEBUG:boto:Token: None 2023-09-29T22:26:01.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c 2023-09-29T22:26:01.737 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FFTkM2EVWmz+zBr9qLEhihFjRCU= 2023-09-29T22:26:01.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FFTkM2EVWmz+zBr9qLEhihFjRCU=', 'Content-Length': '0'} 2023-09-29T22:26:01.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065421a151875f92d-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.782 DEBUG:boto:path=/hcnphz-54/c 2023-09-29T22:26:01.783 DEBUG:boto:auth_path=/hcnphz-54/c 2023-09-29T22:26:01.783 DEBUG:boto:Method: GET 2023-09-29T22:26:01.783 DEBUG:boto:Path: /hcnphz-54/c 2023-09-29T22:26:01.783 DEBUG:boto:Data: 2023-09-29T22:26:01.783 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.783 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.783 DEBUG:boto:Params: {} 2023-09-29T22:26:01.783 DEBUG:boto:Token: None 2023-09-29T22:26:01.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/c 2023-09-29T22:26:01.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FFTkM2EVWmz+zBr9qLEhihFjRCU= 2023-09-29T22:26:01.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FFTkM2EVWmz+zBr9qLEhihFjRCU=', 'Content-Length': '0'} 2023-09-29T22:26:01.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c32e451a18bfa035-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.787 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.787 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.787 DEBUG:boto:path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.787 DEBUG:boto:auth_path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.787 DEBUG:boto:Method: GET 2023-09-29T22:26:01.787 DEBUG:boto:Path: /hcnphz-54/?prefix=c 2023-09-29T22:26:01.788 DEBUG:boto:Data: 2023-09-29T22:26:01.788 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.788 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.788 DEBUG:boto:Params: {} 2023-09-29T22:26:01.788 DEBUG:boto:Token: None 2023-09-29T22:26:01.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005fe0592b847dba4-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.793 DEBUG:boto:b'hcnphz-54c1000falsec2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.794 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.794 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.794 DEBUG:boto:path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.794 DEBUG:boto:auth_path=/hcnphz-54/?prefix=c 2023-09-29T22:26:01.794 DEBUG:boto:Method: GET 2023-09-29T22:26:01.794 DEBUG:boto:Path: /hcnphz-54/?prefix=c 2023-09-29T22:26:01.794 DEBUG:boto:Data: 2023-09-29T22:26:01.795 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.795 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.795 DEBUG:boto:Params: {} 2023-09-29T22:26:01.795 DEBUG:boto:Token: None 2023-09-29T22:26:01.795 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.795 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.800 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ef1bd3b25ef336-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.800 DEBUG:boto:b'hcnphz-54c1000falsec2023-09-29T22:25:45.626Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:26:01.801 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.801 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.801 DEBUG:boto:path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.801 DEBUG:boto:auth_path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.801 DEBUG:boto:Method: GET 2023-09-29T22:26:01.801 DEBUG:boto:Path: /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.801 DEBUG:boto:Data: 2023-09-29T22:26:01.802 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.802 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.802 DEBUG:boto:Params: {} 2023-09-29T22:26:01.802 DEBUG:boto:Token: None 2023-09-29T22:26:01.802 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg= 2023-09-29T22:26:01.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg=', 'Content-Length': '0'} 2023-09-29T22:26:01.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2206fcef699b0ad-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.805 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.805 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.805 DEBUG:boto:path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.805 DEBUG:boto:auth_path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.806 DEBUG:boto:Method: GET 2023-09-29T22:26:01.806 DEBUG:boto:Path: /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.806 DEBUG:boto:Data: 2023-09-29T22:26:01.806 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.806 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.806 DEBUG:boto:Params: {} 2023-09-29T22:26:01.806 DEBUG:boto:Token: None 2023-09-29T22:26:01.806 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg= 2023-09-29T22:26:01.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg=', 'Content-Length': '0'} 2023-09-29T22:26:01.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c164d42c7fc5f4c-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.809 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.809 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.809 DEBUG:boto:path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.809 DEBUG:boto:auth_path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.809 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.809 DEBUG:boto:Path: /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.810 DEBUG:boto:Data: 2023-09-29T22:26:01.810 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.810 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.810 DEBUG:boto:Params: {} 2023-09-29T22:26:01.810 DEBUG:boto:Token: None 2023-09-29T22:26:01.810 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:51GS0TH2BafXD9I0ZymcxdpET2g= 2023-09-29T22:26:01.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:51GS0TH2BafXD9I0ZymcxdpET2g=', 'Content-Length': '0'} 2023-09-29T22:26:01.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1320c19f4abe286-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.813 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.813 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.813 DEBUG:boto:path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.813 DEBUG:boto:auth_path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.813 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.813 DEBUG:boto:Path: /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.814 DEBUG:boto:Data: 2023-09-29T22:26:01.814 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.814 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.814 DEBUG:boto:Params: {} 2023-09-29T22:26:01.814 DEBUG:boto:Token: None 2023-09-29T22:26:01.814 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.814 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:51GS0TH2BafXD9I0ZymcxdpET2g= 2023-09-29T22:26:01.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:51GS0TH2BafXD9I0ZymcxdpET2g=', 'Content-Length': '0'} 2023-09-29T22:26:01.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b85b0949e2f2c8-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:26:01.817 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.817 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.817 DEBUG:boto:path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.817 DEBUG:boto:auth_path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.817 DEBUG:boto:Method: GET 2023-09-29T22:26:01.817 DEBUG:boto:Path: /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.818 DEBUG:boto:Data: 2023-09-29T22:26:01.818 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.818 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.818 DEBUG:boto:Params: {} 2023-09-29T22:26:01.818 DEBUG:boto:Token: None 2023-09-29T22:26:01.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg= 2023-09-29T22:26:01.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg=', 'Content-Length': '0'} 2023-09-29T22:26:01.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000579c55c553163c0d-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.821 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.821 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.821 DEBUG:boto:path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.821 DEBUG:boto:auth_path=/hcnphz-54/d?versionId=null 2023-09-29T22:26:01.822 DEBUG:boto:Method: GET 2023-09-29T22:26:01.822 DEBUG:boto:Path: /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.822 DEBUG:boto:Data: 2023-09-29T22:26:01.822 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.822 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.822 DEBUG:boto:Params: {} 2023-09-29T22:26:01.822 DEBUG:boto:Token: None 2023-09-29T22:26:01.822 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d?versionId=null 2023-09-29T22:26:01.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg= 2023-09-29T22:26:01.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gI8Rt1xIoWi+84+HlthDr5DukXg=', 'Content-Length': '0'} 2023-09-29T22:26:01.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fab993dcf052a832-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.825 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.825 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.825 DEBUG:boto:path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.826 DEBUG:boto:auth_path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.826 DEBUG:boto:Method: GET 2023-09-29T22:26:01.826 DEBUG:boto:Path: /hcnphz-54/?prefix=d 2023-09-29T22:26:01.826 DEBUG:boto:Data: 2023-09-29T22:26:01.826 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.826 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.826 DEBUG:boto:Params: {} 2023-09-29T22:26:01.826 DEBUG:boto:Token: None 2023-09-29T22:26:01.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086bad0068a75cb1e-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.831 DEBUG:boto:b'hcnphz-54d1000falsed2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.831 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.832 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.832 DEBUG:boto:path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.832 DEBUG:boto:auth_path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.832 DEBUG:boto:Method: GET 2023-09-29T22:26:01.832 DEBUG:boto:Path: /hcnphz-54/?prefix=d 2023-09-29T22:26:01.832 DEBUG:boto:Data: 2023-09-29T22:26:01.832 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.832 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.832 DEBUG:boto:Params: {} 2023-09-29T22:26:01.833 DEBUG:boto:Token: None 2023-09-29T22:26:01.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdb30e4296118a34-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.838 DEBUG:boto:b'hcnphz-54d1000falsed2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.838 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.838 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.839 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.839 DEBUG:boto:Path: /hcnphz-54/d 2023-09-29T22:26:01.839 DEBUG:boto:Data: 2023-09-29T22:26:01.839 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.839 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.839 DEBUG:boto:Params: {} 2023-09-29T22:26:01.839 DEBUG:boto:Token: None 2023-09-29T22:26:01.839 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d 2023-09-29T22:26:01.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CwarJ5qqjqkA2OA1upHpO2ZTa6Q= 2023-09-29T22:26:01.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CwarJ5qqjqkA2OA1upHpO2ZTa6Q=', 'Content-Length': '0'} 2023-09-29T22:26:01.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e55312b1ccb42e7-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.842 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.842 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.842 DEBUG:boto:Method: HEAD 2023-09-29T22:26:01.842 DEBUG:boto:Path: /hcnphz-54/d 2023-09-29T22:26:01.842 DEBUG:boto:Data: 2023-09-29T22:26:01.843 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.843 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.843 DEBUG:boto:Params: {} 2023-09-29T22:26:01.843 DEBUG:boto:Token: None 2023-09-29T22:26:01.843 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d 2023-09-29T22:26:01.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CwarJ5qqjqkA2OA1upHpO2ZTa6Q= 2023-09-29T22:26:01.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CwarJ5qqjqkA2OA1upHpO2ZTa6Q=', 'Content-Length': '0'} 2023-09-29T22:26:01.845 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba77a3451f248f0c-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:26:01.846 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.846 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.846 DEBUG:boto:Method: GET 2023-09-29T22:26:01.846 DEBUG:boto:Path: /hcnphz-54/d 2023-09-29T22:26:01.846 DEBUG:boto:Data: 2023-09-29T22:26:01.846 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.846 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.847 DEBUG:boto:Params: {} 2023-09-29T22:26:01.847 DEBUG:boto:Token: None 2023-09-29T22:26:01.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d 2023-09-29T22:26:01.847 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y9vCnxAXfE5s+BGJhho8ejJ1wEo= 2023-09-29T22:26:01.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y9vCnxAXfE5s+BGJhho8ejJ1wEo=', 'Content-Length': '0'} 2023-09-29T22:26:01.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-54:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.19'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000014295b4efdae69d-0065174ef9-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.890 DEBUG:boto:path=/hcnphz-54/d 2023-09-29T22:26:01.890 DEBUG:boto:auth_path=/hcnphz-54/d 2023-09-29T22:26:01.891 DEBUG:boto:Method: GET 2023-09-29T22:26:01.891 DEBUG:boto:Path: /hcnphz-54/d 2023-09-29T22:26:01.891 DEBUG:boto:Data: 2023-09-29T22:26:01.891 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.891 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.891 DEBUG:boto:Params: {} 2023-09-29T22:26:01.891 DEBUG:boto:Token: None 2023-09-29T22:26:01.891 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/d 2023-09-29T22:26:01.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y9vCnxAXfE5s+BGJhho8ejJ1wEo= 2023-09-29T22:26:01.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y9vCnxAXfE5s+BGJhho8ejJ1wEo=', 'Content-Length': '0'} 2023-09-29T22:26:01.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:25:45 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c965b9d6458398ed-0065174ef9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.894 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.895 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.895 DEBUG:boto:path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.895 DEBUG:boto:auth_path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.895 DEBUG:boto:Method: GET 2023-09-29T22:26:01.895 DEBUG:boto:Path: /hcnphz-54/?prefix=d 2023-09-29T22:26:01.895 DEBUG:boto:Data: 2023-09-29T22:26:01.895 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:01.895 DEBUG:boto:Port: 8001 2023-09-29T22:26:01.895 DEBUG:boto:Params: {} 2023-09-29T22:26:01.896 DEBUG:boto:Token: None 2023-09-29T22:26:01.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b07a31f78ae6da40-0065174ef9-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.901 DEBUG:boto:b'hcnphz-54d1000falsed2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.902 DEBUG:boto:path=/hcnphz-54/ 2023-09-29T22:26:01.902 DEBUG:boto:auth_path=/hcnphz-54/ 2023-09-29T22:26:01.902 DEBUG:boto:path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.902 DEBUG:boto:auth_path=/hcnphz-54/?prefix=d 2023-09-29T22:26:01.902 DEBUG:boto:Method: GET 2023-09-29T22:26:01.902 DEBUG:boto:Path: /hcnphz-54/?prefix=d 2023-09-29T22:26:01.902 DEBUG:boto:Data: 2023-09-29T22:26:01.902 DEBUG:boto:Headers: {} 2023-09-29T22:26:01.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:01.903 DEBUG:boto:Port: 8000 2023-09-29T22:26:01.903 DEBUG:boto:Params: {} 2023-09-29T22:26:01.903 DEBUG:boto:Token: None 2023-09-29T22:26:01.903 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:01 GMT /hcnphz-54/ 2023-09-29T22:26:01.903 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU= 2023-09-29T22:26:01.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFE6SqV7Ae3tApEUloQVFFAAAEU=', 'Content-Length': '0'} 2023-09-29T22:26:01.907 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087e1b68647a7aba7-0065174ef9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:01.907 DEBUG:boto:b'hcnphz-54d1000falsed2023-09-29T22:25:45.643Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:01.908 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-54 zones={a2, a1} 2023-09-29T22:26:01.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:01.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:01.908 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:26:01.949 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:01.949 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:02.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:02.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-54', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:02.049 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-54 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:02.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:02.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:02.213 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:26:02.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:02.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:02.216 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:02.258 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:02.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:02.339 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:26:02.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "1#00000000001.18.6", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.17450818123242794188", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "a", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:13.645833920Z", 2023-09-29T22:26:02.422 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:02.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18" 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "5#00000000001.37.6", 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.8966969916720486674", 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "d", 2023-09-29T22:26:02.424 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:13.725739337Z", 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:02.425 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18" 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:02.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "8#00000000001.31.6", 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.358362862501071360", 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "c", 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:02.427 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:13.712359065Z", 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:02.428 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18" 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "10#00000000001.48.6", 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5973061701003397876", 2023-09-29T22:26:02.429 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "b", 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:25:13.661257062Z", 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:26:02.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-53:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.18" 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:02.431 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:02.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:02.458 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:26:02.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:02.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:02.459 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:26:02.503 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:02.503 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:02.598 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.597+0000 7f38708bb700 -1 req 4024704393771876847 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-26:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.26,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-26:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.26,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.599 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.597+0000 7f3958a8b700 -1 req 1589567931812962985 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-3:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.3,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-3:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.3,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.599 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.597+0000 7f38688ab700 -1 req 5654963791240920203 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-24:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.24,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-24:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.24,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.604 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.605+0000 7f3956286700 -1 req 12198024758479012630 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-29:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.29,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-29:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.29,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.616 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.617+0000 7f392922c700 -1 req 7335685817935256227 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-20:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.20,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-20:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.20,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.617 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.617+0000 7f393f258700 -1 req 10154625349745735839 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-51:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.16,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-51:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.16,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.629 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.629+0000 7f3915204700 -1 req 9060617551688022381 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-5:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.5,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-5:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.5,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.670 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.669+0000 7f39121fe700 -1 req 8627846536549040976 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-28:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.28,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-28:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.28,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.676 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.677+0000 7f38e49a3700 -1 req 8040422013547519623 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-39:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.4,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-39:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.4,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.688 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.689+0000 7f38e99ad700 -1 req 8225436193468793624 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-30:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.30,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-30:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.30,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.691 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.689+0000 7f38e219e700 -1 req 18246889654184695623 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-4:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.4,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-4:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.4,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.692 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.693+0000 7f38f01ba700 -1 req 4591289407018674063 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-37:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.2,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-37:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.2,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.704 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.705+0000 7f38ca16e700 -1 req 7617074569701945196 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-38:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.3,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-38:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.3,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.747 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.745+0000 7f38a3120700 -1 req 2260283998629337388 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-42:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.7,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-42:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.7,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.753 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.753+0000 7f3894102700 -1 req 8392242488000617250 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-21:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.21,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-21:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.21,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.758 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.757+0000 7f38790cc700 -1 req 17459678639244895031 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-27:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.27,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-27:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.27,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.766 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.765+0000 7f389d915700 -1 req 13150567699367045215 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-2:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.2,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-2:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.2,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.766 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.765+0000 7f386209e700 -1 req 11456369029284083291 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-22:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.22,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-22:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.22,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.770 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.769+0000 7f38d918c700 -1 req 1704194865929710411 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-43:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.8,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-43:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.8,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.773 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.773+0000 7f38f79c9700 -1 req 4400164871043110834 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-41:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.6,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-41:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.6,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.784 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.785+0000 7f387f8d9700 -1 req 13675707336858696457 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-40:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.5,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-40:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.5,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.806 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.805+0000 7f38640a2700 -1 req 10385219320539533470 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-36:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.1,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-36:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.1,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.811 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.813+0000 7f38d6987700 -1 req 13399451817702629033 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-19:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.19,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-19:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.19,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.832 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.833+0000 7f393fa59700 -1 req 19896969658351566 0.003999892s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-25:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.25,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-25:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.25,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.837 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.837+0000 7f394f278700 -1 req 9900324803619775654 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-23:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.23,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-23:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.23,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.837 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:26:02.837+0000 7f3937248700 -1 req 7345535455466072470 0.003999893s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-1:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.1,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-1:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.1,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:26:02.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:02.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-53', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:02.920 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-53 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:02.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:02.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.041 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:26:03.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2023-09-29T22:26:03.046 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-55 2023-09-29T22:26:03.046 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:03.046 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:03.047 DEBUG:boto:Method: PUT 2023-09-29T22:26:03.047 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:03.047 DEBUG:boto:Data: 2023-09-29T22:26:03.047 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.047 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.047 DEBUG:boto:Params: {} 2023-09-29T22:26:03.048 DEBUG:boto:Token: None 2023-09-29T22:26:03.048 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/ 2023-09-29T22:26:03.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q5nS88jpzhz+cDuOEtsVB9LEt8c= 2023-09-29T22:26:03.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q5nS88jpzhz+cDuOEtsVB9LEt8c=', 'Content-Length': '0'} 2023-09-29T22:26:03.078 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000a5af1a94ff3aa535-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.079 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:03.079 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:03.079 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.079 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:03.079 DEBUG:boto:Data: 2023-09-29T22:26:03.079 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.080 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.080 DEBUG:boto:Params: {} 2023-09-29T22:26:03.080 DEBUG:boto:Token: None 2023-09-29T22:26:03.080 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/ 2023-09-29T22:26:03.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T22:26:03.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T22:26:03.084 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d1833ca4b8204b7-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.085 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:03.085 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:03.085 DEBUG:boto:Method: PUT 2023-09-29T22:26:03.085 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:26:03.085 DEBUG:boto:Data: 2023-09-29T22:26:03.085 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:03.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.086 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.086 DEBUG:boto:Params: {} 2023-09-29T22:26:03.086 DEBUG:boto:Token: None 2023-09-29T22:26:03.086 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/a 2023-09-29T22:26:03.086 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nzYxennEmgYSJngboQg2GsyTQGs= 2023-09-29T22:26:03.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nzYxennEmgYSJngboQg2GsyTQGs='} 2023-09-29T22:26:03.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e565076d0204a9f1-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.135 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:03.135 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:03.135 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.135 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:03.135 DEBUG:boto:Data: 2023-09-29T22:26:03.136 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.136 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.136 DEBUG:boto:Params: {} 2023-09-29T22:26:03.136 DEBUG:boto:Token: None 2023-09-29T22:26:03.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/ 2023-09-29T22:26:03.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T22:26:03.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T22:26:03.140 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fec53a1311be9b9a-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.140 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:03.141 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:03.141 DEBUG:boto:Method: PUT 2023-09-29T22:26:03.141 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:26:03.141 DEBUG:boto:Data: 2023-09-29T22:26:03.141 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:03.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.142 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.142 DEBUG:boto:Params: {} 2023-09-29T22:26:03.142 DEBUG:boto:Token: None 2023-09-29T22:26:03.142 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/b 2023-09-29T22:26:03.142 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s5fVA9Ks+vPJhj6l+2Cih1G8VTM= 2023-09-29T22:26:03.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s5fVA9Ks+vPJhj6l+2Cih1G8VTM='} 2023-09-29T22:26:03.153 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009320a2b0bab1fc9f-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.154 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:03.154 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:03.154 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.154 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:03.154 DEBUG:boto:Data: 2023-09-29T22:26:03.154 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.154 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.155 DEBUG:boto:Params: {} 2023-09-29T22:26:03.155 DEBUG:boto:Token: None 2023-09-29T22:26:03.155 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/ 2023-09-29T22:26:03.155 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T22:26:03.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T22:26:03.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff7b148255168d6a-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.159 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:03.159 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:03.159 DEBUG:boto:Method: PUT 2023-09-29T22:26:03.159 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:26:03.159 DEBUG:boto:Data: 2023-09-29T22:26:03.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:03.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.160 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.160 DEBUG:boto:Params: {} 2023-09-29T22:26:03.160 DEBUG:boto:Token: None 2023-09-29T22:26:03.160 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/c 2023-09-29T22:26:03.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cT5AS5bcHaRRRceVlqM3jYpIbd4= 2023-09-29T22:26:03.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cT5AS5bcHaRRRceVlqM3jYpIbd4='} 2023-09-29T22:26:03.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094fb1f8a91bd61e4-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.203 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:03.203 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:03.203 DEBUG:boto:Method: HEAD 2023-09-29T22:26:03.203 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:03.203 DEBUG:boto:Data: 2023-09-29T22:26:03.203 DEBUG:boto:Headers: {} 2023-09-29T22:26:03.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.204 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.204 DEBUG:boto:Params: {} 2023-09-29T22:26:03.204 DEBUG:boto:Token: None 2023-09-29T22:26:03.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/ 2023-09-29T22:26:03.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T22:26:03.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T22:26:03.208 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004bbc4b673f086c2-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.209 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:03.209 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:03.209 DEBUG:boto:Method: PUT 2023-09-29T22:26:03.209 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:26:03.209 DEBUG:boto:Data: 2023-09-29T22:26:03.209 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:03.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:03.210 DEBUG:boto:Port: 8000 2023-09-29T22:26:03.210 DEBUG:boto:Params: {} 2023-09-29T22:26:03.210 DEBUG:boto:Token: None 2023-09-29T22:26:03.210 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:03 GMT /hcnphz-55/d 2023-09-29T22:26:03.210 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lXa4qQ5E0MkupWuq/eFrB4U0W7Y= 2023-09-29T22:26:03.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lXa4qQ5E0MkupWuq/eFrB4U0W7Y='} 2023-09-29T22:26:03.221 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064af09faacb5c16d-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:03.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:03.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:03.221 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:03.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.264 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.321 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:26:03.348 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026363.077186_423.1", 2023-09-29T22:26:03.349 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:26:03.077186Z" 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:26:03.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:03.351 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:26:03.352 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026363.077186_423.1', 2: '', 3: ''} 2023-09-29T22:26:03.352 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:26:03.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:03.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:03.352 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.394 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.449 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:03.449+0000 7fecc817df40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:03.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:03.449+0000 7fecc817df40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:26:03.475 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:26:03.475 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:26:03.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:26:03.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:26:03.478 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026363.077186_423.1", 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:26:03.077186Z", 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:03.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:26:03.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:26:03.481 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:03.482 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:26:03Z" 2023-09-29T22:26:03.483 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:26:03.484 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026363.077186_423.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:26:03.077186Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:26:03Z" } 2023-09-29T22:26:03.485 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026363.077186_423.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:03.077186Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:26:03.485 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026363.077186_423.1', 2: '', 3: ''} 2023-09-29T22:26:03.485 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026363.077186_423.1', 2: '', 3: ''} 2023-09-29T22:26:03.485 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:26:03.485 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-55 2023-09-29T22:26:03.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:03.486 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:03.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:03.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:03.601+0000 7ff9f42a5f40 1 waiting to reach incremental sync.. 2023-09-29T22:26:08.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:08.605+0000 7ff9f42a5f40 1 waiting to reach incremental sync.. 2023-09-29T22:26:13.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:13.605+0000 7ff9f42a5f40 1 bucket sync caught up with source: 2023-09-29T22:26:13.607 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.39.6, , , , 00000000001.45.6, , , 00000000001.44.6, , 00000000001.62.6] 2023-09-29T22:26:13.608 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.39.6, , , , 00000000001.45.6, , , 00000000001.44.6, , 00000000001.62.6] 2023-09-29T22:26:13.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:13.605+0000 7ff9f42a5f40 0 bucket checkpoint complete 2023-09-29T22:26:13.636 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-55 2023-09-29T22:26:13.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:13.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:13.637 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:13.688 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:13.689 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:13.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:13.757+0000 7fb89ff0ef40 1 waiting to reach incremental sync.. 2023-09-29T22:26:18.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:18.757+0000 7fb89ff0ef40 1 waiting to reach incremental sync.. 2023-09-29T22:26:23.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:23.761+0000 7fb89ff0ef40 1 bucket sync caught up with source: 2023-09-29T22:26:23.762 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.162.6, , , , 00000000001.98.6, , , 00000000001.151.6, , 00000000001.282.6] 2023-09-29T22:26:23.762 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.162.6, , , , 00000000001.98.6, , , 00000000001.151.6, , 00000000001.282.6] 2023-09-29T22:26:23.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:23.761+0000 7fb89ff0ef40 0 bucket checkpoint complete 2023-09-29T22:26:23.791 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-55 zones={a2, a1} 2023-09-29T22:26:23.791 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.791 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.792 DEBUG:boto:Method: HEAD 2023-09-29T22:26:23.792 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:23.792 DEBUG:boto:Data: 2023-09-29T22:26:23.792 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.792 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.792 DEBUG:boto:Params: {} 2023-09-29T22:26:23.792 DEBUG:boto:Token: None 2023-09-29T22:26:23.793 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/ 2023-09-29T22:26:23.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE= 2023-09-29T22:26:23.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE=', 'Content-Length': '0'} 2023-09-29T22:26:23.797 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0c4b6b3425d6ce5-0065174f0f-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.797 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.797 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.797 DEBUG:boto:Method: HEAD 2023-09-29T22:26:23.797 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:23.797 DEBUG:boto:Data: 2023-09-29T22:26:23.797 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.798 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.798 DEBUG:boto:Params: {} 2023-09-29T22:26:23.798 DEBUG:boto:Token: None 2023-09-29T22:26:23.798 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/ 2023-09-29T22:26:23.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE= 2023-09-29T22:26:23.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE=', 'Content-Length': '0'} 2023-09-29T22:26:23.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021708d1e42bfca44-0065174f0f-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.802 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:26:23.802 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.802 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.802 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.802 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.802 DEBUG:boto:Method: GET 2023-09-29T22:26:23.802 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.802 DEBUG:boto:Data: 2023-09-29T22:26:23.803 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.803 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.803 DEBUG:boto:Params: {} 2023-09-29T22:26:23.803 DEBUG:boto:Token: None 2023-09-29T22:26:23.803 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/?versions 2023-09-29T22:26:23.803 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T22:26:23.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T22:26:23.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c3688c9cfee7cb-0065174f0f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.808 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:23.808 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:26:23.808 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:26:23.808 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:26:23.808 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:26:23.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:26:23.809 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.809 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.809 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.809 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.809 DEBUG:boto:Method: GET 2023-09-29T22:26:23.809 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.809 DEBUG:boto:Data: 2023-09-29T22:26:23.809 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.810 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.810 DEBUG:boto:Params: {} 2023-09-29T22:26:23.810 DEBUG:boto:Token: None 2023-09-29T22:26:23.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/?versions 2023-09-29T22:26:23.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T22:26:23.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T22:26:23.813 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfdcd8181897afa-0065174f0f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.814 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:26:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:26:23.814 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:26:23.815 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:26:23.815 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.815 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.815 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.815 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.815 DEBUG:boto:Method: GET 2023-09-29T22:26:23.815 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.815 DEBUG:boto:Data: 2023-09-29T22:26:23.815 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.816 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.816 DEBUG:boto:Params: {} 2023-09-29T22:26:23.816 DEBUG:boto:Token: None 2023-09-29T22:26:23.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/?versions 2023-09-29T22:26:23.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T22:26:23.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T22:26:23.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbb8db47dbdc58d-0065174f0f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.819 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:23.819 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.820 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.820 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.820 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.820 DEBUG:boto:Method: GET 2023-09-29T22:26:23.820 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:26:23.820 DEBUG:boto:Data: 2023-09-29T22:26:23.820 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.820 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.821 DEBUG:boto:Params: {} 2023-09-29T22:26:23.821 DEBUG:boto:Token: None 2023-09-29T22:26:23.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/?versions 2023-09-29T22:26:23.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T22:26:23.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T22:26:23.824 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064747bd9250d8702-0065174f0f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.824 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:23.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:26:23.825 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.825 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.825 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.825 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.825 DEBUG:boto:Method: GET 2023-09-29T22:26:23.825 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.825 DEBUG:boto:Data: 2023-09-29T22:26:23.826 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.826 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.826 DEBUG:boto:Params: {} 2023-09-29T22:26:23.826 DEBUG:boto:Token: None 2023-09-29T22:26:23.826 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.826 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T22:26:23.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T22:26:23.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000556e9e71cd0f812e-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.870 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.870 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.871 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.871 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.871 DEBUG:boto:Method: GET 2023-09-29T22:26:23.871 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.871 DEBUG:boto:Data: 2023-09-29T22:26:23.871 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.871 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.871 DEBUG:boto:Params: {} 2023-09-29T22:26:23.872 DEBUG:boto:Token: None 2023-09-29T22:26:23.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T22:26:23.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T22:26:23.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000186e148764912c-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.875 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.875 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.876 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.876 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.876 DEBUG:boto:Method: HEAD 2023-09-29T22:26:23.876 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.876 DEBUG:boto:Data: 2023-09-29T22:26:23.876 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.876 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.876 DEBUG:boto:Params: {} 2023-09-29T22:26:23.876 DEBUG:boto:Token: None 2023-09-29T22:26:23.877 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY= 2023-09-29T22:26:23.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY=', 'Content-Length': '0'} 2023-09-29T22:26:23.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a4f295e1d3f6d4-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.880 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.880 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.880 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.880 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.880 DEBUG:boto:Method: HEAD 2023-09-29T22:26:23.880 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.881 DEBUG:boto:Data: 2023-09-29T22:26:23.881 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.881 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.881 DEBUG:boto:Params: {} 2023-09-29T22:26:23.881 DEBUG:boto:Token: None 2023-09-29T22:26:23.881 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY= 2023-09-29T22:26:23.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY=', 'Content-Length': '0'} 2023-09-29T22:26:23.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004793b3d245b7c0b7-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:26:23.885 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.885 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.885 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.885 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.885 DEBUG:boto:Method: GET 2023-09-29T22:26:23.885 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.886 DEBUG:boto:Data: 2023-09-29T22:26:23.886 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.886 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.886 DEBUG:boto:Params: {} 2023-09-29T22:26:23.886 DEBUG:boto:Token: None 2023-09-29T22:26:23.886 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T22:26:23.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T22:26:23.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3dd6b462c0db88-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.889 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.889 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.890 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.890 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:26:23.890 DEBUG:boto:Method: GET 2023-09-29T22:26:23.890 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.890 DEBUG:boto:Data: 2023-09-29T22:26:23.890 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.890 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.891 DEBUG:boto:Params: {} 2023-09-29T22:26:23.891 DEBUG:boto:Token: None 2023-09-29T22:26:23.891 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:26:23.891 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T22:26:23.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T22:26:23.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f121ac886f09826e-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.934 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.935 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.935 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:26:23.935 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:26:23.935 DEBUG:boto:Method: GET 2023-09-29T22:26:23.935 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:26:23.935 DEBUG:boto:Data: 2023-09-29T22:26:23.935 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.935 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.936 DEBUG:boto:Params: {} 2023-09-29T22:26:23.936 DEBUG:boto:Token: None 2023-09-29T22:26:23.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/ 2023-09-29T22:26:23.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU= 2023-09-29T22:26:23.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU=', 'Content-Length': '0'} 2023-09-29T22:26:23.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36d88d68ab2aa73-0065174f0f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.941 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:23.941 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:23.941 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:23.941 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:26:23.942 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:26:23.942 DEBUG:boto:Method: GET 2023-09-29T22:26:23.942 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:26:23.942 DEBUG:boto:Data: 2023-09-29T22:26:23.942 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.942 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.943 DEBUG:boto:Params: {} 2023-09-29T22:26:23.943 DEBUG:boto:Token: None 2023-09-29T22:26:23.943 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/ 2023-09-29T22:26:23.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU= 2023-09-29T22:26:23.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU=', 'Content-Length': '0'} 2023-09-29T22:26:23.948 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de61e609e107117c-0065174f0f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.948 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:23.948 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.948 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.948 DEBUG:boto:Method: HEAD 2023-09-29T22:26:23.949 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:26:23.949 DEBUG:boto:Data: 2023-09-29T22:26:23.949 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.949 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.949 DEBUG:boto:Params: {} 2023-09-29T22:26:23.949 DEBUG:boto:Token: None 2023-09-29T22:26:23.949 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a 2023-09-29T22:26:23.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI= 2023-09-29T22:26:23.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI=', 'Content-Length': '0'} 2023-09-29T22:26:23.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aff4f40335cb08d-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.952 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.952 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.952 DEBUG:boto:Method: HEAD 2023-09-29T22:26:23.952 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:26:23.952 DEBUG:boto:Data: 2023-09-29T22:26:23.953 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:23.953 DEBUG:boto:Port: 8000 2023-09-29T22:26:23.953 DEBUG:boto:Params: {} 2023-09-29T22:26:23.953 DEBUG:boto:Token: None 2023-09-29T22:26:23.953 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a 2023-09-29T22:26:23.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI= 2023-09-29T22:26:23.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI=', 'Content-Length': '0'} 2023-09-29T22:26:23.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6572db531d4c9e-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:23.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:26:23.956 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:23.956 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:23.956 DEBUG:boto:Method: GET 2023-09-29T22:26:23.956 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:26:23.956 DEBUG:boto:Data: 2023-09-29T22:26:23.956 DEBUG:boto:Headers: {} 2023-09-29T22:26:23.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:23.956 DEBUG:boto:Port: 8001 2023-09-29T22:26:23.957 DEBUG:boto:Params: {} 2023-09-29T22:26:23.957 DEBUG:boto:Token: None 2023-09-29T22:26:23.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:23 GMT /hcnphz-55/a 2023-09-29T22:26:23.957 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ZPiF+ixRoqydDlOtNHqKUYOm0c= 2023-09-29T22:26:23.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ZPiF+ixRoqydDlOtNHqKUYOm0c=', 'Content-Length': '0'} 2023-09-29T22:26:23.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8896b5089a6f76d-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.006 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:26:24.006 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:26:24.007 DEBUG:boto:Method: GET 2023-09-29T22:26:24.007 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:26:24.007 DEBUG:boto:Data: 2023-09-29T22:26:24.007 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.007 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.007 DEBUG:boto:Params: {} 2023-09-29T22:26:24.007 DEBUG:boto:Token: None 2023-09-29T22:26:24.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/a 2023-09-29T22:26:24.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y/vGev4Yrisqgzvdy/Ebfzzrawo= 2023-09-29T22:26:24.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y/vGev4Yrisqgzvdy/Ebfzzrawo=', 'Content-Length': '0'} 2023-09-29T22:26:24.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c92c5ebbdce28a-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.011 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.011 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.011 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:26:24.011 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:26:24.011 DEBUG:boto:Method: GET 2023-09-29T22:26:24.012 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:26:24.012 DEBUG:boto:Data: 2023-09-29T22:26:24.012 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.012 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.012 DEBUG:boto:Params: {} 2023-09-29T22:26:24.012 DEBUG:boto:Token: None 2023-09-29T22:26:24.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a11705bf58bf82b-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.017 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.018 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.018 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.018 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:26:24.018 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:26:24.018 DEBUG:boto:Method: GET 2023-09-29T22:26:24.018 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:26:24.018 DEBUG:boto:Data: 2023-09-29T22:26:24.018 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.019 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.019 DEBUG:boto:Params: {} 2023-09-29T22:26:24.019 DEBUG:boto:Token: None 2023-09-29T22:26:24.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000323136e231503a1b-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.066 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:26:24.067 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.067 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.067 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.067 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.067 DEBUG:boto:Method: GET 2023-09-29T22:26:24.067 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.068 DEBUG:boto:Data: 2023-09-29T22:26:24.068 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.068 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.068 DEBUG:boto:Params: {} 2023-09-29T22:26:24.068 DEBUG:boto:Token: None 2023-09-29T22:26:24.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T22:26:24.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T22:26:24.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e2f04a2c91a429-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.071 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.071 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.071 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.071 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.071 DEBUG:boto:Method: GET 2023-09-29T22:26:24.071 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.071 DEBUG:boto:Data: 2023-09-29T22:26:24.071 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.072 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.072 DEBUG:boto:Params: {} 2023-09-29T22:26:24.072 DEBUG:boto:Token: None 2023-09-29T22:26:24.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T22:26:24.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T22:26:24.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fecef1af8d82c461-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.075 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.075 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.075 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.075 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.075 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.075 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.075 DEBUG:boto:Data: 2023-09-29T22:26:24.075 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.076 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.076 DEBUG:boto:Params: {} 2023-09-29T22:26:24.076 DEBUG:boto:Token: None 2023-09-29T22:26:24.076 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo= 2023-09-29T22:26:24.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo=', 'Content-Length': '0'} 2023-09-29T22:26:24.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081df48b74cc2faa7-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.079 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.079 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.079 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.079 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.079 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.079 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.079 DEBUG:boto:Data: 2023-09-29T22:26:24.079 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.080 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.080 DEBUG:boto:Params: {} 2023-09-29T22:26:24.080 DEBUG:boto:Token: None 2023-09-29T22:26:24.080 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo= 2023-09-29T22:26:24.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo=', 'Content-Length': '0'} 2023-09-29T22:26:24.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026e103fe6216cc98-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:26:24.083 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.083 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.083 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.084 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.084 DEBUG:boto:Method: GET 2023-09-29T22:26:24.084 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.084 DEBUG:boto:Data: 2023-09-29T22:26:24.084 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.084 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.084 DEBUG:boto:Params: {} 2023-09-29T22:26:24.084 DEBUG:boto:Token: None 2023-09-29T22:26:24.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T22:26:24.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T22:26:24.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f59075dfe8399c0-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.130 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.130 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.131 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.131 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:26:24.131 DEBUG:boto:Method: GET 2023-09-29T22:26:24.131 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.131 DEBUG:boto:Data: 2023-09-29T22:26:24.131 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.131 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.131 DEBUG:boto:Params: {} 2023-09-29T22:26:24.132 DEBUG:boto:Token: None 2023-09-29T22:26:24.132 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:26:24.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T22:26:24.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T22:26:24.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426e6653b247995f-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.135 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.135 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.135 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.135 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.136 DEBUG:boto:Method: GET 2023-09-29T22:26:24.136 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:26:24.136 DEBUG:boto:Data: 2023-09-29T22:26:24.136 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.136 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.136 DEBUG:boto:Params: {} 2023-09-29T22:26:24.136 DEBUG:boto:Token: None 2023-09-29T22:26:24.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b5558a6732076e-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.141 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.141 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.142 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.142 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.142 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.142 DEBUG:boto:Method: GET 2023-09-29T22:26:24.142 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:26:24.142 DEBUG:boto:Data: 2023-09-29T22:26:24.142 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.142 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.143 DEBUG:boto:Params: {} 2023-09-29T22:26:24.143 DEBUG:boto:Token: None 2023-09-29T22:26:24.143 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.143 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.147 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2c6bd8d60a04e8-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.190 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.191 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.191 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.191 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.191 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:26:24.191 DEBUG:boto:Data: 2023-09-29T22:26:24.191 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.191 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.192 DEBUG:boto:Params: {} 2023-09-29T22:26:24.192 DEBUG:boto:Token: None 2023-09-29T22:26:24.192 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b 2023-09-29T22:26:24.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU= 2023-09-29T22:26:24.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU=', 'Content-Length': '0'} 2023-09-29T22:26:24.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d98010c367bce3f-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.195 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.195 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.195 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.195 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:26:24.196 DEBUG:boto:Data: 2023-09-29T22:26:24.196 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.196 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.196 DEBUG:boto:Params: {} 2023-09-29T22:26:24.196 DEBUG:boto:Token: None 2023-09-29T22:26:24.196 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b 2023-09-29T22:26:24.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU= 2023-09-29T22:26:24.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU=', 'Content-Length': '0'} 2023-09-29T22:26:24.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f14e85d2f0a37c-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:26:24.199 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.199 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.199 DEBUG:boto:Method: GET 2023-09-29T22:26:24.200 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:26:24.200 DEBUG:boto:Data: 2023-09-29T22:26:24.200 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.200 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.200 DEBUG:boto:Params: {} 2023-09-29T22:26:24.200 DEBUG:boto:Token: None 2023-09-29T22:26:24.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b 2023-09-29T22:26:24.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg= 2023-09-29T22:26:24.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg=', 'Content-Length': '0'} 2023-09-29T22:26:24.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d696f6af47f5c57-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.203 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:26:24.204 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:26:24.204 DEBUG:boto:Method: GET 2023-09-29T22:26:24.204 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:26:24.204 DEBUG:boto:Data: 2023-09-29T22:26:24.204 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.204 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.204 DEBUG:boto:Params: {} 2023-09-29T22:26:24.204 DEBUG:boto:Token: None 2023-09-29T22:26:24.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/b 2023-09-29T22:26:24.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg= 2023-09-29T22:26:24.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg=', 'Content-Length': '0'} 2023-09-29T22:26:24.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf938eb9e2eaa58-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.208 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.208 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.208 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.208 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.208 DEBUG:boto:Method: GET 2023-09-29T22:26:24.208 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:26:24.208 DEBUG:boto:Data: 2023-09-29T22:26:24.208 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.209 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.209 DEBUG:boto:Params: {} 2023-09-29T22:26:24.209 DEBUG:boto:Token: None 2023-09-29T22:26:24.209 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.209 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002084c5edd7aa63cd-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.254 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.255 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.255 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.255 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.255 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:26:24.255 DEBUG:boto:Method: GET 2023-09-29T22:26:24.255 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:26:24.255 DEBUG:boto:Data: 2023-09-29T22:26:24.256 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.256 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.256 DEBUG:boto:Params: {} 2023-09-29T22:26:24.256 DEBUG:boto:Token: None 2023-09-29T22:26:24.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1f811b342c770a9-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.261 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:26:24.261 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.262 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.262 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.262 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.262 DEBUG:boto:Method: GET 2023-09-29T22:26:24.262 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.262 DEBUG:boto:Data: 2023-09-29T22:26:24.262 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.262 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.262 DEBUG:boto:Params: {} 2023-09-29T22:26:24.263 DEBUG:boto:Token: None 2023-09-29T22:26:24.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.263 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T22:26:24.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T22:26:24.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92545b697765bf2-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.266 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.266 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.266 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.266 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.266 DEBUG:boto:Method: GET 2023-09-29T22:26:24.267 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.267 DEBUG:boto:Data: 2023-09-29T22:26:24.267 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.267 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.267 DEBUG:boto:Params: {} 2023-09-29T22:26:24.267 DEBUG:boto:Token: None 2023-09-29T22:26:24.267 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.267 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T22:26:24.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T22:26:24.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a06443c7abb6dcd-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.310 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.310 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.311 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.311 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.311 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.311 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.311 DEBUG:boto:Data: 2023-09-29T22:26:24.311 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.311 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.311 DEBUG:boto:Params: {} 2023-09-29T22:26:24.311 DEBUG:boto:Token: None 2023-09-29T22:26:24.312 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY= 2023-09-29T22:26:24.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY=', 'Content-Length': '0'} 2023-09-29T22:26:24.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d5116f0f831a9d-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.315 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.315 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.315 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.315 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.315 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.315 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.315 DEBUG:boto:Data: 2023-09-29T22:26:24.315 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.316 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.316 DEBUG:boto:Params: {} 2023-09-29T22:26:24.316 DEBUG:boto:Token: None 2023-09-29T22:26:24.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY= 2023-09-29T22:26:24.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY=', 'Content-Length': '0'} 2023-09-29T22:26:24.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b3507874034230-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:26:24.319 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.320 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.320 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.320 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.320 DEBUG:boto:Method: GET 2023-09-29T22:26:24.320 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.320 DEBUG:boto:Data: 2023-09-29T22:26:24.320 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.320 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.320 DEBUG:boto:Params: {} 2023-09-29T22:26:24.321 DEBUG:boto:Token: None 2023-09-29T22:26:24.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T22:26:24.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T22:26:24.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb4934f0b943c006-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.324 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.324 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.324 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.324 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:26:24.324 DEBUG:boto:Method: GET 2023-09-29T22:26:24.325 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.325 DEBUG:boto:Data: 2023-09-29T22:26:24.325 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.325 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.325 DEBUG:boto:Params: {} 2023-09-29T22:26:24.325 DEBUG:boto:Token: None 2023-09-29T22:26:24.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:26:24.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T22:26:24.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T22:26:24.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015647a6d19b18219-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.329 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.329 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.329 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.329 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.329 DEBUG:boto:Method: GET 2023-09-29T22:26:24.329 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:26:24.329 DEBUG:boto:Data: 2023-09-29T22:26:24.329 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.330 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.330 DEBUG:boto:Params: {} 2023-09-29T22:26:24.330 DEBUG:boto:Token: None 2023-09-29T22:26:24.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b01c79d27a94b02e-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.378 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.379 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.379 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.379 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.379 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.379 DEBUG:boto:Method: GET 2023-09-29T22:26:24.379 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:26:24.379 DEBUG:boto:Data: 2023-09-29T22:26:24.380 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.380 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.380 DEBUG:boto:Params: {} 2023-09-29T22:26:24.380 DEBUG:boto:Token: None 2023-09-29T22:26:24.381 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b6cad884a72be5-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.386 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.386 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.386 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.386 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.386 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:26:24.387 DEBUG:boto:Data: 2023-09-29T22:26:24.387 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.387 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.387 DEBUG:boto:Params: {} 2023-09-29T22:26:24.387 DEBUG:boto:Token: None 2023-09-29T22:26:24.387 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c 2023-09-29T22:26:24.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk= 2023-09-29T22:26:24.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk=', 'Content-Length': '0'} 2023-09-29T22:26:24.390 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e70affb96a3515-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.390 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.390 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.390 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.391 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:26:24.391 DEBUG:boto:Data: 2023-09-29T22:26:24.391 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.391 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.391 DEBUG:boto:Params: {} 2023-09-29T22:26:24.391 DEBUG:boto:Token: None 2023-09-29T22:26:24.391 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c 2023-09-29T22:26:24.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk= 2023-09-29T22:26:24.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk=', 'Content-Length': '0'} 2023-09-29T22:26:24.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc156c9e54c94f6e-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:26:24.394 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.395 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.395 DEBUG:boto:Method: GET 2023-09-29T22:26:24.395 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:26:24.395 DEBUG:boto:Data: 2023-09-29T22:26:24.395 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.395 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.395 DEBUG:boto:Params: {} 2023-09-29T22:26:24.395 DEBUG:boto:Token: None 2023-09-29T22:26:24.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c 2023-09-29T22:26:24.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4= 2023-09-29T22:26:24.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4=', 'Content-Length': '0'} 2023-09-29T22:26:24.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072bcb71cbc428351-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.399 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:26:24.399 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:26:24.399 DEBUG:boto:Method: GET 2023-09-29T22:26:24.399 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:26:24.399 DEBUG:boto:Data: 2023-09-29T22:26:24.399 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.399 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.399 DEBUG:boto:Params: {} 2023-09-29T22:26:24.400 DEBUG:boto:Token: None 2023-09-29T22:26:24.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/c 2023-09-29T22:26:24.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4= 2023-09-29T22:26:24.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4=', 'Content-Length': '0'} 2023-09-29T22:26:24.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa14c897cce4f76-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.446 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.447 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.447 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.447 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.447 DEBUG:boto:Method: GET 2023-09-29T22:26:24.447 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:26:24.447 DEBUG:boto:Data: 2023-09-29T22:26:24.447 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.447 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.448 DEBUG:boto:Params: {} 2023-09-29T22:26:24.448 DEBUG:boto:Token: None 2023-09-29T22:26:24.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.452 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06fe1fc20feff31-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.453 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.453 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.453 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.453 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.454 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:26:24.454 DEBUG:boto:Method: GET 2023-09-29T22:26:24.454 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:26:24.454 DEBUG:boto:Data: 2023-09-29T22:26:24.454 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.454 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.454 DEBUG:boto:Params: {} 2023-09-29T22:26:24.454 DEBUG:boto:Token: None 2023-09-29T22:26:24.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038b747dfaec32439-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.459 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:26:24.460 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.460 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.460 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.460 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.460 DEBUG:boto:Method: GET 2023-09-29T22:26:24.460 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.460 DEBUG:boto:Data: 2023-09-29T22:26:24.460 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.461 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.461 DEBUG:boto:Params: {} 2023-09-29T22:26:24.461 DEBUG:boto:Token: None 2023-09-29T22:26:24.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T22:26:24.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T22:26:24.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f02da3c948053d3e-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.464 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.464 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.465 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.465 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.465 DEBUG:boto:Method: GET 2023-09-29T22:26:24.465 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.465 DEBUG:boto:Data: 2023-09-29T22:26:24.465 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.465 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.465 DEBUG:boto:Params: {} 2023-09-29T22:26:24.465 DEBUG:boto:Token: None 2023-09-29T22:26:24.466 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.466 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T22:26:24.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T22:26:24.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97ed4f365a5ea87-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.469 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.469 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.469 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.469 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.469 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.469 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.469 DEBUG:boto:Data: 2023-09-29T22:26:24.470 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.470 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.470 DEBUG:boto:Params: {} 2023-09-29T22:26:24.470 DEBUG:boto:Token: None 2023-09-29T22:26:24.470 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.470 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc= 2023-09-29T22:26:24.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc=', 'Content-Length': '0'} 2023-09-29T22:26:24.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce46f0dbedeb085-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.473 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.473 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.473 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.473 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.473 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.474 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.474 DEBUG:boto:Data: 2023-09-29T22:26:24.474 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.474 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.474 DEBUG:boto:Params: {} 2023-09-29T22:26:24.474 DEBUG:boto:Token: None 2023-09-29T22:26:24.474 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc= 2023-09-29T22:26:24.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc=', 'Content-Length': '0'} 2023-09-29T22:26:24.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099bc4a89ca949320-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:26:24.477 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.477 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.478 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.478 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.478 DEBUG:boto:Method: GET 2023-09-29T22:26:24.478 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.478 DEBUG:boto:Data: 2023-09-29T22:26:24.478 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.478 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.478 DEBUG:boto:Params: {} 2023-09-29T22:26:24.479 DEBUG:boto:Token: None 2023-09-29T22:26:24.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T22:26:24.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T22:26:24.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b64df04e148228bb-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.522 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.523 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.523 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.523 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:26:24.523 DEBUG:boto:Method: GET 2023-09-29T22:26:24.523 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.523 DEBUG:boto:Data: 2023-09-29T22:26:24.523 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.523 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.523 DEBUG:boto:Params: {} 2023-09-29T22:26:24.524 DEBUG:boto:Token: None 2023-09-29T22:26:24.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:26:24.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T22:26:24.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T22:26:24.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058304e0e35e95cbf-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.527 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.527 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.528 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.528 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.528 DEBUG:boto:Method: GET 2023-09-29T22:26:24.528 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:26:24.528 DEBUG:boto:Data: 2023-09-29T22:26:24.528 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.528 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.528 DEBUG:boto:Params: {} 2023-09-29T22:26:24.528 DEBUG:boto:Token: None 2023-09-29T22:26:24.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005287aae847adbbf5-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.534 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.534 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.535 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.535 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.535 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.535 DEBUG:boto:Method: GET 2023-09-29T22:26:24.535 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:26:24.535 DEBUG:boto:Data: 2023-09-29T22:26:24.535 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.535 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.535 DEBUG:boto:Params: {} 2023-09-29T22:26:24.536 DEBUG:boto:Token: None 2023-09-29T22:26:24.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d40fbd475c9a7f-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.582 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.583 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.583 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.583 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.583 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:26:24.583 DEBUG:boto:Data: 2023-09-29T22:26:24.584 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.584 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.584 DEBUG:boto:Params: {} 2023-09-29T22:26:24.584 DEBUG:boto:Token: None 2023-09-29T22:26:24.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d 2023-09-29T22:26:24.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI= 2023-09-29T22:26:24.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI=', 'Content-Length': '0'} 2023-09-29T22:26:24.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000152b37d0bc0e6cc7-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.587 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.587 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.587 DEBUG:boto:Method: HEAD 2023-09-29T22:26:24.587 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:26:24.588 DEBUG:boto:Data: 2023-09-29T22:26:24.588 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.588 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.588 DEBUG:boto:Params: {} 2023-09-29T22:26:24.588 DEBUG:boto:Token: None 2023-09-29T22:26:24.588 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d 2023-09-29T22:26:24.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI= 2023-09-29T22:26:24.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI=', 'Content-Length': '0'} 2023-09-29T22:26:24.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eafd0360e5e16036-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:26:24.591 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.592 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.592 DEBUG:boto:Method: GET 2023-09-29T22:26:24.592 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:26:24.592 DEBUG:boto:Data: 2023-09-29T22:26:24.592 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.592 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.592 DEBUG:boto:Params: {} 2023-09-29T22:26:24.592 DEBUG:boto:Token: None 2023-09-29T22:26:24.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d 2023-09-29T22:26:24.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo= 2023-09-29T22:26:24.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo=', 'Content-Length': '0'} 2023-09-29T22:26:24.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094353bd415032ad8-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.596 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:26:24.596 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:26:24.596 DEBUG:boto:Method: GET 2023-09-29T22:26:24.596 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:26:24.596 DEBUG:boto:Data: 2023-09-29T22:26:24.596 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.596 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.597 DEBUG:boto:Params: {} 2023-09-29T22:26:24.597 DEBUG:boto:Token: None 2023-09-29T22:26:24.597 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/d 2023-09-29T22:26:24.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo= 2023-09-29T22:26:24.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo=', 'Content-Length': '0'} 2023-09-29T22:26:24.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996fb76b05292edb-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.600 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.600 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.600 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.600 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.601 DEBUG:boto:Method: GET 2023-09-29T22:26:24.601 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:26:24.601 DEBUG:boto:Data: 2023-09-29T22:26:24.601 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:26:24.601 DEBUG:boto:Port: 8001 2023-09-29T22:26:24.601 DEBUG:boto:Params: {} 2023-09-29T22:26:24.601 DEBUG:boto:Token: None 2023-09-29T22:26:24.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.602 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.606 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1cd47ded15a8b2-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.646 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.647 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:24.647 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:24.647 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.647 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:26:24.647 DEBUG:boto:Method: GET 2023-09-29T22:26:24.648 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:26:24.648 DEBUG:boto:Data: 2023-09-29T22:26:24.648 DEBUG:boto:Headers: {} 2023-09-29T22:26:24.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:24.648 DEBUG:boto:Port: 8000 2023-09-29T22:26:24.648 DEBUG:boto:Params: {} 2023-09-29T22:26:24.648 DEBUG:boto:Token: None 2023-09-29T22:26:24.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:26:24 GMT /hcnphz-55/ 2023-09-29T22:26:24.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T22:26:24.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T22:26:24.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96c07f1214e4fd1-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:24.653 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:26:24.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-55 zones={a2, a1} 2023-09-29T22:26:24.654 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:24.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:24.654 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:26:24.696 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:24.696 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:24.779 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:26:24.779 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:26:24.780 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:26:24.781 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.782 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:26:24.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:24.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:24.784 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:24.825 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:24.825 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:24.907 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:26:24.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "1#00000000001.39.6", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16529626172099963377", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "a", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:26:03.091518792Z", 2023-09-29T22:26:24.908 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.909 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "5#00000000001.45.6", 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.7255028496961225069", 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "d", 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:24.910 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:26:03.214573239Z", 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:24.911 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "8#00000000001.44.6", 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10735208816778764772", 2023-09-29T22:26:24.912 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "c", 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:26:03.164775568Z", 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:24.913 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.914 INFO:teuthology.orchestra.run.smithi112.stdout: "op_id": "10#00000000001.62.6", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10601652402760318111", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "op": "write", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "object": "b", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "state": "complete", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "index_ver": 1, 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:26:03.146528163Z", 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:26:24.915 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: "bilog_flags": 0, 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned": false, 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "", 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "", 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: "zones_trace": [ 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:24.916 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:24.917 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:26:24.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:24.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:24.918 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:26:24.960 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:24.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:25.038 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:26:25.038 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-55 2023-09-29T22:26:25.048 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-09-29T22:26:25.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:25.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:26:25.128 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:26:25.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:25.213 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:25.294 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:26:25.294 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:25.294 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:25.296 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:26:25.297 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:26:25.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:26:25.299 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:26:25.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:25.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:26:25.300 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:26:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:25.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:25.423 INFO:teuthology.orchestra.run.smithi112.stdout:[] 2023-09-29T22:26:25.426 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:25.426 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:25.426 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.426 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:25.426 DEBUG:boto:Data: 2023-09-29T22:26:25.426 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.427 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.427 DEBUG:boto:Params: {} 2023-09-29T22:26:25.427 DEBUG:boto:Token: None 2023-09-29T22:26:25.427 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/ 2023-09-29T22:26:25.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T22:26:25.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T22:26:25.430 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b69294f5c8a4f4e-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.430 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:26:25.430 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:26:25.430 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.430 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:26:25.431 DEBUG:boto:Data: 2023-09-29T22:26:25.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:25.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.431 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.431 DEBUG:boto:Params: {} 2023-09-29T22:26:25.431 DEBUG:boto:Token: None 2023-09-29T22:26:25.431 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/e 2023-09-29T22:26:25.431 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Svm409yhFgka/uwmhV1UeO/nV+w= 2023-09-29T22:26:25.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Svm409yhFgka/uwmhV1UeO/nV+w='} 2023-09-29T22:26:25.441 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eba0800a3dede4b7-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.442 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:25.442 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:25.442 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.442 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:25.442 DEBUG:boto:Data: 2023-09-29T22:26:25.442 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.442 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.443 DEBUG:boto:Params: {} 2023-09-29T22:26:25.443 DEBUG:boto:Token: None 2023-09-29T22:26:25.443 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/ 2023-09-29T22:26:25.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T22:26:25.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T22:26:25.446 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6a473b7d2c41841-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.446 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:26:25.446 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:26:25.446 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.446 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:26:25.446 DEBUG:boto:Data: 2023-09-29T22:26:25.447 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:25.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.447 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.447 DEBUG:boto:Params: {} 2023-09-29T22:26:25.447 DEBUG:boto:Token: None 2023-09-29T22:26:25.447 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/f 2023-09-29T22:26:25.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bGcFvmTC8RQ0eyZ88hb/DOJyNos= 2023-09-29T22:26:25.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bGcFvmTC8RQ0eyZ88hb/DOJyNos='} 2023-09-29T22:26:25.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4844afac09f9871-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.495 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:25.495 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:25.495 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.495 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:25.495 DEBUG:boto:Data: 2023-09-29T22:26:25.495 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.495 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.495 DEBUG:boto:Params: {} 2023-09-29T22:26:25.496 DEBUG:boto:Token: None 2023-09-29T22:26:25.496 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/ 2023-09-29T22:26:25.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T22:26:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T22:26:25.499 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079d2d0f797e0d33e-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.499 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:26:25.500 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:26:25.500 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.500 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:26:25.500 DEBUG:boto:Data: 2023-09-29T22:26:25.500 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:25.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.500 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.500 DEBUG:boto:Params: {} 2023-09-29T22:26:25.500 DEBUG:boto:Token: None 2023-09-29T22:26:25.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/g 2023-09-29T22:26:25.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vxRRysFk5R7Sh7pWE1fs65OhyBE= 2023-09-29T22:26:25.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vxRRysFk5R7Sh7pWE1fs65OhyBE='} 2023-09-29T22:26:25.512 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076a5fb8e3bb664a3-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.512 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:26:25.512 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:26:25.512 DEBUG:boto:Method: HEAD 2023-09-29T22:26:25.512 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:26:25.512 DEBUG:boto:Data: 2023-09-29T22:26:25.513 DEBUG:boto:Headers: {} 2023-09-29T22:26:25.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.513 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.513 DEBUG:boto:Params: {} 2023-09-29T22:26:25.513 DEBUG:boto:Token: None 2023-09-29T22:26:25.513 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/ 2023-09-29T22:26:25.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T22:26:25.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T22:26:25.516 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088dde7299d71666d-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.516 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:26:25.516 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:26:25.516 DEBUG:boto:Method: PUT 2023-09-29T22:26:25.516 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:26:25.517 DEBUG:boto:Data: 2023-09-29T22:26:25.517 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:26:25.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:26:25.517 DEBUG:boto:Port: 8000 2023-09-29T22:26:25.517 DEBUG:boto:Params: {} 2023-09-29T22:26:25.517 DEBUG:boto:Token: None 2023-09-29T22:26:25.517 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:26:25 GMT /hcnphz-55/h 2023-09-29T22:26:25.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tg3VGe3/MykbD4TlX5WE7F4WKdk= 2023-09-29T22:26:25.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tg3VGe3/MykbD4TlX5WE7F4WKdk='} 2023-09-29T22:26:25.562 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001765f7ee71591017-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:26:25.564 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-55 2023-09-29T22:26:25.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:25.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:26:25.565 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:26:25.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:26:25.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:26:25.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:25.680+0000 7fa658d2ff40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T22:26:30.683 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:30.680+0000 7fa658d2ff40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T22:26:35.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:35.684+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:26:35.686 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , ] 2023-09-29T22:26:35.686 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:26:40.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:40.684+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:26:40.687 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , ] 2023-09-29T22:26:40.687 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:26:45.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:45.688+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:26:45.689 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , ] 2023-09-29T22:26:45.689 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:26:50.129 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:26:50.125+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:26:50.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:50.688+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:26:50.690 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , ] 2023-09-29T22:26:50.690 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:26:55.691 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:26:55.688+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:26:55.691 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T22:26:55.692 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:27:00.692 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:00.692+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:27:00.693 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T22:27:00.693 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:27:05.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:05.691+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:27:05.694 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T22:27:05.694 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:27:10.695 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:10.691+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:27:10.695 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T22:27:10.695 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:27:15.696 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:15.695+0000 7fa658d2ff40 1 bucket sync caught up with source: 2023-09-29T22:27:15.697 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:27:15.697 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T22:27:15.697 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:15.695+0000 7fa658d2ff40 0 bucket checkpoint complete 2023-09-29T22:27:15.728 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-55 2023-09-29T22:27:15.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:15.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:15.729 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:27:15.782 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:15.782 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:15.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:15.843+0000 7f66bfe8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:27:15.847 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.162.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:15.847 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:20.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:20.847+0000 7f66bfe8bf40 1 waiting for incremental sync to catch up: 2023-09-29T22:27:20.850 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.162.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:20.850 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:25.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:25.851+0000 7f66bfe8bf40 1 bucket sync caught up with source: 2023-09-29T22:27:25.853 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:25.853 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:25.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:25.851+0000 7f66bfe8bf40 0 bucket checkpoint complete 2023-09-29T22:27:25.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-55 zones={a2, a1} 2023-09-29T22:27:25.884 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.884 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.884 DEBUG:boto:Method: HEAD 2023-09-29T22:27:25.884 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:27:25.884 DEBUG:boto:Data: 2023-09-29T22:27:25.884 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.885 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.885 DEBUG:boto:Params: {} 2023-09-29T22:27:25.885 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:27:25.885 DEBUG:boto:Token: None 2023-09-29T22:27:25.885 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/ 2023-09-29T22:27:25.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA= 2023-09-29T22:27:25.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA=', 'Content-Length': '0'} 2023-09-29T22:27:25.891 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fecfb967f412c96-0065174f4d-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.891 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.892 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.892 DEBUG:boto:Method: HEAD 2023-09-29T22:27:25.892 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:27:25.892 DEBUG:boto:Data: 2023-09-29T22:27:25.892 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:25.892 DEBUG:boto:Port: 8000 2023-09-29T22:27:25.892 DEBUG:boto:Params: {} 2023-09-29T22:27:25.892 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:27:25.893 DEBUG:boto:Token: None 2023-09-29T22:27:25.893 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/ 2023-09-29T22:27:25.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA= 2023-09-29T22:27:25.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA=', 'Content-Length': '0'} 2023-09-29T22:27:25.897 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e128b751eb976b25-0065174f4d-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:27:25.897 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.897 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.897 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.897 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.897 DEBUG:boto:Method: GET 2023-09-29T22:27:25.897 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.898 DEBUG:boto:Data: 2023-09-29T22:27:25.898 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.898 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.898 DEBUG:boto:Params: {} 2023-09-29T22:27:25.898 DEBUG:boto:Token: None 2023-09-29T22:27:25.898 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/?versions 2023-09-29T22:27:25.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T22:27:25.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T22:27:25.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e548903fe8cdee2c-0065174f4d-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.902 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:25.903 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:27:25.903 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:27:25.904 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:27:25.904 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.904 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.905 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.905 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.905 DEBUG:boto:Method: GET 2023-09-29T22:27:25.905 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.905 DEBUG:boto:Data: 2023-09-29T22:27:25.905 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:25.905 DEBUG:boto:Port: 8000 2023-09-29T22:27:25.905 DEBUG:boto:Params: {} 2023-09-29T22:27:25.905 DEBUG:boto:Token: None 2023-09-29T22:27:25.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/?versions 2023-09-29T22:27:25.906 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T22:27:25.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T22:27:25.909 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a03ba692d82dfb-0065174f4d-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.909 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:25.909 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:27:25.909 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:27:25.910 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:27:25.910 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:27:25.910 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:27:25.910 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:27:25.910 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:27:25.910 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:27:25.910 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.910 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.910 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.911 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.911 DEBUG:boto:Method: GET 2023-09-29T22:27:25.911 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.911 DEBUG:boto:Data: 2023-09-29T22:27:25.911 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.911 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.911 DEBUG:boto:Params: {} 2023-09-29T22:27:25.911 DEBUG:boto:Token: None 2023-09-29T22:27:25.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/?versions 2023-09-29T22:27:25.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T22:27:25.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T22:27:25.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009237124e724bd7b6-0065174f4d-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.916 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:25.917 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.917 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.918 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.918 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.918 DEBUG:boto:Method: GET 2023-09-29T22:27:25.918 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:25.919 DEBUG:boto:Data: 2023-09-29T22:27:25.919 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:25.919 DEBUG:boto:Port: 8000 2023-09-29T22:27:25.919 DEBUG:boto:Params: {} 2023-09-29T22:27:25.920 DEBUG:boto:Token: None 2023-09-29T22:27:25.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/?versions 2023-09-29T22:27:25.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T22:27:25.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T22:27:25.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee75dd4fbe556e36-0065174f4d-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.924 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:25.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:27:25.925 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:25.926 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:25.926 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.927 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.927 DEBUG:boto:Method: GET 2023-09-29T22:27:25.927 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.929 DEBUG:boto:Data: 2023-09-29T22:27:25.929 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.929 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.929 DEBUG:boto:Params: {} 2023-09-29T22:27:25.929 DEBUG:boto:Token: None 2023-09-29T22:27:25.929 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.930 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T22:27:25.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T22:27:25.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de4fe1d723c45fe-0065174f4d-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.974 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:25.975 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:25.975 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.975 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.975 DEBUG:boto:Method: GET 2023-09-29T22:27:25.975 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.975 DEBUG:boto:Data: 2023-09-29T22:27:25.975 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:25.975 DEBUG:boto:Port: 8000 2023-09-29T22:27:25.976 DEBUG:boto:Params: {} 2023-09-29T22:27:25.976 DEBUG:boto:Token: None 2023-09-29T22:27:25.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T22:27:25.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T22:27:25.980 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081c168677a57750e-0065174f4d-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.980 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:25.980 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:25.980 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.980 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.980 DEBUG:boto:Method: HEAD 2023-09-29T22:27:25.981 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.981 DEBUG:boto:Data: 2023-09-29T22:27:25.981 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.981 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.981 DEBUG:boto:Params: {} 2023-09-29T22:27:25.981 DEBUG:boto:Token: None 2023-09-29T22:27:25.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g= 2023-09-29T22:27:25.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g=', 'Content-Length': '0'} 2023-09-29T22:27:25.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f891bfac70d0b1-0065174f4d-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.984 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:25.984 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:25.984 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.985 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.985 DEBUG:boto:Method: HEAD 2023-09-29T22:27:25.985 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.985 DEBUG:boto:Data: 2023-09-29T22:27:25.985 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:25.985 DEBUG:boto:Port: 8000 2023-09-29T22:27:25.985 DEBUG:boto:Params: {} 2023-09-29T22:27:25.985 DEBUG:boto:Token: None 2023-09-29T22:27:25.985 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.986 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g= 2023-09-29T22:27:25.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g=', 'Content-Length': '0'} 2023-09-29T22:27:25.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005901b8b89c07725f-0065174f4d-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:27:25.989 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:25.989 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:25.989 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.989 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.989 DEBUG:boto:Method: GET 2023-09-29T22:27:25.989 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.989 DEBUG:boto:Data: 2023-09-29T22:27:25.989 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.990 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.990 DEBUG:boto:Params: {} 2023-09-29T22:27:25.990 DEBUG:boto:Token: None 2023-09-29T22:27:25.990 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.990 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T22:27:25.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T22:27:25.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed84c3ad98861e6b-0065174f4d-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.993 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:25.993 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:25.993 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.993 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:25.994 DEBUG:boto:Method: GET 2023-09-29T22:27:25.994 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.994 DEBUG:boto:Data: 2023-09-29T22:27:25.994 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:25.994 DEBUG:boto:Port: 8000 2023-09-29T22:27:25.994 DEBUG:boto:Params: {} 2023-09-29T22:27:25.994 DEBUG:boto:Token: None 2023-09-29T22:27:25.994 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:25.995 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T22:27:25.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T22:27:25.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051472a26e9257e3d-0065174f4d-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:25.998 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:25.998 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:25.998 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:25.998 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:25.998 DEBUG:boto:Method: GET 2023-09-29T22:27:25.998 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:25.998 DEBUG:boto:Data: 2023-09-29T22:27:25.998 DEBUG:boto:Headers: {} 2023-09-29T22:27:25.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:25.999 DEBUG:boto:Port: 8001 2023-09-29T22:27:25.999 DEBUG:boto:Params: {} 2023-09-29T22:27:25.999 DEBUG:boto:Token: None 2023-09-29T22:27:25.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:25 GMT /hcnphz-55/ 2023-09-29T22:27:25.999 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fzsOvyEi/mRElMsoGSaihtzGKW8= 2023-09-29T22:27:25.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fzsOvyEi/mRElMsoGSaihtzGKW8=', 'Content-Length': '0'} 2023-09-29T22:27:26.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa954738329afa59-0065174f4d-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.004 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.004 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.004 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.004 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:26.004 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:26.005 DEBUG:boto:Method: GET 2023-09-29T22:27:26.005 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:26.005 DEBUG:boto:Data: 2023-09-29T22:27:26.005 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.005 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.005 DEBUG:boto:Params: {} 2023-09-29T22:27:26.005 DEBUG:boto:Token: None 2023-09-29T22:27:26.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.006 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3c1c0281b8c12d-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.009 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.009 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:26.009 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:26.009 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.009 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:26.009 DEBUG:boto:Data: 2023-09-29T22:27:26.010 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.010 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.010 DEBUG:boto:Params: {} 2023-09-29T22:27:26.010 DEBUG:boto:Token: None 2023-09-29T22:27:26.010 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/a 2023-09-29T22:27:26.010 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI= 2023-09-29T22:27:26.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI=', 'Content-Length': '0'} 2023-09-29T22:27:26.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f6383da5f67cfb-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.013 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:26.013 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:26.013 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.014 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:26.014 DEBUG:boto:Data: 2023-09-29T22:27:26.014 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.014 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.014 DEBUG:boto:Params: {} 2023-09-29T22:27:26.014 DEBUG:boto:Token: None 2023-09-29T22:27:26.014 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/a 2023-09-29T22:27:26.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI= 2023-09-29T22:27:26.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI=', 'Content-Length': '0'} 2023-09-29T22:27:26.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f31b1ddd2a022a-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:27:26.017 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:26.018 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:26.018 DEBUG:boto:Method: GET 2023-09-29T22:27:26.018 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:26.018 DEBUG:boto:Data: 2023-09-29T22:27:26.018 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.018 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.018 DEBUG:boto:Params: {} 2023-09-29T22:27:26.018 DEBUG:boto:Token: None 2023-09-29T22:27:26.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/a 2023-09-29T22:27:26.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA= 2023-09-29T22:27:26.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA=', 'Content-Length': '0'} 2023-09-29T22:27:26.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001514818696359ba6-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.062 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:26.062 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:26.063 DEBUG:boto:Method: GET 2023-09-29T22:27:26.063 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:26.063 DEBUG:boto:Data: 2023-09-29T22:27:26.063 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.063 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.063 DEBUG:boto:Params: {} 2023-09-29T22:27:26.063 DEBUG:boto:Token: None 2023-09-29T22:27:26.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/a 2023-09-29T22:27:26.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA= 2023-09-29T22:27:26.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA=', 'Content-Length': '0'} 2023-09-29T22:27:26.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0e5c6d01743389-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.067 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.067 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.067 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:26.067 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:26.067 DEBUG:boto:Method: GET 2023-09-29T22:27:26.067 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:26.067 DEBUG:boto:Data: 2023-09-29T22:27:26.067 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.068 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.068 DEBUG:boto:Params: {} 2023-09-29T22:27:26.068 DEBUG:boto:Token: None 2023-09-29T22:27:26.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf77a5dd9293029-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.073 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.073 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.073 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.073 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:26.073 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:26.073 DEBUG:boto:Method: GET 2023-09-29T22:27:26.074 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:26.074 DEBUG:boto:Data: 2023-09-29T22:27:26.074 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.074 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.074 DEBUG:boto:Params: {} 2023-09-29T22:27:26.074 DEBUG:boto:Token: None 2023-09-29T22:27:26.074 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.074 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c4b7587caf5f694-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.122 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:27:26.123 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.123 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.123 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.123 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.123 DEBUG:boto:Method: GET 2023-09-29T22:27:26.124 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.124 DEBUG:boto:Data: 2023-09-29T22:27:26.124 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.124 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.124 DEBUG:boto:Params: {} 2023-09-29T22:27:26.124 DEBUG:boto:Token: None 2023-09-29T22:27:26.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.125 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T22:27:26.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T22:27:26.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e02c34fd6c300c-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.128 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.128 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.128 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.128 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.128 DEBUG:boto:Method: GET 2023-09-29T22:27:26.128 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.128 DEBUG:boto:Data: 2023-09-29T22:27:26.128 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.128 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.129 DEBUG:boto:Params: {} 2023-09-29T22:27:26.129 DEBUG:boto:Token: None 2023-09-29T22:27:26.129 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T22:27:26.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T22:27:26.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956cfeea77f61419-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.133 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.133 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.133 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.134 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.134 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.134 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.134 DEBUG:boto:Data: 2023-09-29T22:27:26.134 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.134 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.134 DEBUG:boto:Params: {} 2023-09-29T22:27:26.134 DEBUG:boto:Token: None 2023-09-29T22:27:26.135 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.135 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc= 2023-09-29T22:27:26.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc=', 'Content-Length': '0'} 2023-09-29T22:27:26.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b87527734526186-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.137 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.137 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.137 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.138 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.138 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.138 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.138 DEBUG:boto:Data: 2023-09-29T22:27:26.138 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.138 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.138 DEBUG:boto:Params: {} 2023-09-29T22:27:26.138 DEBUG:boto:Token: None 2023-09-29T22:27:26.139 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.139 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc= 2023-09-29T22:27:26.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc=', 'Content-Length': '0'} 2023-09-29T22:27:26.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035faf4ce85d129db-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:27:26.142 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.142 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.142 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.142 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.142 DEBUG:boto:Method: GET 2023-09-29T22:27:26.142 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.142 DEBUG:boto:Data: 2023-09-29T22:27:26.142 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.143 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.143 DEBUG:boto:Params: {} 2023-09-29T22:27:26.143 DEBUG:boto:Token: None 2023-09-29T22:27:26.143 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.143 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T22:27:26.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T22:27:26.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f62fd143a200ddd-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.146 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.146 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.146 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.146 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:26.146 DEBUG:boto:Method: GET 2023-09-29T22:27:26.147 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.147 DEBUG:boto:Data: 2023-09-29T22:27:26.147 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.147 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.147 DEBUG:boto:Params: {} 2023-09-29T22:27:26.147 DEBUG:boto:Token: None 2023-09-29T22:27:26.147 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:26.147 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T22:27:26.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T22:27:26.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825689ff75a3b1f1-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.150 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.151 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.151 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.151 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.151 DEBUG:boto:Method: GET 2023-09-29T22:27:26.151 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:26.151 DEBUG:boto:Data: 2023-09-29T22:27:26.151 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.151 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.151 DEBUG:boto:Params: {} 2023-09-29T22:27:26.152 DEBUG:boto:Token: None 2023-09-29T22:27:26.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e359ddd0b490fb3-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.156 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.157 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.157 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.157 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.157 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.157 DEBUG:boto:Method: GET 2023-09-29T22:27:26.158 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:26.158 DEBUG:boto:Data: 2023-09-29T22:27:26.158 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.158 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.158 DEBUG:boto:Params: {} 2023-09-29T22:27:26.158 DEBUG:boto:Token: None 2023-09-29T22:27:26.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.161 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d83ef19eb7ec19f-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.202 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.203 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.203 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.203 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.203 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:26.204 DEBUG:boto:Data: 2023-09-29T22:27:26.204 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.204 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.204 DEBUG:boto:Params: {} 2023-09-29T22:27:26.204 DEBUG:boto:Token: None 2023-09-29T22:27:26.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b 2023-09-29T22:27:26.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q= 2023-09-29T22:27:26.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q=', 'Content-Length': '0'} 2023-09-29T22:27:26.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cb748ee176493e-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.208 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.208 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.208 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.208 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:26.208 DEBUG:boto:Data: 2023-09-29T22:27:26.208 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.208 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.208 DEBUG:boto:Params: {} 2023-09-29T22:27:26.209 DEBUG:boto:Token: None 2023-09-29T22:27:26.209 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b 2023-09-29T22:27:26.209 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q= 2023-09-29T22:27:26.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q=', 'Content-Length': '0'} 2023-09-29T22:27:26.212 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc2f81c3554ca7d-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:27:26.212 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.212 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.212 DEBUG:boto:Method: GET 2023-09-29T22:27:26.212 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:26.212 DEBUG:boto:Data: 2023-09-29T22:27:26.213 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.213 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.213 DEBUG:boto:Params: {} 2023-09-29T22:27:26.213 DEBUG:boto:Token: None 2023-09-29T22:27:26.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b 2023-09-29T22:27:26.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c= 2023-09-29T22:27:26.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c=', 'Content-Length': '0'} 2023-09-29T22:27:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbe54629e3484c8-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.258 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:26.259 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:26.259 DEBUG:boto:Method: GET 2023-09-29T22:27:26.259 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:26.259 DEBUG:boto:Data: 2023-09-29T22:27:26.259 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.259 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.259 DEBUG:boto:Params: {} 2023-09-29T22:27:26.260 DEBUG:boto:Token: None 2023-09-29T22:27:26.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/b 2023-09-29T22:27:26.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c= 2023-09-29T22:27:26.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c=', 'Content-Length': '0'} 2023-09-29T22:27:26.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d067dbd563a095a-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.263 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.264 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.264 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.264 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.264 DEBUG:boto:Method: GET 2023-09-29T22:27:26.264 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:26.264 DEBUG:boto:Data: 2023-09-29T22:27:26.264 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.264 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.265 DEBUG:boto:Params: {} 2023-09-29T22:27:26.265 DEBUG:boto:Token: None 2023-09-29T22:27:26.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb46ea7c0c809cd7-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.270 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.270 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.270 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.270 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.271 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:26.271 DEBUG:boto:Method: GET 2023-09-29T22:27:26.271 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:26.271 DEBUG:boto:Data: 2023-09-29T22:27:26.271 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.271 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.271 DEBUG:boto:Params: {} 2023-09-29T22:27:26.272 DEBUG:boto:Token: None 2023-09-29T22:27:26.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505702c7bf947ed1-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.275 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:27:26.275 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.276 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.276 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.276 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.276 DEBUG:boto:Method: GET 2023-09-29T22:27:26.276 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.276 DEBUG:boto:Data: 2023-09-29T22:27:26.276 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.276 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.276 DEBUG:boto:Params: {} 2023-09-29T22:27:26.277 DEBUG:boto:Token: None 2023-09-29T22:27:26.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T22:27:26.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T22:27:26.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef5d6abffebdc30-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.280 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.280 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.280 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.280 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.280 DEBUG:boto:Method: GET 2023-09-29T22:27:26.280 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.281 DEBUG:boto:Data: 2023-09-29T22:27:26.281 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.281 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.281 DEBUG:boto:Params: {} 2023-09-29T22:27:26.281 DEBUG:boto:Token: None 2023-09-29T22:27:26.281 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T22:27:26.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T22:27:26.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f8d666e7a7c494-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.326 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.327 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.327 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.327 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.327 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.327 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.328 DEBUG:boto:Data: 2023-09-29T22:27:26.328 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.328 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.328 DEBUG:boto:Params: {} 2023-09-29T22:27:26.329 DEBUG:boto:Token: None 2023-09-29T22:27:26.329 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI= 2023-09-29T22:27:26.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI=', 'Content-Length': '0'} 2023-09-29T22:27:26.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bbbe4784a5ece9-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.332 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.333 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.333 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.333 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.333 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.333 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.333 DEBUG:boto:Data: 2023-09-29T22:27:26.333 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.334 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.334 DEBUG:boto:Params: {} 2023-09-29T22:27:26.334 DEBUG:boto:Token: None 2023-09-29T22:27:26.334 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI= 2023-09-29T22:27:26.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI=', 'Content-Length': '0'} 2023-09-29T22:27:26.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2b98505f8b9543-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:27:26.337 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.337 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.337 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.338 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.338 DEBUG:boto:Method: GET 2023-09-29T22:27:26.338 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.338 DEBUG:boto:Data: 2023-09-29T22:27:26.338 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.338 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.338 DEBUG:boto:Params: {} 2023-09-29T22:27:26.338 DEBUG:boto:Token: None 2023-09-29T22:27:26.339 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.339 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T22:27:26.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T22:27:26.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bac9aabb312772-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.386 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.387 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.387 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.387 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:26.387 DEBUG:boto:Method: GET 2023-09-29T22:27:26.387 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.387 DEBUG:boto:Data: 2023-09-29T22:27:26.387 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.388 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.388 DEBUG:boto:Params: {} 2023-09-29T22:27:26.388 DEBUG:boto:Token: None 2023-09-29T22:27:26.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:26.389 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T22:27:26.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T22:27:26.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d61bb48e117d412-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.392 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.392 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.392 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.393 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.393 DEBUG:boto:Method: GET 2023-09-29T22:27:26.393 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:26.393 DEBUG:boto:Data: 2023-09-29T22:27:26.393 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.393 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.393 DEBUG:boto:Params: {} 2023-09-29T22:27:26.393 DEBUG:boto:Token: None 2023-09-29T22:27:26.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005790c1cce3054cac-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.399 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.399 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.399 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.400 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.400 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.400 DEBUG:boto:Method: GET 2023-09-29T22:27:26.400 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:26.400 DEBUG:boto:Data: 2023-09-29T22:27:26.400 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.400 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.400 DEBUG:boto:Params: {} 2023-09-29T22:27:26.400 DEBUG:boto:Token: None 2023-09-29T22:27:26.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.404 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5560d4c73b0c4ce-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.404 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.405 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.405 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.405 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.405 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:26.405 DEBUG:boto:Data: 2023-09-29T22:27:26.405 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.405 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.405 DEBUG:boto:Params: {} 2023-09-29T22:27:26.406 DEBUG:boto:Token: None 2023-09-29T22:27:26.406 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c 2023-09-29T22:27:26.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE= 2023-09-29T22:27:26.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE=', 'Content-Length': '0'} 2023-09-29T22:27:26.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d2cd8ad61d8d0d2-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.408 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.409 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.409 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.409 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:26.409 DEBUG:boto:Data: 2023-09-29T22:27:26.409 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.409 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.409 DEBUG:boto:Params: {} 2023-09-29T22:27:26.409 DEBUG:boto:Token: None 2023-09-29T22:27:26.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c 2023-09-29T22:27:26.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE= 2023-09-29T22:27:26.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE=', 'Content-Length': '0'} 2023-09-29T22:27:26.412 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b44a53162626f34-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:27:26.413 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.413 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.413 DEBUG:boto:Method: GET 2023-09-29T22:27:26.413 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:26.413 DEBUG:boto:Data: 2023-09-29T22:27:26.413 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.414 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.414 DEBUG:boto:Params: {} 2023-09-29T22:27:26.414 DEBUG:boto:Token: None 2023-09-29T22:27:26.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c 2023-09-29T22:27:26.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo= 2023-09-29T22:27:26.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo=', 'Content-Length': '0'} 2023-09-29T22:27:26.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e125f6468202fcf5-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.417 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:26.418 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:26.418 DEBUG:boto:Method: GET 2023-09-29T22:27:26.418 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:26.418 DEBUG:boto:Data: 2023-09-29T22:27:26.418 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.418 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.418 DEBUG:boto:Params: {} 2023-09-29T22:27:26.418 DEBUG:boto:Token: None 2023-09-29T22:27:26.419 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/c 2023-09-29T22:27:26.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo= 2023-09-29T22:27:26.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo=', 'Content-Length': '0'} 2023-09-29T22:27:26.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130f40b75086f0be-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.462 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.463 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.463 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.463 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.463 DEBUG:boto:Method: GET 2023-09-29T22:27:26.463 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:26.463 DEBUG:boto:Data: 2023-09-29T22:27:26.463 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.463 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.464 DEBUG:boto:Params: {} 2023-09-29T22:27:26.464 DEBUG:boto:Token: None 2023-09-29T22:27:26.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034656365072c233c-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.469 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.470 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.470 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.470 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.470 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:26.470 DEBUG:boto:Method: GET 2023-09-29T22:27:26.470 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:26.470 DEBUG:boto:Data: 2023-09-29T22:27:26.471 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.471 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.471 DEBUG:boto:Params: {} 2023-09-29T22:27:26.471 DEBUG:boto:Token: None 2023-09-29T22:27:26.471 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.477 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aff89b531561cb8-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.477 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:27:26.477 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.478 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.478 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.478 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.478 DEBUG:boto:Method: GET 2023-09-29T22:27:26.478 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.478 DEBUG:boto:Data: 2023-09-29T22:27:26.478 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.478 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.478 DEBUG:boto:Params: {} 2023-09-29T22:27:26.479 DEBUG:boto:Token: None 2023-09-29T22:27:26.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T22:27:26.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T22:27:26.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531c778e176994e4-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.522 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.523 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.523 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.523 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.523 DEBUG:boto:Method: GET 2023-09-29T22:27:26.523 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.523 DEBUG:boto:Data: 2023-09-29T22:27:26.523 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.524 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.524 DEBUG:boto:Params: {} 2023-09-29T22:27:26.524 DEBUG:boto:Token: None 2023-09-29T22:27:26.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T22:27:26.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T22:27:26.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a6141d745053a3-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.527 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.528 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.528 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.528 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.528 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.528 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.528 DEBUG:boto:Data: 2023-09-29T22:27:26.528 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.528 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.529 DEBUG:boto:Params: {} 2023-09-29T22:27:26.529 DEBUG:boto:Token: None 2023-09-29T22:27:26.529 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ= 2023-09-29T22:27:26.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ=', 'Content-Length': '0'} 2023-09-29T22:27:26.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007cd125531a91538-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.532 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.532 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.532 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.532 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.532 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.533 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.533 DEBUG:boto:Data: 2023-09-29T22:27:26.533 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.533 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.533 DEBUG:boto:Params: {} 2023-09-29T22:27:26.533 DEBUG:boto:Token: None 2023-09-29T22:27:26.533 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ= 2023-09-29T22:27:26.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ=', 'Content-Length': '0'} 2023-09-29T22:27:26.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f89fbab39c4c3f-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:27:26.537 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.537 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.537 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.537 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.537 DEBUG:boto:Method: GET 2023-09-29T22:27:26.537 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.537 DEBUG:boto:Data: 2023-09-29T22:27:26.537 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.538 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.538 DEBUG:boto:Params: {} 2023-09-29T22:27:26.538 DEBUG:boto:Token: None 2023-09-29T22:27:26.538 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.538 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T22:27:26.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T22:27:26.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1a7840c93025b1f-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.541 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.541 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.542 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.542 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:26.542 DEBUG:boto:Method: GET 2023-09-29T22:27:26.542 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.542 DEBUG:boto:Data: 2023-09-29T22:27:26.543 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.543 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.543 DEBUG:boto:Params: {} 2023-09-29T22:27:26.543 DEBUG:boto:Token: None 2023-09-29T22:27:26.543 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:26.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T22:27:26.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T22:27:26.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d161916174affe-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.590 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.591 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.591 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.591 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.591 DEBUG:boto:Method: GET 2023-09-29T22:27:26.591 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:26.591 DEBUG:boto:Data: 2023-09-29T22:27:26.591 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.591 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.592 DEBUG:boto:Params: {} 2023-09-29T22:27:26.592 DEBUG:boto:Token: None 2023-09-29T22:27:26.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006412da55932e8c1b-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.597 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.597 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.597 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.597 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.597 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.597 DEBUG:boto:Method: GET 2023-09-29T22:27:26.598 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:26.598 DEBUG:boto:Data: 2023-09-29T22:27:26.598 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.598 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.598 DEBUG:boto:Params: {} 2023-09-29T22:27:26.598 DEBUG:boto:Token: None 2023-09-29T22:27:26.598 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd453a2fa2d4a79-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.602 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.602 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.602 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.603 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.603 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:26.603 DEBUG:boto:Data: 2023-09-29T22:27:26.603 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.603 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.603 DEBUG:boto:Params: {} 2023-09-29T22:27:26.603 DEBUG:boto:Token: None 2023-09-29T22:27:26.604 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d 2023-09-29T22:27:26.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo= 2023-09-29T22:27:26.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo=', 'Content-Length': '0'} 2023-09-29T22:27:26.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005268d57ef78f0ed6-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.607 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.607 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.607 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.607 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:26.607 DEBUG:boto:Data: 2023-09-29T22:27:26.607 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.608 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.608 DEBUG:boto:Params: {} 2023-09-29T22:27:26.608 DEBUG:boto:Token: None 2023-09-29T22:27:26.609 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d 2023-09-29T22:27:26.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo= 2023-09-29T22:27:26.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo=', 'Content-Length': '0'} 2023-09-29T22:27:26.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c44482f9cafd9b5-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:27:26.612 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.612 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.613 DEBUG:boto:Method: GET 2023-09-29T22:27:26.613 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:26.613 DEBUG:boto:Data: 2023-09-29T22:27:26.613 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.614 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.614 DEBUG:boto:Params: {} 2023-09-29T22:27:26.614 DEBUG:boto:Token: None 2023-09-29T22:27:26.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d 2023-09-29T22:27:26.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI= 2023-09-29T22:27:26.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI=', 'Content-Length': '0'} 2023-09-29T22:27:26.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b178b5269199d3db-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.658 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:26.659 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:26.659 DEBUG:boto:Method: GET 2023-09-29T22:27:26.659 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:26.659 DEBUG:boto:Data: 2023-09-29T22:27:26.659 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.660 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.660 DEBUG:boto:Params: {} 2023-09-29T22:27:26.660 DEBUG:boto:Token: None 2023-09-29T22:27:26.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/d 2023-09-29T22:27:26.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI= 2023-09-29T22:27:26.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI=', 'Content-Length': '0'} 2023-09-29T22:27:26.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0dddf2aa984e4d-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.664 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.667 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.667 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.667 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.667 DEBUG:boto:Method: GET 2023-09-29T22:27:26.667 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:26.667 DEBUG:boto:Data: 2023-09-29T22:27:26.668 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.668 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.668 DEBUG:boto:Params: {} 2023-09-29T22:27:26.668 DEBUG:boto:Token: None 2023-09-29T22:27:26.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b9592e0e365e592-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.673 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.673 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.674 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.674 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.674 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:26.674 DEBUG:boto:Method: GET 2023-09-29T22:27:26.674 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:26.674 DEBUG:boto:Data: 2023-09-29T22:27:26.674 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.675 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.675 DEBUG:boto:Params: {} 2023-09-29T22:27:26.675 DEBUG:boto:Token: None 2023-09-29T22:27:26.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005100f854695c5f9c-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.678 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:27:26.679 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.679 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.679 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.679 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.679 DEBUG:boto:Method: GET 2023-09-29T22:27:26.679 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.679 DEBUG:boto:Data: 2023-09-29T22:27:26.679 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.680 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.680 DEBUG:boto:Params: {} 2023-09-29T22:27:26.680 DEBUG:boto:Token: None 2023-09-29T22:27:26.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T22:27:26.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T22:27:26.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d85062c1a9cee4b-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.683 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.683 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.683 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.683 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.683 DEBUG:boto:Method: GET 2023-09-29T22:27:26.684 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.684 DEBUG:boto:Data: 2023-09-29T22:27:26.684 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.684 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.684 DEBUG:boto:Params: {} 2023-09-29T22:27:26.684 DEBUG:boto:Token: None 2023-09-29T22:27:26.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.685 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T22:27:26.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T22:27:26.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030ee317f882f42d7-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.730 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.731 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.731 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.731 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.731 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.731 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.731 DEBUG:boto:Data: 2023-09-29T22:27:26.731 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.732 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.732 DEBUG:boto:Params: {} 2023-09-29T22:27:26.732 DEBUG:boto:Token: None 2023-09-29T22:27:26.732 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U= 2023-09-29T22:27:26.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U=', 'Content-Length': '0'} 2023-09-29T22:27:26.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037050c2ffcb2d19b-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.736 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.736 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.737 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.737 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.737 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.737 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.738 DEBUG:boto:Data: 2023-09-29T22:27:26.738 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.738 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.738 DEBUG:boto:Params: {} 2023-09-29T22:27:26.739 DEBUG:boto:Token: None 2023-09-29T22:27:26.739 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U= 2023-09-29T22:27:26.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U=', 'Content-Length': '0'} 2023-09-29T22:27:26.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20b2d438f685a02-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:27:26.742 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.742 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.742 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.743 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.743 DEBUG:boto:Method: GET 2023-09-29T22:27:26.743 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.743 DEBUG:boto:Data: 2023-09-29T22:27:26.743 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.743 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.745 DEBUG:boto:Params: {} 2023-09-29T22:27:26.745 DEBUG:boto:Token: None 2023-09-29T22:27:26.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T22:27:26.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T22:27:26.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e88abb1c388686-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.790 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.791 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.791 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.791 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:26.791 DEBUG:boto:Method: GET 2023-09-29T22:27:26.791 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.791 DEBUG:boto:Data: 2023-09-29T22:27:26.791 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.792 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.792 DEBUG:boto:Params: {} 2023-09-29T22:27:26.792 DEBUG:boto:Token: None 2023-09-29T22:27:26.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:26.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T22:27:26.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T22:27:26.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc33487ca4eebff6-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.795 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.795 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.796 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.796 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.796 DEBUG:boto:Method: GET 2023-09-29T22:27:26.796 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:26.796 DEBUG:boto:Data: 2023-09-29T22:27:26.796 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.796 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.797 DEBUG:boto:Params: {} 2023-09-29T22:27:26.797 DEBUG:boto:Token: None 2023-09-29T22:27:26.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038bea6fcf69079a2-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.802 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.802 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.802 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.802 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.803 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.803 DEBUG:boto:Method: GET 2023-09-29T22:27:26.803 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:26.803 DEBUG:boto:Data: 2023-09-29T22:27:26.803 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.803 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.803 DEBUG:boto:Params: {} 2023-09-29T22:27:26.803 DEBUG:boto:Token: None 2023-09-29T22:27:26.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001666328bc5689918-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.807 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.807 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.807 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.807 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.807 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:26.808 DEBUG:boto:Data: 2023-09-29T22:27:26.808 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.808 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.808 DEBUG:boto:Params: {} 2023-09-29T22:27:26.808 DEBUG:boto:Token: None 2023-09-29T22:27:26.808 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e 2023-09-29T22:27:26.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ= 2023-09-29T22:27:26.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ=', 'Content-Length': '0'} 2023-09-29T22:27:26.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b5671c6e3733f5-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.811 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.811 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.811 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.811 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:26.811 DEBUG:boto:Data: 2023-09-29T22:27:26.812 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.812 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.812 DEBUG:boto:Params: {} 2023-09-29T22:27:26.812 DEBUG:boto:Token: None 2023-09-29T22:27:26.812 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e 2023-09-29T22:27:26.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ= 2023-09-29T22:27:26.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ=', 'Content-Length': '0'} 2023-09-29T22:27:26.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003825abf381e3d75-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:27:26.815 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.815 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.815 DEBUG:boto:Method: GET 2023-09-29T22:27:26.816 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:26.816 DEBUG:boto:Data: 2023-09-29T22:27:26.816 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.816 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.816 DEBUG:boto:Params: {} 2023-09-29T22:27:26.816 DEBUG:boto:Token: None 2023-09-29T22:27:26.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e 2023-09-29T22:27:26.817 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E= 2023-09-29T22:27:26.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E=', 'Content-Length': '0'} 2023-09-29T22:27:26.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047880adfcae4d312-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.862 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:26.863 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:26.863 DEBUG:boto:Method: GET 2023-09-29T22:27:26.863 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:26.863 DEBUG:boto:Data: 2023-09-29T22:27:26.863 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.863 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.863 DEBUG:boto:Params: {} 2023-09-29T22:27:26.863 DEBUG:boto:Token: None 2023-09-29T22:27:26.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/e 2023-09-29T22:27:26.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E= 2023-09-29T22:27:26.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E=', 'Content-Length': '0'} 2023-09-29T22:27:26.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f177f588330b76-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.867 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.867 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.868 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.868 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.868 DEBUG:boto:Method: GET 2023-09-29T22:27:26.868 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:26.868 DEBUG:boto:Data: 2023-09-29T22:27:26.868 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.868 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.868 DEBUG:boto:Params: {} 2023-09-29T22:27:26.868 DEBUG:boto:Token: None 2023-09-29T22:27:26.869 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.875 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e01403d303d1f3d-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.875 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.875 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.875 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.876 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.876 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:26.876 DEBUG:boto:Method: GET 2023-09-29T22:27:26.876 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:26.876 DEBUG:boto:Data: 2023-09-29T22:27:26.876 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.876 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.877 DEBUG:boto:Params: {} 2023-09-29T22:27:26.877 DEBUG:boto:Token: None 2023-09-29T22:27:26.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420c74cbf75f2faa-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.880 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:27:26.881 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.881 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.881 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.881 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.881 DEBUG:boto:Method: GET 2023-09-29T22:27:26.882 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.882 DEBUG:boto:Data: 2023-09-29T22:27:26.882 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.882 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.882 DEBUG:boto:Params: {} 2023-09-29T22:27:26.882 DEBUG:boto:Token: None 2023-09-29T22:27:26.882 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.882 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T22:27:26.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T22:27:26.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2360ce10c46a3a-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.885 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.886 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.886 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.886 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.886 DEBUG:boto:Method: GET 2023-09-29T22:27:26.886 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.886 DEBUG:boto:Data: 2023-09-29T22:27:26.886 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.886 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.886 DEBUG:boto:Params: {} 2023-09-29T22:27:26.887 DEBUG:boto:Token: None 2023-09-29T22:27:26.887 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.887 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T22:27:26.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T22:27:26.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad1bb6c538c8646-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.890 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.890 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.890 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.890 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.890 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.890 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.890 DEBUG:boto:Data: 2023-09-29T22:27:26.890 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.891 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.891 DEBUG:boto:Params: {} 2023-09-29T22:27:26.891 DEBUG:boto:Token: None 2023-09-29T22:27:26.891 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.891 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ= 2023-09-29T22:27:26.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ=', 'Content-Length': '0'} 2023-09-29T22:27:26.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003052224fff4d8ffc-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.893 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.893 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.893 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.893 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.894 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.894 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.894 DEBUG:boto:Data: 2023-09-29T22:27:26.894 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.894 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.894 DEBUG:boto:Params: {} 2023-09-29T22:27:26.894 DEBUG:boto:Token: None 2023-09-29T22:27:26.894 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ= 2023-09-29T22:27:26.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ=', 'Content-Length': '0'} 2023-09-29T22:27:26.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089cd842e800e59d5-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:27:26.897 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.897 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.898 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.898 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.898 DEBUG:boto:Method: GET 2023-09-29T22:27:26.898 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.898 DEBUG:boto:Data: 2023-09-29T22:27:26.898 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.898 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.898 DEBUG:boto:Params: {} 2023-09-29T22:27:26.899 DEBUG:boto:Token: None 2023-09-29T22:27:26.899 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.899 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T22:27:26.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T22:27:26.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594a49ca533d5638-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.902 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.902 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.902 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.902 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:26.902 DEBUG:boto:Method: GET 2023-09-29T22:27:26.902 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.902 DEBUG:boto:Data: 2023-09-29T22:27:26.903 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.903 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.903 DEBUG:boto:Params: {} 2023-09-29T22:27:26.903 DEBUG:boto:Token: None 2023-09-29T22:27:26.903 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:26.903 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T22:27:26.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T22:27:26.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab65709473f1456c-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.906 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.906 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.906 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.906 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.906 DEBUG:boto:Method: GET 2023-09-29T22:27:26.907 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:26.907 DEBUG:boto:Data: 2023-09-29T22:27:26.907 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.907 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.907 DEBUG:boto:Params: {} 2023-09-29T22:27:26.907 DEBUG:boto:Token: None 2023-09-29T22:27:26.907 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.912 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0ef0328d3b9210-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.913 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.913 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.913 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.913 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.913 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.914 DEBUG:boto:Method: GET 2023-09-29T22:27:26.914 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:26.914 DEBUG:boto:Data: 2023-09-29T22:27:26.914 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.914 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.914 DEBUG:boto:Params: {} 2023-09-29T22:27:26.914 DEBUG:boto:Token: None 2023-09-29T22:27:26.915 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.915 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.918 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eadcce9ce195ac2-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.918 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.918 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.919 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.919 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.919 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:26.919 DEBUG:boto:Data: 2023-09-29T22:27:26.919 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.919 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.919 DEBUG:boto:Params: {} 2023-09-29T22:27:26.920 DEBUG:boto:Token: None 2023-09-29T22:27:26.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f 2023-09-29T22:27:26.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY= 2023-09-29T22:27:26.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY=', 'Content-Length': '0'} 2023-09-29T22:27:26.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605a1d19f6f220f1-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.923 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.923 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.923 DEBUG:boto:Method: HEAD 2023-09-29T22:27:26.923 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:26.923 DEBUG:boto:Data: 2023-09-29T22:27:26.923 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.923 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.924 DEBUG:boto:Params: {} 2023-09-29T22:27:26.924 DEBUG:boto:Token: None 2023-09-29T22:27:26.924 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f 2023-09-29T22:27:26.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY= 2023-09-29T22:27:26.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY=', 'Content-Length': '0'} 2023-09-29T22:27:26.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cbbc72845336575-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:27:26.927 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.927 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.927 DEBUG:boto:Method: GET 2023-09-29T22:27:26.927 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:26.927 DEBUG:boto:Data: 2023-09-29T22:27:26.927 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.928 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.928 DEBUG:boto:Params: {} 2023-09-29T22:27:26.928 DEBUG:boto:Token: None 2023-09-29T22:27:26.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f 2023-09-29T22:27:26.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU= 2023-09-29T22:27:26.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU=', 'Content-Length': '0'} 2023-09-29T22:27:26.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67d6c250e9b8dc8-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.978 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:26.979 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:26.979 DEBUG:boto:Method: GET 2023-09-29T22:27:26.979 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:26.980 DEBUG:boto:Data: 2023-09-29T22:27:26.980 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.980 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.980 DEBUG:boto:Params: {} 2023-09-29T22:27:26.980 DEBUG:boto:Token: None 2023-09-29T22:27:26.981 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/f 2023-09-29T22:27:26.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU= 2023-09-29T22:27:26.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU=', 'Content-Length': '0'} 2023-09-29T22:27:26.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df76d1ed65c8acdb-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.985 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.985 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.985 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.985 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.985 DEBUG:boto:Method: GET 2023-09-29T22:27:26.986 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:26.986 DEBUG:boto:Data: 2023-09-29T22:27:26.986 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:26.986 DEBUG:boto:Port: 8001 2023-09-29T22:27:26.986 DEBUG:boto:Params: {} 2023-09-29T22:27:26.987 DEBUG:boto:Token: None 2023-09-29T22:27:26.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.987 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.992 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cccb1e00a3145032-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:26.992 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:26.993 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:26.993 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:26.993 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.993 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:26.993 DEBUG:boto:Method: GET 2023-09-29T22:27:26.994 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:26.994 DEBUG:boto:Data: 2023-09-29T22:27:26.994 DEBUG:boto:Headers: {} 2023-09-29T22:27:26.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:26.994 DEBUG:boto:Port: 8000 2023-09-29T22:27:26.994 DEBUG:boto:Params: {} 2023-09-29T22:27:26.995 DEBUG:boto:Token: None 2023-09-29T22:27:26.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:26 GMT /hcnphz-55/ 2023-09-29T22:27:26.995 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T22:27:26.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T22:27:26.998 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a364616318dc8467-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.042 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:27:27.043 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.043 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.043 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.043 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.043 DEBUG:boto:Method: GET 2023-09-29T22:27:27.044 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.044 DEBUG:boto:Data: 2023-09-29T22:27:27.044 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.044 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.044 DEBUG:boto:Params: {} 2023-09-29T22:27:27.044 DEBUG:boto:Token: None 2023-09-29T22:27:27.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.045 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T22:27:27.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T22:27:27.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ebc229ba835f57-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.048 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.048 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.049 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.049 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.049 DEBUG:boto:Method: GET 2023-09-29T22:27:27.049 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.049 DEBUG:boto:Data: 2023-09-29T22:27:27.049 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.050 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.050 DEBUG:boto:Params: {} 2023-09-29T22:27:27.050 DEBUG:boto:Token: None 2023-09-29T22:27:27.050 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.051 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T22:27:27.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T22:27:27.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8b45b530f1da18-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.054 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.054 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.055 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.055 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.055 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.055 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.055 DEBUG:boto:Data: 2023-09-29T22:27:27.055 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.056 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.056 DEBUG:boto:Params: {} 2023-09-29T22:27:27.056 DEBUG:boto:Token: None 2023-09-29T22:27:27.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y= 2023-09-29T22:27:27.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y=', 'Content-Length': '0'} 2023-09-29T22:27:27.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389f9457070a2386-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.061 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.061 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.061 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.062 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.062 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.062 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.062 DEBUG:boto:Data: 2023-09-29T22:27:27.062 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.063 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.063 DEBUG:boto:Params: {} 2023-09-29T22:27:27.063 DEBUG:boto:Token: None 2023-09-29T22:27:27.063 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y= 2023-09-29T22:27:27.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y=', 'Content-Length': '0'} 2023-09-29T22:27:27.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b3df3bc53b49d7-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:27:27.067 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.067 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.068 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.068 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.068 DEBUG:boto:Method: GET 2023-09-29T22:27:27.068 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.068 DEBUG:boto:Data: 2023-09-29T22:27:27.068 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.069 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.069 DEBUG:boto:Params: {} 2023-09-29T22:27:27.069 DEBUG:boto:Token: None 2023-09-29T22:27:27.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T22:27:27.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T22:27:27.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072eeb96a61e4d66b-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.114 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.115 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.115 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.115 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:27.115 DEBUG:boto:Method: GET 2023-09-29T22:27:27.115 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.115 DEBUG:boto:Data: 2023-09-29T22:27:27.115 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.115 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.116 DEBUG:boto:Params: {} 2023-09-29T22:27:27.116 DEBUG:boto:Token: None 2023-09-29T22:27:27.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:27.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T22:27:27.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T22:27:27.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000120494d18d7a8d30-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.119 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.119 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.119 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.119 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.119 DEBUG:boto:Method: GET 2023-09-29T22:27:27.119 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:27.119 DEBUG:boto:Data: 2023-09-29T22:27:27.120 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.120 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.120 DEBUG:boto:Params: {} 2023-09-29T22:27:27.120 DEBUG:boto:Token: None 2023-09-29T22:27:27.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd40d9ff6459504-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.124 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.124 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.124 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.124 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.124 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.124 DEBUG:boto:Method: GET 2023-09-29T22:27:27.125 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:27.125 DEBUG:boto:Data: 2023-09-29T22:27:27.125 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.125 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.125 DEBUG:boto:Params: {} 2023-09-29T22:27:27.125 DEBUG:boto:Token: None 2023-09-29T22:27:27.125 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd885df891846494-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.128 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.128 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.129 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.129 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.129 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:27.129 DEBUG:boto:Data: 2023-09-29T22:27:27.129 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.129 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.129 DEBUG:boto:Params: {} 2023-09-29T22:27:27.129 DEBUG:boto:Token: None 2023-09-29T22:27:27.130 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g 2023-09-29T22:27:27.130 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU= 2023-09-29T22:27:27.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU=', 'Content-Length': '0'} 2023-09-29T22:27:27.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829e015d9c618f0c-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.132 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.132 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.132 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.132 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:27.133 DEBUG:boto:Data: 2023-09-29T22:27:27.133 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.133 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.133 DEBUG:boto:Params: {} 2023-09-29T22:27:27.133 DEBUG:boto:Token: None 2023-09-29T22:27:27.133 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g 2023-09-29T22:27:27.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU= 2023-09-29T22:27:27.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU=', 'Content-Length': '0'} 2023-09-29T22:27:27.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801eb992c1d9eace-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:27:27.136 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.136 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.136 DEBUG:boto:Method: GET 2023-09-29T22:27:27.136 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:27.136 DEBUG:boto:Data: 2023-09-29T22:27:27.136 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.136 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.136 DEBUG:boto:Params: {} 2023-09-29T22:27:27.137 DEBUG:boto:Token: None 2023-09-29T22:27:27.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g 2023-09-29T22:27:27.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08= 2023-09-29T22:27:27.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08=', 'Content-Length': '0'} 2023-09-29T22:27:27.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daba551c938daea-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.139 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:27.140 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:27.140 DEBUG:boto:Method: GET 2023-09-29T22:27:27.140 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:27.140 DEBUG:boto:Data: 2023-09-29T22:27:27.140 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.140 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.140 DEBUG:boto:Params: {} 2023-09-29T22:27:27.140 DEBUG:boto:Token: None 2023-09-29T22:27:27.141 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/g 2023-09-29T22:27:27.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08= 2023-09-29T22:27:27.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08=', 'Content-Length': '0'} 2023-09-29T22:27:27.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c047c6a3cf686b7a-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.190 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.191 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.191 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.191 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.191 DEBUG:boto:Method: GET 2023-09-29T22:27:27.192 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:27.192 DEBUG:boto:Data: 2023-09-29T22:27:27.192 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.192 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.192 DEBUG:boto:Params: {} 2023-09-29T22:27:27.193 DEBUG:boto:Token: None 2023-09-29T22:27:27.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc8d416ec95c3db-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.200 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.200 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.200 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.200 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.200 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:27.201 DEBUG:boto:Method: GET 2023-09-29T22:27:27.201 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:27.201 DEBUG:boto:Data: 2023-09-29T22:27:27.201 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.201 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.201 DEBUG:boto:Params: {} 2023-09-29T22:27:27.202 DEBUG:boto:Token: None 2023-09-29T22:27:27.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.202 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5795ae68e9b0176-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.205 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:27:27.206 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.206 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.206 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.206 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.207 DEBUG:boto:Method: GET 2023-09-29T22:27:27.207 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.207 DEBUG:boto:Data: 2023-09-29T22:27:27.207 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.207 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.207 DEBUG:boto:Params: {} 2023-09-29T22:27:27.207 DEBUG:boto:Token: None 2023-09-29T22:27:27.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T22:27:27.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T22:27:27.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a4e43ab0774ee9-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.254 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.255 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.255 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.255 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.255 DEBUG:boto:Method: GET 2023-09-29T22:27:27.255 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.255 DEBUG:boto:Data: 2023-09-29T22:27:27.255 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.255 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.256 DEBUG:boto:Params: {} 2023-09-29T22:27:27.256 DEBUG:boto:Token: None 2023-09-29T22:27:27.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T22:27:27.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T22:27:27.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b62a86d66ec79f-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.259 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.260 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.260 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.260 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.260 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.260 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.260 DEBUG:boto:Data: 2023-09-29T22:27:27.260 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.261 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.261 DEBUG:boto:Params: {} 2023-09-29T22:27:27.261 DEBUG:boto:Token: None 2023-09-29T22:27:27.261 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI= 2023-09-29T22:27:27.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI=', 'Content-Length': '0'} 2023-09-29T22:27:27.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086df69ba4fb113a6-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.264 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.264 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.264 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.264 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.264 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.265 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.265 DEBUG:boto:Data: 2023-09-29T22:27:27.265 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.265 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.265 DEBUG:boto:Params: {} 2023-09-29T22:27:27.265 DEBUG:boto:Token: None 2023-09-29T22:27:27.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI= 2023-09-29T22:27:27.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI=', 'Content-Length': '0'} 2023-09-29T22:27:27.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2919d9261bcec53-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:27:27.269 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.269 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.269 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.269 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.269 DEBUG:boto:Method: GET 2023-09-29T22:27:27.269 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.269 DEBUG:boto:Data: 2023-09-29T22:27:27.271 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.271 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.271 DEBUG:boto:Params: {} 2023-09-29T22:27:27.271 DEBUG:boto:Token: None 2023-09-29T22:27:27.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T22:27:27.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T22:27:27.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b27edd8686bcb4-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.275 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.275 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.275 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.275 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:27.276 DEBUG:boto:Method: GET 2023-09-29T22:27:27.276 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.276 DEBUG:boto:Data: 2023-09-29T22:27:27.276 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.276 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.276 DEBUG:boto:Params: {} 2023-09-29T22:27:27.276 DEBUG:boto:Token: None 2023-09-29T22:27:27.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:27.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T22:27:27.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T22:27:27.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005163f5402d65fe94-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.322 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.323 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.323 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.323 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.323 DEBUG:boto:Method: GET 2023-09-29T22:27:27.323 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:27.323 DEBUG:boto:Data: 2023-09-29T22:27:27.323 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.323 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.323 DEBUG:boto:Params: {} 2023-09-29T22:27:27.324 DEBUG:boto:Token: None 2023-09-29T22:27:27.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.328 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f019894178a1f9d9-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.329 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.330 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.330 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.330 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.330 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.330 DEBUG:boto:Method: GET 2023-09-29T22:27:27.331 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:27.331 DEBUG:boto:Data: 2023-09-29T22:27:27.331 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.331 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.331 DEBUG:boto:Params: {} 2023-09-29T22:27:27.331 DEBUG:boto:Token: None 2023-09-29T22:27:27.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.336 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9ee89020f1edc6-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.336 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.336 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.337 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.337 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.337 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:27.337 DEBUG:boto:Data: 2023-09-29T22:27:27.337 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.338 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.338 DEBUG:boto:Params: {} 2023-09-29T22:27:27.338 DEBUG:boto:Token: None 2023-09-29T22:27:27.338 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h 2023-09-29T22:27:27.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E= 2023-09-29T22:27:27.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E=', 'Content-Length': '0'} 2023-09-29T22:27:27.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d54e28a4a7f62e-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.341 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.341 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.342 DEBUG:boto:Method: HEAD 2023-09-29T22:27:27.342 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:27.342 DEBUG:boto:Data: 2023-09-29T22:27:27.342 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.342 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.343 DEBUG:boto:Params: {} 2023-09-29T22:27:27.343 DEBUG:boto:Token: None 2023-09-29T22:27:27.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h 2023-09-29T22:27:27.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E= 2023-09-29T22:27:27.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E=', 'Content-Length': '0'} 2023-09-29T22:27:27.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e200e5f1ca06d40-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:27:27.347 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.347 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.348 DEBUG:boto:Method: GET 2023-09-29T22:27:27.348 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:27.348 DEBUG:boto:Data: 2023-09-29T22:27:27.348 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.348 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.349 DEBUG:boto:Params: {} 2023-09-29T22:27:27.349 DEBUG:boto:Token: None 2023-09-29T22:27:27.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h 2023-09-29T22:27:27.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc= 2023-09-29T22:27:27.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc=', 'Content-Length': '0'} 2023-09-29T22:27:27.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e32019c516a4a4-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.401 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:27.401 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:27.401 DEBUG:boto:Method: GET 2023-09-29T22:27:27.401 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:27.401 DEBUG:boto:Data: 2023-09-29T22:27:27.401 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.402 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.402 DEBUG:boto:Params: {} 2023-09-29T22:27:27.402 DEBUG:boto:Token: None 2023-09-29T22:27:27.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/h 2023-09-29T22:27:27.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc= 2023-09-29T22:27:27.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc=', 'Content-Length': '0'} 2023-09-29T22:27:27.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe439fa79a7b602-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.405 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.406 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.406 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.406 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.406 DEBUG:boto:Method: GET 2023-09-29T22:27:27.406 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:27.406 DEBUG:boto:Data: 2023-09-29T22:27:27.406 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:27.407 DEBUG:boto:Port: 8001 2023-09-29T22:27:27.407 DEBUG:boto:Params: {} 2023-09-29T22:27:27.407 DEBUG:boto:Token: None 2023-09-29T22:27:27.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.407 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e155b123fc6e59ad-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.412 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.412 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:27.412 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:27.413 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.413 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:27.413 DEBUG:boto:Method: GET 2023-09-29T22:27:27.413 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:27.413 DEBUG:boto:Data: 2023-09-29T22:27:27.413 DEBUG:boto:Headers: {} 2023-09-29T22:27:27.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:27.413 DEBUG:boto:Port: 8000 2023-09-29T22:27:27.413 DEBUG:boto:Params: {} 2023-09-29T22:27:27.414 DEBUG:boto:Token: None 2023-09-29T22:27:27.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:27 GMT /hcnphz-55/ 2023-09-29T22:27:27.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T22:27:27.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T22:27:27.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c527c45bd6fcefaa-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:27.417 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:27.418 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-55 zones={a2, a1} 2023-09-29T22:27:27.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:27.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:27.418 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:27:27.463 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:27.463 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:27.535 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:27:27.535 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-55 2023-09-29T22:27:27.538 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 8 2023-09-29T22:27:27.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:27.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:27.591 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:27:27.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:27.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:27.719 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:27:27.719 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.719 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-09-29T22:27:27.719 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-09-29T22:27:27.719 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:27.719 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:27.720 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 0, 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 11, 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.721 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:27.722 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:27.723 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:27.724 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:27:27.725 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-55 2023-09-29T22:27:27.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:27.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:27.726 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:27:27.774 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:27.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:27.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:27.835+0000 7ff139370f40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:27:32.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:32.839+0000 7ff139370f40 1 bucket sync caught up with empty source 2023-09-29T22:27:32.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:32.839+0000 7ff139370f40 0 bucket checkpoint complete 2023-09-29T22:27:32.872 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-55 2023-09-29T22:27:32.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:32.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:32.873 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:27:32.923 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:32.923 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:32.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:32.987+0000 7efe74591f40 1 bucket sync caught up with source: 2023-09-29T22:27:32.988 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:32.988 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T22:27:32.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:32.987+0000 7efe74591f40 0 bucket checkpoint complete 2023-09-29T22:27:33.015 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-55 zones={a2, a1} 2023-09-29T22:27:33.015 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.015 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.015 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.015 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:27:33.015 DEBUG:boto:Data: 2023-09-29T22:27:33.015 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.016 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.016 DEBUG:boto:Params: {} 2023-09-29T22:27:33.016 DEBUG:boto:Token: None 2023-09-29T22:27:33.016 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI= 2023-09-29T22:27:33.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI=', 'Content-Length': '0'} 2023-09-29T22:27:33.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f3a0f631caa7a12-0065174f55-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.022 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.022 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.022 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.023 DEBUG:boto:Path: /hcnphz-55/ 2023-09-29T22:27:33.023 DEBUG:boto:Data: 2023-09-29T22:27:33.023 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.023 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.023 DEBUG:boto:Params: {} 2023-09-29T22:27:33.023 DEBUG:boto:Token: None 2023-09-29T22:27:33.023 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI= 2023-09-29T22:27:33.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI=', 'Content-Length': '0'} 2023-09-29T22:27:33.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001a359af101cbcea-0065174f55-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.029 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:27:33.029 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.029 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.029 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.029 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.030 DEBUG:boto:Method: GET 2023-09-29T22:27:33.030 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.030 DEBUG:boto:Data: 2023-09-29T22:27:33.030 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.030 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.030 DEBUG:boto:Params: {} 2023-09-29T22:27:33.031 DEBUG:boto:Token: None 2023-09-29T22:27:33.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/?versions 2023-09-29T22:27:33.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T22:27:33.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T22:27:33.034 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4d02f120c84444-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.035 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.035 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:27:33.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:27:33.036 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.036 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.037 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.037 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.037 DEBUG:boto:Method: GET 2023-09-29T22:27:33.037 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.037 DEBUG:boto:Data: 2023-09-29T22:27:33.037 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.037 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.037 DEBUG:boto:Params: {} 2023-09-29T22:27:33.037 DEBUG:boto:Token: None 2023-09-29T22:27:33.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/?versions 2023-09-29T22:27:33.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T22:27:33.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T22:27:33.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491dd0bce4705f3a-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.041 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.042 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:27:33.042 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:27:33.042 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:27:33.042 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:27:33.043 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:27:33.043 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:27:33.043 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:27:33.043 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:27:33.043 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.043 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.043 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.043 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.043 DEBUG:boto:Method: GET 2023-09-29T22:27:33.044 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.044 DEBUG:boto:Data: 2023-09-29T22:27:33.044 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.044 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.044 DEBUG:boto:Params: {} 2023-09-29T22:27:33.044 DEBUG:boto:Token: None 2023-09-29T22:27:33.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/?versions 2023-09-29T22:27:33.044 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T22:27:33.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T22:27:33.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000795d564a5824d5c4-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.049 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.049 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.050 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.050 DEBUG:boto:path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.050 DEBUG:boto:auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.050 DEBUG:boto:Method: GET 2023-09-29T22:27:33.050 DEBUG:boto:Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T22:27:33.050 DEBUG:boto:Data: 2023-09-29T22:27:33.050 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.051 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.051 DEBUG:boto:Params: {} 2023-09-29T22:27:33.051 DEBUG:boto:Token: None 2023-09-29T22:27:33.051 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/?versions 2023-09-29T22:27:33.051 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T22:27:33.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T22:27:33.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd49bf121420788-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.054 DEBUG:boto:b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:27:33.055 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.055 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.055 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.055 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.056 DEBUG:boto:Method: GET 2023-09-29T22:27:33.056 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.056 DEBUG:boto:Data: 2023-09-29T22:27:33.056 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.056 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.056 DEBUG:boto:Params: {} 2023-09-29T22:27:33.056 DEBUG:boto:Token: None 2023-09-29T22:27:33.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T22:27:33.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T22:27:33.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c58cf9523e5e071-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.102 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.103 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.103 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.103 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.103 DEBUG:boto:Method: GET 2023-09-29T22:27:33.103 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.103 DEBUG:boto:Data: 2023-09-29T22:27:33.103 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.103 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.103 DEBUG:boto:Params: {} 2023-09-29T22:27:33.104 DEBUG:boto:Token: None 2023-09-29T22:27:33.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T22:27:33.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T22:27:33.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a444942493bfa61-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.107 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.107 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.107 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.107 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.108 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.108 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.108 DEBUG:boto:Data: 2023-09-29T22:27:33.108 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.108 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.108 DEBUG:boto:Params: {} 2023-09-29T22:27:33.108 DEBUG:boto:Token: None 2023-09-29T22:27:33.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c= 2023-09-29T22:27:33.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c=', 'Content-Length': '0'} 2023-09-29T22:27:33.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1402cca137b1364-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.112 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.112 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.112 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.112 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.112 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.112 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.112 DEBUG:boto:Data: 2023-09-29T22:27:33.112 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.113 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.113 DEBUG:boto:Params: {} 2023-09-29T22:27:33.113 DEBUG:boto:Token: None 2023-09-29T22:27:33.113 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c= 2023-09-29T22:27:33.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c=', 'Content-Length': '0'} 2023-09-29T22:27:33.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dee43a1937de05e-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:27:33.116 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.116 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.116 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.116 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.117 DEBUG:boto:Method: GET 2023-09-29T22:27:33.117 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.117 DEBUG:boto:Data: 2023-09-29T22:27:33.117 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.117 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.117 DEBUG:boto:Params: {} 2023-09-29T22:27:33.117 DEBUG:boto:Token: None 2023-09-29T22:27:33.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.118 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T22:27:33.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T22:27:33.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa07df144d96b96-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.120 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.121 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.121 DEBUG:boto:path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.121 DEBUG:boto:auth_path=/hcnphz-55/a?versionId=null 2023-09-29T22:27:33.121 DEBUG:boto:Method: GET 2023-09-29T22:27:33.121 DEBUG:boto:Path: /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.121 DEBUG:boto:Data: 2023-09-29T22:27:33.121 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.121 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.121 DEBUG:boto:Params: {} 2023-09-29T22:27:33.122 DEBUG:boto:Token: None 2023-09-29T22:27:33.122 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a?versionId=null 2023-09-29T22:27:33.122 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T22:27:33.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T22:27:33.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec5c240879168889-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.125 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.125 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.125 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.125 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.125 DEBUG:boto:Method: GET 2023-09-29T22:27:33.125 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:33.126 DEBUG:boto:Data: 2023-09-29T22:27:33.126 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.126 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.126 DEBUG:boto:Params: {} 2023-09-29T22:27:33.126 DEBUG:boto:Token: None 2023-09-29T22:27:33.126 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54a625277d36a7e-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.132 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.132 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.132 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.132 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.132 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.133 DEBUG:boto:Method: GET 2023-09-29T22:27:33.133 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:33.133 DEBUG:boto:Data: 2023-09-29T22:27:33.133 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.133 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.133 DEBUG:boto:Params: {} 2023-09-29T22:27:33.133 DEBUG:boto:Token: None 2023-09-29T22:27:33.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.134 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.134 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.137 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6cd91a9492b45c-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.137 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.137 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.137 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.137 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.138 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:33.138 DEBUG:boto:Data: 2023-09-29T22:27:33.138 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.138 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.138 DEBUG:boto:Params: {} 2023-09-29T22:27:33.138 DEBUG:boto:Token: None 2023-09-29T22:27:33.138 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a 2023-09-29T22:27:33.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0= 2023-09-29T22:27:33.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0=', 'Content-Length': '0'} 2023-09-29T22:27:33.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005683bfaeed2d54ec-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.141 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.141 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.141 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.141 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:33.142 DEBUG:boto:Data: 2023-09-29T22:27:33.142 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.142 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.142 DEBUG:boto:Params: {} 2023-09-29T22:27:33.142 DEBUG:boto:Token: None 2023-09-29T22:27:33.142 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a 2023-09-29T22:27:33.142 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0= 2023-09-29T22:27:33.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0=', 'Content-Length': '0'} 2023-09-29T22:27:33.145 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee593d161011b0fa-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:27:33.145 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.145 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.145 DEBUG:boto:Method: GET 2023-09-29T22:27:33.146 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:33.146 DEBUG:boto:Data: 2023-09-29T22:27:33.146 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.146 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.146 DEBUG:boto:Params: {} 2023-09-29T22:27:33.146 DEBUG:boto:Token: None 2023-09-29T22:27:33.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a 2023-09-29T22:27:33.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw= 2023-09-29T22:27:33.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw=', 'Content-Length': '0'} 2023-09-29T22:27:33.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca32b5a3e59ecd4c-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.190 DEBUG:boto:path=/hcnphz-55/a 2023-09-29T22:27:33.191 DEBUG:boto:auth_path=/hcnphz-55/a 2023-09-29T22:27:33.191 DEBUG:boto:Method: GET 2023-09-29T22:27:33.191 DEBUG:boto:Path: /hcnphz-55/a 2023-09-29T22:27:33.191 DEBUG:boto:Data: 2023-09-29T22:27:33.191 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.191 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.191 DEBUG:boto:Params: {} 2023-09-29T22:27:33.192 DEBUG:boto:Token: None 2023-09-29T22:27:33.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/a 2023-09-29T22:27:33.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw= 2023-09-29T22:27:33.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw=', 'Content-Length': '0'} 2023-09-29T22:27:33.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072538d194d5c9362-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.195 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.195 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.195 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.195 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.195 DEBUG:boto:Method: GET 2023-09-29T22:27:33.195 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:33.195 DEBUG:boto:Data: 2023-09-29T22:27:33.195 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.195 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.196 DEBUG:boto:Params: {} 2023-09-29T22:27:33.196 DEBUG:boto:Token: None 2023-09-29T22:27:33.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8d9106c825a0a8-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.200 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.201 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.201 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.201 DEBUG:boto:path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.201 DEBUG:boto:auth_path=/hcnphz-55/?prefix=a 2023-09-29T22:27:33.202 DEBUG:boto:Method: GET 2023-09-29T22:27:33.202 DEBUG:boto:Path: /hcnphz-55/?prefix=a 2023-09-29T22:27:33.202 DEBUG:boto:Data: 2023-09-29T22:27:33.202 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.202 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.202 DEBUG:boto:Params: {} 2023-09-29T22:27:33.202 DEBUG:boto:Token: None 2023-09-29T22:27:33.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.203 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292459b27e68f298-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.246 DEBUG:boto:b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:27:33.247 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.247 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.247 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.247 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.247 DEBUG:boto:Method: GET 2023-09-29T22:27:33.248 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.248 DEBUG:boto:Data: 2023-09-29T22:27:33.248 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.248 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.248 DEBUG:boto:Params: {} 2023-09-29T22:27:33.248 DEBUG:boto:Token: None 2023-09-29T22:27:33.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T22:27:33.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T22:27:33.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004005a556e53e56c4-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.252 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.252 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.252 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.252 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.252 DEBUG:boto:Method: GET 2023-09-29T22:27:33.252 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.252 DEBUG:boto:Data: 2023-09-29T22:27:33.253 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.253 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.253 DEBUG:boto:Params: {} 2023-09-29T22:27:33.253 DEBUG:boto:Token: None 2023-09-29T22:27:33.253 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.253 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T22:27:33.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T22:27:33.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020790208cf05948f-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.256 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.256 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.257 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.257 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.257 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.257 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.257 DEBUG:boto:Data: 2023-09-29T22:27:33.257 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.257 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.257 DEBUG:boto:Params: {} 2023-09-29T22:27:33.258 DEBUG:boto:Token: None 2023-09-29T22:27:33.258 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.258 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk= 2023-09-29T22:27:33.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk=', 'Content-Length': '0'} 2023-09-29T22:27:33.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d26e03a51fb1ff6-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.261 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.261 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.261 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.261 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.261 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.261 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.261 DEBUG:boto:Data: 2023-09-29T22:27:33.261 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.262 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.262 DEBUG:boto:Params: {} 2023-09-29T22:27:33.262 DEBUG:boto:Token: None 2023-09-29T22:27:33.262 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk= 2023-09-29T22:27:33.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk=', 'Content-Length': '0'} 2023-09-29T22:27:33.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8729410931f48a-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.265 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:27:33.265 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.265 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.265 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.265 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.266 DEBUG:boto:Method: GET 2023-09-29T22:27:33.266 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.266 DEBUG:boto:Data: 2023-09-29T22:27:33.266 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.266 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.266 DEBUG:boto:Params: {} 2023-09-29T22:27:33.266 DEBUG:boto:Token: None 2023-09-29T22:27:33.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.267 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T22:27:33.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T22:27:33.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d73058abf39afc-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.270 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.270 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.270 DEBUG:boto:path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.270 DEBUG:boto:auth_path=/hcnphz-55/b?versionId=null 2023-09-29T22:27:33.270 DEBUG:boto:Method: GET 2023-09-29T22:27:33.270 DEBUG:boto:Path: /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.270 DEBUG:boto:Data: 2023-09-29T22:27:33.270 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.271 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.271 DEBUG:boto:Params: {} 2023-09-29T22:27:33.271 DEBUG:boto:Token: None 2023-09-29T22:27:33.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b?versionId=null 2023-09-29T22:27:33.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T22:27:33.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T22:27:33.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ea020abeb24c76-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.274 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.274 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.274 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.275 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.275 DEBUG:boto:Method: GET 2023-09-29T22:27:33.275 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:33.275 DEBUG:boto:Data: 2023-09-29T22:27:33.275 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.275 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.275 DEBUG:boto:Params: {} 2023-09-29T22:27:33.275 DEBUG:boto:Token: None 2023-09-29T22:27:33.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c8d45918b94599-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.280 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.280 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.280 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.281 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.281 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.281 DEBUG:boto:Method: GET 2023-09-29T22:27:33.281 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:33.281 DEBUG:boto:Data: 2023-09-29T22:27:33.281 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.281 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.281 DEBUG:boto:Params: {} 2023-09-29T22:27:33.281 DEBUG:boto:Token: None 2023-09-29T22:27:33.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d12f3e87a4aad0b9-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.285 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.285 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.286 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.286 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.286 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:33.286 DEBUG:boto:Data: 2023-09-29T22:27:33.286 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.286 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.286 DEBUG:boto:Params: {} 2023-09-29T22:27:33.286 DEBUG:boto:Token: None 2023-09-29T22:27:33.287 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b 2023-09-29T22:27:33.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo= 2023-09-29T22:27:33.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo=', 'Content-Length': '0'} 2023-09-29T22:27:33.288 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfdb4630aa502f26-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.289 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.289 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.289 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.289 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:33.289 DEBUG:boto:Data: 2023-09-29T22:27:33.289 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.290 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.290 DEBUG:boto:Params: {} 2023-09-29T22:27:33.290 DEBUG:boto:Token: None 2023-09-29T22:27:33.290 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b 2023-09-29T22:27:33.290 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo= 2023-09-29T22:27:33.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo=', 'Content-Length': '0'} 2023-09-29T22:27:33.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e22e0d84f48f53-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:27:33.292 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.293 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.293 DEBUG:boto:Method: GET 2023-09-29T22:27:33.293 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:33.293 DEBUG:boto:Data: 2023-09-29T22:27:33.293 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.293 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.293 DEBUG:boto:Params: {} 2023-09-29T22:27:33.293 DEBUG:boto:Token: None 2023-09-29T22:27:33.294 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b 2023-09-29T22:27:33.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw= 2023-09-29T22:27:33.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw=', 'Content-Length': '0'} 2023-09-29T22:27:33.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69ed9bf335b0a6e-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.338 DEBUG:boto:path=/hcnphz-55/b 2023-09-29T22:27:33.339 DEBUG:boto:auth_path=/hcnphz-55/b 2023-09-29T22:27:33.339 DEBUG:boto:Method: GET 2023-09-29T22:27:33.339 DEBUG:boto:Path: /hcnphz-55/b 2023-09-29T22:27:33.339 DEBUG:boto:Data: 2023-09-29T22:27:33.339 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.339 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.339 DEBUG:boto:Params: {} 2023-09-29T22:27:33.340 DEBUG:boto:Token: None 2023-09-29T22:27:33.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/b 2023-09-29T22:27:33.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw= 2023-09-29T22:27:33.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw=', 'Content-Length': '0'} 2023-09-29T22:27:33.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5c629759295b37-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.343 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.343 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.344 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.344 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.344 DEBUG:boto:Method: GET 2023-09-29T22:27:33.344 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:33.344 DEBUG:boto:Data: 2023-09-29T22:27:33.344 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.344 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.344 DEBUG:boto:Params: {} 2023-09-29T22:27:33.344 DEBUG:boto:Token: None 2023-09-29T22:27:33.345 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.349 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f84529e72985c8-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.349 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.350 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.350 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.350 DEBUG:boto:path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.350 DEBUG:boto:auth_path=/hcnphz-55/?prefix=b 2023-09-29T22:27:33.350 DEBUG:boto:Method: GET 2023-09-29T22:27:33.350 DEBUG:boto:Path: /hcnphz-55/?prefix=b 2023-09-29T22:27:33.350 DEBUG:boto:Data: 2023-09-29T22:27:33.350 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.351 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.351 DEBUG:boto:Params: {} 2023-09-29T22:27:33.351 DEBUG:boto:Token: None 2023-09-29T22:27:33.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.354 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003174b99586b5dbbd-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.398 DEBUG:boto:b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:27:33.399 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.399 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.399 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.399 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.399 DEBUG:boto:Method: GET 2023-09-29T22:27:33.400 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.400 DEBUG:boto:Data: 2023-09-29T22:27:33.400 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.400 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.400 DEBUG:boto:Params: {} 2023-09-29T22:27:33.400 DEBUG:boto:Token: None 2023-09-29T22:27:33.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T22:27:33.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T22:27:33.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267be75d8736644c-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.404 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.404 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.404 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.404 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.404 DEBUG:boto:Method: GET 2023-09-29T22:27:33.404 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.404 DEBUG:boto:Data: 2023-09-29T22:27:33.405 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.405 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.405 DEBUG:boto:Params: {} 2023-09-29T22:27:33.405 DEBUG:boto:Token: None 2023-09-29T22:27:33.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T22:27:33.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T22:27:33.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e96c5c7fecae4d6-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.409 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.409 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.409 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.409 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.409 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.409 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.409 DEBUG:boto:Data: 2023-09-29T22:27:33.409 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.410 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.410 DEBUG:boto:Params: {} 2023-09-29T22:27:33.410 DEBUG:boto:Token: None 2023-09-29T22:27:33.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I= 2023-09-29T22:27:33.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I=', 'Content-Length': '0'} 2023-09-29T22:27:33.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9d7ae0676a292f-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.413 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.413 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.413 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.414 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.414 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.414 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.414 DEBUG:boto:Data: 2023-09-29T22:27:33.414 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.414 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.414 DEBUG:boto:Params: {} 2023-09-29T22:27:33.414 DEBUG:boto:Token: None 2023-09-29T22:27:33.415 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I= 2023-09-29T22:27:33.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I=', 'Content-Length': '0'} 2023-09-29T22:27:33.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb4c65fedc62f7f-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:27:33.418 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.418 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.418 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.418 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.418 DEBUG:boto:Method: GET 2023-09-29T22:27:33.418 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.418 DEBUG:boto:Data: 2023-09-29T22:27:33.418 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.418 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.419 DEBUG:boto:Params: {} 2023-09-29T22:27:33.419 DEBUG:boto:Token: None 2023-09-29T22:27:33.419 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T22:27:33.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T22:27:33.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062246877fffd4799-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.422 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.422 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.422 DEBUG:boto:path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.422 DEBUG:boto:auth_path=/hcnphz-55/c?versionId=null 2023-09-29T22:27:33.422 DEBUG:boto:Method: GET 2023-09-29T22:27:33.423 DEBUG:boto:Path: /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.423 DEBUG:boto:Data: 2023-09-29T22:27:33.423 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.423 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.423 DEBUG:boto:Params: {} 2023-09-29T22:27:33.423 DEBUG:boto:Token: None 2023-09-29T22:27:33.423 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c?versionId=null 2023-09-29T22:27:33.423 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T22:27:33.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T22:27:33.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0b01a62f06bb02a-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.426 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.427 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.427 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.427 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.427 DEBUG:boto:Method: GET 2023-09-29T22:27:33.427 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:33.427 DEBUG:boto:Data: 2023-09-29T22:27:33.427 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.427 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.427 DEBUG:boto:Params: {} 2023-09-29T22:27:33.428 DEBUG:boto:Token: None 2023-09-29T22:27:33.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b50f0bed0adbfa-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.432 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.433 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.433 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.433 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.433 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.433 DEBUG:boto:Method: GET 2023-09-29T22:27:33.433 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:33.433 DEBUG:boto:Data: 2023-09-29T22:27:33.434 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.434 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.434 DEBUG:boto:Params: {} 2023-09-29T22:27:33.434 DEBUG:boto:Token: None 2023-09-29T22:27:33.434 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.435 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.438 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c894924404b38d-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.478 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.479 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.479 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.479 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.479 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:33.479 DEBUG:boto:Data: 2023-09-29T22:27:33.479 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.480 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.480 DEBUG:boto:Params: {} 2023-09-29T22:27:33.480 DEBUG:boto:Token: None 2023-09-29T22:27:33.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c 2023-09-29T22:27:33.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q= 2023-09-29T22:27:33.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q=', 'Content-Length': '0'} 2023-09-29T22:27:33.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d241dadd7a0c04af-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.483 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.483 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.484 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.484 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:33.484 DEBUG:boto:Data: 2023-09-29T22:27:33.484 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.484 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.484 DEBUG:boto:Params: {} 2023-09-29T22:27:33.484 DEBUG:boto:Token: None 2023-09-29T22:27:33.485 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c 2023-09-29T22:27:33.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q= 2023-09-29T22:27:33.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q=', 'Content-Length': '0'} 2023-09-29T22:27:33.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee43edd9a27e2a3-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:27:33.488 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.488 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.488 DEBUG:boto:Method: GET 2023-09-29T22:27:33.489 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:33.489 DEBUG:boto:Data: 2023-09-29T22:27:33.489 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.489 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.489 DEBUG:boto:Params: {} 2023-09-29T22:27:33.489 DEBUG:boto:Token: None 2023-09-29T22:27:33.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c 2023-09-29T22:27:33.490 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU= 2023-09-29T22:27:33.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU=', 'Content-Length': '0'} 2023-09-29T22:27:33.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27dd016d9dd6475-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.534 DEBUG:boto:path=/hcnphz-55/c 2023-09-29T22:27:33.535 DEBUG:boto:auth_path=/hcnphz-55/c 2023-09-29T22:27:33.535 DEBUG:boto:Method: GET 2023-09-29T22:27:33.535 DEBUG:boto:Path: /hcnphz-55/c 2023-09-29T22:27:33.536 DEBUG:boto:Data: 2023-09-29T22:27:33.536 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.536 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.536 DEBUG:boto:Params: {} 2023-09-29T22:27:33.537 DEBUG:boto:Token: None 2023-09-29T22:27:33.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/c 2023-09-29T22:27:33.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU= 2023-09-29T22:27:33.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU=', 'Content-Length': '0'} 2023-09-29T22:27:33.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48ef98a7b095ba4-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.541 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.541 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.541 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.541 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.541 DEBUG:boto:Method: GET 2023-09-29T22:27:33.541 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:33.542 DEBUG:boto:Data: 2023-09-29T22:27:33.542 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.542 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.542 DEBUG:boto:Params: {} 2023-09-29T22:27:33.542 DEBUG:boto:Token: None 2023-09-29T22:27:33.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09c060b4434121c-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.547 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.547 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.547 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.547 DEBUG:boto:path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.548 DEBUG:boto:auth_path=/hcnphz-55/?prefix=c 2023-09-29T22:27:33.548 DEBUG:boto:Method: GET 2023-09-29T22:27:33.548 DEBUG:boto:Path: /hcnphz-55/?prefix=c 2023-09-29T22:27:33.548 DEBUG:boto:Data: 2023-09-29T22:27:33.548 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.548 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.548 DEBUG:boto:Params: {} 2023-09-29T22:27:33.548 DEBUG:boto:Token: None 2023-09-29T22:27:33.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.552 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042397a830e4fbd64-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.552 DEBUG:boto:b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:27:33.553 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.553 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.553 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.554 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.554 DEBUG:boto:Method: GET 2023-09-29T22:27:33.554 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.554 DEBUG:boto:Data: 2023-09-29T22:27:33.554 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.555 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.555 DEBUG:boto:Params: {} 2023-09-29T22:27:33.555 DEBUG:boto:Token: None 2023-09-29T22:27:33.555 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T22:27:33.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T22:27:33.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69ee7c73a8d30be-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.559 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.559 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.559 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.560 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.560 DEBUG:boto:Method: GET 2023-09-29T22:27:33.560 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.560 DEBUG:boto:Data: 2023-09-29T22:27:33.560 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.561 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.561 DEBUG:boto:Params: {} 2023-09-29T22:27:33.561 DEBUG:boto:Token: None 2023-09-29T22:27:33.561 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T22:27:33.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T22:27:33.564 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ceea3c43af123f-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.565 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.565 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.565 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.565 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.565 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.566 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.566 DEBUG:boto:Data: 2023-09-29T22:27:33.566 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.566 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.566 DEBUG:boto:Params: {} 2023-09-29T22:27:33.566 DEBUG:boto:Token: None 2023-09-29T22:27:33.566 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g= 2023-09-29T22:27:33.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g=', 'Content-Length': '0'} 2023-09-29T22:27:33.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b860e2d160158f5f-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.570 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.570 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.570 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.570 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.570 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.570 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.570 DEBUG:boto:Data: 2023-09-29T22:27:33.570 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.571 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.571 DEBUG:boto:Params: {} 2023-09-29T22:27:33.571 DEBUG:boto:Token: None 2023-09-29T22:27:33.571 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g= 2023-09-29T22:27:33.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g=', 'Content-Length': '0'} 2023-09-29T22:27:33.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039859f20810e5f31-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:27:33.575 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.575 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.575 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.575 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.575 DEBUG:boto:Method: GET 2023-09-29T22:27:33.575 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.575 DEBUG:boto:Data: 2023-09-29T22:27:33.575 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.575 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.576 DEBUG:boto:Params: {} 2023-09-29T22:27:33.576 DEBUG:boto:Token: None 2023-09-29T22:27:33.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T22:27:33.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T22:27:33.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c1a824e50598d52-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.626 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.627 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.627 DEBUG:boto:path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.627 DEBUG:boto:auth_path=/hcnphz-55/d?versionId=null 2023-09-29T22:27:33.627 DEBUG:boto:Method: GET 2023-09-29T22:27:33.627 DEBUG:boto:Path: /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.627 DEBUG:boto:Data: 2023-09-29T22:27:33.627 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.627 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.627 DEBUG:boto:Params: {} 2023-09-29T22:27:33.628 DEBUG:boto:Token: None 2023-09-29T22:27:33.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d?versionId=null 2023-09-29T22:27:33.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T22:27:33.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T22:27:33.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000927b41e8ea5daa25-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.631 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.631 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.631 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.632 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.632 DEBUG:boto:Method: GET 2023-09-29T22:27:33.632 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:33.632 DEBUG:boto:Data: 2023-09-29T22:27:33.632 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.632 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.632 DEBUG:boto:Params: {} 2023-09-29T22:27:33.632 DEBUG:boto:Token: None 2023-09-29T22:27:33.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019336b2898b30eb4-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.637 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.638 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.638 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.638 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.638 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.638 DEBUG:boto:Method: GET 2023-09-29T22:27:33.638 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:33.638 DEBUG:boto:Data: 2023-09-29T22:27:33.638 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.639 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.639 DEBUG:boto:Params: {} 2023-09-29T22:27:33.639 DEBUG:boto:Token: None 2023-09-29T22:27:33.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba4fc564ad14073-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.690 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.691 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.691 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.691 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.691 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:33.691 DEBUG:boto:Data: 2023-09-29T22:27:33.691 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.692 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.692 DEBUG:boto:Params: {} 2023-09-29T22:27:33.692 DEBUG:boto:Token: None 2023-09-29T22:27:33.692 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d 2023-09-29T22:27:33.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs= 2023-09-29T22:27:33.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs=', 'Content-Length': '0'} 2023-09-29T22:27:33.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082bea5cdd9f6678d-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.695 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.695 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.695 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.695 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:33.696 DEBUG:boto:Data: 2023-09-29T22:27:33.696 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.696 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.696 DEBUG:boto:Params: {} 2023-09-29T22:27:33.696 DEBUG:boto:Token: None 2023-09-29T22:27:33.696 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d 2023-09-29T22:27:33.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs= 2023-09-29T22:27:33.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs=', 'Content-Length': '0'} 2023-09-29T22:27:33.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd6542d0d086061-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:27:33.700 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.700 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.700 DEBUG:boto:Method: GET 2023-09-29T22:27:33.700 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:33.700 DEBUG:boto:Data: 2023-09-29T22:27:33.701 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.701 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.701 DEBUG:boto:Params: {} 2023-09-29T22:27:33.701 DEBUG:boto:Token: None 2023-09-29T22:27:33.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d 2023-09-29T22:27:33.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8= 2023-09-29T22:27:33.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8=', 'Content-Length': '0'} 2023-09-29T22:27:33.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c722580826b23d-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.704 DEBUG:boto:path=/hcnphz-55/d 2023-09-29T22:27:33.705 DEBUG:boto:auth_path=/hcnphz-55/d 2023-09-29T22:27:33.705 DEBUG:boto:Method: GET 2023-09-29T22:27:33.705 DEBUG:boto:Path: /hcnphz-55/d 2023-09-29T22:27:33.705 DEBUG:boto:Data: 2023-09-29T22:27:33.705 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.705 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.705 DEBUG:boto:Params: {} 2023-09-29T22:27:33.705 DEBUG:boto:Token: None 2023-09-29T22:27:33.706 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/d 2023-09-29T22:27:33.706 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8= 2023-09-29T22:27:33.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8=', 'Content-Length': '0'} 2023-09-29T22:27:33.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021b8f28ae393a16b-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.709 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.709 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.709 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.709 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.709 DEBUG:boto:Method: GET 2023-09-29T22:27:33.709 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:33.709 DEBUG:boto:Data: 2023-09-29T22:27:33.709 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.710 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.710 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.710 DEBUG:boto:Params: {} 2023-09-29T22:27:33.710 DEBUG:boto:Token: None 2023-09-29T22:27:33.710 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720fb72a03472d61-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.758 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.759 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.759 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.759 DEBUG:boto:path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.759 DEBUG:boto:auth_path=/hcnphz-55/?prefix=d 2023-09-29T22:27:33.759 DEBUG:boto:Method: GET 2023-09-29T22:27:33.760 DEBUG:boto:Path: /hcnphz-55/?prefix=d 2023-09-29T22:27:33.760 DEBUG:boto:Data: 2023-09-29T22:27:33.760 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.760 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.760 DEBUG:boto:Params: {} 2023-09-29T22:27:33.760 DEBUG:boto:Token: None 2023-09-29T22:27:33.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eab18f1ab39cf88-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.764 DEBUG:boto:b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:27:33.765 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.765 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.765 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.765 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.765 DEBUG:boto:Method: GET 2023-09-29T22:27:33.765 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.765 DEBUG:boto:Data: 2023-09-29T22:27:33.765 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.765 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.766 DEBUG:boto:Params: {} 2023-09-29T22:27:33.766 DEBUG:boto:Token: None 2023-09-29T22:27:33.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.766 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T22:27:33.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T22:27:33.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac882e158ef96ce-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.769 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.769 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.769 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.769 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.770 DEBUG:boto:Method: GET 2023-09-29T22:27:33.770 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.770 DEBUG:boto:Data: 2023-09-29T22:27:33.770 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.770 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.770 DEBUG:boto:Params: {} 2023-09-29T22:27:33.770 DEBUG:boto:Token: None 2023-09-29T22:27:33.771 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.771 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T22:27:33.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T22:27:33.774 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd9210fd2af0d74-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.818 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.818 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.819 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.819 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.819 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.819 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.819 DEBUG:boto:Data: 2023-09-29T22:27:33.819 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.819 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.819 DEBUG:boto:Params: {} 2023-09-29T22:27:33.820 DEBUG:boto:Token: None 2023-09-29T22:27:33.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg= 2023-09-29T22:27:33.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg=', 'Content-Length': '0'} 2023-09-29T22:27:33.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028be9a0d4feee27-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.823 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.823 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.823 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.823 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.823 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.823 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.824 DEBUG:boto:Data: 2023-09-29T22:27:33.824 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.824 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.824 DEBUG:boto:Params: {} 2023-09-29T22:27:33.824 DEBUG:boto:Token: None 2023-09-29T22:27:33.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg= 2023-09-29T22:27:33.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg=', 'Content-Length': '0'} 2023-09-29T22:27:33.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c564e959b2979b55-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:27:33.827 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.827 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.828 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.828 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.828 DEBUG:boto:Method: GET 2023-09-29T22:27:33.828 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.828 DEBUG:boto:Data: 2023-09-29T22:27:33.828 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.828 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.828 DEBUG:boto:Params: {} 2023-09-29T22:27:33.828 DEBUG:boto:Token: None 2023-09-29T22:27:33.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T22:27:33.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T22:27:33.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705ca5d3e3a526c7-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.832 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.832 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.832 DEBUG:boto:path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.832 DEBUG:boto:auth_path=/hcnphz-55/e?versionId=null 2023-09-29T22:27:33.832 DEBUG:boto:Method: GET 2023-09-29T22:27:33.832 DEBUG:boto:Path: /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.833 DEBUG:boto:Data: 2023-09-29T22:27:33.833 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.833 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.833 DEBUG:boto:Params: {} 2023-09-29T22:27:33.833 DEBUG:boto:Token: None 2023-09-29T22:27:33.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e?versionId=null 2023-09-29T22:27:33.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T22:27:33.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T22:27:33.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae592a3b866567b9-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.837 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.837 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.837 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.837 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.837 DEBUG:boto:Method: GET 2023-09-29T22:27:33.837 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:33.837 DEBUG:boto:Data: 2023-09-29T22:27:33.838 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.838 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.838 DEBUG:boto:Params: {} 2023-09-29T22:27:33.838 DEBUG:boto:Token: None 2023-09-29T22:27:33.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.842 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041738e0d0d19007d-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.886 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.887 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.887 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.887 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.887 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.887 DEBUG:boto:Method: GET 2023-09-29T22:27:33.887 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:33.888 DEBUG:boto:Data: 2023-09-29T22:27:33.888 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.888 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.888 DEBUG:boto:Params: {} 2023-09-29T22:27:33.888 DEBUG:boto:Token: None 2023-09-29T22:27:33.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013bf3d5c91ae6824-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.892 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.892 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.892 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.892 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.893 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:33.893 DEBUG:boto:Data: 2023-09-29T22:27:33.893 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.893 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.893 DEBUG:boto:Params: {} 2023-09-29T22:27:33.893 DEBUG:boto:Token: None 2023-09-29T22:27:33.893 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e 2023-09-29T22:27:33.894 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg= 2023-09-29T22:27:33.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg=', 'Content-Length': '0'} 2023-09-29T22:27:33.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507b0893b8f54e48-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.897 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.897 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.897 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.897 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:33.897 DEBUG:boto:Data: 2023-09-29T22:27:33.897 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.897 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.898 DEBUG:boto:Params: {} 2023-09-29T22:27:33.898 DEBUG:boto:Token: None 2023-09-29T22:27:33.898 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e 2023-09-29T22:27:33.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg= 2023-09-29T22:27:33.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg=', 'Content-Length': '0'} 2023-09-29T22:27:33.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abb3156e45c14ce-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:27:33.901 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.901 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.901 DEBUG:boto:Method: GET 2023-09-29T22:27:33.901 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:33.901 DEBUG:boto:Data: 2023-09-29T22:27:33.901 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.902 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.902 DEBUG:boto:Params: {} 2023-09-29T22:27:33.902 DEBUG:boto:Token: None 2023-09-29T22:27:33.902 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e 2023-09-29T22:27:33.902 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8= 2023-09-29T22:27:33.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8=', 'Content-Length': '0'} 2023-09-29T22:27:33.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009f95ba8fbfc45bb-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.905 DEBUG:boto:path=/hcnphz-55/e 2023-09-29T22:27:33.906 DEBUG:boto:auth_path=/hcnphz-55/e 2023-09-29T22:27:33.906 DEBUG:boto:Method: GET 2023-09-29T22:27:33.906 DEBUG:boto:Path: /hcnphz-55/e 2023-09-29T22:27:33.906 DEBUG:boto:Data: 2023-09-29T22:27:33.906 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.906 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.906 DEBUG:boto:Params: {} 2023-09-29T22:27:33.907 DEBUG:boto:Token: None 2023-09-29T22:27:33.907 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/e 2023-09-29T22:27:33.907 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8= 2023-09-29T22:27:33.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8=', 'Content-Length': '0'} 2023-09-29T22:27:33.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000884de15f40dbebf-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.950 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.951 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.951 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.951 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.951 DEBUG:boto:Method: GET 2023-09-29T22:27:33.952 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:33.952 DEBUG:boto:Data: 2023-09-29T22:27:33.952 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.952 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.953 DEBUG:boto:Params: {} 2023-09-29T22:27:33.953 DEBUG:boto:Token: None 2023-09-29T22:27:33.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.958 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b0acaaf16d82baf-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.958 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.959 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:33.959 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:33.959 DEBUG:boto:path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.959 DEBUG:boto:auth_path=/hcnphz-55/?prefix=e 2023-09-29T22:27:33.960 DEBUG:boto:Method: GET 2023-09-29T22:27:33.960 DEBUG:boto:Path: /hcnphz-55/?prefix=e 2023-09-29T22:27:33.960 DEBUG:boto:Data: 2023-09-29T22:27:33.960 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.960 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.960 DEBUG:boto:Params: {} 2023-09-29T22:27:33.960 DEBUG:boto:Token: None 2023-09-29T22:27:33.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/ 2023-09-29T22:27:33.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T22:27:33.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T22:27:33.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080e8150405d9b09-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.964 DEBUG:boto:b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:33.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:27:33.965 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:33.965 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:33.965 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.965 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.965 DEBUG:boto:Method: GET 2023-09-29T22:27:33.965 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.965 DEBUG:boto:Data: 2023-09-29T22:27:33.965 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.966 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.966 DEBUG:boto:Params: {} 2023-09-29T22:27:33.966 DEBUG:boto:Token: None 2023-09-29T22:27:33.966 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.966 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T22:27:33.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T22:27:33.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080844cde59962189-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.969 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:33.969 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:33.969 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.970 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.970 DEBUG:boto:Method: GET 2023-09-29T22:27:33.970 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.970 DEBUG:boto:Data: 2023-09-29T22:27:33.970 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.970 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.970 DEBUG:boto:Params: {} 2023-09-29T22:27:33.970 DEBUG:boto:Token: None 2023-09-29T22:27:33.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.971 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T22:27:33.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T22:27:33.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e797f529b46b850-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.974 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:33.974 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:33.974 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.974 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.975 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.975 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.975 DEBUG:boto:Data: 2023-09-29T22:27:33.975 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.975 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.975 DEBUG:boto:Params: {} 2023-09-29T22:27:33.975 DEBUG:boto:Token: None 2023-09-29T22:27:33.975 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA= 2023-09-29T22:27:33.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA=', 'Content-Length': '0'} 2023-09-29T22:27:33.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478f9b2fabed7a3a-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.978 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:33.978 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:33.979 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.979 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.979 DEBUG:boto:Method: HEAD 2023-09-29T22:27:33.979 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.979 DEBUG:boto:Data: 2023-09-29T22:27:33.979 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.979 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.979 DEBUG:boto:Params: {} 2023-09-29T22:27:33.980 DEBUG:boto:Token: None 2023-09-29T22:27:33.980 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA= 2023-09-29T22:27:33.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA=', 'Content-Length': '0'} 2023-09-29T22:27:33.983 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a28a4697c06185-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:27:33.983 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:33.983 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:33.983 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.983 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.984 DEBUG:boto:Method: GET 2023-09-29T22:27:33.984 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.984 DEBUG:boto:Data: 2023-09-29T22:27:33.984 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:33.984 DEBUG:boto:Port: 8001 2023-09-29T22:27:33.984 DEBUG:boto:Params: {} 2023-09-29T22:27:33.984 DEBUG:boto:Token: None 2023-09-29T22:27:33.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.985 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T22:27:33.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T22:27:33.988 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad80a629fc3a454c-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:33.988 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:33.988 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:33.988 DEBUG:boto:path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.988 DEBUG:boto:auth_path=/hcnphz-55/f?versionId=null 2023-09-29T22:27:33.989 DEBUG:boto:Method: GET 2023-09-29T22:27:33.989 DEBUG:boto:Path: /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.989 DEBUG:boto:Data: 2023-09-29T22:27:33.989 DEBUG:boto:Headers: {} 2023-09-29T22:27:33.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:33.989 DEBUG:boto:Port: 8000 2023-09-29T22:27:33.989 DEBUG:boto:Params: {} 2023-09-29T22:27:33.989 DEBUG:boto:Token: None 2023-09-29T22:27:33.989 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:33 GMT /hcnphz-55/f?versionId=null 2023-09-29T22:27:33.990 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T22:27:33.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T22:27:33.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003144f66e080a3ba1-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.038 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.038 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.039 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.039 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.039 DEBUG:boto:Method: GET 2023-09-29T22:27:34.039 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:34.039 DEBUG:boto:Data: 2023-09-29T22:27:34.039 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.039 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.039 DEBUG:boto:Params: {} 2023-09-29T22:27:34.040 DEBUG:boto:Token: None 2023-09-29T22:27:34.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000662395ca3bf15b66-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.045 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.045 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.046 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.046 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.046 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.046 DEBUG:boto:Method: GET 2023-09-29T22:27:34.046 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:34.046 DEBUG:boto:Data: 2023-09-29T22:27:34.046 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.046 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.047 DEBUG:boto:Params: {} 2023-09-29T22:27:34.047 DEBUG:boto:Token: None 2023-09-29T22:27:34.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7f010ee643c7d2-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.050 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.051 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:34.051 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:34.051 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.051 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:34.051 DEBUG:boto:Data: 2023-09-29T22:27:34.051 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.052 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.052 DEBUG:boto:Params: {} 2023-09-29T22:27:34.052 DEBUG:boto:Token: None 2023-09-29T22:27:34.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/f 2023-09-29T22:27:34.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc= 2023-09-29T22:27:34.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc=', 'Content-Length': '0'} 2023-09-29T22:27:34.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0eaf47e7f0abe70-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.055 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:34.055 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:34.055 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.055 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:34.055 DEBUG:boto:Data: 2023-09-29T22:27:34.055 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.055 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.055 DEBUG:boto:Params: {} 2023-09-29T22:27:34.056 DEBUG:boto:Token: None 2023-09-29T22:27:34.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/f 2023-09-29T22:27:34.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc= 2023-09-29T22:27:34.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc=', 'Content-Length': '0'} 2023-09-29T22:27:34.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d76e567a3c291c60-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:27:34.059 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:34.059 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:34.059 DEBUG:boto:Method: GET 2023-09-29T22:27:34.059 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:34.059 DEBUG:boto:Data: 2023-09-29T22:27:34.060 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.060 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.060 DEBUG:boto:Params: {} 2023-09-29T22:27:34.060 DEBUG:boto:Token: None 2023-09-29T22:27:34.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/f 2023-09-29T22:27:34.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU= 2023-09-29T22:27:34.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU=', 'Content-Length': '0'} 2023-09-29T22:27:34.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032362e3dbd53ceb9-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.106 DEBUG:boto:path=/hcnphz-55/f 2023-09-29T22:27:34.107 DEBUG:boto:auth_path=/hcnphz-55/f 2023-09-29T22:27:34.107 DEBUG:boto:Method: GET 2023-09-29T22:27:34.107 DEBUG:boto:Path: /hcnphz-55/f 2023-09-29T22:27:34.107 DEBUG:boto:Data: 2023-09-29T22:27:34.107 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.108 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.108 DEBUG:boto:Params: {} 2023-09-29T22:27:34.108 DEBUG:boto:Token: None 2023-09-29T22:27:34.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/f 2023-09-29T22:27:34.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU= 2023-09-29T22:27:34.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU=', 'Content-Length': '0'} 2023-09-29T22:27:34.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045100783aa7bc3f2-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.111 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.112 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.112 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.112 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.112 DEBUG:boto:Method: GET 2023-09-29T22:27:34.112 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:34.113 DEBUG:boto:Data: 2023-09-29T22:27:34.113 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.113 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.113 DEBUG:boto:Params: {} 2023-09-29T22:27:34.114 DEBUG:boto:Token: None 2023-09-29T22:27:34.114 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf26b803718f3a8-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.117 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.118 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.118 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.118 DEBUG:boto:path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.118 DEBUG:boto:auth_path=/hcnphz-55/?prefix=f 2023-09-29T22:27:34.119 DEBUG:boto:Method: GET 2023-09-29T22:27:34.119 DEBUG:boto:Path: /hcnphz-55/?prefix=f 2023-09-29T22:27:34.119 DEBUG:boto:Data: 2023-09-29T22:27:34.119 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.120 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.120 DEBUG:boto:Params: {} 2023-09-29T22:27:34.120 DEBUG:boto:Token: None 2023-09-29T22:27:34.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001393a5ce3686eeec-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.166 DEBUG:boto:b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:27:34.167 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.167 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.167 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.167 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.168 DEBUG:boto:Method: GET 2023-09-29T22:27:34.168 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.168 DEBUG:boto:Data: 2023-09-29T22:27:34.168 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.168 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.168 DEBUG:boto:Params: {} 2023-09-29T22:27:34.168 DEBUG:boto:Token: None 2023-09-29T22:27:34.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T22:27:34.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T22:27:34.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db05efbd596c8f2-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.172 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.172 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.172 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.173 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.173 DEBUG:boto:Method: GET 2023-09-29T22:27:34.173 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.173 DEBUG:boto:Data: 2023-09-29T22:27:34.173 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.173 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.173 DEBUG:boto:Params: {} 2023-09-29T22:27:34.173 DEBUG:boto:Token: None 2023-09-29T22:27:34.174 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T22:27:34.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T22:27:34.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c9a4e0f1ab92ab-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.177 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.177 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.177 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.177 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.178 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.178 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.178 DEBUG:boto:Data: 2023-09-29T22:27:34.179 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.179 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.179 DEBUG:boto:Params: {} 2023-09-29T22:27:34.179 DEBUG:boto:Token: None 2023-09-29T22:27:34.180 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk= 2023-09-29T22:27:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk=', 'Content-Length': '0'} 2023-09-29T22:27:34.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c736e4f2af7158-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.183 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.183 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.183 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.183 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.184 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.184 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.184 DEBUG:boto:Data: 2023-09-29T22:27:34.184 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.184 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.184 DEBUG:boto:Params: {} 2023-09-29T22:27:34.185 DEBUG:boto:Token: None 2023-09-29T22:27:34.185 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.185 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk= 2023-09-29T22:27:34.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk=', 'Content-Length': '0'} 2023-09-29T22:27:34.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539186061fa72c0c-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:27:34.188 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.189 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.189 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.189 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.189 DEBUG:boto:Method: GET 2023-09-29T22:27:34.189 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.189 DEBUG:boto:Data: 2023-09-29T22:27:34.190 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.190 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.190 DEBUG:boto:Params: {} 2023-09-29T22:27:34.190 DEBUG:boto:Token: None 2023-09-29T22:27:34.191 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T22:27:34.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T22:27:34.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000046bb0656264f260-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.238 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.239 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.239 DEBUG:boto:path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.239 DEBUG:boto:auth_path=/hcnphz-55/g?versionId=null 2023-09-29T22:27:34.239 DEBUG:boto:Method: GET 2023-09-29T22:27:34.239 DEBUG:boto:Path: /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.239 DEBUG:boto:Data: 2023-09-29T22:27:34.239 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.239 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.240 DEBUG:boto:Params: {} 2023-09-29T22:27:34.240 DEBUG:boto:Token: None 2023-09-29T22:27:34.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g?versionId=null 2023-09-29T22:27:34.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T22:27:34.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T22:27:34.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6dec1395953e24-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.243 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.243 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.243 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.244 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.244 DEBUG:boto:Method: GET 2023-09-29T22:27:34.244 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:34.244 DEBUG:boto:Data: 2023-09-29T22:27:34.244 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.244 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.244 DEBUG:boto:Params: {} 2023-09-29T22:27:34.245 DEBUG:boto:Token: None 2023-09-29T22:27:34.245 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.245 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da30a0dbf46283e3-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.249 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.250 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.250 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.250 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.250 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.250 DEBUG:boto:Method: GET 2023-09-29T22:27:34.250 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:34.251 DEBUG:boto:Data: 2023-09-29T22:27:34.251 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.251 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.251 DEBUG:boto:Params: {} 2023-09-29T22:27:34.251 DEBUG:boto:Token: None 2023-09-29T22:27:34.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000510c8ef9dacc9bff-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.255 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.255 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.255 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.255 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.255 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:34.255 DEBUG:boto:Data: 2023-09-29T22:27:34.256 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.256 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.256 DEBUG:boto:Params: {} 2023-09-29T22:27:34.256 DEBUG:boto:Token: None 2023-09-29T22:27:34.256 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g 2023-09-29T22:27:34.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ= 2023-09-29T22:27:34.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ=', 'Content-Length': '0'} 2023-09-29T22:27:34.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd504c77707300b-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.259 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.259 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.259 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.260 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:34.260 DEBUG:boto:Data: 2023-09-29T22:27:34.260 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.260 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.260 DEBUG:boto:Params: {} 2023-09-29T22:27:34.260 DEBUG:boto:Token: None 2023-09-29T22:27:34.260 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g 2023-09-29T22:27:34.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ= 2023-09-29T22:27:34.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ=', 'Content-Length': '0'} 2023-09-29T22:27:34.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b5e79fc61c3d9b-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:27:34.263 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.263 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.264 DEBUG:boto:Method: GET 2023-09-29T22:27:34.264 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:34.264 DEBUG:boto:Data: 2023-09-29T22:27:34.264 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.264 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.264 DEBUG:boto:Params: {} 2023-09-29T22:27:34.264 DEBUG:boto:Token: None 2023-09-29T22:27:34.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g 2023-09-29T22:27:34.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ= 2023-09-29T22:27:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ=', 'Content-Length': '0'} 2023-09-29T22:27:34.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb1084e4c6f518b-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.310 DEBUG:boto:path=/hcnphz-55/g 2023-09-29T22:27:34.311 DEBUG:boto:auth_path=/hcnphz-55/g 2023-09-29T22:27:34.311 DEBUG:boto:Method: GET 2023-09-29T22:27:34.311 DEBUG:boto:Path: /hcnphz-55/g 2023-09-29T22:27:34.311 DEBUG:boto:Data: 2023-09-29T22:27:34.311 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.311 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.311 DEBUG:boto:Params: {} 2023-09-29T22:27:34.311 DEBUG:boto:Token: None 2023-09-29T22:27:34.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/g 2023-09-29T22:27:34.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ= 2023-09-29T22:27:34.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ=', 'Content-Length': '0'} 2023-09-29T22:27:34.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca2ab5744b72311-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.315 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.315 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.315 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.316 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.316 DEBUG:boto:Method: GET 2023-09-29T22:27:34.316 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:34.316 DEBUG:boto:Data: 2023-09-29T22:27:34.316 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.316 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.316 DEBUG:boto:Params: {} 2023-09-29T22:27:34.316 DEBUG:boto:Token: None 2023-09-29T22:27:34.317 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2331719a79a8470-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.321 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.322 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.322 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.322 DEBUG:boto:path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.322 DEBUG:boto:auth_path=/hcnphz-55/?prefix=g 2023-09-29T22:27:34.322 DEBUG:boto:Method: GET 2023-09-29T22:27:34.323 DEBUG:boto:Path: /hcnphz-55/?prefix=g 2023-09-29T22:27:34.323 DEBUG:boto:Data: 2023-09-29T22:27:34.323 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.323 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.323 DEBUG:boto:Params: {} 2023-09-29T22:27:34.323 DEBUG:boto:Token: None 2023-09-29T22:27:34.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.323 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.327 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd637e13a8497d3c-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.327 DEBUG:boto:b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:27:34.327 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.327 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.328 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.328 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.328 DEBUG:boto:Method: GET 2023-09-29T22:27:34.328 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.328 DEBUG:boto:Data: 2023-09-29T22:27:34.328 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.328 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.328 DEBUG:boto:Params: {} 2023-09-29T22:27:34.329 DEBUG:boto:Token: None 2023-09-29T22:27:34.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T22:27:34.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T22:27:34.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8783bde5966f439-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.332 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.332 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.332 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.332 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.332 DEBUG:boto:Method: GET 2023-09-29T22:27:34.332 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.332 DEBUG:boto:Data: 2023-09-29T22:27:34.333 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.333 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.333 DEBUG:boto:Params: {} 2023-09-29T22:27:34.333 DEBUG:boto:Token: None 2023-09-29T22:27:34.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T22:27:34.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T22:27:34.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be4b155f6da92b5-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.337 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.337 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.337 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.337 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.337 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.337 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.337 DEBUG:boto:Data: 2023-09-29T22:27:34.337 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.338 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.338 DEBUG:boto:Params: {} 2023-09-29T22:27:34.338 DEBUG:boto:Token: None 2023-09-29T22:27:34.338 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4= 2023-09-29T22:27:34.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4=', 'Content-Length': '0'} 2023-09-29T22:27:34.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74675bf08a18f64-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.341 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.341 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.341 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.341 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.341 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.341 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.341 DEBUG:boto:Data: 2023-09-29T22:27:34.341 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.342 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.342 DEBUG:boto:Params: {} 2023-09-29T22:27:34.342 DEBUG:boto:Token: None 2023-09-29T22:27:34.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4= 2023-09-29T22:27:34.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4=', 'Content-Length': '0'} 2023-09-29T22:27:34.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbe49d9cecc04803-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:27:34.345 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.345 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.345 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.346 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.346 DEBUG:boto:Method: GET 2023-09-29T22:27:34.346 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.346 DEBUG:boto:Data: 2023-09-29T22:27:34.346 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.346 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.346 DEBUG:boto:Params: {} 2023-09-29T22:27:34.346 DEBUG:boto:Token: None 2023-09-29T22:27:34.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T22:27:34.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T22:27:34.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1ee306251c5e09-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.394 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.395 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.395 DEBUG:boto:path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.395 DEBUG:boto:auth_path=/hcnphz-55/h?versionId=null 2023-09-29T22:27:34.395 DEBUG:boto:Method: GET 2023-09-29T22:27:34.395 DEBUG:boto:Path: /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.396 DEBUG:boto:Data: 2023-09-29T22:27:34.396 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.396 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.396 DEBUG:boto:Params: {} 2023-09-29T22:27:34.397 DEBUG:boto:Token: None 2023-09-29T22:27:34.397 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h?versionId=null 2023-09-29T22:27:34.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T22:27:34.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T22:27:34.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846d75b3cae133c3-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.401 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.401 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.401 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.402 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.402 DEBUG:boto:Method: GET 2023-09-29T22:27:34.402 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:34.402 DEBUG:boto:Data: 2023-09-29T22:27:34.402 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.403 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.403 DEBUG:boto:Params: {} 2023-09-29T22:27:34.403 DEBUG:boto:Token: None 2023-09-29T22:27:34.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.403 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017a70cd951e0711-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.409 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.409 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.409 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.410 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.410 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.410 DEBUG:boto:Method: GET 2023-09-29T22:27:34.410 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:34.410 DEBUG:boto:Data: 2023-09-29T22:27:34.410 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.411 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.411 DEBUG:boto:Params: {} 2023-09-29T22:27:34.411 DEBUG:boto:Token: None 2023-09-29T22:27:34.411 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.411 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.414 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d993db298e12e0-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.415 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.415 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.415 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.416 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.416 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:34.416 DEBUG:boto:Data: 2023-09-29T22:27:34.416 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.416 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.416 DEBUG:boto:Params: {} 2023-09-29T22:27:34.417 DEBUG:boto:Token: None 2023-09-29T22:27:34.417 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h 2023-09-29T22:27:34.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk= 2023-09-29T22:27:34.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk=', 'Content-Length': '0'} 2023-09-29T22:27:34.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006284388aa328e4d4-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.420 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.420 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.420 DEBUG:boto:Method: HEAD 2023-09-29T22:27:34.420 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:34.421 DEBUG:boto:Data: 2023-09-29T22:27:34.421 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.421 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.421 DEBUG:boto:Params: {} 2023-09-29T22:27:34.421 DEBUG:boto:Token: None 2023-09-29T22:27:34.422 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h 2023-09-29T22:27:34.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk= 2023-09-29T22:27:34.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk=', 'Content-Length': '0'} 2023-09-29T22:27:34.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010047e836e0a2e97-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:27:34.425 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.425 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.425 DEBUG:boto:Method: GET 2023-09-29T22:27:34.425 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:34.425 DEBUG:boto:Data: 2023-09-29T22:27:34.426 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.426 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.426 DEBUG:boto:Params: {} 2023-09-29T22:27:34.426 DEBUG:boto:Token: None 2023-09-29T22:27:34.426 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h 2023-09-29T22:27:34.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4= 2023-09-29T22:27:34.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4=', 'Content-Length': '0'} 2023-09-29T22:27:34.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001244e8d30b0f9a61-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.470 DEBUG:boto:path=/hcnphz-55/h 2023-09-29T22:27:34.470 DEBUG:boto:auth_path=/hcnphz-55/h 2023-09-29T22:27:34.471 DEBUG:boto:Method: GET 2023-09-29T22:27:34.471 DEBUG:boto:Path: /hcnphz-55/h 2023-09-29T22:27:34.471 DEBUG:boto:Data: 2023-09-29T22:27:34.471 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.472 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.472 DEBUG:boto:Params: {} 2023-09-29T22:27:34.472 DEBUG:boto:Token: None 2023-09-29T22:27:34.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/h 2023-09-29T22:27:34.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4= 2023-09-29T22:27:34.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4=', 'Content-Length': '0'} 2023-09-29T22:27:34.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78b3cedee70c3f7-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.476 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.476 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.476 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.476 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.476 DEBUG:boto:Method: GET 2023-09-29T22:27:34.476 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:34.476 DEBUG:boto:Data: 2023-09-29T22:27:34.476 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:27:34.477 DEBUG:boto:Port: 8001 2023-09-29T22:27:34.477 DEBUG:boto:Params: {} 2023-09-29T22:27:34.477 DEBUG:boto:Token: None 2023-09-29T22:27:34.477 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cb879d673e2974-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.485 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.485 DEBUG:boto:path=/hcnphz-55/ 2023-09-29T22:27:34.486 DEBUG:boto:auth_path=/hcnphz-55/ 2023-09-29T22:27:34.486 DEBUG:boto:path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.486 DEBUG:boto:auth_path=/hcnphz-55/?prefix=h 2023-09-29T22:27:34.486 DEBUG:boto:Method: GET 2023-09-29T22:27:34.486 DEBUG:boto:Path: /hcnphz-55/?prefix=h 2023-09-29T22:27:34.486 DEBUG:boto:Data: 2023-09-29T22:27:34.486 DEBUG:boto:Headers: {} 2023-09-29T22:27:34.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:34.487 DEBUG:boto:Port: 8000 2023-09-29T22:27:34.487 DEBUG:boto:Params: {} 2023-09-29T22:27:34.487 DEBUG:boto:Token: None 2023-09-29T22:27:34.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:27:34 GMT /hcnphz-55/ 2023-09-29T22:27:34.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T22:27:34.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T22:27:34.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b26cb66cb4c90511-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:34.530 DEBUG:boto:b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:27:34.531 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-55 zones={a2, a1} 2023-09-29T22:27:34.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:34.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:34.531 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:27:34.572 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:34.573 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:34.659 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:34.655+0000 7f39009db700 -1 req 17317900272779064789 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-3:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.3,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-3:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.3,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:34.659 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:34.655+0000 7f38e49a3700 -1 req 2720698406258693920 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-24:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.24,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-24:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.24,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:34.667 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:34.663+0000 7f3935244700 -1 req 7328178435526173355 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-26:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.26,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-26:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.26,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:34.667 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:34.663+0000 7f38ec9b3700 -1 req 8345277035345697252 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-29:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.29,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-29:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.29,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:34.681 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:34.679+0000 7f38ce977700 -1 req 7922580692162518313 0.003999894s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-20:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.20,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-20:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.20,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:34.682 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:34.679+0000 7f38bd154700 -1 req 17929215227216565450 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-51:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.16,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-51:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.16,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:34.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:34.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:34.773 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:27:34.815 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:34.815 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:34.899 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:27:34.899 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:34.899 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-09-29T22:27:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-09-29T22:27:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-09-29T22:27:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-09-29T22:27:34.900 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:34.901 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:34.903 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:34.904 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:27:34.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:34.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:34.906 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:27:34.944 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:34.944 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:35.030 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.027+0000 7f389810a700 -1 req 16906345383139285555 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-5:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.5,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-5:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.5,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.030 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.027+0000 7f38a291f700 -1 req 3358093351381680342 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-39:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.4,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-39:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.4,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.030 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.027+0000 7f38a111c700 -1 req 6006448330050620975 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-28:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.28,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-28:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.28,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.039 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.035+0000 7f38a011a700 -1 req 11845390325519182613 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-30:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.30,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-30:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.30,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.043 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.039+0000 7f389890b700 -1 req 10070599165353365148 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-4:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.4,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-4:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.4,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.045 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.043+0000 7f38908fb700 -1 req 16402390693735243024 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-37:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.2,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-37:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.2,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.048 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:27:35.047+0000 7f38768c7700 -1 req 14765125288640428672 0.003999894s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-38:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.3,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-38:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.3,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T22:27:35.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:35.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:27:35.136 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:27:35.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:35.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:35.297 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:27:35.298 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:35.298 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding": "None", 2023-09-29T22:27:35.298 INFO:teuthology.orchestra.run.smithi112.stdout: "current_index": { 2023-09-29T22:27:35.298 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:35.298 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:35.299 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "Normal", 2023-09-29T22:27:35.299 INFO:teuthology.orchestra.run.smithi112.stdout: "normal": { 2023-09-29T22:27:35.299 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:35.299 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:35.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.300 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.300 INFO:teuthology.orchestra.run.smithi112.stdout: "logs": [ 2023-09-29T22:27:35.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.301 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:27:35.301 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:35.301 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:35.302 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:35.302 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 1, 2023-09-29T22:27:35.302 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:35.302 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:35.302 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:35.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.304 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:35.304 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:35.304 INFO:teuthology.orchestra.run.smithi112.stdout: "type": "InIndex", 2023-09-29T22:27:35.304 INFO:teuthology.orchestra.run.smithi112.stdout: "in_index": { 2023-09-29T22:27:35.305 INFO:teuthology.orchestra.run.smithi112.stdout: "gen": 2, 2023-09-29T22:27:35.305 INFO:teuthology.orchestra.run.smithi112.stdout: "layout": { 2023-09-29T22:27:35.305 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 3, 2023-09-29T22:27:35.305 INFO:teuthology.orchestra.run.smithi112.stdout: "hash_type": "Mod" 2023-09-29T22:27:35.305 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.306 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:27:35.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.307 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:27:35.325 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_index_log_trim ... FAIL 2023-09-29T22:27:35.326 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:27:35.327 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:27:35.327 DEBUG:boto:Method: PUT 2023-09-29T22:27:35.327 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:27:35.327 DEBUG:boto:Data: 2023-09-29T22:27:35.328 DEBUG:boto:Headers: {} 2023-09-29T22:27:35.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:35.328 DEBUG:boto:Port: 8000 2023-09-29T22:27:35.328 DEBUG:boto:Params: {} 2023-09-29T22:27:35.328 DEBUG:boto:Token: None 2023-09-29T22:27:35.329 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:27:35 GMT /hcnphz-56/ 2023-09-29T22:27:35.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7dWE0Mh7YEGlvNXi75bBucTnd5Y= 2023-09-29T22:27:35.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7dWE0Mh7YEGlvNXi75bBucTnd5Y=', 'Content-Length': '0'} 2023-09-29T22:27:35.363 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000006420d6253f1484a4-0065174f57-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:35.363 DEBUG:rgw_multi.tests:created bucket=hcnphz-56 2023-09-29T22:27:35.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:27:35.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:27:35.365 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:27:35.410 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:35.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:35.463 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:27:35.490 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:27:35.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.490 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:27:35.490 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026455.360632_437.1", 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:27:35.360632Z" 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.492 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:27:35.492 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:27:35.492 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.492 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:27:35.493 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026455.360632_437.1', 2: '', 3: ''} 2023-09-29T22:27:35.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:27:35.493 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:27:35.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:27:35.494 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:27:35.534 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:35.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:35.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:35.595+0000 7f31ba8fbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:27:35.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:35.595+0000 7f31ba8fbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:27:35.619 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:27:35.619 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.620 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:35.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026447.526749_428.1", 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:27:35.622 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:27:27.526749Z", 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:27:35.623 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:27:35.624 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:27:35Z" 2023-09-29T22:27:35.626 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:27:35.627 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026447.526749_428.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:27:27.526749Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:27:35Z" } 2023-09-29T22:27:35.627 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026447.526749_428.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:27:27.526749Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:27:35.628 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026455.360632_437.1', 2: '', 3: ''} 2023-09-29T22:27:35.628 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026447.526749_428.1', 2: '', 3: ''} 2023-09-29T22:27:35.628 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026455.360632_437.1 target=1_1696026447.526749_428.1 2023-09-29T22:27:40.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:27:40.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:27:40.629 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:27:40.676 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:40.676 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:40.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:40.726+0000 7f0e88acdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:27:40.732 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:40.730+0000 7f0e88acdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:27:40.758 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:27:40.758 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:27:40.758 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:27:40.758 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:40.759 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:40.760 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026455.360632_437.1", 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:27:35.360632Z", 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:40.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:27:40.762 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:27:40.763 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:27:40.764 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:27:40.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:27:40.765 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:27:40Z" 2023-09-29T22:27:40.765 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:27:40.766 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026455.360632_437.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:27:35.360632Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:27:40Z" } 2023-09-29T22:27:40.766 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026455.360632_437.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:27:35.360632Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:27:40.766 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026455.360632_437.1', 2: '', 3: ''} 2023-09-29T22:27:40.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026455.360632_437.1', 2: '', 3: ''} 2023-09-29T22:27:40.766 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:27:40.766 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:27:40.767 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:27:40.767 DEBUG:boto:Method: HEAD 2023-09-29T22:27:40.767 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:27:40.767 DEBUG:boto:Data: 2023-09-29T22:27:40.767 DEBUG:boto:Headers: {} 2023-09-29T22:27:40.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.767 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.767 DEBUG:boto:Params: {} 2023-09-29T22:27:40.768 DEBUG:boto:Token: None 2023-09-29T22:27:40.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/ 2023-09-29T22:27:40.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU= 2023-09-29T22:27:40.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU=', 'Content-Length': '0'} 2023-09-29T22:27:40.771 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b81d7ee0507e4b8-0065174f5c-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.772 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:27:40.772 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:27:40.772 DEBUG:boto:Method: PUT 2023-09-29T22:27:40.772 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:27:40.772 DEBUG:boto:Data: 2023-09-29T22:27:40.772 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:27:40.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.773 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.773 DEBUG:boto:Params: {} 2023-09-29T22:27:40.773 DEBUG:boto:Token: None 2023-09-29T22:27:40.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/a 2023-09-29T22:27:40.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g9hA0gu2mR64u9yv0vtMJydk3sE= 2023-09-29T22:27:40.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g9hA0gu2mR64u9yv0vtMJydk3sE='} 2023-09-29T22:27:40.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000956b16ec7db20aab-0065174f5c-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.789 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:27:40.790 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:27:40.790 DEBUG:boto:Method: HEAD 2023-09-29T22:27:40.790 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:27:40.790 DEBUG:boto:Data: 2023-09-29T22:27:40.790 DEBUG:boto:Headers: {} 2023-09-29T22:27:40.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.790 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.790 DEBUG:boto:Params: {} 2023-09-29T22:27:40.790 DEBUG:boto:Token: None 2023-09-29T22:27:40.791 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/ 2023-09-29T22:27:40.791 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU= 2023-09-29T22:27:40.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU=', 'Content-Length': '0'} 2023-09-29T22:27:40.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c0d8bf392884c7c6-0065174f5c-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.794 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:27:40.794 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:27:40.794 DEBUG:boto:Method: PUT 2023-09-29T22:27:40.794 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:27:40.794 DEBUG:boto:Data: 2023-09-29T22:27:40.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:27:40.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.794 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.794 DEBUG:boto:Params: {} 2023-09-29T22:27:40.795 DEBUG:boto:Token: None 2023-09-29T22:27:40.795 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/b 2023-09-29T22:27:40.795 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H4r4zkqiRCTAOjTw/IUiXdUFFEI= 2023-09-29T22:27:40.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H4r4zkqiRCTAOjTw/IUiXdUFFEI='} 2023-09-29T22:27:40.804 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009040f6e2f11fe620-0065174f5c-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.804 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:27:40.804 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:27:40.804 DEBUG:boto:Method: HEAD 2023-09-29T22:27:40.804 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:27:40.804 DEBUG:boto:Data: 2023-09-29T22:27:40.804 DEBUG:boto:Headers: {} 2023-09-29T22:27:40.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.805 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.805 DEBUG:boto:Params: {} 2023-09-29T22:27:40.805 DEBUG:boto:Token: None 2023-09-29T22:27:40.805 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/ 2023-09-29T22:27:40.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU= 2023-09-29T22:27:40.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU=', 'Content-Length': '0'} 2023-09-29T22:27:40.808 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ae2eeccf5cc16256-0065174f5c-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.808 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:27:40.809 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:27:40.809 DEBUG:boto:Method: PUT 2023-09-29T22:27:40.809 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:27:40.809 DEBUG:boto:Data: 2023-09-29T22:27:40.809 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:27:40.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.809 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.809 DEBUG:boto:Params: {} 2023-09-29T22:27:40.809 DEBUG:boto:Token: None 2023-09-29T22:27:40.810 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/c 2023-09-29T22:27:40.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3SeDyUa7RTNLNmC1lYafOUHbx1c= 2023-09-29T22:27:40.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3SeDyUa7RTNLNmC1lYafOUHbx1c='} 2023-09-29T22:27:40.855 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000c54592aba05649-0065174f5c-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.855 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:27:40.855 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:27:40.855 DEBUG:boto:Method: HEAD 2023-09-29T22:27:40.855 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:27:40.855 DEBUG:boto:Data: 2023-09-29T22:27:40.855 DEBUG:boto:Headers: {} 2023-09-29T22:27:40.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.856 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.856 DEBUG:boto:Params: {} 2023-09-29T22:27:40.856 DEBUG:boto:Token: None 2023-09-29T22:27:40.856 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/ 2023-09-29T22:27:40.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU= 2023-09-29T22:27:40.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y9nWHB04mjC3UOyn6cpUIM0/LIU=', 'Content-Length': '0'} 2023-09-29T22:27:40.860 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c7aa10b3bdd37c0-0065174f5c-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.860 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:27:40.860 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:27:40.861 DEBUG:boto:Method: PUT 2023-09-29T22:27:40.861 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:27:40.861 DEBUG:boto:Data: 2023-09-29T22:27:40.861 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:27:40.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:27:40.861 DEBUG:boto:Port: 8000 2023-09-29T22:27:40.861 DEBUG:boto:Params: {} 2023-09-29T22:27:40.861 DEBUG:boto:Token: None 2023-09-29T22:27:40.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:27:40 GMT /hcnphz-56/d 2023-09-29T22:27:40.862 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oUj5JbmuNk9rtY/rewzwtkSHmMg= 2023-09-29T22:27:40.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:27:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oUj5JbmuNk9rtY/rewzwtkSHmMg='} 2023-09-29T22:27:40.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a1b57ea69e6c27d-0065174f5c-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:27:40.873 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-56 2023-09-29T22:27:40.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:40.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:40.874 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:27:40.926 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:40.926 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:40.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:40.990+0000 7f5eec906f40 1 waiting to reach incremental sync.. 2023-09-29T22:27:45.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:45.990+0000 7f5eec906f40 1 waiting to reach incremental sync.. 2023-09-29T22:27:50.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:50.990+0000 7f5eec906f40 1 waiting to reach incremental sync.. 2023-09-29T22:27:55.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:55.994+0000 7f5eec906f40 1 bucket sync caught up with source: 2023-09-29T22:27:55.998 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.22.6, , , , 00000000001.43.6, , , 00000000001.38.6, , 00000000001.54.6] 2023-09-29T22:27:55.998 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.22.6, , , , 00000000001.43.6, , , 00000000001.38.6, , 00000000001.54.6] 2023-09-29T22:27:55.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:55.994+0000 7f5eec906f40 0 bucket checkpoint complete 2023-09-29T22:27:56.029 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-56 2023-09-29T22:27:56.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:56.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:27:56.030 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:27:56.080 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:27:56.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:27:56.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:27:56.134+0000 7f5d71754f40 1 waiting to reach incremental sync.. 2023-09-29T22:28:01.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:01.138+0000 7f5d71754f40 1 bucket sync caught up with source: 2023-09-29T22:28:01.142 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.55.6, , , , 00000000001.149.6, , , 00000000001.514.6, , 00000000001.125.6] 2023-09-29T22:28:01.142 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.55.6, , , , 00000000001.149.6, , , 00000000001.514.6, , 00000000001.125.6] 2023-09-29T22:28:01.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:01.138+0000 7f5d71754f40 0 bucket checkpoint complete 2023-09-29T22:28:01.168 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-56 zones={a2, a1} 2023-09-29T22:28:01.168 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.168 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.169 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.169 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:28:01.169 DEBUG:boto:Data: 2023-09-29T22:28:01.169 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.169 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.169 DEBUG:boto:Params: {} 2023-09-29T22:28:01.169 DEBUG:boto:Token: None 2023-09-29T22:28:01.170 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.170 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ioDdjrOsEYzb341HC7pkP1xEy2U= 2023-09-29T22:28:01.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ioDdjrOsEYzb341HC7pkP1xEy2U=', 'Content-Length': '0'} 2023-09-29T22:28:01.176 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085dad96bae87306a-0065174f71-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.176 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.176 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.176 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.177 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:28:01.177 DEBUG:boto:Data: 2023-09-29T22:28:01.177 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.177 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.177 DEBUG:boto:Params: {} 2023-09-29T22:28:01.177 DEBUG:boto:Token: None 2023-09-29T22:28:01.177 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.177 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ioDdjrOsEYzb341HC7pkP1xEy2U= 2023-09-29T22:28:01.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ioDdjrOsEYzb341HC7pkP1xEy2U=', 'Content-Length': '0'} 2023-09-29T22:28:01.180 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ee3be84a0532c9e4-0065174f71-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.181 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:28:01.181 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.181 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.181 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.181 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.181 DEBUG:boto:Method: GET 2023-09-29T22:28:01.181 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.181 DEBUG:boto:Data: 2023-09-29T22:28:01.182 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.182 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.182 DEBUG:boto:Params: {} 2023-09-29T22:28:01.182 DEBUG:boto:Token: None 2023-09-29T22:28:01.182 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/?versions 2023-09-29T22:28:01.182 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA= 2023-09-29T22:28:01.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA=', 'Content-Length': '0'} 2023-09-29T22:28:01.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c66198cc4969eaf1-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.185 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.186 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:28:01.187 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:28:01.187 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:28:01.187 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:28:01.187 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:28:01.188 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.188 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.188 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.188 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.188 DEBUG:boto:Method: GET 2023-09-29T22:28:01.188 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.188 DEBUG:boto:Data: 2023-09-29T22:28:01.188 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.189 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.189 DEBUG:boto:Params: {} 2023-09-29T22:28:01.189 DEBUG:boto:Token: None 2023-09-29T22:28:01.189 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/?versions 2023-09-29T22:28:01.189 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA= 2023-09-29T22:28:01.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA=', 'Content-Length': '0'} 2023-09-29T22:28:01.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd0e97285d94de71-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.193 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.194 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:28:01.194 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:28:01.194 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:28:01.194 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:28:01.195 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.195 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.195 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.195 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.195 DEBUG:boto:Method: GET 2023-09-29T22:28:01.195 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.195 DEBUG:boto:Data: 2023-09-29T22:28:01.195 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.195 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.196 DEBUG:boto:Params: {} 2023-09-29T22:28:01.196 DEBUG:boto:Token: None 2023-09-29T22:28:01.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/?versions 2023-09-29T22:28:01.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA= 2023-09-29T22:28:01.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA=', 'Content-Length': '0'} 2023-09-29T22:28:01.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5b05f5aa5ca9d65-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.200 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.200 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.200 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.200 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.201 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.201 DEBUG:boto:Method: GET 2023-09-29T22:28:01.201 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:28:01.201 DEBUG:boto:Data: 2023-09-29T22:28:01.201 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.201 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.201 DEBUG:boto:Params: {} 2023-09-29T22:28:01.201 DEBUG:boto:Token: None 2023-09-29T22:28:01.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/?versions 2023-09-29T22:28:01.202 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA= 2023-09-29T22:28:01.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DLbECqqfU0We9zmmAuRckErxSgA=', 'Content-Length': '0'} 2023-09-29T22:28:01.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a41d3bd8e615cb9b-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.205 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:28:01.206 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.206 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.206 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.206 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.207 DEBUG:boto:Method: GET 2023-09-29T22:28:01.207 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.207 DEBUG:boto:Data: 2023-09-29T22:28:01.207 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.207 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.207 DEBUG:boto:Params: {} 2023-09-29T22:28:01.207 DEBUG:boto:Token: None 2023-09-29T22:28:01.207 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8= 2023-09-29T22:28:01.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8=', 'Content-Length': '0'} 2023-09-29T22:28:01.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef919d51c39c3793-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.254 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.254 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.255 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.255 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.255 DEBUG:boto:Method: GET 2023-09-29T22:28:01.255 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.255 DEBUG:boto:Data: 2023-09-29T22:28:01.255 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.255 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.255 DEBUG:boto:Params: {} 2023-09-29T22:28:01.255 DEBUG:boto:Token: None 2023-09-29T22:28:01.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8= 2023-09-29T22:28:01.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8=', 'Content-Length': '0'} 2023-09-29T22:28:01.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6489b8678962441-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.259 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.259 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.259 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.259 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.260 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.260 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.260 DEBUG:boto:Data: 2023-09-29T22:28:01.260 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.260 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.260 DEBUG:boto:Params: {} 2023-09-29T22:28:01.260 DEBUG:boto:Token: None 2023-09-29T22:28:01.260 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3T3rsOpmdMbQotp432zyvNQGmv0= 2023-09-29T22:28:01.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3T3rsOpmdMbQotp432zyvNQGmv0=', 'Content-Length': '0'} 2023-09-29T22:28:01.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e776503e735a6a6-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.263 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.264 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.264 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.264 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.264 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.264 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.264 DEBUG:boto:Data: 2023-09-29T22:28:01.264 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.264 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.264 DEBUG:boto:Params: {} 2023-09-29T22:28:01.265 DEBUG:boto:Token: None 2023-09-29T22:28:01.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3T3rsOpmdMbQotp432zyvNQGmv0= 2023-09-29T22:28:01.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3T3rsOpmdMbQotp432zyvNQGmv0=', 'Content-Length': '0'} 2023-09-29T22:28:01.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c6803d35006a403-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:28:01.268 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.268 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.268 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.268 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.268 DEBUG:boto:Method: GET 2023-09-29T22:28:01.268 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.268 DEBUG:boto:Data: 2023-09-29T22:28:01.269 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.269 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.269 DEBUG:boto:Params: {} 2023-09-29T22:28:01.269 DEBUG:boto:Token: None 2023-09-29T22:28:01.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8= 2023-09-29T22:28:01.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8=', 'Content-Length': '0'} 2023-09-29T22:28:01.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000917d491a2aa1d04a-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.273 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.273 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.273 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.273 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:28:01.273 DEBUG:boto:Method: GET 2023-09-29T22:28:01.273 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.273 DEBUG:boto:Data: 2023-09-29T22:28:01.273 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.274 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.274 DEBUG:boto:Params: {} 2023-09-29T22:28:01.274 DEBUG:boto:Token: None 2023-09-29T22:28:01.274 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:28:01.274 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8= 2023-09-29T22:28:01.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wAmKN+vZOdSrzUeuWweievMJnD8=', 'Content-Length': '0'} 2023-09-29T22:28:01.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c0a0a23be18bf69-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.277 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.277 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.278 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.278 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.278 DEBUG:boto:Method: GET 2023-09-29T22:28:01.278 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:28:01.278 DEBUG:boto:Data: 2023-09-29T22:28:01.278 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.278 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.278 DEBUG:boto:Params: {} 2023-09-29T22:28:01.279 DEBUG:boto:Token: None 2023-09-29T22:28:01.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000603538448b2efc3a-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.326 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.327 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.327 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.327 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.327 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.327 DEBUG:boto:Method: GET 2023-09-29T22:28:01.327 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:28:01.327 DEBUG:boto:Data: 2023-09-29T22:28:01.327 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.328 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.328 DEBUG:boto:Params: {} 2023-09-29T22:28:01.328 DEBUG:boto:Token: None 2023-09-29T22:28:01.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000014c4dcd63de9f075-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.333 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.334 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.334 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.334 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.334 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:28:01.334 DEBUG:boto:Data: 2023-09-29T22:28:01.334 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.334 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.334 DEBUG:boto:Params: {} 2023-09-29T22:28:01.335 DEBUG:boto:Token: None 2023-09-29T22:28:01.335 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a 2023-09-29T22:28:01.335 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7c7GyWW7lS6CJmp2oGDnLwl/vYs= 2023-09-29T22:28:01.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7c7GyWW7lS6CJmp2oGDnLwl/vYs=', 'Content-Length': '0'} 2023-09-29T22:28:01.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b8cecff1404a4dc-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.338 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.338 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.338 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.338 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:28:01.338 DEBUG:boto:Data: 2023-09-29T22:28:01.338 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.339 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.339 DEBUG:boto:Params: {} 2023-09-29T22:28:01.339 DEBUG:boto:Token: None 2023-09-29T22:28:01.339 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a 2023-09-29T22:28:01.339 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7c7GyWW7lS6CJmp2oGDnLwl/vYs= 2023-09-29T22:28:01.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7c7GyWW7lS6CJmp2oGDnLwl/vYs=', 'Content-Length': '0'} 2023-09-29T22:28:01.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001f5292204ea8d7c-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:28:01.342 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.342 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.342 DEBUG:boto:Method: GET 2023-09-29T22:28:01.343 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:28:01.343 DEBUG:boto:Data: 2023-09-29T22:28:01.343 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.343 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.343 DEBUG:boto:Params: {} 2023-09-29T22:28:01.343 DEBUG:boto:Token: None 2023-09-29T22:28:01.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a 2023-09-29T22:28:01.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7BR5iogww1iyQyFx7O1tB8FgBDo= 2023-09-29T22:28:01.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7BR5iogww1iyQyFx7O1tB8FgBDo=', 'Content-Length': '0'} 2023-09-29T22:28:01.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ae7c70301d6bf2c-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.346 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:28:01.347 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:28:01.347 DEBUG:boto:Method: GET 2023-09-29T22:28:01.347 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:28:01.347 DEBUG:boto:Data: 2023-09-29T22:28:01.347 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.347 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.347 DEBUG:boto:Params: {} 2023-09-29T22:28:01.347 DEBUG:boto:Token: None 2023-09-29T22:28:01.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/a 2023-09-29T22:28:01.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7BR5iogww1iyQyFx7O1tB8FgBDo= 2023-09-29T22:28:01.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7BR5iogww1iyQyFx7O1tB8FgBDo=', 'Content-Length': '0'} 2023-09-29T22:28:01.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f9f07d3a4fa31c7-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.394 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.394 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.395 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.395 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.395 DEBUG:boto:Method: GET 2023-09-29T22:28:01.395 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:28:01.395 DEBUG:boto:Data: 2023-09-29T22:28:01.395 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.395 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.395 DEBUG:boto:Params: {} 2023-09-29T22:28:01.395 DEBUG:boto:Token: None 2023-09-29T22:28:01.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.400 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a788602f36a6939e-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.400 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.401 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.401 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.401 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.401 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:28:01.401 DEBUG:boto:Method: GET 2023-09-29T22:28:01.401 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:28:01.401 DEBUG:boto:Data: 2023-09-29T22:28:01.401 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.402 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.402 DEBUG:boto:Params: {} 2023-09-29T22:28:01.402 DEBUG:boto:Token: None 2023-09-29T22:28:01.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.407 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd3a26c1f4a84ff5-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.407 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:28:01.408 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.408 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.408 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.408 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.408 DEBUG:boto:Method: GET 2023-09-29T22:28:01.408 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.408 DEBUG:boto:Data: 2023-09-29T22:28:01.408 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.409 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.409 DEBUG:boto:Params: {} 2023-09-29T22:28:01.409 DEBUG:boto:Token: None 2023-09-29T22:28:01.409 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc= 2023-09-29T22:28:01.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc=', 'Content-Length': '0'} 2023-09-29T22:28:01.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801ec698c887c08a-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.412 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.412 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.412 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.412 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.412 DEBUG:boto:Method: GET 2023-09-29T22:28:01.412 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.412 DEBUG:boto:Data: 2023-09-29T22:28:01.412 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.413 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.413 DEBUG:boto:Params: {} 2023-09-29T22:28:01.413 DEBUG:boto:Token: None 2023-09-29T22:28:01.413 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc= 2023-09-29T22:28:01.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc=', 'Content-Length': '0'} 2023-09-29T22:28:01.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa3ce493fe95a198-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.416 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.416 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.416 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.416 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.417 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.417 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.417 DEBUG:boto:Data: 2023-09-29T22:28:01.417 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.417 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.417 DEBUG:boto:Params: {} 2023-09-29T22:28:01.417 DEBUG:boto:Token: None 2023-09-29T22:28:01.417 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:feEyiudHrsmevSRBp+aHtdmObZU= 2023-09-29T22:28:01.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:feEyiudHrsmevSRBp+aHtdmObZU=', 'Content-Length': '0'} 2023-09-29T22:28:01.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2472fbe36056f19-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.420 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.420 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.420 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.420 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.421 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.421 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.421 DEBUG:boto:Data: 2023-09-29T22:28:01.421 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.421 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.421 DEBUG:boto:Params: {} 2023-09-29T22:28:01.421 DEBUG:boto:Token: None 2023-09-29T22:28:01.421 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:feEyiudHrsmevSRBp+aHtdmObZU= 2023-09-29T22:28:01.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:feEyiudHrsmevSRBp+aHtdmObZU=', 'Content-Length': '0'} 2023-09-29T22:28:01.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d268f42207476024-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:28:01.424 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.424 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.424 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.425 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.425 DEBUG:boto:Method: GET 2023-09-29T22:28:01.425 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.425 DEBUG:boto:Data: 2023-09-29T22:28:01.425 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.425 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.425 DEBUG:boto:Params: {} 2023-09-29T22:28:01.425 DEBUG:boto:Token: None 2023-09-29T22:28:01.426 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc= 2023-09-29T22:28:01.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc=', 'Content-Length': '0'} 2023-09-29T22:28:01.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000090a10f10b9462c29-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.470 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.470 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.471 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.471 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:28:01.471 DEBUG:boto:Method: GET 2023-09-29T22:28:01.471 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.471 DEBUG:boto:Data: 2023-09-29T22:28:01.471 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.471 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.471 DEBUG:boto:Params: {} 2023-09-29T22:28:01.472 DEBUG:boto:Token: None 2023-09-29T22:28:01.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:28:01.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc= 2023-09-29T22:28:01.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IycpI5Mxu9qqxpeAWJt8kXzVQgc=', 'Content-Length': '0'} 2023-09-29T22:28:01.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5276aea491e526a-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.475 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.475 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.475 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.475 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.476 DEBUG:boto:Method: GET 2023-09-29T22:28:01.476 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:28:01.476 DEBUG:boto:Data: 2023-09-29T22:28:01.476 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.476 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.476 DEBUG:boto:Params: {} 2023-09-29T22:28:01.476 DEBUG:boto:Token: None 2023-09-29T22:28:01.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eee6213c518b582c-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.481 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.481 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.482 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.482 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.482 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.482 DEBUG:boto:Method: GET 2023-09-29T22:28:01.482 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:28:01.482 DEBUG:boto:Data: 2023-09-29T22:28:01.482 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.482 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.483 DEBUG:boto:Params: {} 2023-09-29T22:28:01.483 DEBUG:boto:Token: None 2023-09-29T22:28:01.483 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.483 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.487 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000624099bb755a165-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.487 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.487 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.487 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.488 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.488 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:28:01.488 DEBUG:boto:Data: 2023-09-29T22:28:01.488 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.488 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.488 DEBUG:boto:Params: {} 2023-09-29T22:28:01.488 DEBUG:boto:Token: None 2023-09-29T22:28:01.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b 2023-09-29T22:28:01.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3YrTD2ijgkOqhU63YVw6qkpNl1g= 2023-09-29T22:28:01.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3YrTD2ijgkOqhU63YVw6qkpNl1g=', 'Content-Length': '0'} 2023-09-29T22:28:01.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000680f53116bf1aa8a-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.491 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.491 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.491 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.491 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:28:01.491 DEBUG:boto:Data: 2023-09-29T22:28:01.492 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.492 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.492 DEBUG:boto:Params: {} 2023-09-29T22:28:01.492 DEBUG:boto:Token: None 2023-09-29T22:28:01.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b 2023-09-29T22:28:01.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3YrTD2ijgkOqhU63YVw6qkpNl1g= 2023-09-29T22:28:01.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3YrTD2ijgkOqhU63YVw6qkpNl1g=', 'Content-Length': '0'} 2023-09-29T22:28:01.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f8cdda2f4244531-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:28:01.495 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.495 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.495 DEBUG:boto:Method: GET 2023-09-29T22:28:01.495 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:28:01.495 DEBUG:boto:Data: 2023-09-29T22:28:01.495 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.496 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.496 DEBUG:boto:Params: {} 2023-09-29T22:28:01.496 DEBUG:boto:Token: None 2023-09-29T22:28:01.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b 2023-09-29T22:28:01.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C5RC+W5UOayuk+6/HQj9MmIfWVA= 2023-09-29T22:28:01.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C5RC+W5UOayuk+6/HQj9MmIfWVA=', 'Content-Length': '0'} 2023-09-29T22:28:01.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff4074224413116e-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.542 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:28:01.542 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:28:01.543 DEBUG:boto:Method: GET 2023-09-29T22:28:01.543 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:28:01.543 DEBUG:boto:Data: 2023-09-29T22:28:01.543 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.543 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.543 DEBUG:boto:Params: {} 2023-09-29T22:28:01.543 DEBUG:boto:Token: None 2023-09-29T22:28:01.543 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/b 2023-09-29T22:28:01.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C5RC+W5UOayuk+6/HQj9MmIfWVA= 2023-09-29T22:28:01.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C5RC+W5UOayuk+6/HQj9MmIfWVA=', 'Content-Length': '0'} 2023-09-29T22:28:01.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2849e1a97409b2c-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.547 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.547 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.547 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.547 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.547 DEBUG:boto:Method: GET 2023-09-29T22:28:01.547 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:28:01.547 DEBUG:boto:Data: 2023-09-29T22:28:01.547 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.548 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.548 DEBUG:boto:Params: {} 2023-09-29T22:28:01.548 DEBUG:boto:Token: None 2023-09-29T22:28:01.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc15ba439c9bc3d6-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.554 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.554 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.554 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.555 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.555 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:28:01.555 DEBUG:boto:Method: GET 2023-09-29T22:28:01.555 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:28:01.555 DEBUG:boto:Data: 2023-09-29T22:28:01.555 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.555 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.555 DEBUG:boto:Params: {} 2023-09-29T22:28:01.555 DEBUG:boto:Token: None 2023-09-29T22:28:01.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.560 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b072233c6908e72a-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.602 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:28:01.603 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.603 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.603 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.603 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.603 DEBUG:boto:Method: GET 2023-09-29T22:28:01.603 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.603 DEBUG:boto:Data: 2023-09-29T22:28:01.603 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.604 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.604 DEBUG:boto:Params: {} 2023-09-29T22:28:01.604 DEBUG:boto:Token: None 2023-09-29T22:28:01.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0= 2023-09-29T22:28:01.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0=', 'Content-Length': '0'} 2023-09-29T22:28:01.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1e3de1c068a5258-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.607 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.608 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.608 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.608 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.608 DEBUG:boto:Method: GET 2023-09-29T22:28:01.608 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.608 DEBUG:boto:Data: 2023-09-29T22:28:01.608 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.608 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.608 DEBUG:boto:Params: {} 2023-09-29T22:28:01.609 DEBUG:boto:Token: None 2023-09-29T22:28:01.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0= 2023-09-29T22:28:01.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0=', 'Content-Length': '0'} 2023-09-29T22:28:01.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68b4d420cd0ea1a-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.612 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.612 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.612 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.612 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.613 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.613 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.613 DEBUG:boto:Data: 2023-09-29T22:28:01.613 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.613 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.613 DEBUG:boto:Params: {} 2023-09-29T22:28:01.613 DEBUG:boto:Token: None 2023-09-29T22:28:01.613 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.614 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F8N4aYo5keLdk4XnOu+50+YOSRU= 2023-09-29T22:28:01.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F8N4aYo5keLdk4XnOu+50+YOSRU=', 'Content-Length': '0'} 2023-09-29T22:28:01.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1e4ac2ab036eb91-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.616 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.617 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.617 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.617 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.617 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.617 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.617 DEBUG:boto:Data: 2023-09-29T22:28:01.617 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.617 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.617 DEBUG:boto:Params: {} 2023-09-29T22:28:01.618 DEBUG:boto:Token: None 2023-09-29T22:28:01.618 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.618 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F8N4aYo5keLdk4XnOu+50+YOSRU= 2023-09-29T22:28:01.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F8N4aYo5keLdk4XnOu+50+YOSRU=', 'Content-Length': '0'} 2023-09-29T22:28:01.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e7c31e35d6f3dc6-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:28:01.621 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.621 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.621 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.621 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.621 DEBUG:boto:Method: GET 2023-09-29T22:28:01.621 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.621 DEBUG:boto:Data: 2023-09-29T22:28:01.621 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.622 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.622 DEBUG:boto:Params: {} 2023-09-29T22:28:01.622 DEBUG:boto:Token: None 2023-09-29T22:28:01.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0= 2023-09-29T22:28:01.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0=', 'Content-Length': '0'} 2023-09-29T22:28:01.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009181e34586daa362-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.625 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.626 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.626 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.626 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:28:01.626 DEBUG:boto:Method: GET 2023-09-29T22:28:01.626 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.626 DEBUG:boto:Data: 2023-09-29T22:28:01.626 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.626 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.626 DEBUG:boto:Params: {} 2023-09-29T22:28:01.627 DEBUG:boto:Token: None 2023-09-29T22:28:01.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:28:01.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0= 2023-09-29T22:28:01.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Lt2slnYyIQ3jrdszdkbgW/t2X0=', 'Content-Length': '0'} 2023-09-29T22:28:01.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006c5a52418a37100-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.630 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.630 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.630 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.630 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.630 DEBUG:boto:Method: GET 2023-09-29T22:28:01.630 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:28:01.630 DEBUG:boto:Data: 2023-09-29T22:28:01.631 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.631 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.631 DEBUG:boto:Params: {} 2023-09-29T22:28:01.631 DEBUG:boto:Token: None 2023-09-29T22:28:01.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed6a24e2719256a2-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.636 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.636 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.637 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.637 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.637 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.637 DEBUG:boto:Method: GET 2023-09-29T22:28:01.637 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:28:01.637 DEBUG:boto:Data: 2023-09-29T22:28:01.637 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.637 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.637 DEBUG:boto:Params: {} 2023-09-29T22:28:01.638 DEBUG:boto:Token: None 2023-09-29T22:28:01.638 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae69f3d9226b47d1-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.642 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.642 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.643 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.643 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.643 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:28:01.643 DEBUG:boto:Data: 2023-09-29T22:28:01.643 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.643 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.643 DEBUG:boto:Params: {} 2023-09-29T22:28:01.643 DEBUG:boto:Token: None 2023-09-29T22:28:01.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c 2023-09-29T22:28:01.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PBnKjaFPFbJ8kXDRX87DGpsgxPo= 2023-09-29T22:28:01.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PBnKjaFPFbJ8kXDRX87DGpsgxPo=', 'Content-Length': '0'} 2023-09-29T22:28:01.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000181f91aa20aab52e-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.646 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.646 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.646 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.646 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:28:01.646 DEBUG:boto:Data: 2023-09-29T22:28:01.647 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.647 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.647 DEBUG:boto:Params: {} 2023-09-29T22:28:01.647 DEBUG:boto:Token: None 2023-09-29T22:28:01.647 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c 2023-09-29T22:28:01.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PBnKjaFPFbJ8kXDRX87DGpsgxPo= 2023-09-29T22:28:01.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PBnKjaFPFbJ8kXDRX87DGpsgxPo=', 'Content-Length': '0'} 2023-09-29T22:28:01.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f8137849ac431f2-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:28:01.650 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.650 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.650 DEBUG:boto:Method: GET 2023-09-29T22:28:01.650 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:28:01.650 DEBUG:boto:Data: 2023-09-29T22:28:01.650 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.651 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.651 DEBUG:boto:Params: {} 2023-09-29T22:28:01.651 DEBUG:boto:Token: None 2023-09-29T22:28:01.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c 2023-09-29T22:28:01.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ar0HixLEsAm43VOEqTihMY7+NV0= 2023-09-29T22:28:01.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ar0HixLEsAm43VOEqTihMY7+NV0=', 'Content-Length': '0'} 2023-09-29T22:28:01.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4ae8181e6b96a0a-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.695 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:28:01.695 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:28:01.695 DEBUG:boto:Method: GET 2023-09-29T22:28:01.695 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:28:01.695 DEBUG:boto:Data: 2023-09-29T22:28:01.695 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.695 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.695 DEBUG:boto:Params: {} 2023-09-29T22:28:01.696 DEBUG:boto:Token: None 2023-09-29T22:28:01.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/c 2023-09-29T22:28:01.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ar0HixLEsAm43VOEqTihMY7+NV0= 2023-09-29T22:28:01.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ar0HixLEsAm43VOEqTihMY7+NV0=', 'Content-Length': '0'} 2023-09-29T22:28:01.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830af52afca23163-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.699 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.699 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.699 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.700 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.700 DEBUG:boto:Method: GET 2023-09-29T22:28:01.700 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:28:01.700 DEBUG:boto:Data: 2023-09-29T22:28:01.700 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.700 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.700 DEBUG:boto:Params: {} 2023-09-29T22:28:01.700 DEBUG:boto:Token: None 2023-09-29T22:28:01.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce8c107660ecca1b-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.705 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.705 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.706 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.706 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.706 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:28:01.706 DEBUG:boto:Method: GET 2023-09-29T22:28:01.706 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:28:01.706 DEBUG:boto:Data: 2023-09-29T22:28:01.706 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.706 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.706 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.707 DEBUG:boto:Params: {} 2023-09-29T22:28:01.707 DEBUG:boto:Token: None 2023-09-29T22:28:01.707 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.707 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a13f060bdc57b70-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.711 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:28:01.712 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.712 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.712 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.712 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.712 DEBUG:boto:Method: GET 2023-09-29T22:28:01.712 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.713 DEBUG:boto:Data: 2023-09-29T22:28:01.713 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.713 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.713 DEBUG:boto:Params: {} 2023-09-29T22:28:01.713 DEBUG:boto:Token: None 2023-09-29T22:28:01.713 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc= 2023-09-29T22:28:01.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc=', 'Content-Length': '0'} 2023-09-29T22:28:01.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aeb59a1f6e68e8f-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.716 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.716 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.716 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.716 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.717 DEBUG:boto:Method: GET 2023-09-29T22:28:01.717 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.717 DEBUG:boto:Data: 2023-09-29T22:28:01.717 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.717 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.717 DEBUG:boto:Params: {} 2023-09-29T22:28:01.717 DEBUG:boto:Token: None 2023-09-29T22:28:01.717 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.717 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc= 2023-09-29T22:28:01.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc=', 'Content-Length': '0'} 2023-09-29T22:28:01.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000463d65473893ea2a-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.720 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.720 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.720 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.720 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.721 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.721 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.721 DEBUG:boto:Data: 2023-09-29T22:28:01.721 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.721 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.721 DEBUG:boto:Params: {} 2023-09-29T22:28:01.721 DEBUG:boto:Token: None 2023-09-29T22:28:01.721 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/P/IiGHnhaLVTTQOdSzavpQESKs= 2023-09-29T22:28:01.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/P/IiGHnhaLVTTQOdSzavpQESKs=', 'Content-Length': '0'} 2023-09-29T22:28:01.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c185336b19ea85f-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.724 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.724 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.724 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.724 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.724 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.725 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.725 DEBUG:boto:Data: 2023-09-29T22:28:01.725 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.725 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.725 DEBUG:boto:Params: {} 2023-09-29T22:28:01.725 DEBUG:boto:Token: None 2023-09-29T22:28:01.725 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/P/IiGHnhaLVTTQOdSzavpQESKs= 2023-09-29T22:28:01.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/P/IiGHnhaLVTTQOdSzavpQESKs=', 'Content-Length': '0'} 2023-09-29T22:28:01.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f194003f10ec1924-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:28:01.728 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.728 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.728 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.729 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.729 DEBUG:boto:Method: GET 2023-09-29T22:28:01.729 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.729 DEBUG:boto:Data: 2023-09-29T22:28:01.729 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.729 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.729 DEBUG:boto:Params: {} 2023-09-29T22:28:01.730 DEBUG:boto:Token: None 2023-09-29T22:28:01.730 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc= 2023-09-29T22:28:01.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc=', 'Content-Length': '0'} 2023-09-29T22:28:01.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7070eeb84cb8f2-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.774 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.774 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.775 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.775 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:28:01.775 DEBUG:boto:Method: GET 2023-09-29T22:28:01.775 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.775 DEBUG:boto:Data: 2023-09-29T22:28:01.775 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.775 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.775 DEBUG:boto:Params: {} 2023-09-29T22:28:01.776 DEBUG:boto:Token: None 2023-09-29T22:28:01.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:28:01.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc= 2023-09-29T22:28:01.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XMkag4G9ujC+pHaHgi/6lkLHbLc=', 'Content-Length': '0'} 2023-09-29T22:28:01.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e22484998e6c9af8-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.779 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.779 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.779 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.779 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.780 DEBUG:boto:Method: GET 2023-09-29T22:28:01.780 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:28:01.780 DEBUG:boto:Data: 2023-09-29T22:28:01.780 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.780 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.780 DEBUG:boto:Params: {} 2023-09-29T22:28:01.780 DEBUG:boto:Token: None 2023-09-29T22:28:01.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dc793ed5c650400-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.785 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.786 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.786 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.786 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.786 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.786 DEBUG:boto:Method: GET 2023-09-29T22:28:01.786 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:28:01.787 DEBUG:boto:Data: 2023-09-29T22:28:01.787 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.787 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.787 DEBUG:boto:Params: {} 2023-09-29T22:28:01.787 DEBUG:boto:Token: None 2023-09-29T22:28:01.787 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.787 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb1ad558d129f1c-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.792 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.792 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.792 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.792 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.792 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:28:01.793 DEBUG:boto:Data: 2023-09-29T22:28:01.793 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.793 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.793 DEBUG:boto:Params: {} 2023-09-29T22:28:01.793 DEBUG:boto:Token: None 2023-09-29T22:28:01.793 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d 2023-09-29T22:28:01.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CgOoC0porPa75y/l+Lx0NdIXJrE= 2023-09-29T22:28:01.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CgOoC0porPa75y/l+Lx0NdIXJrE=', 'Content-Length': '0'} 2023-09-29T22:28:01.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000479dfcac0bf85dc6-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.796 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.796 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.796 DEBUG:boto:Method: HEAD 2023-09-29T22:28:01.796 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:28:01.796 DEBUG:boto:Data: 2023-09-29T22:28:01.796 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.797 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.797 DEBUG:boto:Params: {} 2023-09-29T22:28:01.797 DEBUG:boto:Token: None 2023-09-29T22:28:01.797 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d 2023-09-29T22:28:01.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CgOoC0porPa75y/l+Lx0NdIXJrE= 2023-09-29T22:28:01.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CgOoC0porPa75y/l+Lx0NdIXJrE=', 'Content-Length': '0'} 2023-09-29T22:28:01.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070e5775a8e698bb4-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:28:01.800 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.800 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.800 DEBUG:boto:Method: GET 2023-09-29T22:28:01.800 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:28:01.801 DEBUG:boto:Data: 2023-09-29T22:28:01.801 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.801 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.801 DEBUG:boto:Params: {} 2023-09-29T22:28:01.801 DEBUG:boto:Token: None 2023-09-29T22:28:01.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d 2023-09-29T22:28:01.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lMEAk4b42pXF7WEzyTmsRROCA8Y= 2023-09-29T22:28:01.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lMEAk4b42pXF7WEzyTmsRROCA8Y=', 'Content-Length': '0'} 2023-09-29T22:28:01.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b3711c50c055cb2-0065174f71-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.846 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:28:01.846 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:28:01.847 DEBUG:boto:Method: GET 2023-09-29T22:28:01.847 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:28:01.847 DEBUG:boto:Data: 2023-09-29T22:28:01.847 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.847 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.847 DEBUG:boto:Params: {} 2023-09-29T22:28:01.847 DEBUG:boto:Token: None 2023-09-29T22:28:01.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/d 2023-09-29T22:28:01.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lMEAk4b42pXF7WEzyTmsRROCA8Y= 2023-09-29T22:28:01.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lMEAk4b42pXF7WEzyTmsRROCA8Y=', 'Content-Length': '0'} 2023-09-29T22:28:01.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000020ec50da71b2229-0065174f71-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.851 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.851 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.851 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.851 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.851 DEBUG:boto:Method: GET 2023-09-29T22:28:01.851 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:28:01.851 DEBUG:boto:Data: 2023-09-29T22:28:01.852 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:28:01.852 DEBUG:boto:Port: 8001 2023-09-29T22:28:01.852 DEBUG:boto:Params: {} 2023-09-29T22:28:01.852 DEBUG:boto:Token: None 2023-09-29T22:28:01.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003522fe97c46619bd-0065174f71-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.857 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.857 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:01.857 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:01.857 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.857 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:28:01.857 DEBUG:boto:Method: GET 2023-09-29T22:28:01.857 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:28:01.858 DEBUG:boto:Data: 2023-09-29T22:28:01.858 DEBUG:boto:Headers: {} 2023-09-29T22:28:01.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:01.858 DEBUG:boto:Port: 8000 2023-09-29T22:28:01.858 DEBUG:boto:Params: {} 2023-09-29T22:28:01.858 DEBUG:boto:Token: None 2023-09-29T22:28:01.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:28:01 GMT /hcnphz-56/ 2023-09-29T22:28:01.858 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50= 2023-09-29T22:28:01.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ow3rJ46Y9NPFvSFnM4yDc3nFB50=', 'Content-Length': '0'} 2023-09-29T22:28:01.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d3ca66e9cb45fbb-0065174f71-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:28:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:01.906 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:28:01.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-56 zones={a2, a1} 2023-09-29T22:28:01.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:28:01.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:28:01.907 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:28:01.955 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:01.955 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.029 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:28:02.030 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-56 2023-09-29T22:28:02.038 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-09-29T22:28:02.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:28:02.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-56', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:28:02.113 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-56 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:28:02.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.233 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:28:02.234 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-56 2023-09-29T22:28:02.236 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-09-29T22:28:02.293 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:02.293 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:02.293 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.293 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:28:02.293 DEBUG:boto:Data: 2023-09-29T22:28:02.293 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.294 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.294 DEBUG:boto:Params: {} 2023-09-29T22:28:02.294 DEBUG:boto:Token: None 2023-09-29T22:28:02.294 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/ 2023-09-29T22:28:02.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo= 2023-09-29T22:28:02.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo=', 'Content-Length': '0'} 2023-09-29T22:28:02.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f7b9da2ab660e1ab-0065174f72-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.297 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:28:02.298 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:28:02.298 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.298 DEBUG:boto:Path: /hcnphz-56/e 2023-09-29T22:28:02.298 DEBUG:boto:Data: 2023-09-29T22:28:02.298 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:28:02.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.298 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.298 DEBUG:boto:Params: {} 2023-09-29T22:28:02.298 DEBUG:boto:Token: None 2023-09-29T22:28:02.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/e 2023-09-29T22:28:02.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TJ0d/1kyKU9CC+gRp25MxJtqubU= 2023-09-29T22:28:02.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TJ0d/1kyKU9CC+gRp25MxJtqubU='} 2023-09-29T22:28:02.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000992192d671da9e1e-0065174f72-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.309 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:02.309 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:02.309 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.309 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:28:02.309 DEBUG:boto:Data: 2023-09-29T22:28:02.309 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.309 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.310 DEBUG:boto:Params: {} 2023-09-29T22:28:02.310 DEBUG:boto:Token: None 2023-09-29T22:28:02.310 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/ 2023-09-29T22:28:02.310 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo= 2023-09-29T22:28:02.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo=', 'Content-Length': '0'} 2023-09-29T22:28:02.312 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000921fea961246f2c9-0065174f72-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.313 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:28:02.313 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:28:02.313 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.313 DEBUG:boto:Path: /hcnphz-56/f 2023-09-29T22:28:02.313 DEBUG:boto:Data: 2023-09-29T22:28:02.313 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:28:02.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.313 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.313 DEBUG:boto:Params: {} 2023-09-29T22:28:02.314 DEBUG:boto:Token: None 2023-09-29T22:28:02.314 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/f 2023-09-29T22:28:02.314 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ktzf0PB6TnLl0Iq3huJy6Ojs3YI= 2023-09-29T22:28:02.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ktzf0PB6TnLl0Iq3huJy6Ojs3YI='} 2023-09-29T22:28:02.323 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b1aee251da76b63-0065174f72-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.323 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:02.323 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:02.323 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.323 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:28:02.323 DEBUG:boto:Data: 2023-09-29T22:28:02.323 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.323 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.324 DEBUG:boto:Params: {} 2023-09-29T22:28:02.324 DEBUG:boto:Token: None 2023-09-29T22:28:02.324 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/ 2023-09-29T22:28:02.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo= 2023-09-29T22:28:02.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo=', 'Content-Length': '0'} 2023-09-29T22:28:02.326 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f59768e478750d99-0065174f72-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.327 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:28:02.327 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:28:02.327 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.327 DEBUG:boto:Path: /hcnphz-56/g 2023-09-29T22:28:02.327 DEBUG:boto:Data: 2023-09-29T22:28:02.327 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:28:02.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.327 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.327 DEBUG:boto:Params: {} 2023-09-29T22:28:02.328 DEBUG:boto:Token: None 2023-09-29T22:28:02.328 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/g 2023-09-29T22:28:02.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JMWIf6sPKjDiI5NaFzFVMttj+u4= 2023-09-29T22:28:02.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JMWIf6sPKjDiI5NaFzFVMttj+u4='} 2023-09-29T22:28:02.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000f2cb42ec3203227-0065174f72-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.370 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:28:02.371 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:28:02.371 DEBUG:boto:Method: HEAD 2023-09-29T22:28:02.371 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:28:02.371 DEBUG:boto:Data: 2023-09-29T22:28:02.371 DEBUG:boto:Headers: {} 2023-09-29T22:28:02.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.371 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.371 DEBUG:boto:Params: {} 2023-09-29T22:28:02.371 DEBUG:boto:Token: None 2023-09-29T22:28:02.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/ 2023-09-29T22:28:02.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo= 2023-09-29T22:28:02.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1u/L7FlNSF8OS2NaSEWsETFXTo=', 'Content-Length': '0'} 2023-09-29T22:28:02.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045de98c21a84bc57-0065174f72-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.375 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:28:02.375 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:28:02.376 DEBUG:boto:Method: PUT 2023-09-29T22:28:02.376 DEBUG:boto:Path: /hcnphz-56/h 2023-09-29T22:28:02.376 DEBUG:boto:Data: 2023-09-29T22:28:02.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:28:02.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:28:02.376 DEBUG:boto:Port: 8000 2023-09-29T22:28:02.376 DEBUG:boto:Params: {} 2023-09-29T22:28:02.376 DEBUG:boto:Token: None 2023-09-29T22:28:02.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:28:02 GMT /hcnphz-56/h 2023-09-29T22:28:02.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TdNNL2JdJHW2BM8wVofdy6+maWE= 2023-09-29T22:28:02.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:28:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TdNNL2JdJHW2BM8wVofdy6+maWE='} 2023-09-29T22:28:02.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c0cc78f25d182ff-0065174f72-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:28:02.385 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-56 2023-09-29T22:28:02.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:28:02.386 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-56 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:28:02.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:28:02.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:28:02.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:02.490+0000 7f027fdd0f40 1 waiting to reach latest gen 2, on 0.. 2023-09-29T22:28:07.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:07.490+0000 7f027fdd0f40 1 waiting to reach latest gen 2, on 0.. 2023-09-29T22:28:12.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:12.494+0000 7f027fdd0f40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:28:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:17.494+0000 7f027fdd0f40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:28:22.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:22.493+0000 7f027fdd0f40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:28:27.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:27.493+0000 7f027fdd0f40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:28:32.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:32.501+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:28:32.502 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:28:32.502 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:28:37.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:37.501+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:28:37.503 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:28:37.503 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:28:42.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:42.505+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:28:42.504 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:28:42.504 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:28:47.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:47.505+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:28:47.505 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:28:47.506 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:28:52.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:52.505+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:28:52.507 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:28:52.507 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:28:57.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:28:57.508+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:28:57.508 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:28:57.508 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:02.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:02.508+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:02.509 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:29:02.510 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:07.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:07.508+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:07.511 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , ] 2023-09-29T22:29:07.511 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:12.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:12.512+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:12.512 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, ] 2023-09-29T22:29:12.512 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:17.512+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:17.513 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, ] 2023-09-29T22:29:17.514 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:22.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:22.512+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:22.515 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, ] 2023-09-29T22:29:22.515 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:27.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:27.512+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:27.516 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, ] 2023-09-29T22:29:27.516 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:32.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:32.512+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:32.513 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:32.514 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:37.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:37.511+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:37.515 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:37.515 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:42.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:42.515+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:42.517 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:42.517 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:47.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:47.515+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:47.518 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:47.518 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:52.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:52.515+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:52.520 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:52.520 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:57.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:29:57.519+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:29:57.523 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:29:57.523 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:02.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:02.519+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:02.522 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:02.522 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:07.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:07.519+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:07.524 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:07.524 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:12.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:12.522+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:12.525 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:12.525 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:17.522+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:17.527 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:17.527 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:22.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:22.526+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:22.528 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:22.528 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:27.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:27.526+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:27.530 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:27.530 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:32.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:32.526+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:32.531 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:32.531 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:37.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:37.530+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:37.533 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:37.533 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:42.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:42.534+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:42.534 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:42.534 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:47.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:47.533+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:47.535 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:47.536 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:52.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:52.533+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:52.537 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:52.537 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:57.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:30:57.537+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:30:57.538 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:30:57.539 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:02.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:02.537+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:31:02.539 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:02.539 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:07.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:07.541+0000 7f027fdd0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:31:07.541 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:07.541 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:12.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:12.541+0000 7f027fdd0f40 1 bucket sync caught up with source: 2023-09-29T22:31:12.543 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:12.543 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.42.6, 00000000002.28.6, 00000000002.40.6] 2023-09-29T22:31:12.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:12.541+0000 7f027fdd0f40 0 bucket checkpoint complete 2023-09-29T22:31:12.573 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-56 2023-09-29T22:31:12.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:31:12.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-56', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:31:12.574 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-56 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:31:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:12.625 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:12.688 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:12.685+0000 7f2516378f40 1 waiting for incremental sync to catch up: 2023-09-29T22:31:12.688 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.113.6, , , 00000000001.202.6, 00000000001.149.6, , , 00000000002.544.6, , 00000000001.125.6] 2023-09-29T22:31:12.688 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.113.6, , , 00000000001.202.6, 00000000001.149.6, , , 00000000002.544.6, , 00000000002.155.6] 2023-09-29T22:31:17.690 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:17.689+0000 7f2516378f40 1 waiting for incremental sync to catch up: 2023-09-29T22:31:17.691 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.113.6, , , 00000000001.202.6, 00000000001.149.6, , , 00000000002.544.6, , 00000000001.125.6] 2023-09-29T22:31:17.691 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.113.6, , , 00000000001.202.6, 00000000001.149.6, , , 00000000002.544.6, , 00000000002.155.6] 2023-09-29T22:31:22.693 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:22.693+0000 7f2516378f40 1 bucket sync caught up with source: 2023-09-29T22:31:22.694 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.113.6, , , 00000000001.202.6, 00000000001.149.6, , , 00000000002.544.6, , 00000000002.155.6] 2023-09-29T22:31:22.694 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.113.6, , , 00000000001.202.6, 00000000001.149.6, , , 00000000002.544.6, , 00000000002.155.6] 2023-09-29T22:31:22.694 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:22.693+0000 7f2516378f40 0 bucket checkpoint complete 2023-09-29T22:31:22.722 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-56 zones={a2, a1} 2023-09-29T22:31:22.722 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.722 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.722 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.722 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:31:22.723 DEBUG:boto:Data: 2023-09-29T22:31:22.723 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.723 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.723 DEBUG:boto:Params: {} 2023-09-29T22:31:22.723 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:31:22.724 DEBUG:boto:Token: None 2023-09-29T22:31:22.724 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/ 2023-09-29T22:31:22.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t2Z/E4H7gGnya4MYuA2F9v4MC7E= 2023-09-29T22:31:22.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t2Z/E4H7gGnya4MYuA2F9v4MC7E=', 'Content-Length': '0'} 2023-09-29T22:31:22.729 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b78b7ecff039e195-006517503a-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.730 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.730 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.730 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.730 DEBUG:boto:Path: /hcnphz-56/ 2023-09-29T22:31:22.730 DEBUG:boto:Data: 2023-09-29T22:31:22.730 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.730 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.731 DEBUG:boto:Params: {} 2023-09-29T22:31:22.731 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:31:22.731 DEBUG:boto:Token: None 2023-09-29T22:31:22.731 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/ 2023-09-29T22:31:22.731 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t2Z/E4H7gGnya4MYuA2F9v4MC7E= 2023-09-29T22:31:22.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t2Z/E4H7gGnya4MYuA2F9v4MC7E=', 'Content-Length': '0'} 2023-09-29T22:31:22.735 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008a4f0b34ed97b5bb-006517503a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:31:22.736 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.736 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.736 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.736 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.736 DEBUG:boto:Method: GET 2023-09-29T22:31:22.737 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.737 DEBUG:boto:Data: 2023-09-29T22:31:22.737 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.737 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.737 DEBUG:boto:Params: {} 2023-09-29T22:31:22.737 DEBUG:boto:Token: None 2023-09-29T22:31:22.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/?versions 2023-09-29T22:31:22.737 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0= 2023-09-29T22:31:22.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0=', 'Content-Length': '0'} 2023-09-29T22:31:22.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009428cdcbb51d862-006517503a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.742 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.743 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:31:22.743 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:31:22.743 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:31:22.744 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:31:22.744 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:31:22.744 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:31:22.744 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:31:22.744 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:31:22.744 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:31:22.744 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.744 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.744 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.745 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.745 DEBUG:boto:Method: GET 2023-09-29T22:31:22.745 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.745 DEBUG:boto:Data: 2023-09-29T22:31:22.745 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.745 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.746 DEBUG:boto:Params: {} 2023-09-29T22:31:22.746 DEBUG:boto:Token: None 2023-09-29T22:31:22.746 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/?versions 2023-09-29T22:31:22.746 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0= 2023-09-29T22:31:22.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0=', 'Content-Length': '0'} 2023-09-29T22:31:22.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000053e0b0c0b967376c-006517503a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.750 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.750 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:31:22.751 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:31:22.752 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.752 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.752 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.752 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.752 DEBUG:boto:Method: GET 2023-09-29T22:31:22.752 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.752 DEBUG:boto:Data: 2023-09-29T22:31:22.753 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.753 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.753 DEBUG:boto:Params: {} 2023-09-29T22:31:22.753 DEBUG:boto:Token: None 2023-09-29T22:31:22.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/?versions 2023-09-29T22:31:22.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0= 2023-09-29T22:31:22.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0=', 'Content-Length': '0'} 2023-09-29T22:31:22.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a3a6b2219b60c70-006517503a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.758 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.758 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.759 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.759 DEBUG:boto:path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.759 DEBUG:boto:auth_path=/hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.759 DEBUG:boto:Method: GET 2023-09-29T22:31:22.759 DEBUG:boto:Path: /hcnphz-56/?versions&max-keys=999 2023-09-29T22:31:22.759 DEBUG:boto:Data: 2023-09-29T22:31:22.759 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.760 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.760 DEBUG:boto:Params: {} 2023-09-29T22:31:22.760 DEBUG:boto:Token: None 2023-09-29T22:31:22.761 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/?versions 2023-09-29T22:31:22.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0= 2023-09-29T22:31:22.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ylue5Sa2C5ZRCkzfi2+bSpzx0=', 'Content-Length': '0'} 2023-09-29T22:31:22.765 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000565d92bf5925c90a-006517503a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.765 DEBUG:boto:b'hcnphz-56999falseanulltrue2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:31:22.766 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.767 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.767 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.767 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.767 DEBUG:boto:Method: GET 2023-09-29T22:31:22.767 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.767 DEBUG:boto:Data: 2023-09-29T22:31:22.767 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.767 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.767 DEBUG:boto:Params: {} 2023-09-29T22:31:22.768 DEBUG:boto:Token: None 2023-09-29T22:31:22.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4= 2023-09-29T22:31:22.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4=', 'Content-Length': '0'} 2023-09-29T22:31:22.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4cdef16e4e34c6b-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.771 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.772 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.772 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.772 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.772 DEBUG:boto:Method: GET 2023-09-29T22:31:22.772 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.772 DEBUG:boto:Data: 2023-09-29T22:31:22.772 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.773 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.773 DEBUG:boto:Params: {} 2023-09-29T22:31:22.773 DEBUG:boto:Token: None 2023-09-29T22:31:22.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4= 2023-09-29T22:31:22.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4=', 'Content-Length': '0'} 2023-09-29T22:31:22.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1d3274a3427a11a-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.776 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.777 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.777 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.777 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.777 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.777 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.777 DEBUG:boto:Data: 2023-09-29T22:31:22.778 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.778 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.778 DEBUG:boto:Params: {} 2023-09-29T22:31:22.778 DEBUG:boto:Token: None 2023-09-29T22:31:22.779 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.779 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7ucA/ecjcpdHaSTILHR9UcnpFJo= 2023-09-29T22:31:22.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7ucA/ecjcpdHaSTILHR9UcnpFJo=', 'Content-Length': '0'} 2023-09-29T22:31:22.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc3cc365819140b4-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.782 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.782 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.782 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.782 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.782 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.782 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.783 DEBUG:boto:Data: 2023-09-29T22:31:22.783 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.783 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.783 DEBUG:boto:Params: {} 2023-09-29T22:31:22.783 DEBUG:boto:Token: None 2023-09-29T22:31:22.783 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7ucA/ecjcpdHaSTILHR9UcnpFJo= 2023-09-29T22:31:22.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7ucA/ecjcpdHaSTILHR9UcnpFJo=', 'Content-Length': '0'} 2023-09-29T22:31:22.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaec2ad7c2346be5-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:31:22.786 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.787 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.787 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.787 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.787 DEBUG:boto:Method: GET 2023-09-29T22:31:22.787 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.787 DEBUG:boto:Data: 2023-09-29T22:31:22.787 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.788 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.788 DEBUG:boto:Params: {} 2023-09-29T22:31:22.788 DEBUG:boto:Token: None 2023-09-29T22:31:22.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4= 2023-09-29T22:31:22.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4=', 'Content-Length': '0'} 2023-09-29T22:31:22.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7cdbcf4692c85e6-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.834 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.835 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.835 DEBUG:boto:path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.835 DEBUG:boto:auth_path=/hcnphz-56/a?versionId=null 2023-09-29T22:31:22.835 DEBUG:boto:Method: GET 2023-09-29T22:31:22.835 DEBUG:boto:Path: /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.835 DEBUG:boto:Data: 2023-09-29T22:31:22.835 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.836 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.836 DEBUG:boto:Params: {} 2023-09-29T22:31:22.836 DEBUG:boto:Token: None 2023-09-29T22:31:22.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a?versionId=null 2023-09-29T22:31:22.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4= 2023-09-29T22:31:22.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZtLKtlx3r9iym/m4f4Ir/8Ihy4=', 'Content-Length': '0'} 2023-09-29T22:31:22.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ab1f623a867aa91-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.840 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.840 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.840 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.840 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.840 DEBUG:boto:Method: GET 2023-09-29T22:31:22.840 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:31:22.840 DEBUG:boto:Data: 2023-09-29T22:31:22.840 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.841 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.841 DEBUG:boto:Params: {} 2023-09-29T22:31:22.841 DEBUG:boto:Token: None 2023-09-29T22:31:22.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/ 2023-09-29T22:31:22.841 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI= 2023-09-29T22:31:22.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI=', 'Content-Length': '0'} 2023-09-29T22:31:22.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fdcb42addfe426c2-006517503a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.846 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.846 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.847 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.847 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.847 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.847 DEBUG:boto:Method: GET 2023-09-29T22:31:22.847 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:31:22.847 DEBUG:boto:Data: 2023-09-29T22:31:22.847 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.848 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.848 DEBUG:boto:Params: {} 2023-09-29T22:31:22.848 DEBUG:boto:Token: None 2023-09-29T22:31:22.848 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/ 2023-09-29T22:31:22.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI= 2023-09-29T22:31:22.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI=', 'Content-Length': '0'} 2023-09-29T22:31:22.852 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051b90769ce48bd10-006517503a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.852 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.853 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.853 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.853 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.853 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:31:22.854 DEBUG:boto:Data: 2023-09-29T22:31:22.854 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.854 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.854 DEBUG:boto:Params: {} 2023-09-29T22:31:22.854 DEBUG:boto:Token: None 2023-09-29T22:31:22.854 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a 2023-09-29T22:31:22.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QjTuRdY/lk2RN3kVqAzdaHgqcHw= 2023-09-29T22:31:22.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QjTuRdY/lk2RN3kVqAzdaHgqcHw=', 'Content-Length': '0'} 2023-09-29T22:31:22.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089900c1179d2ca34-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.857 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.858 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.858 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.858 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:31:22.858 DEBUG:boto:Data: 2023-09-29T22:31:22.858 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.858 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.858 DEBUG:boto:Params: {} 2023-09-29T22:31:22.859 DEBUG:boto:Token: None 2023-09-29T22:31:22.859 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a 2023-09-29T22:31:22.859 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QjTuRdY/lk2RN3kVqAzdaHgqcHw= 2023-09-29T22:31:22.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QjTuRdY/lk2RN3kVqAzdaHgqcHw=', 'Content-Length': '0'} 2023-09-29T22:31:22.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c2adebed461e7d1-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:31:22.862 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.862 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.862 DEBUG:boto:Method: GET 2023-09-29T22:31:22.862 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:31:22.862 DEBUG:boto:Data: 2023-09-29T22:31:22.863 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.863 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.863 DEBUG:boto:Params: {} 2023-09-29T22:31:22.863 DEBUG:boto:Token: None 2023-09-29T22:31:22.863 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a 2023-09-29T22:31:22.863 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fj5608p30JXz2HpmGUXAR6J056M= 2023-09-29T22:31:22.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fj5608p30JXz2HpmGUXAR6J056M=', 'Content-Length': '0'} 2023-09-29T22:31:22.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099d87c70c0443503-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.866 DEBUG:boto:path=/hcnphz-56/a 2023-09-29T22:31:22.867 DEBUG:boto:auth_path=/hcnphz-56/a 2023-09-29T22:31:22.867 DEBUG:boto:Method: GET 2023-09-29T22:31:22.867 DEBUG:boto:Path: /hcnphz-56/a 2023-09-29T22:31:22.867 DEBUG:boto:Data: 2023-09-29T22:31:22.867 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.867 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.867 DEBUG:boto:Params: {} 2023-09-29T22:31:22.867 DEBUG:boto:Token: None 2023-09-29T22:31:22.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/a 2023-09-29T22:31:22.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fj5608p30JXz2HpmGUXAR6J056M= 2023-09-29T22:31:22.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fj5608p30JXz2HpmGUXAR6J056M=', 'Content-Length': '0'} 2023-09-29T22:31:22.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5288a9a1ca8ac1e-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.871 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.871 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.871 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.871 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.871 DEBUG:boto:Method: GET 2023-09-29T22:31:22.871 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:31:22.871 DEBUG:boto:Data: 2023-09-29T22:31:22.871 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.872 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.872 DEBUG:boto:Params: {} 2023-09-29T22:31:22.872 DEBUG:boto:Token: None 2023-09-29T22:31:22.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/ 2023-09-29T22:31:22.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI= 2023-09-29T22:31:22.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI=', 'Content-Length': '0'} 2023-09-29T22:31:22.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5051da051a72d4c-006517503a-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.918 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.919 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.919 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.919 DEBUG:boto:path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.919 DEBUG:boto:auth_path=/hcnphz-56/?prefix=a 2023-09-29T22:31:22.919 DEBUG:boto:Method: GET 2023-09-29T22:31:22.919 DEBUG:boto:Path: /hcnphz-56/?prefix=a 2023-09-29T22:31:22.919 DEBUG:boto:Data: 2023-09-29T22:31:22.920 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.920 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.920 DEBUG:boto:Params: {} 2023-09-29T22:31:22.920 DEBUG:boto:Token: None 2023-09-29T22:31:22.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/ 2023-09-29T22:31:22.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI= 2023-09-29T22:31:22.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PMGz8VHT1GlfYAIzsqawBP4FFnI=', 'Content-Length': '0'} 2023-09-29T22:31:22.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000763bb07ab891b613-006517503a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.924 DEBUG:boto:b'hcnphz-56a1000falsea2023-09-29T22:27:40.777Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:22.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:31:22.925 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:22.925 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:22.925 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.925 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.925 DEBUG:boto:Method: GET 2023-09-29T22:31:22.925 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.925 DEBUG:boto:Data: 2023-09-29T22:31:22.926 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.926 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.926 DEBUG:boto:Params: {} 2023-09-29T22:31:22.926 DEBUG:boto:Token: None 2023-09-29T22:31:22.926 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.926 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ= 2023-09-29T22:31:22.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ=', 'Content-Length': '0'} 2023-09-29T22:31:22.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcad13441ae7749e-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.929 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:22.930 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:22.930 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.930 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.930 DEBUG:boto:Method: GET 2023-09-29T22:31:22.930 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.930 DEBUG:boto:Data: 2023-09-29T22:31:22.930 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.931 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.931 DEBUG:boto:Params: {} 2023-09-29T22:31:22.931 DEBUG:boto:Token: None 2023-09-29T22:31:22.931 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ= 2023-09-29T22:31:22.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ=', 'Content-Length': '0'} 2023-09-29T22:31:22.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba8fd27b8666ca6-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.974 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:22.974 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:22.975 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.975 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.975 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.975 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.975 DEBUG:boto:Data: 2023-09-29T22:31:22.975 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.975 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.975 DEBUG:boto:Params: {} 2023-09-29T22:31:22.975 DEBUG:boto:Token: None 2023-09-29T22:31:22.976 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ICYfMWJZHUvqmKoXr9Ut9wGZC0= 2023-09-29T22:31:22.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ICYfMWJZHUvqmKoXr9Ut9wGZC0=', 'Content-Length': '0'} 2023-09-29T22:31:22.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4d579c003285a0-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.979 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:22.979 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:22.979 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.979 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.980 DEBUG:boto:Method: HEAD 2023-09-29T22:31:22.980 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.980 DEBUG:boto:Data: 2023-09-29T22:31:22.980 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.981 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.981 DEBUG:boto:Params: {} 2023-09-29T22:31:22.981 DEBUG:boto:Token: None 2023-09-29T22:31:22.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ICYfMWJZHUvqmKoXr9Ut9wGZC0= 2023-09-29T22:31:22.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ICYfMWJZHUvqmKoXr9Ut9wGZC0=', 'Content-Length': '0'} 2023-09-29T22:31:22.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000619eb3856dbe9fd1-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:31:22.985 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:22.985 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:22.986 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.986 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.986 DEBUG:boto:Method: GET 2023-09-29T22:31:22.986 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.986 DEBUG:boto:Data: 2023-09-29T22:31:22.987 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:22.987 DEBUG:boto:Port: 8001 2023-09-29T22:31:22.987 DEBUG:boto:Params: {} 2023-09-29T22:31:22.987 DEBUG:boto:Token: None 2023-09-29T22:31:22.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ= 2023-09-29T22:31:22.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ=', 'Content-Length': '0'} 2023-09-29T22:31:22.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec752d7e502aa6bf-006517503a-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.991 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:22.991 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:22.992 DEBUG:boto:path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.992 DEBUG:boto:auth_path=/hcnphz-56/b?versionId=null 2023-09-29T22:31:22.992 DEBUG:boto:Method: GET 2023-09-29T22:31:22.992 DEBUG:boto:Path: /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.992 DEBUG:boto:Data: 2023-09-29T22:31:22.993 DEBUG:boto:Headers: {} 2023-09-29T22:31:22.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:22.993 DEBUG:boto:Port: 8000 2023-09-29T22:31:22.993 DEBUG:boto:Params: {} 2023-09-29T22:31:22.993 DEBUG:boto:Token: None 2023-09-29T22:31:22.994 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:22 GMT /hcnphz-56/b?versionId=null 2023-09-29T22:31:22.994 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ= 2023-09-29T22:31:22.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:22 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AYxllihNiV8DG8p5gJod8H2GWGQ=', 'Content-Length': '0'} 2023-09-29T22:31:22.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003312c41e5dc90400-006517503a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:22 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:22.998 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:22.998 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:22.998 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:31:22.998 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:31:22.999 DEBUG:boto:Method: GET 2023-09-29T22:31:22.999 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:31:22.999 DEBUG:boto:Data: 2023-09-29T22:31:22.999 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.000 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.000 DEBUG:boto:Params: {} 2023-09-29T22:31:23.000 DEBUG:boto:Token: None 2023-09-29T22:31:23.001 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.001 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094e4e223a9e3ed07-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.046 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.047 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.047 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.047 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:31:23.047 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:31:23.047 DEBUG:boto:Method: GET 2023-09-29T22:31:23.048 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:31:23.048 DEBUG:boto:Data: 2023-09-29T22:31:23.048 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.048 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.048 DEBUG:boto:Params: {} 2023-09-29T22:31:23.048 DEBUG:boto:Token: None 2023-09-29T22:31:23.049 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6132e29722d5c72-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.053 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.054 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:23.054 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:23.054 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.054 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:31:23.055 DEBUG:boto:Data: 2023-09-29T22:31:23.055 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.055 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.055 DEBUG:boto:Params: {} 2023-09-29T22:31:23.055 DEBUG:boto:Token: None 2023-09-29T22:31:23.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/b 2023-09-29T22:31:23.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j3iR5dOQr0ZxmhU9XPe3LNnGJ8A= 2023-09-29T22:31:23.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j3iR5dOQr0ZxmhU9XPe3LNnGJ8A=', 'Content-Length': '0'} 2023-09-29T22:31:23.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3b4f3760afb4d2-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.059 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:23.059 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:23.060 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.060 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:31:23.060 DEBUG:boto:Data: 2023-09-29T22:31:23.060 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.061 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.061 DEBUG:boto:Params: {} 2023-09-29T22:31:23.061 DEBUG:boto:Token: None 2023-09-29T22:31:23.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/b 2023-09-29T22:31:23.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j3iR5dOQr0ZxmhU9XPe3LNnGJ8A= 2023-09-29T22:31:23.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j3iR5dOQr0ZxmhU9XPe3LNnGJ8A=', 'Content-Length': '0'} 2023-09-29T22:31:23.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e990449a56d26439-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:31:23.065 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:23.065 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:23.065 DEBUG:boto:Method: GET 2023-09-29T22:31:23.065 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:31:23.065 DEBUG:boto:Data: 2023-09-29T22:31:23.065 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.066 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.066 DEBUG:boto:Params: {} 2023-09-29T22:31:23.066 DEBUG:boto:Token: None 2023-09-29T22:31:23.066 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/b 2023-09-29T22:31:23.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q2bK+WxTU3y6PamgMuMQB5pTA2g= 2023-09-29T22:31:23.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q2bK+WxTU3y6PamgMuMQB5pTA2g=', 'Content-Length': '0'} 2023-09-29T22:31:23.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002eb3d331850e1402-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.070 DEBUG:boto:path=/hcnphz-56/b 2023-09-29T22:31:23.070 DEBUG:boto:auth_path=/hcnphz-56/b 2023-09-29T22:31:23.070 DEBUG:boto:Method: GET 2023-09-29T22:31:23.070 DEBUG:boto:Path: /hcnphz-56/b 2023-09-29T22:31:23.070 DEBUG:boto:Data: 2023-09-29T22:31:23.070 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.071 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.071 DEBUG:boto:Params: {} 2023-09-29T22:31:23.071 DEBUG:boto:Token: None 2023-09-29T22:31:23.071 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/b 2023-09-29T22:31:23.071 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q2bK+WxTU3y6PamgMuMQB5pTA2g= 2023-09-29T22:31:23.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q2bK+WxTU3y6PamgMuMQB5pTA2g=', 'Content-Length': '0'} 2023-09-29T22:31:23.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e9ce3d2e208776-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.074 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.074 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.074 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:31:23.075 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:31:23.075 DEBUG:boto:Method: GET 2023-09-29T22:31:23.075 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:31:23.075 DEBUG:boto:Data: 2023-09-29T22:31:23.075 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.075 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.075 DEBUG:boto:Params: {} 2023-09-29T22:31:23.075 DEBUG:boto:Token: None 2023-09-29T22:31:23.075 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.080 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dc3383b60350377-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.080 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.081 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.081 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.081 DEBUG:boto:path=/hcnphz-56/?prefix=b 2023-09-29T22:31:23.081 DEBUG:boto:auth_path=/hcnphz-56/?prefix=b 2023-09-29T22:31:23.082 DEBUG:boto:Method: GET 2023-09-29T22:31:23.082 DEBUG:boto:Path: /hcnphz-56/?prefix=b 2023-09-29T22:31:23.082 DEBUG:boto:Data: 2023-09-29T22:31:23.082 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.082 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.082 DEBUG:boto:Params: {} 2023-09-29T22:31:23.082 DEBUG:boto:Token: None 2023-09-29T22:31:23.083 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.083 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047db2a9c6c29b70a-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.087 DEBUG:boto:b'hcnphz-56b1000falseb2023-09-29T22:27:40.798Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:31:23.087 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.088 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.088 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.088 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.088 DEBUG:boto:Method: GET 2023-09-29T22:31:23.088 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.088 DEBUG:boto:Data: 2023-09-29T22:31:23.088 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.089 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.089 DEBUG:boto:Params: {} 2023-09-29T22:31:23.089 DEBUG:boto:Token: None 2023-09-29T22:31:23.089 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q= 2023-09-29T22:31:23.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q=', 'Content-Length': '0'} 2023-09-29T22:31:23.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b0dde592acbd8d6-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.134 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.134 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.135 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.135 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.135 DEBUG:boto:Method: GET 2023-09-29T22:31:23.135 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.135 DEBUG:boto:Data: 2023-09-29T22:31:23.135 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.135 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.135 DEBUG:boto:Params: {} 2023-09-29T22:31:23.136 DEBUG:boto:Token: None 2023-09-29T22:31:23.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q= 2023-09-29T22:31:23.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q=', 'Content-Length': '0'} 2023-09-29T22:31:23.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee0b90c2ba6f2607-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.139 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.139 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.139 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.140 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.140 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.140 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.140 DEBUG:boto:Data: 2023-09-29T22:31:23.140 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.140 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.140 DEBUG:boto:Params: {} 2023-09-29T22:31:23.140 DEBUG:boto:Token: None 2023-09-29T22:31:23.141 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lzZg9SR+vpa786clydZ10UjOyQ0= 2023-09-29T22:31:23.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lzZg9SR+vpa786clydZ10UjOyQ0=', 'Content-Length': '0'} 2023-09-29T22:31:23.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccb4a74c9dd7e413-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.144 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.144 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.144 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.144 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.144 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.144 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.145 DEBUG:boto:Data: 2023-09-29T22:31:23.145 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.145 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.145 DEBUG:boto:Params: {} 2023-09-29T22:31:23.145 DEBUG:boto:Token: None 2023-09-29T22:31:23.145 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lzZg9SR+vpa786clydZ10UjOyQ0= 2023-09-29T22:31:23.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lzZg9SR+vpa786clydZ10UjOyQ0=', 'Content-Length': '0'} 2023-09-29T22:31:23.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a3101eea6293f32-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:31:23.149 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.149 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.149 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.149 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.149 DEBUG:boto:Method: GET 2023-09-29T22:31:23.149 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.149 DEBUG:boto:Data: 2023-09-29T22:31:23.149 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.150 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.150 DEBUG:boto:Params: {} 2023-09-29T22:31:23.150 DEBUG:boto:Token: None 2023-09-29T22:31:23.150 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.150 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q= 2023-09-29T22:31:23.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q=', 'Content-Length': '0'} 2023-09-29T22:31:23.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbfb609e9952fcf-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.154 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.154 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.154 DEBUG:boto:path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.154 DEBUG:boto:auth_path=/hcnphz-56/c?versionId=null 2023-09-29T22:31:23.154 DEBUG:boto:Method: GET 2023-09-29T22:31:23.154 DEBUG:boto:Path: /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.154 DEBUG:boto:Data: 2023-09-29T22:31:23.154 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.155 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.155 DEBUG:boto:Params: {} 2023-09-29T22:31:23.155 DEBUG:boto:Token: None 2023-09-29T22:31:23.155 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c?versionId=null 2023-09-29T22:31:23.155 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q= 2023-09-29T22:31:23.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pc0KDhHY+6Pgz5zcVGDQJYJMp4Q=', 'Content-Length': '0'} 2023-09-29T22:31:23.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c1ed9944a42f574-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.198 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.198 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.199 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.199 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.199 DEBUG:boto:Method: GET 2023-09-29T22:31:23.199 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:31:23.199 DEBUG:boto:Data: 2023-09-29T22:31:23.199 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.199 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.200 DEBUG:boto:Params: {} 2023-09-29T22:31:23.200 DEBUG:boto:Token: None 2023-09-29T22:31:23.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.204 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e369deade990e20d-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.205 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.205 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.205 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.205 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.205 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.205 DEBUG:boto:Method: GET 2023-09-29T22:31:23.206 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:31:23.206 DEBUG:boto:Data: 2023-09-29T22:31:23.206 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.206 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.206 DEBUG:boto:Params: {} 2023-09-29T22:31:23.206 DEBUG:boto:Token: None 2023-09-29T22:31:23.206 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585ab4ddf4c74365-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.210 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.211 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.211 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.211 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.211 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:31:23.211 DEBUG:boto:Data: 2023-09-29T22:31:23.212 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.212 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.212 DEBUG:boto:Params: {} 2023-09-29T22:31:23.212 DEBUG:boto:Token: None 2023-09-29T22:31:23.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c 2023-09-29T22:31:23.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q+iME/CmIEqT4BaIM+x04d6g5cE= 2023-09-29T22:31:23.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q+iME/CmIEqT4BaIM+x04d6g5cE=', 'Content-Length': '0'} 2023-09-29T22:31:23.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b117730e30aa2ee2-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.215 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.215 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.215 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.215 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:31:23.216 DEBUG:boto:Data: 2023-09-29T22:31:23.216 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.216 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.216 DEBUG:boto:Params: {} 2023-09-29T22:31:23.216 DEBUG:boto:Token: None 2023-09-29T22:31:23.216 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c 2023-09-29T22:31:23.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q+iME/CmIEqT4BaIM+x04d6g5cE= 2023-09-29T22:31:23.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q+iME/CmIEqT4BaIM+x04d6g5cE=', 'Content-Length': '0'} 2023-09-29T22:31:23.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000941c478248c6d6b0-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:31:23.219 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.219 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.220 DEBUG:boto:Method: GET 2023-09-29T22:31:23.220 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:31:23.220 DEBUG:boto:Data: 2023-09-29T22:31:23.220 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.221 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.221 DEBUG:boto:Params: {} 2023-09-29T22:31:23.221 DEBUG:boto:Token: None 2023-09-29T22:31:23.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c 2023-09-29T22:31:23.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oXBNdkJe7aNaX56p6DB8WWionhM= 2023-09-29T22:31:23.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oXBNdkJe7aNaX56p6DB8WWionhM=', 'Content-Length': '0'} 2023-09-29T22:31:23.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000519316be592968d6-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.266 DEBUG:boto:path=/hcnphz-56/c 2023-09-29T22:31:23.266 DEBUG:boto:auth_path=/hcnphz-56/c 2023-09-29T22:31:23.267 DEBUG:boto:Method: GET 2023-09-29T22:31:23.267 DEBUG:boto:Path: /hcnphz-56/c 2023-09-29T22:31:23.267 DEBUG:boto:Data: 2023-09-29T22:31:23.267 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.267 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.267 DEBUG:boto:Params: {} 2023-09-29T22:31:23.267 DEBUG:boto:Token: None 2023-09-29T22:31:23.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/c 2023-09-29T22:31:23.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oXBNdkJe7aNaX56p6DB8WWionhM= 2023-09-29T22:31:23.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oXBNdkJe7aNaX56p6DB8WWionhM=', 'Content-Length': '0'} 2023-09-29T22:31:23.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077c032fd77a838da-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.271 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.272 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.272 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.272 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.272 DEBUG:boto:Method: GET 2023-09-29T22:31:23.272 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:31:23.272 DEBUG:boto:Data: 2023-09-29T22:31:23.272 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.273 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.273 DEBUG:boto:Params: {} 2023-09-29T22:31:23.273 DEBUG:boto:Token: None 2023-09-29T22:31:23.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4d949fc77af65c6-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.278 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.278 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.279 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.279 DEBUG:boto:path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.279 DEBUG:boto:auth_path=/hcnphz-56/?prefix=c 2023-09-29T22:31:23.279 DEBUG:boto:Method: GET 2023-09-29T22:31:23.279 DEBUG:boto:Path: /hcnphz-56/?prefix=c 2023-09-29T22:31:23.279 DEBUG:boto:Data: 2023-09-29T22:31:23.279 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.279 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.279 DEBUG:boto:Params: {} 2023-09-29T22:31:23.280 DEBUG:boto:Token: None 2023-09-29T22:31:23.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.283 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024ee3c3d15b149cf-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.330 DEBUG:boto:b'hcnphz-56c1000falsec2023-09-29T22:27:40.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:31:23.331 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.331 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.331 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.331 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.331 DEBUG:boto:Method: GET 2023-09-29T22:31:23.332 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.332 DEBUG:boto:Data: 2023-09-29T22:31:23.332 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.332 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.332 DEBUG:boto:Params: {} 2023-09-29T22:31:23.332 DEBUG:boto:Token: None 2023-09-29T22:31:23.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0= 2023-09-29T22:31:23.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0=', 'Content-Length': '0'} 2023-09-29T22:31:23.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813a50f4b24ad445-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.336 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.336 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.336 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.336 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.336 DEBUG:boto:Method: GET 2023-09-29T22:31:23.336 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.337 DEBUG:boto:Data: 2023-09-29T22:31:23.337 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.337 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.337 DEBUG:boto:Params: {} 2023-09-29T22:31:23.337 DEBUG:boto:Token: None 2023-09-29T22:31:23.337 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0= 2023-09-29T22:31:23.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0=', 'Content-Length': '0'} 2023-09-29T22:31:23.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4936906815e6bf9-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.341 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.341 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.341 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.341 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.341 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.341 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.341 DEBUG:boto:Data: 2023-09-29T22:31:23.341 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.342 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.342 DEBUG:boto:Params: {} 2023-09-29T22:31:23.342 DEBUG:boto:Token: None 2023-09-29T22:31:23.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:69vPe9BQF+uuDU+JKhBolyJVfpg= 2023-09-29T22:31:23.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:69vPe9BQF+uuDU+JKhBolyJVfpg=', 'Content-Length': '0'} 2023-09-29T22:31:23.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952c218fb3f0c82e-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.345 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.345 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.345 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.346 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.346 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.346 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.346 DEBUG:boto:Data: 2023-09-29T22:31:23.346 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.346 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.346 DEBUG:boto:Params: {} 2023-09-29T22:31:23.346 DEBUG:boto:Token: None 2023-09-29T22:31:23.347 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:69vPe9BQF+uuDU+JKhBolyJVfpg= 2023-09-29T22:31:23.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:69vPe9BQF+uuDU+JKhBolyJVfpg=', 'Content-Length': '0'} 2023-09-29T22:31:23.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b205abfa5cc420a-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:31:23.350 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.350 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.350 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.350 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.350 DEBUG:boto:Method: GET 2023-09-29T22:31:23.350 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.350 DEBUG:boto:Data: 2023-09-29T22:31:23.351 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.351 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.351 DEBUG:boto:Params: {} 2023-09-29T22:31:23.351 DEBUG:boto:Token: None 2023-09-29T22:31:23.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0= 2023-09-29T22:31:23.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0=', 'Content-Length': '0'} 2023-09-29T22:31:23.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a03160bf92dd5fb-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.354 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.355 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.355 DEBUG:boto:path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.355 DEBUG:boto:auth_path=/hcnphz-56/d?versionId=null 2023-09-29T22:31:23.355 DEBUG:boto:Method: GET 2023-09-29T22:31:23.355 DEBUG:boto:Path: /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.355 DEBUG:boto:Data: 2023-09-29T22:31:23.355 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.355 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.355 DEBUG:boto:Params: {} 2023-09-29T22:31:23.356 DEBUG:boto:Token: None 2023-09-29T22:31:23.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d?versionId=null 2023-09-29T22:31:23.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0= 2023-09-29T22:31:23.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2hVJA99Wx+WJQZYh+3ahR2aCiY0=', 'Content-Length': '0'} 2023-09-29T22:31:23.359 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000079545261388b869-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.359 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.359 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.359 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.359 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.359 DEBUG:boto:Method: GET 2023-09-29T22:31:23.360 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:31:23.360 DEBUG:boto:Data: 2023-09-29T22:31:23.360 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.360 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.360 DEBUG:boto:Params: {} 2023-09-29T22:31:23.360 DEBUG:boto:Token: None 2023-09-29T22:31:23.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9825b7a948d1aa-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.410 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.411 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.411 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.411 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.411 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.411 DEBUG:boto:Method: GET 2023-09-29T22:31:23.412 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:31:23.412 DEBUG:boto:Data: 2023-09-29T22:31:23.412 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.412 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.412 DEBUG:boto:Params: {} 2023-09-29T22:31:23.412 DEBUG:boto:Token: None 2023-09-29T22:31:23.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f70d4ef83945cce1-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.417 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.417 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.417 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.417 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.417 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:31:23.417 DEBUG:boto:Data: 2023-09-29T22:31:23.417 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.418 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.418 DEBUG:boto:Params: {} 2023-09-29T22:31:23.418 DEBUG:boto:Token: None 2023-09-29T22:31:23.418 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d 2023-09-29T22:31:23.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OIMcBP8oCf0a1M+p6WSsukXhkmg= 2023-09-29T22:31:23.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OIMcBP8oCf0a1M+p6WSsukXhkmg=', 'Content-Length': '0'} 2023-09-29T22:31:23.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056ba73f7fb3ecd3e-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.421 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.422 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.422 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.422 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:31:23.422 DEBUG:boto:Data: 2023-09-29T22:31:23.422 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.422 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.422 DEBUG:boto:Params: {} 2023-09-29T22:31:23.423 DEBUG:boto:Token: None 2023-09-29T22:31:23.423 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d 2023-09-29T22:31:23.423 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OIMcBP8oCf0a1M+p6WSsukXhkmg= 2023-09-29T22:31:23.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OIMcBP8oCf0a1M+p6WSsukXhkmg=', 'Content-Length': '0'} 2023-09-29T22:31:23.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f195e7438ab02df3-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:31:23.426 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.426 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.426 DEBUG:boto:Method: GET 2023-09-29T22:31:23.426 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:31:23.426 DEBUG:boto:Data: 2023-09-29T22:31:23.427 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.427 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.427 DEBUG:boto:Params: {} 2023-09-29T22:31:23.427 DEBUG:boto:Token: None 2023-09-29T22:31:23.427 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d 2023-09-29T22:31:23.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b5u7mgvyKWT0jZIuvmB3FxANYDo= 2023-09-29T22:31:23.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b5u7mgvyKWT0jZIuvmB3FxANYDo=', 'Content-Length': '0'} 2023-09-29T22:31:23.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ad9b0bc88658a3-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.430 DEBUG:boto:path=/hcnphz-56/d 2023-09-29T22:31:23.430 DEBUG:boto:auth_path=/hcnphz-56/d 2023-09-29T22:31:23.430 DEBUG:boto:Method: GET 2023-09-29T22:31:23.430 DEBUG:boto:Path: /hcnphz-56/d 2023-09-29T22:31:23.431 DEBUG:boto:Data: 2023-09-29T22:31:23.431 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.431 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.431 DEBUG:boto:Params: {} 2023-09-29T22:31:23.431 DEBUG:boto:Token: None 2023-09-29T22:31:23.431 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/d 2023-09-29T22:31:23.431 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b5u7mgvyKWT0jZIuvmB3FxANYDo= 2023-09-29T22:31:23.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b5u7mgvyKWT0jZIuvmB3FxANYDo=', 'Content-Length': '0'} 2023-09-29T22:31:23.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:27:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036cf022ec5f56a1a-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.478 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.478 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.479 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.479 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.479 DEBUG:boto:Method: GET 2023-09-29T22:31:23.479 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:31:23.479 DEBUG:boto:Data: 2023-09-29T22:31:23.479 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.480 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.480 DEBUG:boto:Params: {} 2023-09-29T22:31:23.480 DEBUG:boto:Token: None 2023-09-29T22:31:23.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.484 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784d6d16e536114a-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.485 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.485 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.485 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.485 DEBUG:boto:path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.485 DEBUG:boto:auth_path=/hcnphz-56/?prefix=d 2023-09-29T22:31:23.486 DEBUG:boto:Method: GET 2023-09-29T22:31:23.486 DEBUG:boto:Path: /hcnphz-56/?prefix=d 2023-09-29T22:31:23.486 DEBUG:boto:Data: 2023-09-29T22:31:23.486 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.486 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.486 DEBUG:boto:Params: {} 2023-09-29T22:31:23.486 DEBUG:boto:Token: None 2023-09-29T22:31:23.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.491 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095743a9cad99ad3a-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.491 DEBUG:boto:b'hcnphz-56d1000falsed2023-09-29T22:27:40.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:31:23.491 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.491 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.492 DEBUG:boto:path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.492 DEBUG:boto:auth_path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.492 DEBUG:boto:Method: GET 2023-09-29T22:31:23.492 DEBUG:boto:Path: /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.492 DEBUG:boto:Data: 2023-09-29T22:31:23.492 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.492 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.492 DEBUG:boto:Params: {} 2023-09-29T22:31:23.493 DEBUG:boto:Token: None 2023-09-29T22:31:23.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE= 2023-09-29T22:31:23.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE=', 'Content-Length': '0'} 2023-09-29T22:31:23.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000887169e979fd717e-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.496 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.496 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.496 DEBUG:boto:path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.496 DEBUG:boto:auth_path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.496 DEBUG:boto:Method: GET 2023-09-29T22:31:23.497 DEBUG:boto:Path: /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.497 DEBUG:boto:Data: 2023-09-29T22:31:23.497 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.498 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.498 DEBUG:boto:Params: {} 2023-09-29T22:31:23.498 DEBUG:boto:Token: None 2023-09-29T22:31:23.498 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.498 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE= 2023-09-29T22:31:23.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE=', 'Content-Length': '0'} 2023-09-29T22:31:23.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003344bc88fe59527c-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.501 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.501 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.501 DEBUG:boto:path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.501 DEBUG:boto:auth_path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.501 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.502 DEBUG:boto:Path: /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.502 DEBUG:boto:Data: 2023-09-29T22:31:23.502 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.502 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.502 DEBUG:boto:Params: {} 2023-09-29T22:31:23.503 DEBUG:boto:Token: None 2023-09-29T22:31:23.503 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6IZsTbhzIRnHpvxvAKUDAKM/mo8= 2023-09-29T22:31:23.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6IZsTbhzIRnHpvxvAKUDAKM/mo8=', 'Content-Length': '0'} 2023-09-29T22:31:23.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043df72e7f6653b47-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.506 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.506 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.506 DEBUG:boto:path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.506 DEBUG:boto:auth_path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.506 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.506 DEBUG:boto:Path: /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.506 DEBUG:boto:Data: 2023-09-29T22:31:23.506 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.507 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.507 DEBUG:boto:Params: {} 2023-09-29T22:31:23.507 DEBUG:boto:Token: None 2023-09-29T22:31:23.507 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.507 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6IZsTbhzIRnHpvxvAKUDAKM/mo8= 2023-09-29T22:31:23.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6IZsTbhzIRnHpvxvAKUDAKM/mo8=', 'Content-Length': '0'} 2023-09-29T22:31:23.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2be6927fadc91af-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:31:23.512 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.512 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.512 DEBUG:boto:path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.512 DEBUG:boto:auth_path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.512 DEBUG:boto:Method: GET 2023-09-29T22:31:23.512 DEBUG:boto:Path: /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.512 DEBUG:boto:Data: 2023-09-29T22:31:23.512 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.513 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.513 DEBUG:boto:Params: {} 2023-09-29T22:31:23.513 DEBUG:boto:Token: None 2023-09-29T22:31:23.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE= 2023-09-29T22:31:23.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE=', 'Content-Length': '0'} 2023-09-29T22:31:23.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000221bd1eb18769536-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.558 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.559 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.559 DEBUG:boto:path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.559 DEBUG:boto:auth_path=/hcnphz-56/e?versionId=null 2023-09-29T22:31:23.559 DEBUG:boto:Method: GET 2023-09-29T22:31:23.559 DEBUG:boto:Path: /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.559 DEBUG:boto:Data: 2023-09-29T22:31:23.559 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.559 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.559 DEBUG:boto:Params: {} 2023-09-29T22:31:23.560 DEBUG:boto:Token: None 2023-09-29T22:31:23.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e?versionId=null 2023-09-29T22:31:23.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE= 2023-09-29T22:31:23.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8gbt5X79pqR82NSGlZqGcE96pxE=', 'Content-Length': '0'} 2023-09-29T22:31:23.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fcf6589d2ea60c3-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.563 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.563 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.564 DEBUG:boto:path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.564 DEBUG:boto:auth_path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.564 DEBUG:boto:Method: GET 2023-09-29T22:31:23.564 DEBUG:boto:Path: /hcnphz-56/?prefix=e 2023-09-29T22:31:23.564 DEBUG:boto:Data: 2023-09-29T22:31:23.564 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.564 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.564 DEBUG:boto:Params: {} 2023-09-29T22:31:23.564 DEBUG:boto:Token: None 2023-09-29T22:31:23.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.570 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ba86960ac9d843f-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.570 DEBUG:boto:b'hcnphz-56e1000falsee2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.570 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.571 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.571 DEBUG:boto:path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.571 DEBUG:boto:auth_path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.571 DEBUG:boto:Method: GET 2023-09-29T22:31:23.571 DEBUG:boto:Path: /hcnphz-56/?prefix=e 2023-09-29T22:31:23.571 DEBUG:boto:Data: 2023-09-29T22:31:23.571 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.571 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.572 DEBUG:boto:Params: {} 2023-09-29T22:31:23.572 DEBUG:boto:Token: None 2023-09-29T22:31:23.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cac90f6381180d2-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.618 DEBUG:boto:b'hcnphz-56e1000falsee2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.619 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.619 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.620 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.620 DEBUG:boto:Path: /hcnphz-56/e 2023-09-29T22:31:23.620 DEBUG:boto:Data: 2023-09-29T22:31:23.620 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.621 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.621 DEBUG:boto:Params: {} 2023-09-29T22:31:23.621 DEBUG:boto:Token: None 2023-09-29T22:31:23.621 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e 2023-09-29T22:31:23.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lmSdOqddRD/LbRVjhKzoX/EioZA= 2023-09-29T22:31:23.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lmSdOqddRD/LbRVjhKzoX/EioZA=', 'Content-Length': '0'} 2023-09-29T22:31:23.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3d2e681a6ee4fb2-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.625 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.625 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.626 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.626 DEBUG:boto:Path: /hcnphz-56/e 2023-09-29T22:31:23.626 DEBUG:boto:Data: 2023-09-29T22:31:23.626 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.626 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.626 DEBUG:boto:Params: {} 2023-09-29T22:31:23.627 DEBUG:boto:Token: None 2023-09-29T22:31:23.627 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e 2023-09-29T22:31:23.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lmSdOqddRD/LbRVjhKzoX/EioZA= 2023-09-29T22:31:23.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lmSdOqddRD/LbRVjhKzoX/EioZA=', 'Content-Length': '0'} 2023-09-29T22:31:23.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b388b9d95ca2cab7-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:31:23.630 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.630 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.630 DEBUG:boto:Method: GET 2023-09-29T22:31:23.631 DEBUG:boto:Path: /hcnphz-56/e 2023-09-29T22:31:23.631 DEBUG:boto:Data: 2023-09-29T22:31:23.631 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.631 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.631 DEBUG:boto:Params: {} 2023-09-29T22:31:23.631 DEBUG:boto:Token: None 2023-09-29T22:31:23.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e 2023-09-29T22:31:23.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d4/QplRBIQFiRfh5BH3dq66hFi4= 2023-09-29T22:31:23.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d4/QplRBIQFiRfh5BH3dq66hFi4=', 'Content-Length': '0'} 2023-09-29T22:31:23.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e705dc74c7851647-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.635 DEBUG:boto:path=/hcnphz-56/e 2023-09-29T22:31:23.635 DEBUG:boto:auth_path=/hcnphz-56/e 2023-09-29T22:31:23.635 DEBUG:boto:Method: GET 2023-09-29T22:31:23.636 DEBUG:boto:Path: /hcnphz-56/e 2023-09-29T22:31:23.636 DEBUG:boto:Data: 2023-09-29T22:31:23.636 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.636 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.636 DEBUG:boto:Params: {} 2023-09-29T22:31:23.637 DEBUG:boto:Token: None 2023-09-29T22:31:23.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/e 2023-09-29T22:31:23.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d4/QplRBIQFiRfh5BH3dq66hFi4= 2023-09-29T22:31:23.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d4/QplRBIQFiRfh5BH3dq66hFi4=', 'Content-Length': '0'} 2023-09-29T22:31:23.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b3a79ea0b5ebc4-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.640 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.640 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.640 DEBUG:boto:path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.641 DEBUG:boto:auth_path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.641 DEBUG:boto:Method: GET 2023-09-29T22:31:23.641 DEBUG:boto:Path: /hcnphz-56/?prefix=e 2023-09-29T22:31:23.641 DEBUG:boto:Data: 2023-09-29T22:31:23.641 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.641 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.641 DEBUG:boto:Params: {} 2023-09-29T22:31:23.642 DEBUG:boto:Token: None 2023-09-29T22:31:23.642 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.642 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d699fadcd3ef496-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.647 DEBUG:boto:b'hcnphz-56e1000falsee2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.647 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.648 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.648 DEBUG:boto:path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.648 DEBUG:boto:auth_path=/hcnphz-56/?prefix=e 2023-09-29T22:31:23.648 DEBUG:boto:Method: GET 2023-09-29T22:31:23.648 DEBUG:boto:Path: /hcnphz-56/?prefix=e 2023-09-29T22:31:23.648 DEBUG:boto:Data: 2023-09-29T22:31:23.648 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.648 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.649 DEBUG:boto:Params: {} 2023-09-29T22:31:23.649 DEBUG:boto:Token: None 2023-09-29T22:31:23.649 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a32f63746b9ff7a-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.653 DEBUG:boto:b'hcnphz-56e1000falsee2023-09-29T22:28:02.303Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:31:23.654 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.654 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.654 DEBUG:boto:path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.654 DEBUG:boto:auth_path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.654 DEBUG:boto:Method: GET 2023-09-29T22:31:23.654 DEBUG:boto:Path: /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.654 DEBUG:boto:Data: 2023-09-29T22:31:23.655 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.655 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.655 DEBUG:boto:Params: {} 2023-09-29T22:31:23.655 DEBUG:boto:Token: None 2023-09-29T22:31:23.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4= 2023-09-29T22:31:23.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4=', 'Content-Length': '0'} 2023-09-29T22:31:23.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007766f6b7b4053be8-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.702 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.703 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.703 DEBUG:boto:path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.703 DEBUG:boto:auth_path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.703 DEBUG:boto:Method: GET 2023-09-29T22:31:23.703 DEBUG:boto:Path: /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.703 DEBUG:boto:Data: 2023-09-29T22:31:23.703 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.703 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.704 DEBUG:boto:Params: {} 2023-09-29T22:31:23.704 DEBUG:boto:Token: None 2023-09-29T22:31:23.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4= 2023-09-29T22:31:23.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4=', 'Content-Length': '0'} 2023-09-29T22:31:23.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083a4a6ce4ef5f41e-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.707 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.707 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.707 DEBUG:boto:path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.707 DEBUG:boto:auth_path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.708 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.708 DEBUG:boto:Path: /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.708 DEBUG:boto:Data: 2023-09-29T22:31:23.708 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.708 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.708 DEBUG:boto:Params: {} 2023-09-29T22:31:23.708 DEBUG:boto:Token: None 2023-09-29T22:31:23.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M2SpmdcID+SjOyWLQ8XnXd00JVY= 2023-09-29T22:31:23.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M2SpmdcID+SjOyWLQ8XnXd00JVY=', 'Content-Length': '0'} 2023-09-29T22:31:23.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039546701c31c8aae-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.712 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.712 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.712 DEBUG:boto:path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.712 DEBUG:boto:auth_path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.712 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.712 DEBUG:boto:Path: /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.712 DEBUG:boto:Data: 2023-09-29T22:31:23.712 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.713 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.713 DEBUG:boto:Params: {} 2023-09-29T22:31:23.713 DEBUG:boto:Token: None 2023-09-29T22:31:23.713 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M2SpmdcID+SjOyWLQ8XnXd00JVY= 2023-09-29T22:31:23.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M2SpmdcID+SjOyWLQ8XnXd00JVY=', 'Content-Length': '0'} 2023-09-29T22:31:23.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e020b5c586850ba-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:31:23.716 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.716 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.717 DEBUG:boto:path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.717 DEBUG:boto:auth_path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.717 DEBUG:boto:Method: GET 2023-09-29T22:31:23.717 DEBUG:boto:Path: /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.717 DEBUG:boto:Data: 2023-09-29T22:31:23.717 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.717 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.717 DEBUG:boto:Params: {} 2023-09-29T22:31:23.718 DEBUG:boto:Token: None 2023-09-29T22:31:23.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4= 2023-09-29T22:31:23.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4=', 'Content-Length': '0'} 2023-09-29T22:31:23.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080a9f07ea7328f95-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.721 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.721 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.721 DEBUG:boto:path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.721 DEBUG:boto:auth_path=/hcnphz-56/f?versionId=null 2023-09-29T22:31:23.721 DEBUG:boto:Method: GET 2023-09-29T22:31:23.722 DEBUG:boto:Path: /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.722 DEBUG:boto:Data: 2023-09-29T22:31:23.722 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.722 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.722 DEBUG:boto:Params: {} 2023-09-29T22:31:23.722 DEBUG:boto:Token: None 2023-09-29T22:31:23.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f?versionId=null 2023-09-29T22:31:23.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4= 2023-09-29T22:31:23.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EW4dtZwzP8IhvriwC31ylcJwHe4=', 'Content-Length': '0'} 2023-09-29T22:31:23.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a711002afe02c5fc-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.770 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.771 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.771 DEBUG:boto:path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.771 DEBUG:boto:auth_path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.771 DEBUG:boto:Method: GET 2023-09-29T22:31:23.771 DEBUG:boto:Path: /hcnphz-56/?prefix=f 2023-09-29T22:31:23.771 DEBUG:boto:Data: 2023-09-29T22:31:23.771 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.772 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.772 DEBUG:boto:Params: {} 2023-09-29T22:31:23.772 DEBUG:boto:Token: None 2023-09-29T22:31:23.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.772 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024e9b228c1b8e25d-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.777 DEBUG:boto:b'hcnphz-56f1000falsef2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.777 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.778 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.778 DEBUG:boto:path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.778 DEBUG:boto:auth_path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.778 DEBUG:boto:Method: GET 2023-09-29T22:31:23.778 DEBUG:boto:Path: /hcnphz-56/?prefix=f 2023-09-29T22:31:23.779 DEBUG:boto:Data: 2023-09-29T22:31:23.779 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.779 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.779 DEBUG:boto:Params: {} 2023-09-29T22:31:23.779 DEBUG:boto:Token: None 2023-09-29T22:31:23.779 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.783 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbd6e880851c68a1-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.784 DEBUG:boto:b'hcnphz-56f1000falsef2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.784 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.785 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.785 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.785 DEBUG:boto:Path: /hcnphz-56/f 2023-09-29T22:31:23.785 DEBUG:boto:Data: 2023-09-29T22:31:23.785 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.785 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.785 DEBUG:boto:Params: {} 2023-09-29T22:31:23.785 DEBUG:boto:Token: None 2023-09-29T22:31:23.786 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f 2023-09-29T22:31:23.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L3YKUY53/yRiejFeENkbF6oxFBs= 2023-09-29T22:31:23.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L3YKUY53/yRiejFeENkbF6oxFBs=', 'Content-Length': '0'} 2023-09-29T22:31:23.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b624f0168d8bb206-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.789 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.789 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.789 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.789 DEBUG:boto:Path: /hcnphz-56/f 2023-09-29T22:31:23.789 DEBUG:boto:Data: 2023-09-29T22:31:23.789 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.789 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.790 DEBUG:boto:Params: {} 2023-09-29T22:31:23.790 DEBUG:boto:Token: None 2023-09-29T22:31:23.790 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f 2023-09-29T22:31:23.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L3YKUY53/yRiejFeENkbF6oxFBs= 2023-09-29T22:31:23.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L3YKUY53/yRiejFeENkbF6oxFBs=', 'Content-Length': '0'} 2023-09-29T22:31:23.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb253a7885b253aa-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:31:23.793 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.793 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.793 DEBUG:boto:Method: GET 2023-09-29T22:31:23.793 DEBUG:boto:Path: /hcnphz-56/f 2023-09-29T22:31:23.794 DEBUG:boto:Data: 2023-09-29T22:31:23.794 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.794 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.794 DEBUG:boto:Params: {} 2023-09-29T22:31:23.794 DEBUG:boto:Token: None 2023-09-29T22:31:23.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f 2023-09-29T22:31:23.794 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JjBp8jNqp/clNjsnA51/bnY12L4= 2023-09-29T22:31:23.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JjBp8jNqp/clNjsnA51/bnY12L4=', 'Content-Length': '0'} 2023-09-29T22:31:23.797 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d915ef9279f2a702-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.838 DEBUG:boto:path=/hcnphz-56/f 2023-09-29T22:31:23.839 DEBUG:boto:auth_path=/hcnphz-56/f 2023-09-29T22:31:23.839 DEBUG:boto:Method: GET 2023-09-29T22:31:23.839 DEBUG:boto:Path: /hcnphz-56/f 2023-09-29T22:31:23.839 DEBUG:boto:Data: 2023-09-29T22:31:23.839 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.839 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.839 DEBUG:boto:Params: {} 2023-09-29T22:31:23.840 DEBUG:boto:Token: None 2023-09-29T22:31:23.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/f 2023-09-29T22:31:23.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JjBp8jNqp/clNjsnA51/bnY12L4= 2023-09-29T22:31:23.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JjBp8jNqp/clNjsnA51/bnY12L4=', 'Content-Length': '0'} 2023-09-29T22:31:23.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e078782e88564153-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.843 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.843 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.844 DEBUG:boto:path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.844 DEBUG:boto:auth_path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.844 DEBUG:boto:Method: GET 2023-09-29T22:31:23.844 DEBUG:boto:Path: /hcnphz-56/?prefix=f 2023-09-29T22:31:23.844 DEBUG:boto:Data: 2023-09-29T22:31:23.844 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.844 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.844 DEBUG:boto:Params: {} 2023-09-29T22:31:23.844 DEBUG:boto:Token: None 2023-09-29T22:31:23.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9e5db565bb17660-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.850 DEBUG:boto:b'hcnphz-56f1000falsef2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.851 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.851 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.851 DEBUG:boto:path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.851 DEBUG:boto:auth_path=/hcnphz-56/?prefix=f 2023-09-29T22:31:23.851 DEBUG:boto:Method: GET 2023-09-29T22:31:23.851 DEBUG:boto:Path: /hcnphz-56/?prefix=f 2023-09-29T22:31:23.851 DEBUG:boto:Data: 2023-09-29T22:31:23.851 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.852 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.852 DEBUG:boto:Params: {} 2023-09-29T22:31:23.852 DEBUG:boto:Token: None 2023-09-29T22:31:23.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.856 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b33a50b4a62f560-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.902 DEBUG:boto:b'hcnphz-56f1000falsef2023-09-29T22:28:02.317Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:31:23.903 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.903 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.903 DEBUG:boto:path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.903 DEBUG:boto:auth_path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.904 DEBUG:boto:Method: GET 2023-09-29T22:31:23.904 DEBUG:boto:Path: /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.904 DEBUG:boto:Data: 2023-09-29T22:31:23.904 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.904 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.904 DEBUG:boto:Params: {} 2023-09-29T22:31:23.904 DEBUG:boto:Token: None 2023-09-29T22:31:23.905 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI= 2023-09-29T22:31:23.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI=', 'Content-Length': '0'} 2023-09-29T22:31:23.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000302d2f2e8c3523a7-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.908 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.909 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.909 DEBUG:boto:path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.909 DEBUG:boto:auth_path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.909 DEBUG:boto:Method: GET 2023-09-29T22:31:23.909 DEBUG:boto:Path: /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.909 DEBUG:boto:Data: 2023-09-29T22:31:23.909 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.910 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.910 DEBUG:boto:Params: {} 2023-09-29T22:31:23.910 DEBUG:boto:Token: None 2023-09-29T22:31:23.910 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI= 2023-09-29T22:31:23.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI=', 'Content-Length': '0'} 2023-09-29T22:31:23.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c70741243e970db-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.914 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.914 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.914 DEBUG:boto:path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.915 DEBUG:boto:auth_path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.915 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.915 DEBUG:boto:Path: /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.915 DEBUG:boto:Data: 2023-09-29T22:31:23.915 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.915 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.915 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.915 DEBUG:boto:Params: {} 2023-09-29T22:31:23.915 DEBUG:boto:Token: None 2023-09-29T22:31:23.916 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.916 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o1cKowczQQIPne03YEK4Uv2WkXM= 2023-09-29T22:31:23.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o1cKowczQQIPne03YEK4Uv2WkXM=', 'Content-Length': '0'} 2023-09-29T22:31:23.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8334358fa4bd373-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.919 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.919 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.920 DEBUG:boto:path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.920 DEBUG:boto:auth_path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.920 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.920 DEBUG:boto:Path: /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.920 DEBUG:boto:Data: 2023-09-29T22:31:23.920 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.920 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.920 DEBUG:boto:Params: {} 2023-09-29T22:31:23.920 DEBUG:boto:Token: None 2023-09-29T22:31:23.921 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.921 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o1cKowczQQIPne03YEK4Uv2WkXM= 2023-09-29T22:31:23.921 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o1cKowczQQIPne03YEK4Uv2WkXM=', 'Content-Length': '0'} 2023-09-29T22:31:23.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2447e03944c0226-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:31:23.924 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.924 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.924 DEBUG:boto:path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.924 DEBUG:boto:auth_path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.924 DEBUG:boto:Method: GET 2023-09-29T22:31:23.924 DEBUG:boto:Path: /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.924 DEBUG:boto:Data: 2023-09-29T22:31:23.924 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.925 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.925 DEBUG:boto:Params: {} 2023-09-29T22:31:23.925 DEBUG:boto:Token: None 2023-09-29T22:31:23.925 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.925 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI= 2023-09-29T22:31:23.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI=', 'Content-Length': '0'} 2023-09-29T22:31:23.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000079e5a98bc22830b-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.970 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.971 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.971 DEBUG:boto:path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.971 DEBUG:boto:auth_path=/hcnphz-56/g?versionId=null 2023-09-29T22:31:23.971 DEBUG:boto:Method: GET 2023-09-29T22:31:23.971 DEBUG:boto:Path: /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.971 DEBUG:boto:Data: 2023-09-29T22:31:23.971 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.971 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.972 DEBUG:boto:Params: {} 2023-09-29T22:31:23.972 DEBUG:boto:Token: None 2023-09-29T22:31:23.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g?versionId=null 2023-09-29T22:31:23.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI= 2023-09-29T22:31:23.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnXoxhZfnO4jZLngl48Jz89zZVI=', 'Content-Length': '0'} 2023-09-29T22:31:23.975 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bdecd099155ce51-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.975 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.976 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.976 DEBUG:boto:path=/hcnphz-56/?prefix=g 2023-09-29T22:31:23.976 DEBUG:boto:auth_path=/hcnphz-56/?prefix=g 2023-09-29T22:31:23.976 DEBUG:boto:Method: GET 2023-09-29T22:31:23.976 DEBUG:boto:Path: /hcnphz-56/?prefix=g 2023-09-29T22:31:23.976 DEBUG:boto:Data: 2023-09-29T22:31:23.976 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.976 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.976 DEBUG:boto:Params: {} 2023-09-29T22:31:23.977 DEBUG:boto:Token: None 2023-09-29T22:31:23.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.977 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d753452dda7bf6a7-006517503b-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.981 DEBUG:boto:b'hcnphz-56g1000falseg2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.982 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:23.982 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:23.982 DEBUG:boto:path=/hcnphz-56/?prefix=g 2023-09-29T22:31:23.982 DEBUG:boto:auth_path=/hcnphz-56/?prefix=g 2023-09-29T22:31:23.982 DEBUG:boto:Method: GET 2023-09-29T22:31:23.982 DEBUG:boto:Path: /hcnphz-56/?prefix=g 2023-09-29T22:31:23.982 DEBUG:boto:Data: 2023-09-29T22:31:23.983 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.983 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.983 DEBUG:boto:Params: {} 2023-09-29T22:31:23.983 DEBUG:boto:Token: None 2023-09-29T22:31:23.983 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/ 2023-09-29T22:31:23.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ= 2023-09-29T22:31:23.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tF9EyNiXgHs4wEiaq7Ug3vP36LQ=', 'Content-Length': '0'} 2023-09-29T22:31:23.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7a7e922030ff77-006517503b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.987 DEBUG:boto:b'hcnphz-56g1000falseg2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:23.988 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.988 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.988 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.988 DEBUG:boto:Path: /hcnphz-56/g 2023-09-29T22:31:23.988 DEBUG:boto:Data: 2023-09-29T22:31:23.988 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.989 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.989 DEBUG:boto:Params: {} 2023-09-29T22:31:23.989 DEBUG:boto:Token: None 2023-09-29T22:31:23.989 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g 2023-09-29T22:31:23.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:co3Qw5aNamnozrF0OH68oja/SuM= 2023-09-29T22:31:23.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:co3Qw5aNamnozrF0OH68oja/SuM=', 'Content-Length': '0'} 2023-09-29T22:31:23.992 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a94945f5edfdd5f1-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.992 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.992 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.992 DEBUG:boto:Method: HEAD 2023-09-29T22:31:23.992 DEBUG:boto:Path: /hcnphz-56/g 2023-09-29T22:31:23.993 DEBUG:boto:Data: 2023-09-29T22:31:23.993 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:23.993 DEBUG:boto:Port: 8000 2023-09-29T22:31:23.993 DEBUG:boto:Params: {} 2023-09-29T22:31:23.993 DEBUG:boto:Token: None 2023-09-29T22:31:23.993 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g 2023-09-29T22:31:23.993 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:co3Qw5aNamnozrF0OH68oja/SuM= 2023-09-29T22:31:23.994 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:co3Qw5aNamnozrF0OH68oja/SuM=', 'Content-Length': '0'} 2023-09-29T22:31:23.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bacce6bdd950b9b8-006517503b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:23.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:31:23.996 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:23.996 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:23.996 DEBUG:boto:Method: GET 2023-09-29T22:31:23.996 DEBUG:boto:Path: /hcnphz-56/g 2023-09-29T22:31:23.997 DEBUG:boto:Data: 2023-09-29T22:31:23.997 DEBUG:boto:Headers: {} 2023-09-29T22:31:23.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:23.997 DEBUG:boto:Port: 8001 2023-09-29T22:31:23.997 DEBUG:boto:Params: {} 2023-09-29T22:31:23.997 DEBUG:boto:Token: None 2023-09-29T22:31:23.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:23 GMT /hcnphz-56/g 2023-09-29T22:31:23.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLdYb/FSm2O9i4A9i7NTBzTwmMU= 2023-09-29T22:31:23.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLdYb/FSm2O9i4A9i7NTBzTwmMU=', 'Content-Length': '0'} 2023-09-29T22:31:24.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed711dca04f4c014-006517503b-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.046 DEBUG:boto:path=/hcnphz-56/g 2023-09-29T22:31:24.047 DEBUG:boto:auth_path=/hcnphz-56/g 2023-09-29T22:31:24.047 DEBUG:boto:Method: GET 2023-09-29T22:31:24.047 DEBUG:boto:Path: /hcnphz-56/g 2023-09-29T22:31:24.047 DEBUG:boto:Data: 2023-09-29T22:31:24.047 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.047 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.048 DEBUG:boto:Params: {} 2023-09-29T22:31:24.048 DEBUG:boto:Token: None 2023-09-29T22:31:24.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/g 2023-09-29T22:31:24.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sJasyp4rfhAu9Yht4Km5mmcqNic= 2023-09-29T22:31:24.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sJasyp4rfhAu9Yht4Km5mmcqNic=', 'Content-Length': '0'} 2023-09-29T22:31:24.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075fa5c9c37c49bff-006517503c-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.051 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:24.051 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:24.051 DEBUG:boto:path=/hcnphz-56/?prefix=g 2023-09-29T22:31:24.052 DEBUG:boto:auth_path=/hcnphz-56/?prefix=g 2023-09-29T22:31:24.052 DEBUG:boto:Method: GET 2023-09-29T22:31:24.052 DEBUG:boto:Path: /hcnphz-56/?prefix=g 2023-09-29T22:31:24.052 DEBUG:boto:Data: 2023-09-29T22:31:24.052 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.052 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.052 DEBUG:boto:Params: {} 2023-09-29T22:31:24.053 DEBUG:boto:Token: None 2023-09-29T22:31:24.053 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/ 2023-09-29T22:31:24.053 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg= 2023-09-29T22:31:24.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg=', 'Content-Length': '0'} 2023-09-29T22:31:24.057 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ba0b607199c8dc9-006517503c-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.058 DEBUG:boto:b'hcnphz-56g1000falseg2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:24.058 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:24.058 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:24.059 DEBUG:boto:path=/hcnphz-56/?prefix=g 2023-09-29T22:31:24.059 DEBUG:boto:auth_path=/hcnphz-56/?prefix=g 2023-09-29T22:31:24.059 DEBUG:boto:Method: GET 2023-09-29T22:31:24.059 DEBUG:boto:Path: /hcnphz-56/?prefix=g 2023-09-29T22:31:24.059 DEBUG:boto:Data: 2023-09-29T22:31:24.059 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.059 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.059 DEBUG:boto:Params: {} 2023-09-29T22:31:24.060 DEBUG:boto:Token: None 2023-09-29T22:31:24.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/ 2023-09-29T22:31:24.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg= 2023-09-29T22:31:24.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg=', 'Content-Length': '0'} 2023-09-29T22:31:24.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008ff5bd2c5cd8d39-006517503c-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.064 DEBUG:boto:b'hcnphz-56g1000falseg2023-09-29T22:28:02.332Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:24.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:31:24.064 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.064 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.064 DEBUG:boto:path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.065 DEBUG:boto:auth_path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.065 DEBUG:boto:Method: GET 2023-09-29T22:31:24.065 DEBUG:boto:Path: /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.065 DEBUG:boto:Data: 2023-09-29T22:31:24.065 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.065 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.065 DEBUG:boto:Params: {} 2023-09-29T22:31:24.065 DEBUG:boto:Token: None 2023-09-29T22:31:24.066 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk= 2023-09-29T22:31:24.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk=', 'Content-Length': '0'} 2023-09-29T22:31:24.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007031457d24cec874-006517503c-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.069 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.069 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.069 DEBUG:boto:path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.069 DEBUG:boto:auth_path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.070 DEBUG:boto:Method: GET 2023-09-29T22:31:24.070 DEBUG:boto:Path: /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.070 DEBUG:boto:Data: 2023-09-29T22:31:24.070 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.070 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.070 DEBUG:boto:Params: {} 2023-09-29T22:31:24.070 DEBUG:boto:Token: None 2023-09-29T22:31:24.070 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.071 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk= 2023-09-29T22:31:24.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk=', 'Content-Length': '0'} 2023-09-29T22:31:24.073 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de292b8ec93438b4-006517503c-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.074 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.074 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.074 DEBUG:boto:path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.074 DEBUG:boto:auth_path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.074 DEBUG:boto:Method: HEAD 2023-09-29T22:31:24.074 DEBUG:boto:Path: /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.074 DEBUG:boto:Data: 2023-09-29T22:31:24.075 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.075 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.075 DEBUG:boto:Params: {} 2023-09-29T22:31:24.075 DEBUG:boto:Token: None 2023-09-29T22:31:24.075 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.075 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w6xTUS7NSREKCmQFfncR6rVX4aI= 2023-09-29T22:31:24.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w6xTUS7NSREKCmQFfncR6rVX4aI=', 'Content-Length': '0'} 2023-09-29T22:31:24.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c818312a3841263-006517503c-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.078 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.078 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.078 DEBUG:boto:path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.078 DEBUG:boto:auth_path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.079 DEBUG:boto:Method: HEAD 2023-09-29T22:31:24.079 DEBUG:boto:Path: /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.079 DEBUG:boto:Data: 2023-09-29T22:31:24.079 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.079 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.079 DEBUG:boto:Params: {} 2023-09-29T22:31:24.079 DEBUG:boto:Token: None 2023-09-29T22:31:24.079 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w6xTUS7NSREKCmQFfncR6rVX4aI= 2023-09-29T22:31:24.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w6xTUS7NSREKCmQFfncR6rVX4aI=', 'Content-Length': '0'} 2023-09-29T22:31:24.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c28007773cd76564-006517503c-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:31:24.083 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.083 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.083 DEBUG:boto:path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.083 DEBUG:boto:auth_path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.083 DEBUG:boto:Method: GET 2023-09-29T22:31:24.083 DEBUG:boto:Path: /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.083 DEBUG:boto:Data: 2023-09-29T22:31:24.083 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.084 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.084 DEBUG:boto:Params: {} 2023-09-29T22:31:24.084 DEBUG:boto:Token: None 2023-09-29T22:31:24.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk= 2023-09-29T22:31:24.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk=', 'Content-Length': '0'} 2023-09-29T22:31:24.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c0f8eaf728461d8-006517503c-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.087 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.087 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.087 DEBUG:boto:path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.087 DEBUG:boto:auth_path=/hcnphz-56/h?versionId=null 2023-09-29T22:31:24.087 DEBUG:boto:Method: GET 2023-09-29T22:31:24.087 DEBUG:boto:Path: /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.087 DEBUG:boto:Data: 2023-09-29T22:31:24.088 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.088 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.088 DEBUG:boto:Params: {} 2023-09-29T22:31:24.088 DEBUG:boto:Token: None 2023-09-29T22:31:24.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h?versionId=null 2023-09-29T22:31:24.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk= 2023-09-29T22:31:24.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:roHY/4tDT/CLtVeuwX2FTqqWCEk=', 'Content-Length': '0'} 2023-09-29T22:31:24.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006328d77a9013a6dd-006517503c-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.091 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:24.091 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:24.092 DEBUG:boto:path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.092 DEBUG:boto:auth_path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.092 DEBUG:boto:Method: GET 2023-09-29T22:31:24.092 DEBUG:boto:Path: /hcnphz-56/?prefix=h 2023-09-29T22:31:24.092 DEBUG:boto:Data: 2023-09-29T22:31:24.092 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.092 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.092 DEBUG:boto:Params: {} 2023-09-29T22:31:24.093 DEBUG:boto:Token: None 2023-09-29T22:31:24.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/ 2023-09-29T22:31:24.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg= 2023-09-29T22:31:24.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg=', 'Content-Length': '0'} 2023-09-29T22:31:24.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000664548e30454ebc3-006517503c-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.142 DEBUG:boto:b'hcnphz-56h1000falseh2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:24.143 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:24.143 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:24.143 DEBUG:boto:path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.143 DEBUG:boto:auth_path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.143 DEBUG:boto:Method: GET 2023-09-29T22:31:24.144 DEBUG:boto:Path: /hcnphz-56/?prefix=h 2023-09-29T22:31:24.144 DEBUG:boto:Data: 2023-09-29T22:31:24.144 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.144 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.144 DEBUG:boto:Params: {} 2023-09-29T22:31:24.144 DEBUG:boto:Token: None 2023-09-29T22:31:24.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/ 2023-09-29T22:31:24.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg= 2023-09-29T22:31:24.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg=', 'Content-Length': '0'} 2023-09-29T22:31:24.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d866f23a3bb962df-006517503c-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.148 DEBUG:boto:b'hcnphz-56h1000falseh2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:24.148 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.150 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.150 DEBUG:boto:Method: HEAD 2023-09-29T22:31:24.150 DEBUG:boto:Path: /hcnphz-56/h 2023-09-29T22:31:24.150 DEBUG:boto:Data: 2023-09-29T22:31:24.150 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.151 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.151 DEBUG:boto:Params: {} 2023-09-29T22:31:24.151 DEBUG:boto:Token: None 2023-09-29T22:31:24.151 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h 2023-09-29T22:31:24.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tlnVIQ9PsnkzesHGl6NE+d/R46w= 2023-09-29T22:31:24.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tlnVIQ9PsnkzesHGl6NE+d/R46w=', 'Content-Length': '0'} 2023-09-29T22:31:24.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab1ab0029dcae4eb-006517503c-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.155 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.155 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.155 DEBUG:boto:Method: HEAD 2023-09-29T22:31:24.156 DEBUG:boto:Path: /hcnphz-56/h 2023-09-29T22:31:24.156 DEBUG:boto:Data: 2023-09-29T22:31:24.156 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.156 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.157 DEBUG:boto:Params: {} 2023-09-29T22:31:24.157 DEBUG:boto:Token: None 2023-09-29T22:31:24.157 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h 2023-09-29T22:31:24.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tlnVIQ9PsnkzesHGl6NE+d/R46w= 2023-09-29T22:31:24.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tlnVIQ9PsnkzesHGl6NE+d/R46w=', 'Content-Length': '0'} 2023-09-29T22:31:24.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adeae82b7ab81ea8-006517503c-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:31:24.161 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.161 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.161 DEBUG:boto:Method: GET 2023-09-29T22:31:24.161 DEBUG:boto:Path: /hcnphz-56/h 2023-09-29T22:31:24.162 DEBUG:boto:Data: 2023-09-29T22:31:24.162 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.162 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.162 DEBUG:boto:Params: {} 2023-09-29T22:31:24.163 DEBUG:boto:Token: None 2023-09-29T22:31:24.163 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h 2023-09-29T22:31:24.163 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q9WuOPUtaOCU9yAaPJaP2cAuLeo= 2023-09-29T22:31:24.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9WuOPUtaOCU9yAaPJaP2cAuLeo=', 'Content-Length': '0'} 2023-09-29T22:31:24.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-56:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.21'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000518318cda5b08e-006517503c-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.167 DEBUG:boto:path=/hcnphz-56/h 2023-09-29T22:31:24.167 DEBUG:boto:auth_path=/hcnphz-56/h 2023-09-29T22:31:24.167 DEBUG:boto:Method: GET 2023-09-29T22:31:24.167 DEBUG:boto:Path: /hcnphz-56/h 2023-09-29T22:31:24.167 DEBUG:boto:Data: 2023-09-29T22:31:24.168 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.168 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.168 DEBUG:boto:Params: {} 2023-09-29T22:31:24.168 DEBUG:boto:Token: None 2023-09-29T22:31:24.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/h 2023-09-29T22:31:24.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q9WuOPUtaOCU9yAaPJaP2cAuLeo= 2023-09-29T22:31:24.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9WuOPUtaOCU9yAaPJaP2cAuLeo=', 'Content-Length': '0'} 2023-09-29T22:31:24.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:28:02 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087029aacdbe95163-006517503c-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.214 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:24.215 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:24.215 DEBUG:boto:path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.215 DEBUG:boto:auth_path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.215 DEBUG:boto:Method: GET 2023-09-29T22:31:24.215 DEBUG:boto:Path: /hcnphz-56/?prefix=h 2023-09-29T22:31:24.216 DEBUG:boto:Data: 2023-09-29T22:31:24.216 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:24.216 DEBUG:boto:Port: 8001 2023-09-29T22:31:24.216 DEBUG:boto:Params: {} 2023-09-29T22:31:24.217 DEBUG:boto:Token: None 2023-09-29T22:31:24.217 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/ 2023-09-29T22:31:24.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg= 2023-09-29T22:31:24.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg=', 'Content-Length': '0'} 2023-09-29T22:31:24.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1ebc64f54a5694b-006517503c-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.223 DEBUG:boto:b'hcnphz-56h1000falseh2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:24.224 DEBUG:boto:path=/hcnphz-56/ 2023-09-29T22:31:24.224 DEBUG:boto:auth_path=/hcnphz-56/ 2023-09-29T22:31:24.224 DEBUG:boto:path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.224 DEBUG:boto:auth_path=/hcnphz-56/?prefix=h 2023-09-29T22:31:24.224 DEBUG:boto:Method: GET 2023-09-29T22:31:24.225 DEBUG:boto:Path: /hcnphz-56/?prefix=h 2023-09-29T22:31:24.225 DEBUG:boto:Data: 2023-09-29T22:31:24.225 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.225 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.225 DEBUG:boto:Params: {} 2023-09-29T22:31:24.226 DEBUG:boto:Token: None 2023-09-29T22:31:24.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-56/ 2023-09-29T22:31:24.226 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg= 2023-09-29T22:31:24.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Qg41oSZ5MYSRn1o1nZyq7dnObg=', 'Content-Length': '0'} 2023-09-29T22:31:24.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062bafc8b3949ceb1-006517503c-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.231 DEBUG:boto:b'hcnphz-56h1000falseh2023-09-29T22:28:02.380Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:31:24.231 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-56 zones={a2, a1} 2023-09-29T22:31:24.232 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_incremental ... ok 2023-09-29T22:31:24.233 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:24.233 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:24.233 DEBUG:boto:Method: PUT 2023-09-29T22:31:24.233 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:24.233 DEBUG:boto:Data: 2023-09-29T22:31:24.233 DEBUG:boto:Headers: {} 2023-09-29T22:31:24.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:24.233 DEBUG:boto:Port: 8000 2023-09-29T22:31:24.234 DEBUG:boto:Params: {} 2023-09-29T22:31:24.234 DEBUG:boto:Token: None 2023-09-29T22:31:24.234 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:31:24 GMT /hcnphz-57/ 2023-09-29T22:31:24.234 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UDvV3S48BQnifPf7Cmj9xkbkfAk= 2023-09-29T22:31:24.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UDvV3S48BQnifPf7Cmj9xkbkfAk=', 'Content-Length': '0'} 2023-09-29T22:31:24.265 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005817c2929f7096cd-006517503c-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:24.265 DEBUG:rgw_multi.tests:created bucket=hcnphz-57 2023-09-29T22:31:24.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:24.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:24.266 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:24.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:24.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:24.361 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:31:24.389 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:31:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:31:24.389 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026684.263616_450.1", 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:31:24.263616Z" 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:31:24.390 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:31:24.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.391 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:31:24.391 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:31:24.391 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:31:24.391 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:31:24.392 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026684.263616_450.1', 2: '', 3: ''} 2023-09-29T22:31:24.392 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:31:24.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:24.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:31:24.393 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:31:24.437 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:24.437 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:24.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:24.492+0000 7f68b0294f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:24.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:24.496+0000 7f68b0294f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:31:24.522 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:31:24.522 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:31:24.522 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:31:24.522 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:31:24.522 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:31:24.522 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:31:24.523 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:31:24.524 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:31:24.525 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026684.263616_450.1", 2023-09-29T22:31:24.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:31:24.525 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:31:24.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:31:24.525 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:31:24.263616Z", 2023-09-29T22:31:24.525 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:31:24.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:31:24.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.530 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:31:24.531 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:31:24.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:31:24.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:31:24.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:31:24.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:31:24.534 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:31:24.535 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:31:24.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:31:24.535 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:31:24Z" 2023-09-29T22:31:24.535 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:31:24.536 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026684.263616_450.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:31:24.263616Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:31:24Z" } 2023-09-29T22:31:24.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026684.263616_450.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:31:24.263616Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:31:24.537 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026684.263616_450.1', 2: '', 3: ''} 2023-09-29T22:31:24.537 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026684.263616_450.1', 2: '', 3: ''} 2023-09-29T22:31:24.537 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:31:24.537 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T22:31:24.537 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:31:24.538 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:31:24.536+0000 7f3987d00700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 36415) UID: 0 2023-09-29T22:31:24.538 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:31:24.536+0000 7f398b55ff80 -1 shutting down 2023-09-29T22:31:25.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T22:31:30.639 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T22:31:30.640 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:30.640 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:30.640 DEBUG:boto:Method: HEAD 2023-09-29T22:31:30.640 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:30.640 DEBUG:boto:Data: 2023-09-29T22:31:30.640 DEBUG:boto:Headers: {} 2023-09-29T22:31:30.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.640 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.641 DEBUG:boto:Params: {} 2023-09-29T22:31:30.641 DEBUG:boto:Token: None 2023-09-29T22:31:30.641 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/ 2023-09-29T22:31:30.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms= 2023-09-29T22:31:30.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms=', 'Content-Length': '0'} 2023-09-29T22:31:30.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002f7fbd7118de49ae-0065175042-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.646 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:31:30.646 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:31:30.646 DEBUG:boto:Method: PUT 2023-09-29T22:31:30.646 DEBUG:boto:Path: /hcnphz-57/a 2023-09-29T22:31:30.646 DEBUG:boto:Data: 2023-09-29T22:31:30.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:30.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.647 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.647 DEBUG:boto:Params: {} 2023-09-29T22:31:30.647 DEBUG:boto:Token: None 2023-09-29T22:31:30.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/a 2023-09-29T22:31:30.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xYllvUnTt/JDVpLhQv/CNtuv8qs= 2023-09-29T22:31:30.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xYllvUnTt/JDVpLhQv/CNtuv8qs='} 2023-09-29T22:31:30.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff2ccd9c302e5c9e-0065175042-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.695 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:30.695 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:30.695 DEBUG:boto:Method: HEAD 2023-09-29T22:31:30.695 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:30.695 DEBUG:boto:Data: 2023-09-29T22:31:30.695 DEBUG:boto:Headers: {} 2023-09-29T22:31:30.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.696 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.696 DEBUG:boto:Params: {} 2023-09-29T22:31:30.696 DEBUG:boto:Token: None 2023-09-29T22:31:30.696 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/ 2023-09-29T22:31:30.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms= 2023-09-29T22:31:30.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms=', 'Content-Length': '0'} 2023-09-29T22:31:30.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b10aad31a3b92798-0065175042-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.700 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:31:30.700 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:31:30.700 DEBUG:boto:Method: PUT 2023-09-29T22:31:30.700 DEBUG:boto:Path: /hcnphz-57/b 2023-09-29T22:31:30.701 DEBUG:boto:Data: 2023-09-29T22:31:30.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:30.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.701 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.701 DEBUG:boto:Params: {} 2023-09-29T22:31:30.701 DEBUG:boto:Token: None 2023-09-29T22:31:30.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/b 2023-09-29T22:31:30.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OdeUHSjYJtXukmPjI3QrN3nBOb8= 2023-09-29T22:31:30.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OdeUHSjYJtXukmPjI3QrN3nBOb8='} 2023-09-29T22:31:30.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009967d0f977f47d84-0065175042-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.714 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:30.714 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:30.714 DEBUG:boto:Method: HEAD 2023-09-29T22:31:30.714 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:30.714 DEBUG:boto:Data: 2023-09-29T22:31:30.715 DEBUG:boto:Headers: {} 2023-09-29T22:31:30.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.715 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.715 DEBUG:boto:Params: {} 2023-09-29T22:31:30.715 DEBUG:boto:Token: None 2023-09-29T22:31:30.715 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/ 2023-09-29T22:31:30.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms= 2023-09-29T22:31:30.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms=', 'Content-Length': '0'} 2023-09-29T22:31:30.719 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098dddeb9b91df81c-0065175042-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.720 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:31:30.720 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:31:30.720 DEBUG:boto:Method: PUT 2023-09-29T22:31:30.720 DEBUG:boto:Path: /hcnphz-57/c 2023-09-29T22:31:30.720 DEBUG:boto:Data: 2023-09-29T22:31:30.720 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:30.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.721 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.721 DEBUG:boto:Params: {} 2023-09-29T22:31:30.721 DEBUG:boto:Token: None 2023-09-29T22:31:30.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/c 2023-09-29T22:31:30.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ppwh/vCWroLYosxN4cW6YfOQCns= 2023-09-29T22:31:30.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ppwh/vCWroLYosxN4cW6YfOQCns='} 2023-09-29T22:31:30.771 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b72b9afa4302146-0065175042-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.771 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:30.771 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:30.771 DEBUG:boto:Method: HEAD 2023-09-29T22:31:30.771 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:30.771 DEBUG:boto:Data: 2023-09-29T22:31:30.771 DEBUG:boto:Headers: {} 2023-09-29T22:31:30.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.772 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.772 DEBUG:boto:Params: {} 2023-09-29T22:31:30.772 DEBUG:boto:Token: None 2023-09-29T22:31:30.772 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/ 2023-09-29T22:31:30.772 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms= 2023-09-29T22:31:30.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nd5PmyHJZDlQgzeBFhVNDUmv6ms=', 'Content-Length': '0'} 2023-09-29T22:31:30.776 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8717840e159d6c3-0065175042-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.776 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:31:30.776 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:31:30.777 DEBUG:boto:Method: PUT 2023-09-29T22:31:30.777 DEBUG:boto:Path: /hcnphz-57/d 2023-09-29T22:31:30.777 DEBUG:boto:Data: 2023-09-29T22:31:30.777 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:30.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:30.777 DEBUG:boto:Port: 8000 2023-09-29T22:31:30.777 DEBUG:boto:Params: {} 2023-09-29T22:31:30.777 DEBUG:boto:Token: None 2023-09-29T22:31:30.778 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:30 GMT /hcnphz-57/d 2023-09-29T22:31:30.778 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:udcxP/ToG0Hv+Hcz/jC7Cp2qosI= 2023-09-29T22:31:30.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:udcxP/ToG0Hv+Hcz/jC7Cp2qosI='} 2023-09-29T22:31:30.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T22:31:30.789 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007a7bfe1ddae2948d-0065175042-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:30.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:31:30.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-57', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:31:30.790 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-57 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:31:30.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:30.835 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:30.916 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:31:30.916 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-57 2023-09-29T22:31:30.926 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 4 2023-09-29T22:31:31.002 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:31.002 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:31.002 DEBUG:boto:Method: HEAD 2023-09-29T22:31:31.002 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:31.002 DEBUG:boto:Data: 2023-09-29T22:31:31.002 DEBUG:boto:Headers: {} 2023-09-29T22:31:31.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.002 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.003 DEBUG:boto:Params: {} 2023-09-29T22:31:31.003 DEBUG:boto:Token: None 2023-09-29T22:31:31.003 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/ 2023-09-29T22:31:31.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA= 2023-09-29T22:31:31.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA=', 'Content-Length': '0'} 2023-09-29T22:31:31.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c3624de24848823a-0065175043-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.006 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:31:31.006 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:31:31.006 DEBUG:boto:Method: PUT 2023-09-29T22:31:31.007 DEBUG:boto:Path: /hcnphz-57/e 2023-09-29T22:31:31.007 DEBUG:boto:Data: 2023-09-29T22:31:31.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:31.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.007 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.007 DEBUG:boto:Params: {} 2023-09-29T22:31:31.007 DEBUG:boto:Token: None 2023-09-29T22:31:31.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/e 2023-09-29T22:31:31.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wXOuboaPz3+sTHFYpkFb5g6YHqE= 2023-09-29T22:31:31.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wXOuboaPz3+sTHFYpkFb5g6YHqE='} 2023-09-29T22:31:31.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000210a9097f50bf77a-0065175043-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.018 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:31.018 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:31.018 DEBUG:boto:Method: HEAD 2023-09-29T22:31:31.019 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:31.019 DEBUG:boto:Data: 2023-09-29T22:31:31.019 DEBUG:boto:Headers: {} 2023-09-29T22:31:31.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.019 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.019 DEBUG:boto:Params: {} 2023-09-29T22:31:31.019 DEBUG:boto:Token: None 2023-09-29T22:31:31.019 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/ 2023-09-29T22:31:31.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA= 2023-09-29T22:31:31.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA=', 'Content-Length': '0'} 2023-09-29T22:31:31.022 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eca937aa0069da07-0065175043-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.022 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:31:31.022 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:31:31.022 DEBUG:boto:Method: PUT 2023-09-29T22:31:31.022 DEBUG:boto:Path: /hcnphz-57/f 2023-09-29T22:31:31.023 DEBUG:boto:Data: 2023-09-29T22:31:31.023 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:31.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.023 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.023 DEBUG:boto:Params: {} 2023-09-29T22:31:31.023 DEBUG:boto:Token: None 2023-09-29T22:31:31.023 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/f 2023-09-29T22:31:31.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zXJI3bez5SjtbMEJd5BV4ao+Z+Y= 2023-09-29T22:31:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zXJI3bez5SjtbMEJd5BV4ao+Z+Y='} 2023-09-29T22:31:31.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005e1735d3134230bf-0065175043-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.071 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:31.071 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:31.071 DEBUG:boto:Method: HEAD 2023-09-29T22:31:31.071 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:31.071 DEBUG:boto:Data: 2023-09-29T22:31:31.071 DEBUG:boto:Headers: {} 2023-09-29T22:31:31.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.072 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.072 DEBUG:boto:Params: {} 2023-09-29T22:31:31.072 DEBUG:boto:Token: None 2023-09-29T22:31:31.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/ 2023-09-29T22:31:31.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA= 2023-09-29T22:31:31.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA=', 'Content-Length': '0'} 2023-09-29T22:31:31.074 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8f72a39f211b446-0065175043-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.075 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:31:31.075 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:31:31.075 DEBUG:boto:Method: PUT 2023-09-29T22:31:31.075 DEBUG:boto:Path: /hcnphz-57/g 2023-09-29T22:31:31.075 DEBUG:boto:Data: 2023-09-29T22:31:31.075 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:31.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.076 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.076 DEBUG:boto:Params: {} 2023-09-29T22:31:31.076 DEBUG:boto:Token: None 2023-09-29T22:31:31.076 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/g 2023-09-29T22:31:31.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3gszIM93AcX1Nz8WcL13yMmCPYo= 2023-09-29T22:31:31.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3gszIM93AcX1Nz8WcL13yMmCPYo='} 2023-09-29T22:31:31.085 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aa943dff2aaf466-0065175043-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.085 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:31:31.085 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:31:31.085 DEBUG:boto:Method: HEAD 2023-09-29T22:31:31.085 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:31:31.085 DEBUG:boto:Data: 2023-09-29T22:31:31.085 DEBUG:boto:Headers: {} 2023-09-29T22:31:31.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.086 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.086 DEBUG:boto:Params: {} 2023-09-29T22:31:31.086 DEBUG:boto:Token: None 2023-09-29T22:31:31.086 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/ 2023-09-29T22:31:31.086 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA= 2023-09-29T22:31:31.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F6NqMxHfZRastb6cIbqKXzUonuA=', 'Content-Length': '0'} 2023-09-29T22:31:31.088 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001b86e537fc4a4812-0065175043-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.089 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:31:31.089 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:31:31.089 DEBUG:boto:Method: PUT 2023-09-29T22:31:31.089 DEBUG:boto:Path: /hcnphz-57/h 2023-09-29T22:31:31.089 DEBUG:boto:Data: 2023-09-29T22:31:31.089 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:31:31.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:31:31.090 DEBUG:boto:Port: 8000 2023-09-29T22:31:31.090 DEBUG:boto:Params: {} 2023-09-29T22:31:31.090 DEBUG:boto:Token: None 2023-09-29T22:31:31.090 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:31:31 GMT /hcnphz-57/h 2023-09-29T22:31:31.090 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2XA9kBNOC9J7GrY4rPssWpLULaU= 2023-09-29T22:31:31.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:31:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2XA9kBNOC9J7GrY4rPssWpLULaU='} 2023-09-29T22:31:31.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000596c6f1dd635dafe-0065175043-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:31:31.135 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:31:31.135 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:31:31.138 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:31:31.138 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:31.154 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:31:31.154 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:31:31.155 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:31:31.156 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:31:31.156 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T22:31:32.157 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:31:32.174 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:31:32.174 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:31:32.177 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T22:31:32.179 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T22:31:32.179 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-57 2023-09-29T22:31:32.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:31:32.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-57', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:31:32.179 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-57 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:31:32.230 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:31:32.230 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:31:32.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:32.292+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:31:37.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:37.292+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:31:42.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:42.292+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:31:47.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:47.292+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:31:50.142 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:31:50.139+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:31:52.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:52.292+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:31:57.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:31:57.296+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:02.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:02.295+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:07.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:07.295+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:12.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:12.299+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:17.299+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:22.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:22.299+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:27.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:27.299+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:32.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:32.299+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:37.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:37.303+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:42.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:42.302+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:47.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:47.302+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:52.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:52.302+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:32:57.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:32:57.306+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:02.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:02.306+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:07.310 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:07.306+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:12.311 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:12.306+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:17.312 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:17.309+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:22.314 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:22.309+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:27.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:27.313+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:32.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:32.317+0000 7f639b519f40 1 waiting to reach incremental sync.. 2023-09-29T22:33:37.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:37.317+0000 7f639b519f40 1 bucket sync caught up with source: 2023-09-29T22:33:37.319 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.50.6, 00000000002.48.6, 00000000002.21.6] 2023-09-29T22:33:37.319 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.50.6, 00000000002.48.6, 00000000002.21.6] 2023-09-29T22:33:37.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:37.317+0000 7f639b519f40 0 bucket checkpoint complete 2023-09-29T22:33:37.350 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-57 2023-09-29T22:33:37.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:33:37.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-57', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:33:37.351 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-57 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:33:37.400 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:37.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:37.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:37.469+0000 7f2a16584f40 1 bucket sync caught up with source: 2023-09-29T22:33:37.469 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.188.6, , , 00000000001.176.6, 00000000001.174.6, , , 00000000002.66.6, , 00000000002.62.6] 2023-09-29T22:33:37.469 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.188.6, , , 00000000001.176.6, 00000000001.174.6, , , 00000000002.66.6, , 00000000002.62.6] 2023-09-29T22:33:37.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:37.469+0000 7f2a16584f40 0 bucket checkpoint complete 2023-09-29T22:33:37.495 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-57 zones={a2, a1} 2023-09-29T22:33:37.495 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.495 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.495 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.495 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:33:37.496 DEBUG:boto:Data: 2023-09-29T22:33:37.496 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.496 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.496 DEBUG:boto:Params: {} 2023-09-29T22:33:37.496 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:33:37.496 DEBUG:boto:Token: None 2023-09-29T22:33:37.497 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+HYxWpg07wEIIDRO2Ou+DD9is1k= 2023-09-29T22:33:37.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+HYxWpg07wEIIDRO2Ou+DD9is1k=', 'Content-Length': '0'} 2023-09-29T22:33:37.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005865e7e0b31e8b4f-00651750c1-1a82-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.505 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.505 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.505 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.505 DEBUG:boto:Path: /hcnphz-57/ 2023-09-29T22:33:37.505 DEBUG:boto:Data: 2023-09-29T22:33:37.506 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.506 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.506 DEBUG:boto:Params: {} 2023-09-29T22:33:37.506 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:33:37.506 DEBUG:boto:Token: None 2023-09-29T22:33:37.507 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.507 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+HYxWpg07wEIIDRO2Ou+DD9is1k= 2023-09-29T22:33:37.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+HYxWpg07wEIIDRO2Ou+DD9is1k=', 'Content-Length': '0'} 2023-09-29T22:33:37.512 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e7b254bf3b06589-00651750c1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:33:37.512 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.513 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.513 DEBUG:boto:path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.513 DEBUG:boto:auth_path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.513 DEBUG:boto:Method: GET 2023-09-29T22:33:37.513 DEBUG:boto:Path: /hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.513 DEBUG:boto:Data: 2023-09-29T22:33:37.513 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.513 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.513 DEBUG:boto:Params: {} 2023-09-29T22:33:37.514 DEBUG:boto:Token: None 2023-09-29T22:33:37.514 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/?versions 2023-09-29T22:33:37.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg= 2023-09-29T22:33:37.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg=', 'Content-Length': '0'} 2023-09-29T22:33:37.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5bdd3c12c75e96-00651750c1-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.518 DEBUG:boto:b'hcnphz-57999falseanulltrue2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:33:37.520 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:33:37.521 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:33:37.521 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.521 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.521 DEBUG:boto:path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.521 DEBUG:boto:auth_path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.521 DEBUG:boto:Method: GET 2023-09-29T22:33:37.521 DEBUG:boto:Path: /hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.521 DEBUG:boto:Data: 2023-09-29T22:33:37.521 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.522 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.522 DEBUG:boto:Params: {} 2023-09-29T22:33:37.522 DEBUG:boto:Token: None 2023-09-29T22:33:37.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/?versions 2023-09-29T22:33:37.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg= 2023-09-29T22:33:37.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg=', 'Content-Length': '0'} 2023-09-29T22:33:37.526 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7a403ce835e8ebf-00651750c1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.527 DEBUG:boto:b'hcnphz-57999falseanulltrue2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.527 DEBUG:tasks.rgw_multi.zone_rados:o=a 2023-09-29T22:33:37.527 DEBUG:tasks.rgw_multi.zone_rados:o=b 2023-09-29T22:33:37.528 DEBUG:tasks.rgw_multi.zone_rados:o=c 2023-09-29T22:33:37.528 DEBUG:tasks.rgw_multi.zone_rados:o=d 2023-09-29T22:33:37.528 DEBUG:tasks.rgw_multi.zone_rados:o=e 2023-09-29T22:33:37.528 DEBUG:tasks.rgw_multi.zone_rados:o=f 2023-09-29T22:33:37.528 DEBUG:tasks.rgw_multi.zone_rados:o=g 2023-09-29T22:33:37.528 DEBUG:tasks.rgw_multi.zone_rados:o=h 2023-09-29T22:33:37.528 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.528 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.528 DEBUG:boto:path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.528 DEBUG:boto:auth_path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.529 DEBUG:boto:Method: GET 2023-09-29T22:33:37.529 DEBUG:boto:Path: /hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.529 DEBUG:boto:Data: 2023-09-29T22:33:37.529 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.529 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.529 DEBUG:boto:Params: {} 2023-09-29T22:33:37.529 DEBUG:boto:Token: None 2023-09-29T22:33:37.530 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/?versions 2023-09-29T22:33:37.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg= 2023-09-29T22:33:37.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg=', 'Content-Length': '0'} 2023-09-29T22:33:37.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b53d5898c81d665-00651750c1-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.535 DEBUG:boto:b'hcnphz-57999falseanulltrue2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.536 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.536 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.536 DEBUG:boto:path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.536 DEBUG:boto:auth_path=/hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.536 DEBUG:boto:Method: GET 2023-09-29T22:33:37.536 DEBUG:boto:Path: /hcnphz-57/?versions&max-keys=999 2023-09-29T22:33:37.536 DEBUG:boto:Data: 2023-09-29T22:33:37.536 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.537 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.537 DEBUG:boto:Params: {} 2023-09-29T22:33:37.537 DEBUG:boto:Token: None 2023-09-29T22:33:37.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/?versions 2023-09-29T22:33:37.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg= 2023-09-29T22:33:37.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b7lQsw4eIf8zAgujMmirEEll8Fg=', 'Content-Length': '0'} 2023-09-29T22:33:37.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009da9c23371756df-00651750c1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.540 DEBUG:boto:b'hcnphz-57999falseanulltrue2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:33:37.541 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.541 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.542 DEBUG:boto:path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.542 DEBUG:boto:auth_path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.542 DEBUG:boto:Method: GET 2023-09-29T22:33:37.542 DEBUG:boto:Path: /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.542 DEBUG:boto:Data: 2023-09-29T22:33:37.542 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.543 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.543 DEBUG:boto:Params: {} 2023-09-29T22:33:37.543 DEBUG:boto:Token: None 2023-09-29T22:33:37.543 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI= 2023-09-29T22:33:37.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI=', 'Content-Length': '0'} 2023-09-29T22:33:37.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b9a0040c8afb40-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.590 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.591 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.591 DEBUG:boto:path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.591 DEBUG:boto:auth_path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.591 DEBUG:boto:Method: GET 2023-09-29T22:33:37.591 DEBUG:boto:Path: /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.591 DEBUG:boto:Data: 2023-09-29T22:33:37.591 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.591 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.592 DEBUG:boto:Params: {} 2023-09-29T22:33:37.592 DEBUG:boto:Token: None 2023-09-29T22:33:37.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI= 2023-09-29T22:33:37.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI=', 'Content-Length': '0'} 2023-09-29T22:33:37.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db49f438bb5d34e6-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.597 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.597 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.597 DEBUG:boto:path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.597 DEBUG:boto:auth_path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.598 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.598 DEBUG:boto:Path: /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.598 DEBUG:boto:Data: 2023-09-29T22:33:37.598 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.598 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.598 DEBUG:boto:Params: {} 2023-09-29T22:33:37.598 DEBUG:boto:Token: None 2023-09-29T22:33:37.599 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TpTrQ9P0/IUztePJC0ElKQHbtXU= 2023-09-29T22:33:37.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TpTrQ9P0/IUztePJC0ElKQHbtXU=', 'Content-Length': '0'} 2023-09-29T22:33:37.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bf2fff62d3ae823-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.602 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.602 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.602 DEBUG:boto:path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.602 DEBUG:boto:auth_path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.602 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.602 DEBUG:boto:Path: /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.603 DEBUG:boto:Data: 2023-09-29T22:33:37.603 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.603 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.603 DEBUG:boto:Params: {} 2023-09-29T22:33:37.603 DEBUG:boto:Token: None 2023-09-29T22:33:37.603 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TpTrQ9P0/IUztePJC0ElKQHbtXU= 2023-09-29T22:33:37.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TpTrQ9P0/IUztePJC0ElKQHbtXU=', 'Content-Length': '0'} 2023-09-29T22:33:37.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab5f398843f9f648-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:33:37.607 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.607 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.607 DEBUG:boto:path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.607 DEBUG:boto:auth_path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.608 DEBUG:boto:Method: GET 2023-09-29T22:33:37.608 DEBUG:boto:Path: /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.608 DEBUG:boto:Data: 2023-09-29T22:33:37.608 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.608 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.609 DEBUG:boto:Params: {} 2023-09-29T22:33:37.609 DEBUG:boto:Token: None 2023-09-29T22:33:37.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI= 2023-09-29T22:33:37.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI=', 'Content-Length': '0'} 2023-09-29T22:33:37.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f803bed91bc20642-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.612 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.613 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.613 DEBUG:boto:path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.613 DEBUG:boto:auth_path=/hcnphz-57/a?versionId=null 2023-09-29T22:33:37.613 DEBUG:boto:Method: GET 2023-09-29T22:33:37.613 DEBUG:boto:Path: /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.613 DEBUG:boto:Data: 2023-09-29T22:33:37.613 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.613 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.614 DEBUG:boto:Params: {} 2023-09-29T22:33:37.614 DEBUG:boto:Token: None 2023-09-29T22:33:37.614 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a?versionId=null 2023-09-29T22:33:37.614 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI= 2023-09-29T22:33:37.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nd31RyR89ZPCT/Yu9EQfy4dzYAI=', 'Content-Length': '0'} 2023-09-29T22:33:37.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee57e6dfe0b9662-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.658 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.659 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.659 DEBUG:boto:path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.659 DEBUG:boto:auth_path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.659 DEBUG:boto:Method: GET 2023-09-29T22:33:37.659 DEBUG:boto:Path: /hcnphz-57/?prefix=a 2023-09-29T22:33:37.659 DEBUG:boto:Data: 2023-09-29T22:33:37.659 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.660 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.660 DEBUG:boto:Params: {} 2023-09-29T22:33:37.660 DEBUG:boto:Token: None 2023-09-29T22:33:37.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a356fec6bbac995-00651750c1-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.664 DEBUG:boto:b'hcnphz-57a1000falsea2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.665 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.665 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.665 DEBUG:boto:path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.665 DEBUG:boto:auth_path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.665 DEBUG:boto:Method: GET 2023-09-29T22:33:37.665 DEBUG:boto:Path: /hcnphz-57/?prefix=a 2023-09-29T22:33:37.666 DEBUG:boto:Data: 2023-09-29T22:33:37.666 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.666 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.666 DEBUG:boto:Params: {} 2023-09-29T22:33:37.666 DEBUG:boto:Token: None 2023-09-29T22:33:37.666 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.667 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009546fe4c7aa90c46-00651750c1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.670 DEBUG:boto:b'hcnphz-57a1000falsea2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.671 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.671 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.671 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.671 DEBUG:boto:Path: /hcnphz-57/a 2023-09-29T22:33:37.671 DEBUG:boto:Data: 2023-09-29T22:33:37.671 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.671 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.671 DEBUG:boto:Params: {} 2023-09-29T22:33:37.672 DEBUG:boto:Token: None 2023-09-29T22:33:37.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a 2023-09-29T22:33:37.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ICu4iMiXxiLZQZ5+huMRsHYgQpA= 2023-09-29T22:33:37.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ICu4iMiXxiLZQZ5+huMRsHYgQpA=', 'Content-Length': '0'} 2023-09-29T22:33:37.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5843eb40452f2ff-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.675 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.675 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.675 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.675 DEBUG:boto:Path: /hcnphz-57/a 2023-09-29T22:33:37.675 DEBUG:boto:Data: 2023-09-29T22:33:37.675 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.675 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.675 DEBUG:boto:Params: {} 2023-09-29T22:33:37.676 DEBUG:boto:Token: None 2023-09-29T22:33:37.676 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a 2023-09-29T22:33:37.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ICu4iMiXxiLZQZ5+huMRsHYgQpA= 2023-09-29T22:33:37.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ICu4iMiXxiLZQZ5+huMRsHYgQpA=', 'Content-Length': '0'} 2023-09-29T22:33:37.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ef41316d854b83c-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2023-09-29T22:33:37.679 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.679 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.679 DEBUG:boto:Method: GET 2023-09-29T22:33:37.679 DEBUG:boto:Path: /hcnphz-57/a 2023-09-29T22:33:37.679 DEBUG:boto:Data: 2023-09-29T22:33:37.679 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.680 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.680 DEBUG:boto:Params: {} 2023-09-29T22:33:37.680 DEBUG:boto:Token: None 2023-09-29T22:33:37.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a 2023-09-29T22:33:37.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z5bsVNPEu/xNPgg/8PXAwK0n74Q= 2023-09-29T22:33:37.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z5bsVNPEu/xNPgg/8PXAwK0n74Q=', 'Content-Length': '0'} 2023-09-29T22:33:37.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da8f94bf26a26e88-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.730 DEBUG:boto:path=/hcnphz-57/a 2023-09-29T22:33:37.731 DEBUG:boto:auth_path=/hcnphz-57/a 2023-09-29T22:33:37.731 DEBUG:boto:Method: GET 2023-09-29T22:33:37.731 DEBUG:boto:Path: /hcnphz-57/a 2023-09-29T22:33:37.731 DEBUG:boto:Data: 2023-09-29T22:33:37.731 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.731 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.731 DEBUG:boto:Params: {} 2023-09-29T22:33:37.731 DEBUG:boto:Token: None 2023-09-29T22:33:37.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/a 2023-09-29T22:33:37.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z5bsVNPEu/xNPgg/8PXAwK0n74Q= 2023-09-29T22:33:37.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z5bsVNPEu/xNPgg/8PXAwK0n74Q=', 'Content-Length': '0'} 2023-09-29T22:33:37.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b182c4513a56e91-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.735 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.735 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.735 DEBUG:boto:path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.735 DEBUG:boto:auth_path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.735 DEBUG:boto:Method: GET 2023-09-29T22:33:37.735 DEBUG:boto:Path: /hcnphz-57/?prefix=a 2023-09-29T22:33:37.736 DEBUG:boto:Data: 2023-09-29T22:33:37.736 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.736 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.736 DEBUG:boto:Params: {} 2023-09-29T22:33:37.736 DEBUG:boto:Token: None 2023-09-29T22:33:37.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.742 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000951e41ae165d6acb-00651750c1-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.742 DEBUG:boto:b'hcnphz-57a1000falsea2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.743 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.743 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.743 DEBUG:boto:path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.743 DEBUG:boto:auth_path=/hcnphz-57/?prefix=a 2023-09-29T22:33:37.744 DEBUG:boto:Method: GET 2023-09-29T22:33:37.744 DEBUG:boto:Path: /hcnphz-57/?prefix=a 2023-09-29T22:33:37.744 DEBUG:boto:Data: 2023-09-29T22:33:37.744 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.745 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.745 DEBUG:boto:Params: {} 2023-09-29T22:33:37.745 DEBUG:boto:Token: None 2023-09-29T22:33:37.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023def61ff6662d57-00651750c1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.794 DEBUG:boto:b'hcnphz-57a1000falsea2023-09-29T22:31:30.653Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:33:37.795 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.795 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.795 DEBUG:boto:path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.795 DEBUG:boto:auth_path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.796 DEBUG:boto:Method: GET 2023-09-29T22:33:37.796 DEBUG:boto:Path: /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.796 DEBUG:boto:Data: 2023-09-29T22:33:37.796 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.796 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.796 DEBUG:boto:Params: {} 2023-09-29T22:33:37.796 DEBUG:boto:Token: None 2023-09-29T22:33:37.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8= 2023-09-29T22:33:37.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8=', 'Content-Length': '0'} 2023-09-29T22:33:37.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a16541a2f7ddd335-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.800 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.800 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.800 DEBUG:boto:path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.800 DEBUG:boto:auth_path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.800 DEBUG:boto:Method: GET 2023-09-29T22:33:37.801 DEBUG:boto:Path: /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.801 DEBUG:boto:Data: 2023-09-29T22:33:37.801 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.801 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.801 DEBUG:boto:Params: {} 2023-09-29T22:33:37.802 DEBUG:boto:Token: None 2023-09-29T22:33:37.802 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8= 2023-09-29T22:33:37.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8=', 'Content-Length': '0'} 2023-09-29T22:33:37.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000891c31028f3cca67-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.805 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.805 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.805 DEBUG:boto:path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.805 DEBUG:boto:auth_path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.805 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.806 DEBUG:boto:Path: /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.806 DEBUG:boto:Data: 2023-09-29T22:33:37.806 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.806 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.806 DEBUG:boto:Params: {} 2023-09-29T22:33:37.806 DEBUG:boto:Token: None 2023-09-29T22:33:37.806 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7o+U0AxSF4bVDPOzPQ/YvU94woU= 2023-09-29T22:33:37.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7o+U0AxSF4bVDPOzPQ/YvU94woU=', 'Content-Length': '0'} 2023-09-29T22:33:37.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5734981f1f38ce3-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.809 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.810 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.810 DEBUG:boto:path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.810 DEBUG:boto:auth_path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.810 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.810 DEBUG:boto:Path: /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.810 DEBUG:boto:Data: 2023-09-29T22:33:37.810 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.810 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.811 DEBUG:boto:Params: {} 2023-09-29T22:33:37.811 DEBUG:boto:Token: None 2023-09-29T22:33:37.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7o+U0AxSF4bVDPOzPQ/YvU94woU= 2023-09-29T22:33:37.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7o+U0AxSF4bVDPOzPQ/YvU94woU=', 'Content-Length': '0'} 2023-09-29T22:33:37.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d85500d98804f9c2-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:33:37.814 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.814 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.814 DEBUG:boto:path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.815 DEBUG:boto:auth_path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.815 DEBUG:boto:Method: GET 2023-09-29T22:33:37.815 DEBUG:boto:Path: /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.815 DEBUG:boto:Data: 2023-09-29T22:33:37.815 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.815 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.815 DEBUG:boto:Params: {} 2023-09-29T22:33:37.815 DEBUG:boto:Token: None 2023-09-29T22:33:37.815 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8= 2023-09-29T22:33:37.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8=', 'Content-Length': '0'} 2023-09-29T22:33:37.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6f770c56253693-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.819 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.819 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.819 DEBUG:boto:path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.819 DEBUG:boto:auth_path=/hcnphz-57/b?versionId=null 2023-09-29T22:33:37.819 DEBUG:boto:Method: GET 2023-09-29T22:33:37.819 DEBUG:boto:Path: /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.819 DEBUG:boto:Data: 2023-09-29T22:33:37.820 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.820 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.820 DEBUG:boto:Params: {} 2023-09-29T22:33:37.820 DEBUG:boto:Token: None 2023-09-29T22:33:37.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b?versionId=null 2023-09-29T22:33:37.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8= 2023-09-29T22:33:37.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TY1kDtXtlvcqYsccnK4XCEd3mx8=', 'Content-Length': '0'} 2023-09-29T22:33:37.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbe7dadaf606223c-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.823 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.823 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.824 DEBUG:boto:path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.824 DEBUG:boto:auth_path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.824 DEBUG:boto:Method: GET 2023-09-29T22:33:37.824 DEBUG:boto:Path: /hcnphz-57/?prefix=b 2023-09-29T22:33:37.824 DEBUG:boto:Data: 2023-09-29T22:33:37.824 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.824 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.824 DEBUG:boto:Params: {} 2023-09-29T22:33:37.825 DEBUG:boto:Token: None 2023-09-29T22:33:37.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000869fe4d23c51f055-00651750c1-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.878 DEBUG:boto:b'hcnphz-57b1000falseb2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.879 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.879 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.879 DEBUG:boto:path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.879 DEBUG:boto:auth_path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.879 DEBUG:boto:Method: GET 2023-09-29T22:33:37.879 DEBUG:boto:Path: /hcnphz-57/?prefix=b 2023-09-29T22:33:37.879 DEBUG:boto:Data: 2023-09-29T22:33:37.880 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.880 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.880 DEBUG:boto:Params: {} 2023-09-29T22:33:37.880 DEBUG:boto:Token: None 2023-09-29T22:33:37.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.884 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063d70371e2ebc1f5-00651750c1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.884 DEBUG:boto:b'hcnphz-57b1000falseb2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.885 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.885 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.885 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.885 DEBUG:boto:Path: /hcnphz-57/b 2023-09-29T22:33:37.885 DEBUG:boto:Data: 2023-09-29T22:33:37.885 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.885 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.886 DEBUG:boto:Params: {} 2023-09-29T22:33:37.886 DEBUG:boto:Token: None 2023-09-29T22:33:37.886 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b 2023-09-29T22:33:37.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KuHfzQDUKAQZuRRviwPxuG7Fzw8= 2023-09-29T22:33:37.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KuHfzQDUKAQZuRRviwPxuG7Fzw8=', 'Content-Length': '0'} 2023-09-29T22:33:37.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165183d0aa0465f5-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.889 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.889 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.889 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.889 DEBUG:boto:Path: /hcnphz-57/b 2023-09-29T22:33:37.889 DEBUG:boto:Data: 2023-09-29T22:33:37.890 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.890 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.890 DEBUG:boto:Params: {} 2023-09-29T22:33:37.890 DEBUG:boto:Token: None 2023-09-29T22:33:37.890 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b 2023-09-29T22:33:37.890 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KuHfzQDUKAQZuRRviwPxuG7Fzw8= 2023-09-29T22:33:37.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KuHfzQDUKAQZuRRviwPxuG7Fzw8=', 'Content-Length': '0'} 2023-09-29T22:33:37.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0bfe7c82b1709e3-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2023-09-29T22:33:37.894 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.894 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.894 DEBUG:boto:Method: GET 2023-09-29T22:33:37.894 DEBUG:boto:Path: /hcnphz-57/b 2023-09-29T22:33:37.894 DEBUG:boto:Data: 2023-09-29T22:33:37.894 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.895 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.895 DEBUG:boto:Params: {} 2023-09-29T22:33:37.895 DEBUG:boto:Token: None 2023-09-29T22:33:37.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b 2023-09-29T22:33:37.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fcvQdGYY9yrmHZtrza0oj5Ye/I0= 2023-09-29T22:33:37.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fcvQdGYY9yrmHZtrza0oj5Ye/I0=', 'Content-Length': '0'} 2023-09-29T22:33:37.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc1afd262f95eb8-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.899 DEBUG:boto:path=/hcnphz-57/b 2023-09-29T22:33:37.899 DEBUG:boto:auth_path=/hcnphz-57/b 2023-09-29T22:33:37.899 DEBUG:boto:Method: GET 2023-09-29T22:33:37.899 DEBUG:boto:Path: /hcnphz-57/b 2023-09-29T22:33:37.899 DEBUG:boto:Data: 2023-09-29T22:33:37.899 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.899 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.900 DEBUG:boto:Params: {} 2023-09-29T22:33:37.900 DEBUG:boto:Token: None 2023-09-29T22:33:37.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/b 2023-09-29T22:33:37.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fcvQdGYY9yrmHZtrza0oj5Ye/I0= 2023-09-29T22:33:37.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fcvQdGYY9yrmHZtrza0oj5Ye/I0=', 'Content-Length': '0'} 2023-09-29T22:33:37.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f1540b93c45bec8-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.904 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.904 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.905 DEBUG:boto:path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.905 DEBUG:boto:auth_path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.905 DEBUG:boto:Method: GET 2023-09-29T22:33:37.905 DEBUG:boto:Path: /hcnphz-57/?prefix=b 2023-09-29T22:33:37.905 DEBUG:boto:Data: 2023-09-29T22:33:37.905 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.905 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.905 DEBUG:boto:Params: {} 2023-09-29T22:33:37.906 DEBUG:boto:Token: None 2023-09-29T22:33:37.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.906 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fdf19380066ad2e-00651750c1-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.911 DEBUG:boto:b'hcnphz-57b1000falseb2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.911 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:37.911 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:37.911 DEBUG:boto:path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.911 DEBUG:boto:auth_path=/hcnphz-57/?prefix=b 2023-09-29T22:33:37.912 DEBUG:boto:Method: GET 2023-09-29T22:33:37.912 DEBUG:boto:Path: /hcnphz-57/?prefix=b 2023-09-29T22:33:37.912 DEBUG:boto:Data: 2023-09-29T22:33:37.912 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.912 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.912 DEBUG:boto:Params: {} 2023-09-29T22:33:37.912 DEBUG:boto:Token: None 2023-09-29T22:33:37.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/ 2023-09-29T22:33:37.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0= 2023-09-29T22:33:37.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VQCLIgoDbnNTRxAwDYLDLCf1bH0=', 'Content-Length': '0'} 2023-09-29T22:33:37.916 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000583388f0ddfa49bb-00651750c1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.962 DEBUG:boto:b'hcnphz-57b1000falseb2023-09-29T22:31:30.706Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:37.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:33:37.963 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:37.963 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:37.963 DEBUG:boto:path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.963 DEBUG:boto:auth_path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.963 DEBUG:boto:Method: GET 2023-09-29T22:33:37.964 DEBUG:boto:Path: /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.964 DEBUG:boto:Data: 2023-09-29T22:33:37.964 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.964 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.964 DEBUG:boto:Params: {} 2023-09-29T22:33:37.964 DEBUG:boto:Token: None 2023-09-29T22:33:37.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.965 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clcb0+Yyjn35uIISWaFKBfW3aLI= 2023-09-29T22:33:37.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clcb0+Yyjn35uIISWaFKBfW3aLI=', 'Content-Length': '0'} 2023-09-29T22:33:37.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3ec09f72b818d5-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.968 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:37.968 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:37.968 DEBUG:boto:path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.968 DEBUG:boto:auth_path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.968 DEBUG:boto:Method: GET 2023-09-29T22:33:37.968 DEBUG:boto:Path: /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.968 DEBUG:boto:Data: 2023-09-29T22:33:37.968 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.969 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.969 DEBUG:boto:Params: {} 2023-09-29T22:33:37.969 DEBUG:boto:Token: None 2023-09-29T22:33:37.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clcb0+Yyjn35uIISWaFKBfW3aLI= 2023-09-29T22:33:37.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clcb0+Yyjn35uIISWaFKBfW3aLI=', 'Content-Length': '0'} 2023-09-29T22:33:37.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d0d1ed3fc91b451-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.972 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:37.973 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:37.973 DEBUG:boto:path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.973 DEBUG:boto:auth_path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.973 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.973 DEBUG:boto:Path: /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.973 DEBUG:boto:Data: 2023-09-29T22:33:37.973 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.974 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.974 DEBUG:boto:Params: {} 2023-09-29T22:33:37.974 DEBUG:boto:Token: None 2023-09-29T22:33:37.974 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.974 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Z5Gdfy//dnEl6b68avQsppzdCk= 2023-09-29T22:33:37.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Z5Gdfy//dnEl6b68avQsppzdCk=', 'Content-Length': '0'} 2023-09-29T22:33:37.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bfb66f41c832f4-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.977 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:37.977 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:37.977 DEBUG:boto:path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.977 DEBUG:boto:auth_path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.978 DEBUG:boto:Method: HEAD 2023-09-29T22:33:37.978 DEBUG:boto:Path: /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.978 DEBUG:boto:Data: 2023-09-29T22:33:37.978 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:37.978 DEBUG:boto:Port: 8000 2023-09-29T22:33:37.978 DEBUG:boto:Params: {} 2023-09-29T22:33:37.978 DEBUG:boto:Token: None 2023-09-29T22:33:37.978 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.979 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Z5Gdfy//dnEl6b68avQsppzdCk= 2023-09-29T22:33:37.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Z5Gdfy//dnEl6b68avQsppzdCk=', 'Content-Length': '0'} 2023-09-29T22:33:37.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb3957bc767ca5d0-00651750c1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:37.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:33:37.982 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:37.982 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:37.982 DEBUG:boto:path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.982 DEBUG:boto:auth_path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:37.982 DEBUG:boto:Method: GET 2023-09-29T22:33:37.982 DEBUG:boto:Path: /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.982 DEBUG:boto:Data: 2023-09-29T22:33:37.982 DEBUG:boto:Headers: {} 2023-09-29T22:33:37.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:37.982 DEBUG:boto:Port: 8001 2023-09-29T22:33:37.983 DEBUG:boto:Params: {} 2023-09-29T22:33:37.983 DEBUG:boto:Token: None 2023-09-29T22:33:37.983 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:37 GMT /hcnphz-57/c?versionId=null 2023-09-29T22:33:37.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clcb0+Yyjn35uIISWaFKBfW3aLI= 2023-09-29T22:33:37.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clcb0+Yyjn35uIISWaFKBfW3aLI=', 'Content-Length': '0'} 2023-09-29T22:33:37.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088300f5375be7fdf-00651750c1-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.030 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:38.031 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:38.031 DEBUG:boto:path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:38.031 DEBUG:boto:auth_path=/hcnphz-57/c?versionId=null 2023-09-29T22:33:38.031 DEBUG:boto:Method: GET 2023-09-29T22:33:38.031 DEBUG:boto:Path: /hcnphz-57/c?versionId=null 2023-09-29T22:33:38.031 DEBUG:boto:Data: 2023-09-29T22:33:38.031 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.032 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.032 DEBUG:boto:Params: {} 2023-09-29T22:33:38.032 DEBUG:boto:Token: None 2023-09-29T22:33:38.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/c?versionId=null 2023-09-29T22:33:38.033 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nPNeYxvD6CyvFObxGpqaBZQaVAw= 2023-09-29T22:33:38.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nPNeYxvD6CyvFObxGpqaBZQaVAw=', 'Content-Length': '0'} 2023-09-29T22:33:38.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa54e0ae2b84b833-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.037 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.037 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.037 DEBUG:boto:path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.037 DEBUG:boto:auth_path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.037 DEBUG:boto:Method: GET 2023-09-29T22:33:38.037 DEBUG:boto:Path: /hcnphz-57/?prefix=c 2023-09-29T22:33:38.037 DEBUG:boto:Data: 2023-09-29T22:33:38.037 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.038 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.038 DEBUG:boto:Params: {} 2023-09-29T22:33:38.038 DEBUG:boto:Token: None 2023-09-29T22:33:38.038 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f95e11ef386a411-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.043 DEBUG:boto:b'hcnphz-57c1000falsec2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.043 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.044 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.044 DEBUG:boto:path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.044 DEBUG:boto:auth_path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.044 DEBUG:boto:Method: GET 2023-09-29T22:33:38.044 DEBUG:boto:Path: /hcnphz-57/?prefix=c 2023-09-29T22:33:38.044 DEBUG:boto:Data: 2023-09-29T22:33:38.044 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.045 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.045 DEBUG:boto:Params: {} 2023-09-29T22:33:38.045 DEBUG:boto:Token: None 2023-09-29T22:33:38.045 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.045 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000117eac9ae0f18966-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.094 DEBUG:boto:b'hcnphz-57c1000falsec2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.095 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:38.095 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:38.095 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.095 DEBUG:boto:Path: /hcnphz-57/c 2023-09-29T22:33:38.095 DEBUG:boto:Data: 2023-09-29T22:33:38.096 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.096 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.096 DEBUG:boto:Params: {} 2023-09-29T22:33:38.096 DEBUG:boto:Token: None 2023-09-29T22:33:38.096 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/c 2023-09-29T22:33:38.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VCCcOIBezxxpyZJPTE2gMdqrPRk= 2023-09-29T22:33:38.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VCCcOIBezxxpyZJPTE2gMdqrPRk=', 'Content-Length': '0'} 2023-09-29T22:33:38.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002941e1e4a3a3e72a-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.100 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:38.101 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:38.101 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.101 DEBUG:boto:Path: /hcnphz-57/c 2023-09-29T22:33:38.101 DEBUG:boto:Data: 2023-09-29T22:33:38.101 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.101 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.101 DEBUG:boto:Params: {} 2023-09-29T22:33:38.102 DEBUG:boto:Token: None 2023-09-29T22:33:38.102 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/c 2023-09-29T22:33:38.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VCCcOIBezxxpyZJPTE2gMdqrPRk= 2023-09-29T22:33:38.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VCCcOIBezxxpyZJPTE2gMdqrPRk=', 'Content-Length': '0'} 2023-09-29T22:33:38.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391c49af7be2d91b-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.104 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2023-09-29T22:33:38.104 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:38.104 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:38.105 DEBUG:boto:Method: GET 2023-09-29T22:33:38.105 DEBUG:boto:Path: /hcnphz-57/c 2023-09-29T22:33:38.105 DEBUG:boto:Data: 2023-09-29T22:33:38.105 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.105 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.105 DEBUG:boto:Params: {} 2023-09-29T22:33:38.105 DEBUG:boto:Token: None 2023-09-29T22:33:38.105 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/c 2023-09-29T22:33:38.106 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:81ew5svunYR3Xqqe0y+yiAuIXHQ= 2023-09-29T22:33:38.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:81ew5svunYR3Xqqe0y+yiAuIXHQ=', 'Content-Length': '0'} 2023-09-29T22:33:38.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdb446bf965489df-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.108 DEBUG:boto:path=/hcnphz-57/c 2023-09-29T22:33:38.108 DEBUG:boto:auth_path=/hcnphz-57/c 2023-09-29T22:33:38.108 DEBUG:boto:Method: GET 2023-09-29T22:33:38.108 DEBUG:boto:Path: /hcnphz-57/c 2023-09-29T22:33:38.108 DEBUG:boto:Data: 2023-09-29T22:33:38.108 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.108 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.109 DEBUG:boto:Params: {} 2023-09-29T22:33:38.109 DEBUG:boto:Token: None 2023-09-29T22:33:38.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/c 2023-09-29T22:33:38.110 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:81ew5svunYR3Xqqe0y+yiAuIXHQ= 2023-09-29T22:33:38.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:81ew5svunYR3Xqqe0y+yiAuIXHQ=', 'Content-Length': '0'} 2023-09-29T22:33:38.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b86817dc927537e-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.113 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.113 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.113 DEBUG:boto:path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.113 DEBUG:boto:auth_path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.114 DEBUG:boto:Method: GET 2023-09-29T22:33:38.114 DEBUG:boto:Path: /hcnphz-57/?prefix=c 2023-09-29T22:33:38.114 DEBUG:boto:Data: 2023-09-29T22:33:38.114 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.115 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.115 DEBUG:boto:Params: {} 2023-09-29T22:33:38.115 DEBUG:boto:Token: None 2023-09-29T22:33:38.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000874aaa17d0a1879b-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.184 DEBUG:boto:b'hcnphz-57c1000falsec2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.184 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.184 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.185 DEBUG:boto:path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.185 DEBUG:boto:auth_path=/hcnphz-57/?prefix=c 2023-09-29T22:33:38.185 DEBUG:boto:Method: GET 2023-09-29T22:33:38.185 DEBUG:boto:Path: /hcnphz-57/?prefix=c 2023-09-29T22:33:38.185 DEBUG:boto:Data: 2023-09-29T22:33:38.185 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.186 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.186 DEBUG:boto:Params: {} 2023-09-29T22:33:38.186 DEBUG:boto:Token: None 2023-09-29T22:33:38.186 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.186 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e748f3efafedd11a-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.190 DEBUG:boto:b'hcnphz-57c1000falsec2023-09-29T22:31:30.725Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:33:38.191 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.192 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.192 DEBUG:boto:path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.192 DEBUG:boto:auth_path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.192 DEBUG:boto:Method: GET 2023-09-29T22:33:38.192 DEBUG:boto:Path: /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.192 DEBUG:boto:Data: 2023-09-29T22:33:38.193 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.193 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.193 DEBUG:boto:Params: {} 2023-09-29T22:33:38.193 DEBUG:boto:Token: None 2023-09-29T22:33:38.194 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI= 2023-09-29T22:33:38.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI=', 'Content-Length': '0'} 2023-09-29T22:33:38.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ceae932313c4d59b-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.242 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.243 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.243 DEBUG:boto:path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.243 DEBUG:boto:auth_path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.244 DEBUG:boto:Method: GET 2023-09-29T22:33:38.244 DEBUG:boto:Path: /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.244 DEBUG:boto:Data: 2023-09-29T22:33:38.244 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.245 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.245 DEBUG:boto:Params: {} 2023-09-29T22:33:38.245 DEBUG:boto:Token: None 2023-09-29T22:33:38.245 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI= 2023-09-29T22:33:38.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI=', 'Content-Length': '0'} 2023-09-29T22:33:38.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022e0d5d3226b7c4-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.249 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.249 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.249 DEBUG:boto:path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.249 DEBUG:boto:auth_path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.249 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.250 DEBUG:boto:Path: /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.250 DEBUG:boto:Data: 2023-09-29T22:33:38.250 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.250 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.250 DEBUG:boto:Params: {} 2023-09-29T22:33:38.251 DEBUG:boto:Token: None 2023-09-29T22:33:38.251 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4aOmeeGHcEg7uBXVSQIv3dmmXEc= 2023-09-29T22:33:38.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4aOmeeGHcEg7uBXVSQIv3dmmXEc=', 'Content-Length': '0'} 2023-09-29T22:33:38.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb28b1f8ecd8aa24-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.254 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.255 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.255 DEBUG:boto:path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.255 DEBUG:boto:auth_path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.255 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.255 DEBUG:boto:Path: /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.255 DEBUG:boto:Data: 2023-09-29T22:33:38.255 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.255 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.255 DEBUG:boto:Params: {} 2023-09-29T22:33:38.256 DEBUG:boto:Token: None 2023-09-29T22:33:38.256 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4aOmeeGHcEg7uBXVSQIv3dmmXEc= 2023-09-29T22:33:38.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4aOmeeGHcEg7uBXVSQIv3dmmXEc=', 'Content-Length': '0'} 2023-09-29T22:33:38.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3c0ea8dadce0190-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:33:38.259 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.259 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.259 DEBUG:boto:path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.260 DEBUG:boto:auth_path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.260 DEBUG:boto:Method: GET 2023-09-29T22:33:38.260 DEBUG:boto:Path: /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.260 DEBUG:boto:Data: 2023-09-29T22:33:38.260 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.260 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.260 DEBUG:boto:Params: {} 2023-09-29T22:33:38.260 DEBUG:boto:Token: None 2023-09-29T22:33:38.261 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI= 2023-09-29T22:33:38.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI=', 'Content-Length': '0'} 2023-09-29T22:33:38.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682cfaef216a892f-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.264 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.264 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.265 DEBUG:boto:path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.265 DEBUG:boto:auth_path=/hcnphz-57/d?versionId=null 2023-09-29T22:33:38.265 DEBUG:boto:Method: GET 2023-09-29T22:33:38.265 DEBUG:boto:Path: /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.265 DEBUG:boto:Data: 2023-09-29T22:33:38.265 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.265 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.265 DEBUG:boto:Params: {} 2023-09-29T22:33:38.265 DEBUG:boto:Token: None 2023-09-29T22:33:38.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d?versionId=null 2023-09-29T22:33:38.266 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI= 2023-09-29T22:33:38.266 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kOYLaYjPHWLGEeQkRiwTW0lGljI=', 'Content-Length': '0'} 2023-09-29T22:33:38.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520bc2c33c11b1c4-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.269 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.269 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.269 DEBUG:boto:path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.269 DEBUG:boto:auth_path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.269 DEBUG:boto:Method: GET 2023-09-29T22:33:38.270 DEBUG:boto:Path: /hcnphz-57/?prefix=d 2023-09-29T22:33:38.270 DEBUG:boto:Data: 2023-09-29T22:33:38.270 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.270 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.270 DEBUG:boto:Params: {} 2023-09-29T22:33:38.270 DEBUG:boto:Token: None 2023-09-29T22:33:38.270 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.275 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa5416aae3ebe156-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.276 DEBUG:boto:b'hcnphz-57d1000falsed2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.276 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.276 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.276 DEBUG:boto:path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.276 DEBUG:boto:auth_path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.276 DEBUG:boto:Method: GET 2023-09-29T22:33:38.277 DEBUG:boto:Path: /hcnphz-57/?prefix=d 2023-09-29T22:33:38.277 DEBUG:boto:Data: 2023-09-29T22:33:38.277 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.277 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.277 DEBUG:boto:Params: {} 2023-09-29T22:33:38.277 DEBUG:boto:Token: None 2023-09-29T22:33:38.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b2fe2007c7e453d-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.281 DEBUG:boto:b'hcnphz-57d1000falsed2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.281 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.282 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.282 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.282 DEBUG:boto:Path: /hcnphz-57/d 2023-09-29T22:33:38.282 DEBUG:boto:Data: 2023-09-29T22:33:38.282 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.282 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.282 DEBUG:boto:Params: {} 2023-09-29T22:33:38.282 DEBUG:boto:Token: None 2023-09-29T22:33:38.283 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d 2023-09-29T22:33:38.283 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sXrNPXxItPomyWcosXLNpsqjQe0= 2023-09-29T22:33:38.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sXrNPXxItPomyWcosXLNpsqjQe0=', 'Content-Length': '0'} 2023-09-29T22:33:38.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e01908779695510b-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.286 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.286 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.286 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.286 DEBUG:boto:Path: /hcnphz-57/d 2023-09-29T22:33:38.287 DEBUG:boto:Data: 2023-09-29T22:33:38.287 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.287 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.287 DEBUG:boto:Params: {} 2023-09-29T22:33:38.287 DEBUG:boto:Token: None 2023-09-29T22:33:38.287 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d 2023-09-29T22:33:38.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sXrNPXxItPomyWcosXLNpsqjQe0= 2023-09-29T22:33:38.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sXrNPXxItPomyWcosXLNpsqjQe0=', 'Content-Length': '0'} 2023-09-29T22:33:38.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e1bd54c52342aec-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2023-09-29T22:33:38.290 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.291 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.291 DEBUG:boto:Method: GET 2023-09-29T22:33:38.291 DEBUG:boto:Path: /hcnphz-57/d 2023-09-29T22:33:38.291 DEBUG:boto:Data: 2023-09-29T22:33:38.291 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.291 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.291 DEBUG:boto:Params: {} 2023-09-29T22:33:38.292 DEBUG:boto:Token: None 2023-09-29T22:33:38.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d 2023-09-29T22:33:38.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fKCoa3j8xUGHG5AOwZ1IgngsnwE= 2023-09-29T22:33:38.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fKCoa3j8xUGHG5AOwZ1IgngsnwE=', 'Content-Length': '0'} 2023-09-29T22:33:38.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005288bd9c548f3ffe-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.338 DEBUG:boto:path=/hcnphz-57/d 2023-09-29T22:33:38.339 DEBUG:boto:auth_path=/hcnphz-57/d 2023-09-29T22:33:38.339 DEBUG:boto:Method: GET 2023-09-29T22:33:38.339 DEBUG:boto:Path: /hcnphz-57/d 2023-09-29T22:33:38.339 DEBUG:boto:Data: 2023-09-29T22:33:38.339 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.339 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.339 DEBUG:boto:Params: {} 2023-09-29T22:33:38.340 DEBUG:boto:Token: None 2023-09-29T22:33:38.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/d 2023-09-29T22:33:38.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fKCoa3j8xUGHG5AOwZ1IgngsnwE= 2023-09-29T22:33:38.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fKCoa3j8xUGHG5AOwZ1IgngsnwE=', 'Content-Length': '0'} 2023-09-29T22:33:38.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:30 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c063b75aa3a9a57d-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.343 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.343 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.344 DEBUG:boto:path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.344 DEBUG:boto:auth_path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.344 DEBUG:boto:Method: GET 2023-09-29T22:33:38.344 DEBUG:boto:Path: /hcnphz-57/?prefix=d 2023-09-29T22:33:38.344 DEBUG:boto:Data: 2023-09-29T22:33:38.344 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.344 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.344 DEBUG:boto:Params: {} 2023-09-29T22:33:38.345 DEBUG:boto:Token: None 2023-09-29T22:33:38.345 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.350 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7139bff54814d46-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.350 DEBUG:boto:b'hcnphz-57d1000falsed2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.350 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.350 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.351 DEBUG:boto:path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.351 DEBUG:boto:auth_path=/hcnphz-57/?prefix=d 2023-09-29T22:33:38.351 DEBUG:boto:Method: GET 2023-09-29T22:33:38.351 DEBUG:boto:Path: /hcnphz-57/?prefix=d 2023-09-29T22:33:38.351 DEBUG:boto:Data: 2023-09-29T22:33:38.351 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.351 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.351 DEBUG:boto:Params: {} 2023-09-29T22:33:38.351 DEBUG:boto:Token: None 2023-09-29T22:33:38.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d21bbfa16f187130-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.355 DEBUG:boto:b'hcnphz-57d1000falsed2023-09-29T22:31:30.781Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:33:38.355 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.356 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.356 DEBUG:boto:path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.356 DEBUG:boto:auth_path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.356 DEBUG:boto:Method: GET 2023-09-29T22:33:38.356 DEBUG:boto:Path: /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.356 DEBUG:boto:Data: 2023-09-29T22:33:38.356 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.357 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.357 DEBUG:boto:Params: {} 2023-09-29T22:33:38.357 DEBUG:boto:Token: None 2023-09-29T22:33:38.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.357 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg= 2023-09-29T22:33:38.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg=', 'Content-Length': '0'} 2023-09-29T22:33:38.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1253b4f22204ae9-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.360 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.360 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.361 DEBUG:boto:path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.361 DEBUG:boto:auth_path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.361 DEBUG:boto:Method: GET 2023-09-29T22:33:38.361 DEBUG:boto:Path: /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.361 DEBUG:boto:Data: 2023-09-29T22:33:38.361 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.361 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.361 DEBUG:boto:Params: {} 2023-09-29T22:33:38.362 DEBUG:boto:Token: None 2023-09-29T22:33:38.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg= 2023-09-29T22:33:38.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg=', 'Content-Length': '0'} 2023-09-29T22:33:38.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038ba2c7caa95909e-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.365 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.365 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.365 DEBUG:boto:path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.365 DEBUG:boto:auth_path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.365 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.365 DEBUG:boto:Path: /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.365 DEBUG:boto:Data: 2023-09-29T22:33:38.365 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.365 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.366 DEBUG:boto:Params: {} 2023-09-29T22:33:38.366 DEBUG:boto:Token: None 2023-09-29T22:33:38.366 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.366 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zFWFg6LJcTkhdMDSot1KiqPw95s= 2023-09-29T22:33:38.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zFWFg6LJcTkhdMDSot1KiqPw95s=', 'Content-Length': '0'} 2023-09-29T22:33:38.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000746044bf67039d78-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.369 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.369 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.369 DEBUG:boto:path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.369 DEBUG:boto:auth_path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.369 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.369 DEBUG:boto:Path: /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.369 DEBUG:boto:Data: 2023-09-29T22:33:38.370 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.370 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.370 DEBUG:boto:Params: {} 2023-09-29T22:33:38.370 DEBUG:boto:Token: None 2023-09-29T22:33:38.370 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.370 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zFWFg6LJcTkhdMDSot1KiqPw95s= 2023-09-29T22:33:38.371 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zFWFg6LJcTkhdMDSot1KiqPw95s=', 'Content-Length': '0'} 2023-09-29T22:33:38.373 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004303fbf369157893-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:33:38.374 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.374 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.375 DEBUG:boto:path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.375 DEBUG:boto:auth_path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.375 DEBUG:boto:Method: GET 2023-09-29T22:33:38.375 DEBUG:boto:Path: /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.375 DEBUG:boto:Data: 2023-09-29T22:33:38.375 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.375 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.375 DEBUG:boto:Params: {} 2023-09-29T22:33:38.376 DEBUG:boto:Token: None 2023-09-29T22:33:38.376 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg= 2023-09-29T22:33:38.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg=', 'Content-Length': '0'} 2023-09-29T22:33:38.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7de1e240fa49051-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.426 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.427 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.427 DEBUG:boto:path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.427 DEBUG:boto:auth_path=/hcnphz-57/e?versionId=null 2023-09-29T22:33:38.427 DEBUG:boto:Method: GET 2023-09-29T22:33:38.427 DEBUG:boto:Path: /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.428 DEBUG:boto:Data: 2023-09-29T22:33:38.428 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.428 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.428 DEBUG:boto:Params: {} 2023-09-29T22:33:38.428 DEBUG:boto:Token: None 2023-09-29T22:33:38.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e?versionId=null 2023-09-29T22:33:38.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg= 2023-09-29T22:33:38.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YDx9xPg//9lpoa5zJ8vhOnB+OZg=', 'Content-Length': '0'} 2023-09-29T22:33:38.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d925831b76f267ac-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.432 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.432 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.433 DEBUG:boto:path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.433 DEBUG:boto:auth_path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.433 DEBUG:boto:Method: GET 2023-09-29T22:33:38.433 DEBUG:boto:Path: /hcnphz-57/?prefix=e 2023-09-29T22:33:38.433 DEBUG:boto:Data: 2023-09-29T22:33:38.433 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.434 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.434 DEBUG:boto:Params: {} 2023-09-29T22:33:38.434 DEBUG:boto:Token: None 2023-09-29T22:33:38.434 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6943990205122af-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.439 DEBUG:boto:b'hcnphz-57e1000falsee2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.440 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.440 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.440 DEBUG:boto:path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.440 DEBUG:boto:auth_path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.441 DEBUG:boto:Method: GET 2023-09-29T22:33:38.441 DEBUG:boto:Path: /hcnphz-57/?prefix=e 2023-09-29T22:33:38.441 DEBUG:boto:Data: 2023-09-29T22:33:38.441 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.441 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.441 DEBUG:boto:Params: {} 2023-09-29T22:33:38.442 DEBUG:boto:Token: None 2023-09-29T22:33:38.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.445 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ede4ec8e693d12d8-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.486 DEBUG:boto:b'hcnphz-57e1000falsee2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.487 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.487 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.487 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.487 DEBUG:boto:Path: /hcnphz-57/e 2023-09-29T22:33:38.487 DEBUG:boto:Data: 2023-09-29T22:33:38.487 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.488 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.488 DEBUG:boto:Params: {} 2023-09-29T22:33:38.488 DEBUG:boto:Token: None 2023-09-29T22:33:38.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e 2023-09-29T22:33:38.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ck0zJF16h2m7m45O8KMCC3Q86cE= 2023-09-29T22:33:38.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ck0zJF16h2m7m45O8KMCC3Q86cE=', 'Content-Length': '0'} 2023-09-29T22:33:38.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000524c0b9cb3edcbbf-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.492 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.492 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.492 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.492 DEBUG:boto:Path: /hcnphz-57/e 2023-09-29T22:33:38.492 DEBUG:boto:Data: 2023-09-29T22:33:38.492 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.492 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.492 DEBUG:boto:Params: {} 2023-09-29T22:33:38.493 DEBUG:boto:Token: None 2023-09-29T22:33:38.493 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e 2023-09-29T22:33:38.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ck0zJF16h2m7m45O8KMCC3Q86cE= 2023-09-29T22:33:38.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ck0zJF16h2m7m45O8KMCC3Q86cE=', 'Content-Length': '0'} 2023-09-29T22:33:38.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92100b146e8876d-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=e 2023-09-29T22:33:38.496 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.496 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.496 DEBUG:boto:Method: GET 2023-09-29T22:33:38.497 DEBUG:boto:Path: /hcnphz-57/e 2023-09-29T22:33:38.497 DEBUG:boto:Data: 2023-09-29T22:33:38.497 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.497 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.497 DEBUG:boto:Params: {} 2023-09-29T22:33:38.497 DEBUG:boto:Token: None 2023-09-29T22:33:38.498 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e 2023-09-29T22:33:38.498 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0ZQ8OhobvcRyT2c74CBkGc04Lcc= 2023-09-29T22:33:38.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0ZQ8OhobvcRyT2c74CBkGc04Lcc=', 'Content-Length': '0'} 2023-09-29T22:33:38.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038467036cfc951eb-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.501 DEBUG:boto:path=/hcnphz-57/e 2023-09-29T22:33:38.501 DEBUG:boto:auth_path=/hcnphz-57/e 2023-09-29T22:33:38.501 DEBUG:boto:Method: GET 2023-09-29T22:33:38.501 DEBUG:boto:Path: /hcnphz-57/e 2023-09-29T22:33:38.501 DEBUG:boto:Data: 2023-09-29T22:33:38.502 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.502 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.502 DEBUG:boto:Params: {} 2023-09-29T22:33:38.502 DEBUG:boto:Token: None 2023-09-29T22:33:38.502 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/e 2023-09-29T22:33:38.502 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0ZQ8OhobvcRyT2c74CBkGc04Lcc= 2023-09-29T22:33:38.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0ZQ8OhobvcRyT2c74CBkGc04Lcc=', 'Content-Length': '0'} 2023-09-29T22:33:38.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c85a009d78582406-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.506 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.506 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.506 DEBUG:boto:path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.506 DEBUG:boto:auth_path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.506 DEBUG:boto:Method: GET 2023-09-29T22:33:38.507 DEBUG:boto:Path: /hcnphz-57/?prefix=e 2023-09-29T22:33:38.507 DEBUG:boto:Data: 2023-09-29T22:33:38.507 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.507 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.507 DEBUG:boto:Params: {} 2023-09-29T22:33:38.507 DEBUG:boto:Token: None 2023-09-29T22:33:38.507 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.507 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.512 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fcb43d49477a2917-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.512 DEBUG:boto:b'hcnphz-57e1000falsee2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.513 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.513 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.513 DEBUG:boto:path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.513 DEBUG:boto:auth_path=/hcnphz-57/?prefix=e 2023-09-29T22:33:38.513 DEBUG:boto:Method: GET 2023-09-29T22:33:38.513 DEBUG:boto:Path: /hcnphz-57/?prefix=e 2023-09-29T22:33:38.513 DEBUG:boto:Data: 2023-09-29T22:33:38.513 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.514 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.514 DEBUG:boto:Params: {} 2023-09-29T22:33:38.514 DEBUG:boto:Token: None 2023-09-29T22:33:38.514 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.517 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045e81222f65e756b-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.518 DEBUG:boto:b'hcnphz-57e1000falsee2023-09-29T22:31:31.011Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:33:38.518 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.518 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.518 DEBUG:boto:path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.518 DEBUG:boto:auth_path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.519 DEBUG:boto:Method: GET 2023-09-29T22:33:38.519 DEBUG:boto:Path: /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.519 DEBUG:boto:Data: 2023-09-29T22:33:38.519 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.519 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.519 DEBUG:boto:Params: {} 2023-09-29T22:33:38.519 DEBUG:boto:Token: None 2023-09-29T22:33:38.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE= 2023-09-29T22:33:38.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE=', 'Content-Length': '0'} 2023-09-29T22:33:38.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590b0424e04f83a0-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.566 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.567 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.567 DEBUG:boto:path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.567 DEBUG:boto:auth_path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.567 DEBUG:boto:Method: GET 2023-09-29T22:33:38.567 DEBUG:boto:Path: /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.567 DEBUG:boto:Data: 2023-09-29T22:33:38.567 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.567 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.567 DEBUG:boto:Params: {} 2023-09-29T22:33:38.568 DEBUG:boto:Token: None 2023-09-29T22:33:38.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE= 2023-09-29T22:33:38.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE=', 'Content-Length': '0'} 2023-09-29T22:33:38.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d0b44728948afa0-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.571 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.571 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.571 DEBUG:boto:path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.572 DEBUG:boto:auth_path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.572 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.572 DEBUG:boto:Path: /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.572 DEBUG:boto:Data: 2023-09-29T22:33:38.572 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.572 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.572 DEBUG:boto:Params: {} 2023-09-29T22:33:38.572 DEBUG:boto:Token: None 2023-09-29T22:33:38.573 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m2aWgysAar7ujpcZuMSbDLaQiIo= 2023-09-29T22:33:38.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m2aWgysAar7ujpcZuMSbDLaQiIo=', 'Content-Length': '0'} 2023-09-29T22:33:38.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d267baaae684debd-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.576 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.576 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.576 DEBUG:boto:path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.576 DEBUG:boto:auth_path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.576 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.576 DEBUG:boto:Path: /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.576 DEBUG:boto:Data: 2023-09-29T22:33:38.577 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.577 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.577 DEBUG:boto:Params: {} 2023-09-29T22:33:38.577 DEBUG:boto:Token: None 2023-09-29T22:33:38.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m2aWgysAar7ujpcZuMSbDLaQiIo= 2023-09-29T22:33:38.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m2aWgysAar7ujpcZuMSbDLaQiIo=', 'Content-Length': '0'} 2023-09-29T22:33:38.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004501e8066c49b2c2-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:33:38.580 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.580 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.581 DEBUG:boto:path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.581 DEBUG:boto:auth_path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.581 DEBUG:boto:Method: GET 2023-09-29T22:33:38.581 DEBUG:boto:Path: /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.581 DEBUG:boto:Data: 2023-09-29T22:33:38.581 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.581 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.582 DEBUG:boto:Params: {} 2023-09-29T22:33:38.582 DEBUG:boto:Token: None 2023-09-29T22:33:38.582 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE= 2023-09-29T22:33:38.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE=', 'Content-Length': '0'} 2023-09-29T22:33:38.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f42cae47721cc3c3-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.585 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.585 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.585 DEBUG:boto:path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.586 DEBUG:boto:auth_path=/hcnphz-57/f?versionId=null 2023-09-29T22:33:38.586 DEBUG:boto:Method: GET 2023-09-29T22:33:38.586 DEBUG:boto:Path: /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.586 DEBUG:boto:Data: 2023-09-29T22:33:38.586 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.586 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.586 DEBUG:boto:Params: {} 2023-09-29T22:33:38.586 DEBUG:boto:Token: None 2023-09-29T22:33:38.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f?versionId=null 2023-09-29T22:33:38.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE= 2023-09-29T22:33:38.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MOsIoT0ZbV5zzRbbNqvKwkBRyLE=', 'Content-Length': '0'} 2023-09-29T22:33:38.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b7f01e350e8bb8-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.630 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.631 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.631 DEBUG:boto:path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.631 DEBUG:boto:auth_path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.631 DEBUG:boto:Method: GET 2023-09-29T22:33:38.631 DEBUG:boto:Path: /hcnphz-57/?prefix=f 2023-09-29T22:33:38.631 DEBUG:boto:Data: 2023-09-29T22:33:38.631 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.631 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.632 DEBUG:boto:Params: {} 2023-09-29T22:33:38.632 DEBUG:boto:Token: None 2023-09-29T22:33:38.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029634327420f0c05-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.639 DEBUG:boto:b'hcnphz-57f1000falsef2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.639 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.639 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.639 DEBUG:boto:path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.639 DEBUG:boto:auth_path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.640 DEBUG:boto:Method: GET 2023-09-29T22:33:38.640 DEBUG:boto:Path: /hcnphz-57/?prefix=f 2023-09-29T22:33:38.640 DEBUG:boto:Data: 2023-09-29T22:33:38.640 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.640 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.640 DEBUG:boto:Params: {} 2023-09-29T22:33:38.640 DEBUG:boto:Token: None 2023-09-29T22:33:38.641 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.644 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066262dc790780330-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.644 DEBUG:boto:b'hcnphz-57f1000falsef2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.645 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.645 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.645 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.645 DEBUG:boto:Path: /hcnphz-57/f 2023-09-29T22:33:38.645 DEBUG:boto:Data: 2023-09-29T22:33:38.645 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.645 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.646 DEBUG:boto:Params: {} 2023-09-29T22:33:38.646 DEBUG:boto:Token: None 2023-09-29T22:33:38.646 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f 2023-09-29T22:33:38.646 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gIrX+mnTS6u/XUYmNb+IaGdM/Gg= 2023-09-29T22:33:38.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gIrX+mnTS6u/XUYmNb+IaGdM/Gg=', 'Content-Length': '0'} 2023-09-29T22:33:38.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720552be2ee2e21e-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.649 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.649 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.650 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.650 DEBUG:boto:Path: /hcnphz-57/f 2023-09-29T22:33:38.650 DEBUG:boto:Data: 2023-09-29T22:33:38.650 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.650 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.650 DEBUG:boto:Params: {} 2023-09-29T22:33:38.650 DEBUG:boto:Token: None 2023-09-29T22:33:38.651 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f 2023-09-29T22:33:38.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gIrX+mnTS6u/XUYmNb+IaGdM/Gg= 2023-09-29T22:33:38.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gIrX+mnTS6u/XUYmNb+IaGdM/Gg=', 'Content-Length': '0'} 2023-09-29T22:33:38.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7b9150851463799-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=f 2023-09-29T22:33:38.654 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.654 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.654 DEBUG:boto:Method: GET 2023-09-29T22:33:38.654 DEBUG:boto:Path: /hcnphz-57/f 2023-09-29T22:33:38.654 DEBUG:boto:Data: 2023-09-29T22:33:38.654 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.655 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.655 DEBUG:boto:Params: {} 2023-09-29T22:33:38.655 DEBUG:boto:Token: None 2023-09-29T22:33:38.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f 2023-09-29T22:33:38.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1WPJ6b+Hrskec2E8Mmm0M4GmHpU= 2023-09-29T22:33:38.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1WPJ6b+Hrskec2E8Mmm0M4GmHpU=', 'Content-Length': '0'} 2023-09-29T22:33:38.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bb47418b994736c-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.702 DEBUG:boto:path=/hcnphz-57/f 2023-09-29T22:33:38.703 DEBUG:boto:auth_path=/hcnphz-57/f 2023-09-29T22:33:38.703 DEBUG:boto:Method: GET 2023-09-29T22:33:38.703 DEBUG:boto:Path: /hcnphz-57/f 2023-09-29T22:33:38.703 DEBUG:boto:Data: 2023-09-29T22:33:38.703 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.703 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.703 DEBUG:boto:Params: {} 2023-09-29T22:33:38.703 DEBUG:boto:Token: None 2023-09-29T22:33:38.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/f 2023-09-29T22:33:38.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1WPJ6b+Hrskec2E8Mmm0M4GmHpU= 2023-09-29T22:33:38.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1WPJ6b+Hrskec2E8Mmm0M4GmHpU=', 'Content-Length': '0'} 2023-09-29T22:33:38.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee09b3f745e930b2-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.707 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.707 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.707 DEBUG:boto:path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.707 DEBUG:boto:auth_path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.708 DEBUG:boto:Method: GET 2023-09-29T22:33:38.708 DEBUG:boto:Path: /hcnphz-57/?prefix=f 2023-09-29T22:33:38.708 DEBUG:boto:Data: 2023-09-29T22:33:38.708 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.708 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.708 DEBUG:boto:Params: {} 2023-09-29T22:33:38.708 DEBUG:boto:Token: None 2023-09-29T22:33:38.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c34571f8fb6fe3e-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.714 DEBUG:boto:b'hcnphz-57f1000falsef2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.714 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.714 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.715 DEBUG:boto:path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.715 DEBUG:boto:auth_path=/hcnphz-57/?prefix=f 2023-09-29T22:33:38.715 DEBUG:boto:Method: GET 2023-09-29T22:33:38.715 DEBUG:boto:Path: /hcnphz-57/?prefix=f 2023-09-29T22:33:38.715 DEBUG:boto:Data: 2023-09-29T22:33:38.715 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.715 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.716 DEBUG:boto:Params: {} 2023-09-29T22:33:38.716 DEBUG:boto:Token: None 2023-09-29T22:33:38.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.719 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1099181723bc4a8-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.762 DEBUG:boto:b'hcnphz-57f1000falsef2023-09-29T22:31:31.027Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:33:38.763 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.763 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.764 DEBUG:boto:path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.764 DEBUG:boto:auth_path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.764 DEBUG:boto:Method: GET 2023-09-29T22:33:38.764 DEBUG:boto:Path: /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.764 DEBUG:boto:Data: 2023-09-29T22:33:38.764 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.765 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.765 DEBUG:boto:Params: {} 2023-09-29T22:33:38.765 DEBUG:boto:Token: None 2023-09-29T22:33:38.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.766 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg= 2023-09-29T22:33:38.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg=', 'Content-Length': '0'} 2023-09-29T22:33:38.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053638d160bfa667d-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.770 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.770 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.770 DEBUG:boto:path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.770 DEBUG:boto:auth_path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.770 DEBUG:boto:Method: GET 2023-09-29T22:33:38.771 DEBUG:boto:Path: /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.771 DEBUG:boto:Data: 2023-09-29T22:33:38.771 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.772 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.772 DEBUG:boto:Params: {} 2023-09-29T22:33:38.772 DEBUG:boto:Token: None 2023-09-29T22:33:38.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg= 2023-09-29T22:33:38.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg=', 'Content-Length': '0'} 2023-09-29T22:33:38.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e233c3fee0459d39-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.776 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.776 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.777 DEBUG:boto:path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.777 DEBUG:boto:auth_path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.777 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.777 DEBUG:boto:Path: /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.777 DEBUG:boto:Data: 2023-09-29T22:33:38.778 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.778 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.778 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.778 DEBUG:boto:Params: {} 2023-09-29T22:33:38.778 DEBUG:boto:Token: None 2023-09-29T22:33:38.779 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.779 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5ivFkQDvBVJtChHNtoBm+kP3TKc= 2023-09-29T22:33:38.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5ivFkQDvBVJtChHNtoBm+kP3TKc=', 'Content-Length': '0'} 2023-09-29T22:33:38.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b39da68517caf6df-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.782 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.783 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.783 DEBUG:boto:path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.783 DEBUG:boto:auth_path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.783 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.783 DEBUG:boto:Path: /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.783 DEBUG:boto:Data: 2023-09-29T22:33:38.784 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.784 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.784 DEBUG:boto:Params: {} 2023-09-29T22:33:38.784 DEBUG:boto:Token: None 2023-09-29T22:33:38.785 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.785 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5ivFkQDvBVJtChHNtoBm+kP3TKc= 2023-09-29T22:33:38.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5ivFkQDvBVJtChHNtoBm+kP3TKc=', 'Content-Length': '0'} 2023-09-29T22:33:38.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f19a95f7f1710ba5-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:33:38.788 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.788 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.789 DEBUG:boto:path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.789 DEBUG:boto:auth_path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.789 DEBUG:boto:Method: GET 2023-09-29T22:33:38.789 DEBUG:boto:Path: /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.789 DEBUG:boto:Data: 2023-09-29T22:33:38.789 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.789 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.789 DEBUG:boto:Params: {} 2023-09-29T22:33:38.790 DEBUG:boto:Token: None 2023-09-29T22:33:38.790 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg= 2023-09-29T22:33:38.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg=', 'Content-Length': '0'} 2023-09-29T22:33:38.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000812025554667998f-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.793 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.793 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.793 DEBUG:boto:path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.793 DEBUG:boto:auth_path=/hcnphz-57/g?versionId=null 2023-09-29T22:33:38.793 DEBUG:boto:Method: GET 2023-09-29T22:33:38.794 DEBUG:boto:Path: /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.794 DEBUG:boto:Data: 2023-09-29T22:33:38.794 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.794 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.794 DEBUG:boto:Params: {} 2023-09-29T22:33:38.794 DEBUG:boto:Token: None 2023-09-29T22:33:38.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g?versionId=null 2023-09-29T22:33:38.795 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg= 2023-09-29T22:33:38.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QHY/PIAue+/p9uVwH8iIfVHWWFg=', 'Content-Length': '0'} 2023-09-29T22:33:38.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc59ba085c57f66-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.839 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.839 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.839 DEBUG:boto:path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.839 DEBUG:boto:auth_path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.840 DEBUG:boto:Method: GET 2023-09-29T22:33:38.840 DEBUG:boto:Path: /hcnphz-57/?prefix=g 2023-09-29T22:33:38.840 DEBUG:boto:Data: 2023-09-29T22:33:38.840 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.840 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.840 DEBUG:boto:Params: {} 2023-09-29T22:33:38.840 DEBUG:boto:Token: None 2023-09-29T22:33:38.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.841 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.845 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cea0e2c50e8fc061-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.845 DEBUG:boto:b'hcnphz-57g1000falseg2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.846 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.846 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.846 DEBUG:boto:path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.846 DEBUG:boto:auth_path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.846 DEBUG:boto:Method: GET 2023-09-29T22:33:38.846 DEBUG:boto:Path: /hcnphz-57/?prefix=g 2023-09-29T22:33:38.846 DEBUG:boto:Data: 2023-09-29T22:33:38.846 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.847 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.847 DEBUG:boto:Params: {} 2023-09-29T22:33:38.847 DEBUG:boto:Token: None 2023-09-29T22:33:38.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.847 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002ca5c3773af1167d-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.851 DEBUG:boto:b'hcnphz-57g1000falseg2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.851 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.851 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.851 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.851 DEBUG:boto:Path: /hcnphz-57/g 2023-09-29T22:33:38.852 DEBUG:boto:Data: 2023-09-29T22:33:38.852 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.852 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.852 DEBUG:boto:Params: {} 2023-09-29T22:33:38.852 DEBUG:boto:Token: None 2023-09-29T22:33:38.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g 2023-09-29T22:33:38.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D19VCQKoukgwct4JPkrbMAKB2OE= 2023-09-29T22:33:38.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D19VCQKoukgwct4JPkrbMAKB2OE=', 'Content-Length': '0'} 2023-09-29T22:33:38.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028ed2934b0a473f3-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.855 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.856 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.856 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.856 DEBUG:boto:Path: /hcnphz-57/g 2023-09-29T22:33:38.856 DEBUG:boto:Data: 2023-09-29T22:33:38.856 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.856 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.856 DEBUG:boto:Params: {} 2023-09-29T22:33:38.856 DEBUG:boto:Token: None 2023-09-29T22:33:38.857 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g 2023-09-29T22:33:38.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D19VCQKoukgwct4JPkrbMAKB2OE= 2023-09-29T22:33:38.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D19VCQKoukgwct4JPkrbMAKB2OE=', 'Content-Length': '0'} 2023-09-29T22:33:38.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb2b7b87dacb1a7d-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=g 2023-09-29T22:33:38.860 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.860 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.860 DEBUG:boto:Method: GET 2023-09-29T22:33:38.860 DEBUG:boto:Path: /hcnphz-57/g 2023-09-29T22:33:38.860 DEBUG:boto:Data: 2023-09-29T22:33:38.860 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.861 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.861 DEBUG:boto:Params: {} 2023-09-29T22:33:38.861 DEBUG:boto:Token: None 2023-09-29T22:33:38.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g 2023-09-29T22:33:38.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ny53lfN0HfCQCvKsGt3wj8vyf0c= 2023-09-29T22:33:38.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ny53lfN0HfCQCvKsGt3wj8vyf0c=', 'Content-Length': '0'} 2023-09-29T22:33:38.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000540e4e781e4b7380-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.907 DEBUG:boto:path=/hcnphz-57/g 2023-09-29T22:33:38.907 DEBUG:boto:auth_path=/hcnphz-57/g 2023-09-29T22:33:38.907 DEBUG:boto:Method: GET 2023-09-29T22:33:38.907 DEBUG:boto:Path: /hcnphz-57/g 2023-09-29T22:33:38.908 DEBUG:boto:Data: 2023-09-29T22:33:38.908 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.908 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.908 DEBUG:boto:Params: {} 2023-09-29T22:33:38.908 DEBUG:boto:Token: None 2023-09-29T22:33:38.909 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/g 2023-09-29T22:33:38.909 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ny53lfN0HfCQCvKsGt3wj8vyf0c= 2023-09-29T22:33:38.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ny53lfN0HfCQCvKsGt3wj8vyf0c=', 'Content-Length': '0'} 2023-09-29T22:33:38.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b52165ec72d2013-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.913 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.913 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.913 DEBUG:boto:path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.913 DEBUG:boto:auth_path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.913 DEBUG:boto:Method: GET 2023-09-29T22:33:38.913 DEBUG:boto:Path: /hcnphz-57/?prefix=g 2023-09-29T22:33:38.913 DEBUG:boto:Data: 2023-09-29T22:33:38.914 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.914 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.914 DEBUG:boto:Params: {} 2023-09-29T22:33:38.914 DEBUG:boto:Token: None 2023-09-29T22:33:38.914 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.919 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008d694b15c1dd8e0-00651750c2-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.920 DEBUG:boto:b'hcnphz-57g1000falseg2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.920 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:38.920 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:38.920 DEBUG:boto:path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.920 DEBUG:boto:auth_path=/hcnphz-57/?prefix=g 2023-09-29T22:33:38.921 DEBUG:boto:Method: GET 2023-09-29T22:33:38.921 DEBUG:boto:Path: /hcnphz-57/?prefix=g 2023-09-29T22:33:38.921 DEBUG:boto:Data: 2023-09-29T22:33:38.921 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.921 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.921 DEBUG:boto:Params: {} 2023-09-29T22:33:38.921 DEBUG:boto:Token: None 2023-09-29T22:33:38.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/ 2023-09-29T22:33:38.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM= 2023-09-29T22:33:38.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:utpOZXrTkEjuDlw8EsjCs8J8FHM=', 'Content-Length': '0'} 2023-09-29T22:33:38.925 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072c95299b3782ada-00651750c2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.970 DEBUG:boto:b'hcnphz-57g1000falseg2023-09-29T22:31:31.079Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:38.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:33:38.971 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:38.971 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:38.972 DEBUG:boto:path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.972 DEBUG:boto:auth_path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.972 DEBUG:boto:Method: GET 2023-09-29T22:33:38.972 DEBUG:boto:Path: /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.972 DEBUG:boto:Data: 2023-09-29T22:33:38.972 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.972 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.972 DEBUG:boto:Params: {} 2023-09-29T22:33:38.973 DEBUG:boto:Token: None 2023-09-29T22:33:38.973 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.973 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84= 2023-09-29T22:33:38.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84=', 'Content-Length': '0'} 2023-09-29T22:33:38.976 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e9c024d0e486f14-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.977 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:38.977 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:38.977 DEBUG:boto:path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.977 DEBUG:boto:auth_path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.977 DEBUG:boto:Method: GET 2023-09-29T22:33:38.977 DEBUG:boto:Path: /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.977 DEBUG:boto:Data: 2023-09-29T22:33:38.977 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.978 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.978 DEBUG:boto:Params: {} 2023-09-29T22:33:38.978 DEBUG:boto:Token: None 2023-09-29T22:33:38.978 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.978 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84= 2023-09-29T22:33:38.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84=', 'Content-Length': '0'} 2023-09-29T22:33:38.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065312ad469228d51-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.982 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:38.982 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:38.982 DEBUG:boto:path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.982 DEBUG:boto:auth_path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.982 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.982 DEBUG:boto:Path: /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.982 DEBUG:boto:Data: 2023-09-29T22:33:38.982 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.983 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.983 DEBUG:boto:Params: {} 2023-09-29T22:33:38.983 DEBUG:boto:Token: None 2023-09-29T22:33:38.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Szjl4hiXJwGWf8x6tgw+XliFy0= 2023-09-29T22:33:38.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Szjl4hiXJwGWf8x6tgw+XliFy0=', 'Content-Length': '0'} 2023-09-29T22:33:38.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e2fdbff176a6b81-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.986 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:38.986 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:38.986 DEBUG:boto:path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.986 DEBUG:boto:auth_path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.987 DEBUG:boto:Method: HEAD 2023-09-29T22:33:38.987 DEBUG:boto:Path: /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.987 DEBUG:boto:Data: 2023-09-29T22:33:38.987 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.987 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.987 DEBUG:boto:Params: {} 2023-09-29T22:33:38.987 DEBUG:boto:Token: None 2023-09-29T22:33:38.987 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Szjl4hiXJwGWf8x6tgw+XliFy0= 2023-09-29T22:33:38.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Szjl4hiXJwGWf8x6tgw+XliFy0=', 'Content-Length': '0'} 2023-09-29T22:33:38.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6b481df2edf649-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:33:38.991 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:38.991 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:38.991 DEBUG:boto:path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.991 DEBUG:boto:auth_path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.991 DEBUG:boto:Method: GET 2023-09-29T22:33:38.991 DEBUG:boto:Path: /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.992 DEBUG:boto:Data: 2023-09-29T22:33:38.992 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:38.992 DEBUG:boto:Port: 8001 2023-09-29T22:33:38.992 DEBUG:boto:Params: {} 2023-09-29T22:33:38.992 DEBUG:boto:Token: None 2023-09-29T22:33:38.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84= 2023-09-29T22:33:38.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84=', 'Content-Length': '0'} 2023-09-29T22:33:38.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006057be7c3fd82ee8-00651750c2-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:38.996 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:38.996 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:38.996 DEBUG:boto:path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.996 DEBUG:boto:auth_path=/hcnphz-57/h?versionId=null 2023-09-29T22:33:38.996 DEBUG:boto:Method: GET 2023-09-29T22:33:38.996 DEBUG:boto:Path: /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.996 DEBUG:boto:Data: 2023-09-29T22:33:38.997 DEBUG:boto:Headers: {} 2023-09-29T22:33:38.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:38.997 DEBUG:boto:Port: 8000 2023-09-29T22:33:38.997 DEBUG:boto:Params: {} 2023-09-29T22:33:38.997 DEBUG:boto:Token: None 2023-09-29T22:33:38.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:38 GMT /hcnphz-57/h?versionId=null 2023-09-29T22:33:38.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84= 2023-09-29T22:33:38.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nv4jYvC6LplGh2bzZcq2JTGrH84=', 'Content-Length': '0'} 2023-09-29T22:33:39.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c977c94f08a6fcfa-00651750c2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.002 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:39.002 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:39.002 DEBUG:boto:path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.002 DEBUG:boto:auth_path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.002 DEBUG:boto:Method: GET 2023-09-29T22:33:39.002 DEBUG:boto:Path: /hcnphz-57/?prefix=h 2023-09-29T22:33:39.002 DEBUG:boto:Data: 2023-09-29T22:33:39.002 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:39.003 DEBUG:boto:Port: 8001 2023-09-29T22:33:39.003 DEBUG:boto:Params: {} 2023-09-29T22:33:39.003 DEBUG:boto:Token: None 2023-09-29T22:33:39.003 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/ 2023-09-29T22:33:39.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM= 2023-09-29T22:33:39.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM=', 'Content-Length': '0'} 2023-09-29T22:33:39.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cd78fade4d75a98-00651750c3-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.054 DEBUG:boto:b'hcnphz-57h1000falseh2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:39.055 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:39.055 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:39.056 DEBUG:boto:path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.056 DEBUG:boto:auth_path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.056 DEBUG:boto:Method: GET 2023-09-29T22:33:39.056 DEBUG:boto:Path: /hcnphz-57/?prefix=h 2023-09-29T22:33:39.056 DEBUG:boto:Data: 2023-09-29T22:33:39.056 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.057 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.057 DEBUG:boto:Params: {} 2023-09-29T22:33:39.057 DEBUG:boto:Token: None 2023-09-29T22:33:39.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/ 2023-09-29T22:33:39.058 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM= 2023-09-29T22:33:39.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM=', 'Content-Length': '0'} 2023-09-29T22:33:39.061 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ad210db9f5a854a-00651750c3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.061 DEBUG:boto:b'hcnphz-57h1000falseh2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:39.062 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:39.062 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:39.062 DEBUG:boto:Method: HEAD 2023-09-29T22:33:39.062 DEBUG:boto:Path: /hcnphz-57/h 2023-09-29T22:33:39.062 DEBUG:boto:Data: 2023-09-29T22:33:39.062 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:39.063 DEBUG:boto:Port: 8001 2023-09-29T22:33:39.063 DEBUG:boto:Params: {} 2023-09-29T22:33:39.063 DEBUG:boto:Token: None 2023-09-29T22:33:39.063 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/h 2023-09-29T22:33:39.063 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uVgMhraViLjmpgwHFkhTMFo/ek4= 2023-09-29T22:33:39.063 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uVgMhraViLjmpgwHFkhTMFo/ek4=', 'Content-Length': '0'} 2023-09-29T22:33:39.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c618fa4b0b50165d-00651750c3-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.066 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:39.067 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:39.067 DEBUG:boto:Method: HEAD 2023-09-29T22:33:39.067 DEBUG:boto:Path: /hcnphz-57/h 2023-09-29T22:33:39.067 DEBUG:boto:Data: 2023-09-29T22:33:39.067 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.067 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.067 DEBUG:boto:Params: {} 2023-09-29T22:33:39.068 DEBUG:boto:Token: None 2023-09-29T22:33:39.068 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/h 2023-09-29T22:33:39.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uVgMhraViLjmpgwHFkhTMFo/ek4= 2023-09-29T22:33:39.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uVgMhraViLjmpgwHFkhTMFo/ek4=', 'Content-Length': '0'} 2023-09-29T22:33:39.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae2222ebcdebb78-00651750c3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=h 2023-09-29T22:33:39.071 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:39.071 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:39.071 DEBUG:boto:Method: GET 2023-09-29T22:33:39.072 DEBUG:boto:Path: /hcnphz-57/h 2023-09-29T22:33:39.072 DEBUG:boto:Data: 2023-09-29T22:33:39.072 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:39.072 DEBUG:boto:Port: 8001 2023-09-29T22:33:39.073 DEBUG:boto:Params: {} 2023-09-29T22:33:39.073 DEBUG:boto:Token: None 2023-09-29T22:33:39.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/h 2023-09-29T22:33:39.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dsbYTCikoshK+1/NIw8B1EB+OLM= 2023-09-29T22:33:39.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dsbYTCikoshK+1/NIw8B1EB+OLM=', 'Content-Length': '0'} 2023-09-29T22:33:39.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-57:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.22'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000830a4fa927c0d3ce-00651750c3-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.077 DEBUG:boto:path=/hcnphz-57/h 2023-09-29T22:33:39.077 DEBUG:boto:auth_path=/hcnphz-57/h 2023-09-29T22:33:39.078 DEBUG:boto:Method: GET 2023-09-29T22:33:39.078 DEBUG:boto:Path: /hcnphz-57/h 2023-09-29T22:33:39.078 DEBUG:boto:Data: 2023-09-29T22:33:39.078 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.078 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.078 DEBUG:boto:Params: {} 2023-09-29T22:33:39.078 DEBUG:boto:Token: None 2023-09-29T22:33:39.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/h 2023-09-29T22:33:39.079 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dsbYTCikoshK+1/NIw8B1EB+OLM= 2023-09-29T22:33:39.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dsbYTCikoshK+1/NIw8B1EB+OLM=', 'Content-Length': '0'} 2023-09-29T22:33:39.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:31:31 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003785439d3e1fe046-00651750c3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.122 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:39.123 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:39.123 DEBUG:boto:path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.123 DEBUG:boto:auth_path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.123 DEBUG:boto:Method: GET 2023-09-29T22:33:39.123 DEBUG:boto:Path: /hcnphz-57/?prefix=h 2023-09-29T22:33:39.123 DEBUG:boto:Data: 2023-09-29T22:33:39.123 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:39.124 DEBUG:boto:Port: 8001 2023-09-29T22:33:39.124 DEBUG:boto:Params: {} 2023-09-29T22:33:39.124 DEBUG:boto:Token: None 2023-09-29T22:33:39.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/ 2023-09-29T22:33:39.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM= 2023-09-29T22:33:39.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM=', 'Content-Length': '0'} 2023-09-29T22:33:39.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b30af2955b4f648e-00651750c3-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.131 DEBUG:boto:b'hcnphz-57h1000falseh2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:39.131 DEBUG:boto:path=/hcnphz-57/ 2023-09-29T22:33:39.132 DEBUG:boto:auth_path=/hcnphz-57/ 2023-09-29T22:33:39.132 DEBUG:boto:path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.132 DEBUG:boto:auth_path=/hcnphz-57/?prefix=h 2023-09-29T22:33:39.132 DEBUG:boto:Method: GET 2023-09-29T22:33:39.132 DEBUG:boto:Path: /hcnphz-57/?prefix=h 2023-09-29T22:33:39.132 DEBUG:boto:Data: 2023-09-29T22:33:39.132 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.132 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.133 DEBUG:boto:Params: {} 2023-09-29T22:33:39.133 DEBUG:boto:Token: None 2023-09-29T22:33:39.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-57/ 2023-09-29T22:33:39.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM= 2023-09-29T22:33:39.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JxF/9Z0ETTpHsbkQ4jR+8jlA5PM=', 'Content-Length': '0'} 2023-09-29T22:33:39.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b4b236676d65ed5-00651750c3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.137 DEBUG:boto:b'hcnphz-57h1000falseh2023-09-29T22:31:31.094Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:33:39.137 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-57 zones={a2, a1} 2023-09-29T22:33:39.138 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_reshard_full ... ok 2023-09-29T22:33:39.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.141 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:39.186 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:39.186 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:39.244 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:33:39.271 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:33:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:33:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:33:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026690.934977_454.1", 2023-09-29T22:33:39.272 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:31:30.934977Z" 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:33:39.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.274 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:33:39.274 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026690.934977_454.1', 2: '', 3: ''} 2023-09-29T22:33:39.275 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:33:39.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.275 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:39.321 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:39.322 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:39.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:39.377+0000 7fdf868b3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:39.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:39.377+0000 7fdf868b3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:39.405 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.407 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026690.934977_454.1", 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:33:39.408 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:31:30.934977Z", 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.409 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.410 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.410 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:33:39.411 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.411 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.412 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:33:39.413 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:33:39.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.414 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:33:39Z" 2023-09-29T22:33:39.414 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:33:39.415 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026690.934977_454.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:31:30.934977Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:39Z" } 2023-09-29T22:33:39.415 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026690.934977_454.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:31:30.934977Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:33:39.415 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026690.934977_454.1', 2: '', 3: ''} 2023-09-29T22:33:39.415 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026690.934977_454.1', 2: '', 3: ''} 2023-09-29T22:33:39.416 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:33:39.416 DEBUG:boto:path=/ 2023-09-29T22:33:39.416 DEBUG:boto:auth_path=/ 2023-09-29T22:33:39.416 DEBUG:boto:Method: GET 2023-09-29T22:33:39.416 DEBUG:boto:Path: / 2023-09-29T22:33:39.416 DEBUG:boto:Data: 2023-09-29T22:33:39.417 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.417 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.417 DEBUG:boto:Params: {} 2023-09-29T22:33:39.417 DEBUG:boto:Token: None 2023-09-29T22:33:39.417 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT / 2023-09-29T22:33:39.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2WHfDDI94DEyJvh7cNlblFd2mk= 2023-09-29T22:33:39.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2WHfDDI94DEyJvh7cNlblFd2mk=', 'Content-Length': '0'} 2023-09-29T22:33:39.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000315f0afa681811d9-00651750c3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.430 DEBUG:boto:path=/ 2023-09-29T22:33:39.430 DEBUG:boto:auth_path=/ 2023-09-29T22:33:39.430 DEBUG:boto:Method: GET 2023-09-29T22:33:39.430 DEBUG:boto:Path: / 2023-09-29T22:33:39.430 DEBUG:boto:Data: 2023-09-29T22:33:39.431 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:33:39.431 DEBUG:boto:Port: 8001 2023-09-29T22:33:39.431 DEBUG:boto:Params: {} 2023-09-29T22:33:39.431 DEBUG:boto:Token: None 2023-09-29T22:33:39.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:33:39 GMT / 2023-09-29T22:33:39.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2WHfDDI94DEyJvh7cNlblFd2mk= 2023-09-29T22:33:39.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2WHfDDI94DEyJvh7cNlblFd2mk=', 'Content-Length': '0'} 2023-09-29T22:33:39.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002897e531763413e-00651750c3-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.440 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2023-09-29T22:33:39.441 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:39.442 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:39.442 DEBUG:boto:Method: PUT 2023-09-29T22:33:39.442 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:39.442 DEBUG:boto:Data: 2023-09-29T22:33:39.442 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.442 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.442 DEBUG:boto:Params: {} 2023-09-29T22:33:39.443 DEBUG:boto:Token: None 2023-09-29T22:33:39.443 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/ 2023-09-29T22:33:39.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p2ahnmJQPOtK62oUbxsPpTLpONs= 2023-09-29T22:33:39.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p2ahnmJQPOtK62oUbxsPpTLpONs=', 'Content-Length': '0'} 2023-09-29T22:33:39.473 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f78e1dc44853f85d-00651750c3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.473 DEBUG:rgw_multi.tests:created bucket=hcnphz-58 2023-09-29T22:33:39.473 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.474 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:39.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:39.515 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:39.574 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:33:39.599 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:33:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:33:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:33:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:33:39.601 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:33:39.471267Z" 2023-09-29T22:33:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.602 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:33:39.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.604 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:33:39.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.605 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:33:39.607 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:33:39.607 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:33:39.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.609 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:39.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:39.696 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:39.745 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:39.745+0000 7fd5abebff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:39.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:39.745+0000 7fd5abebff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:33:39.771 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:33:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:33:39.771 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:33:39.772 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.775 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:33:39.776 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.777 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:33:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:33:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:33:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:33:39.778 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:33:39Z" 2023-09-29T22:33:39.778 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:33:39.779 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:33:39Z" } 2023-09-29T22:33:39.779 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:33:39.779 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:33:39.780 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:33:39.780 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:33:39.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:33:39.780 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:33:39.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:39.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:39.903 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:33:39.903 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:33:39.903 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:33:39.903 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-09-29T22:33:39.903 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006" 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:33:39.904 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:33:39.906 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:39.906 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:39.906 DEBUG:boto:Method: HEAD 2023-09-29T22:33:39.907 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:39.907 DEBUG:boto:Data: 2023-09-29T22:33:39.907 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.907 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.907 DEBUG:boto:Params: {} 2023-09-29T22:33:39.907 DEBUG:boto:Token: None 2023-09-29T22:33:39.907 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/ 2023-09-29T22:33:39.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a5CyZ2+3EokGddMO4UiPwb0/VZg= 2023-09-29T22:33:39.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a5CyZ2+3EokGddMO4UiPwb0/VZg=', 'Content-Length': '0'} 2023-09-29T22:33:39.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008763f7d04ab7688a-00651750c3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.911 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:33:39.912 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:33:39.912 DEBUG:boto:Method: PUT 2023-09-29T22:33:39.912 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:33:39.912 DEBUG:boto:Data: 2023-09-29T22:33:39.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:39.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.912 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.912 DEBUG:boto:Params: {} 2023-09-29T22:33:39.912 DEBUG:boto:Token: None 2023-09-29T22:33:39.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:33:39.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0vXn65afH6jElbkE91i6bY+Ng0s= 2023-09-29T22:33:39.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0vXn65afH6jElbkE91i6bY+Ng0s='} 2023-09-29T22:33:39.962 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020ad6be33056fd6e-00651750c3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.963 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:39.963 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:39.963 DEBUG:boto:Method: HEAD 2023-09-29T22:33:39.963 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:39.963 DEBUG:boto:Data: 2023-09-29T22:33:39.963 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.964 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.964 DEBUG:boto:Params: {} 2023-09-29T22:33:39.964 DEBUG:boto:Token: None 2023-09-29T22:33:39.964 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/ 2023-09-29T22:33:39.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a5CyZ2+3EokGddMO4UiPwb0/VZg= 2023-09-29T22:33:39.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a5CyZ2+3EokGddMO4UiPwb0/VZg=', 'Content-Length': '0'} 2023-09-29T22:33:39.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000c7969543bfebcb9-00651750c3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.969 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:33:39.969 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:33:39.969 DEBUG:boto:Method: PUT 2023-09-29T22:33:39.969 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:33:39.969 DEBUG:boto:Data: 2023-09-29T22:33:39.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:39.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.969 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.970 DEBUG:boto:Params: {} 2023-09-29T22:33:39.970 DEBUG:boto:Token: None 2023-09-29T22:33:39.970 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:33:39.970 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a4qrj/5bhdoavO3w8KS1I4M6BlE= 2023-09-29T22:33:39.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a4qrj/5bhdoavO3w8KS1I4M6BlE='} 2023-09-29T22:33:39.982 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d34b274f08da330e-00651750c3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.982 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:39.982 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:39.982 DEBUG:boto:Method: HEAD 2023-09-29T22:33:39.982 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:39.982 DEBUG:boto:Data: 2023-09-29T22:33:39.982 DEBUG:boto:Headers: {} 2023-09-29T22:33:39.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.983 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.983 DEBUG:boto:Params: {} 2023-09-29T22:33:39.983 DEBUG:boto:Token: None 2023-09-29T22:33:39.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/ 2023-09-29T22:33:39.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a5CyZ2+3EokGddMO4UiPwb0/VZg= 2023-09-29T22:33:39.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a5CyZ2+3EokGddMO4UiPwb0/VZg=', 'Content-Length': '0'} 2023-09-29T22:33:39.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000063613711f949cf62-00651750c3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:39.987 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:33:39.987 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:33:39.987 DEBUG:boto:Method: PUT 2023-09-29T22:33:39.987 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:33:39.987 DEBUG:boto:Data: 2023-09-29T22:33:39.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:39.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:39.988 DEBUG:boto:Port: 8000 2023-09-29T22:33:39.988 DEBUG:boto:Params: {} 2023-09-29T22:33:39.988 DEBUG:boto:Token: None 2023-09-29T22:33:39.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:39 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:33:39.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r+toKfgHeW1skluu1fuIRmAIR3U= 2023-09-29T22:33:39.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:39 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r+toKfgHeW1skluu1fuIRmAIR3U='} 2023-09-29T22:33:40.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004c83c5628c19cb0d-00651750c3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.035 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.035 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.035 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.035 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.035 DEBUG:boto:Data: 2023-09-29T22:33:40.035 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.036 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.036 DEBUG:boto:Params: {} 2023-09-29T22:33:40.036 DEBUG:boto:Token: None 2023-09-29T22:33:40.036 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.040 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d40752d70998de6-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.041 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:33:40.041 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:33:40.041 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.041 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:33:40.041 DEBUG:boto:Data: 2023-09-29T22:33:40.041 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.041 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.041 DEBUG:boto:Params: {} 2023-09-29T22:33:40.042 DEBUG:boto:Token: None 2023-09-29T22:33:40.042 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:33:40.042 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QE2HtD6/FES8PhvoYO6O3Hnpdzc= 2023-09-29T22:33:40.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QE2HtD6/FES8PhvoYO6O3Hnpdzc='} 2023-09-29T22:33:40.054 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000315877ae88d199da-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.054 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.055 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.055 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.055 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.055 DEBUG:boto:Data: 2023-09-29T22:33:40.055 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.056 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.056 DEBUG:boto:Params: {} 2023-09-29T22:33:40.056 DEBUG:boto:Token: None 2023-09-29T22:33:40.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000590f0d32f21c4bb0-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.062 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:33:40.062 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:33:40.062 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.062 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:33:40.062 DEBUG:boto:Data: 2023-09-29T22:33:40.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.062 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.062 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.063 DEBUG:boto:Params: {} 2023-09-29T22:33:40.063 DEBUG:boto:Token: None 2023-09-29T22:33:40.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:33:40.063 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:651589NdIZKKmD3iym//DDlpfyY= 2023-09-29T22:33:40.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:651589NdIZKKmD3iym//DDlpfyY='} 2023-09-29T22:33:40.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e0584cbb08cd974f-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.107 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.107 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.107 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.107 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.107 DEBUG:boto:Data: 2023-09-29T22:33:40.107 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.108 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.108 DEBUG:boto:Params: {} 2023-09-29T22:33:40.108 DEBUG:boto:Token: None 2023-09-29T22:33:40.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.111 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fac7fbb2d39f2e1d-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.112 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:33:40.112 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:33:40.112 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.112 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:33:40.112 DEBUG:boto:Data: 2023-09-29T22:33:40.112 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.112 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.112 DEBUG:boto:Params: {} 2023-09-29T22:33:40.113 DEBUG:boto:Token: None 2023-09-29T22:33:40.113 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:33:40.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gaadvnEZIjLqiIVqIUv/V/lyQ0o= 2023-09-29T22:33:40.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gaadvnEZIjLqiIVqIUv/V/lyQ0o='} 2023-09-29T22:33:40.120 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000016b989fe833f7806-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.120 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.120 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.120 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.120 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.120 DEBUG:boto:Data: 2023-09-29T22:33:40.121 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.121 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.121 DEBUG:boto:Params: {} 2023-09-29T22:33:40.121 DEBUG:boto:Token: None 2023-09-29T22:33:40.121 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000071e3fd299675278f-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.124 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:33:40.124 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:33:40.124 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.124 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:33:40.125 DEBUG:boto:Data: 2023-09-29T22:33:40.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.125 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.125 DEBUG:boto:Params: {} 2023-09-29T22:33:40.125 DEBUG:boto:Token: None 2023-09-29T22:33:40.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:33:40.125 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D3b8hLJWCX2vq23km1QB6f0K19M= 2023-09-29T22:33:40.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D3b8hLJWCX2vq23km1QB6f0K19M='} 2023-09-29T22:33:40.170 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfb62d00e3b3fb3f-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.171 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.171 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.171 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.171 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.171 DEBUG:boto:Data: 2023-09-29T22:33:40.171 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.172 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.172 DEBUG:boto:Params: {} 2023-09-29T22:33:40.172 DEBUG:boto:Token: None 2023-09-29T22:33:40.172 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.177 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005dd0c611f3b5de07-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.177 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:33:40.177 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:33:40.177 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.177 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:33:40.177 DEBUG:boto:Data: 2023-09-29T22:33:40.178 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.178 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.178 DEBUG:boto:Params: {} 2023-09-29T22:33:40.178 DEBUG:boto:Token: None 2023-09-29T22:33:40.178 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:33:40.178 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NWaRmaiYO8SaiGuxAiL2rCHTM8w= 2023-09-29T22:33:40.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NWaRmaiYO8SaiGuxAiL2rCHTM8w='} 2023-09-29T22:33:40.191 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bf36bbc847ee7e5-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.191 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.191 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.191 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.191 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.192 DEBUG:boto:Data: 2023-09-29T22:33:40.192 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.192 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.192 DEBUG:boto:Params: {} 2023-09-29T22:33:40.192 DEBUG:boto:Token: None 2023-09-29T22:33:40.192 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008ff8bb8065db53dd-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.196 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:33:40.197 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:33:40.197 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.197 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:33:40.197 DEBUG:boto:Data: 2023-09-29T22:33:40.197 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.197 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.197 DEBUG:boto:Params: {} 2023-09-29T22:33:40.198 DEBUG:boto:Token: None 2023-09-29T22:33:40.198 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:33:40.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9XSnA8Q3345gPbeaN8Z3+ITkkrQ= 2023-09-29T22:33:40.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9XSnA8Q3345gPbeaN8Z3+ITkkrQ='} 2023-09-29T22:33:40.247 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c110137f08bb947-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.247 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:33:40.247 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:33:40.247 DEBUG:boto:Method: HEAD 2023-09-29T22:33:40.247 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:33:40.247 DEBUG:boto:Data: 2023-09-29T22:33:40.248 DEBUG:boto:Headers: {} 2023-09-29T22:33:40.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.248 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.248 DEBUG:boto:Params: {} 2023-09-29T22:33:40.248 DEBUG:boto:Token: None 2023-09-29T22:33:40.248 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/ 2023-09-29T22:33:40.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c= 2023-09-29T22:33:40.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cFfYb1L/PIOhCa/3NKkJi6Zc84c=', 'Content-Length': '0'} 2023-09-29T22:33:40.253 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e83af2954b49819-00651750c4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.254 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:33:40.254 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:33:40.254 DEBUG:boto:Method: PUT 2023-09-29T22:33:40.254 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:33:40.254 DEBUG:boto:Data: 2023-09-29T22:33:40.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:33:40.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:33:40.255 DEBUG:boto:Port: 8000 2023-09-29T22:33:40.255 DEBUG:boto:Params: {} 2023-09-29T22:33:40.255 DEBUG:boto:Token: None 2023-09-29T22:33:40.255 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:33:40 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:33:40.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W1daK+ZYgh+i8nfTyjUXLMPdS/E= 2023-09-29T22:33:40.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:33:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W1daK+ZYgh+i8nfTyjUXLMPdS/E='} 2023-09-29T22:33:40.267 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb308aa712feea2e-00651750c4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:33:40.267 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-58 2023-09-29T22:33:40.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:33:40.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:33:40.268 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:33:40.318 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:40.318 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:40.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:40.385+0000 7f5aea35af40 1 waiting to reach incremental sync.. 2023-09-29T22:33:45.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:45.389+0000 7f5aea35af40 1 bucket sync caught up with source: 2023-09-29T22:33:45.389 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.136.6, 00000000001.53.6, 00000000001.57.6, 00000000001.23.6, 00000000001.55.6, 00000000001.55.6, 00000000001.45.6, 00000000001.69.6, 00000000001.25.6, 00000000001.60.6] 2023-09-29T22:33:45.389 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.136.6, 00000000001.53.6, 00000000001.57.6, 00000000001.23.6, 00000000001.55.6, 00000000001.55.6, 00000000001.45.6, 00000000001.69.6, 00000000001.25.6, 00000000001.60.6] 2023-09-29T22:33:45.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:45.389+0000 7f5aea35af40 0 bucket checkpoint complete 2023-09-29T22:33:45.418 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-58 2023-09-29T22:33:45.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:33:45.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:33:45.419 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:33:45.470 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:33:45.471 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:33:45.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:45.537+0000 7f7f15e3bf40 1 waiting to reach incremental sync.. 2023-09-29T22:33:50.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:50.537+0000 7f7f15e3bf40 1 waiting to reach incremental sync.. 2023-09-29T22:33:55.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:33:55.536+0000 7f7f15e3bf40 1 waiting to reach incremental sync.. 2023-09-29T22:34:00.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:00.540+0000 7f7f15e3bf40 1 bucket sync caught up with source: 2023-09-29T22:34:00.543 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.184.6, 00000000001.198.6, 00000000001.120.6, 00000000001.122.6, 00000000001.199.6, 00000000001.121.6, 00000000001.547.6, 00000000001.304.6, 00000000001.121.6, 00000000001.163.6] 2023-09-29T22:34:00.543 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.184.6, 00000000001.198.6, 00000000001.120.6, 00000000001.122.6, 00000000001.199.6, 00000000001.121.6, 00000000001.547.6, 00000000001.304.6, 00000000001.121.6, 00000000001.163.6] 2023-09-29T22:34:00.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:00.540+0000 7f7f15e3bf40 0 bucket checkpoint complete 2023-09-29T22:34:00.572 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-58 zones={a2, a1} 2023-09-29T22:34:00.572 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.572 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.572 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.572 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:00.573 DEBUG:boto:Data: 2023-09-29T22:34:00.573 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.573 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.573 DEBUG:boto:Params: {} 2023-09-29T22:34:00.573 DEBUG:boto:Token: None 2023-09-29T22:34:00.573 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q97c9V44d02wDQl9QduV7yBN+/s= 2023-09-29T22:34:00.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q97c9V44d02wDQl9QduV7yBN+/s=', 'Content-Length': '0'} 2023-09-29T22:34:00.577 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004bd1dbbf4e1d9693-00651750d8-1a82-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.577 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.578 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.578 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.578 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:00.578 DEBUG:boto:Data: 2023-09-29T22:34:00.578 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.578 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.578 DEBUG:boto:Params: {} 2023-09-29T22:34:00.579 DEBUG:boto:Token: None 2023-09-29T22:34:00.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q97c9V44d02wDQl9QduV7yBN+/s= 2023-09-29T22:34:00.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q97c9V44d02wDQl9QduV7yBN+/s=', 'Content-Length': '0'} 2023-09-29T22:34:00.583 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8edfdfc9dc7618c-00651750d8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:34:00.583 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.583 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.583 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.583 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.583 DEBUG:boto:Method: GET 2023-09-29T22:34:00.584 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.584 DEBUG:boto:Data: 2023-09-29T22:34:00.584 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.584 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.584 DEBUG:boto:Params: {} 2023-09-29T22:34:00.584 DEBUG:boto:Token: None 2023-09-29T22:34:00.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/?versions 2023-09-29T22:34:00.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ= 2023-09-29T22:34:00.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029a85453adcfc8a9-00651750d8-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.590 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-29T22:34:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-29T22:34:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-29T22:34:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-09-29T22:34:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-29T22:34:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-29T22:34:00.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-29T22:34:00.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-29T22:34:00.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-29T22:34:00.592 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-29T22:34:00.592 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:34:00.592 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.592 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.592 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.592 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.593 DEBUG:boto:Method: GET 2023-09-29T22:34:00.593 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.593 DEBUG:boto:Data: 2023-09-29T22:34:00.593 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.593 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.593 DEBUG:boto:Params: {} 2023-09-29T22:34:00.593 DEBUG:boto:Token: None 2023-09-29T22:34:00.594 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/?versions 2023-09-29T22:34:00.594 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ= 2023-09-29T22:34:00.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.597 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586a2072dd499b87-00651750d8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.598 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-29T22:34:00.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-29T22:34:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-29T22:34:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-29T22:34:00.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-29T22:34:00.600 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.600 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.600 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.600 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.601 DEBUG:boto:Method: GET 2023-09-29T22:34:00.601 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.601 DEBUG:boto:Data: 2023-09-29T22:34:00.601 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.601 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.601 DEBUG:boto:Params: {} 2023-09-29T22:34:00.601 DEBUG:boto:Token: None 2023-09-29T22:34:00.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/?versions 2023-09-29T22:34:00.602 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ= 2023-09-29T22:34:00.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a8ec96bde592c2b-00651750d8-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.606 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.607 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.607 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.607 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.607 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.607 DEBUG:boto:Method: GET 2023-09-29T22:34:00.607 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:00.607 DEBUG:boto:Data: 2023-09-29T22:34:00.607 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.608 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.608 DEBUG:boto:Params: {} 2023-09-29T22:34:00.608 DEBUG:boto:Token: None 2023-09-29T22:34:00.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/?versions 2023-09-29T22:34:00.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ= 2023-09-29T22:34:00.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zq/pTrTR5isOpt5xkYB48wwVSVQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1f0e95ab7108df6-00651750d8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.612 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-29T22:34:00.613 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.613 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.613 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.613 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.614 DEBUG:boto:Method: GET 2023-09-29T22:34:00.614 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.614 DEBUG:boto:Data: 2023-09-29T22:34:00.614 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.614 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.614 DEBUG:boto:Params: {} 2023-09-29T22:34:00.614 DEBUG:boto:Token: None 2023-09-29T22:34:00.614 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ= 2023-09-29T22:34:00.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000380e8789a38f88b0-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.617 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.618 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.618 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.618 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.618 DEBUG:boto:Method: GET 2023-09-29T22:34:00.618 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.618 DEBUG:boto:Data: 2023-09-29T22:34:00.618 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.618 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.619 DEBUG:boto:Params: {} 2023-09-29T22:34:00.619 DEBUG:boto:Token: None 2023-09-29T22:34:00.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ= 2023-09-29T22:34:00.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dda17136ad9d52b9-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.662 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.663 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.663 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.663 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.663 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.663 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.663 DEBUG:boto:Data: 2023-09-29T22:34:00.663 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.664 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.664 DEBUG:boto:Params: {} 2023-09-29T22:34:00.664 DEBUG:boto:Token: None 2023-09-29T22:34:00.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggW7Rpo+Zw9zHGJAPWG2VUp05oQ= 2023-09-29T22:34:00.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggW7Rpo+Zw9zHGJAPWG2VUp05oQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1d09b0c85b7122-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.668 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.668 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.668 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.668 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.668 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.668 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.668 DEBUG:boto:Data: 2023-09-29T22:34:00.668 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.669 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.669 DEBUG:boto:Params: {} 2023-09-29T22:34:00.669 DEBUG:boto:Token: None 2023-09-29T22:34:00.669 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggW7Rpo+Zw9zHGJAPWG2VUp05oQ= 2023-09-29T22:34:00.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggW7Rpo+Zw9zHGJAPWG2VUp05oQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce15e416c61852d-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-29T22:34:00.672 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.672 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.672 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.673 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.673 DEBUG:boto:Method: GET 2023-09-29T22:34:00.673 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.673 DEBUG:boto:Data: 2023-09-29T22:34:00.673 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.673 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.673 DEBUG:boto:Params: {} 2023-09-29T22:34:00.673 DEBUG:boto:Token: None 2023-09-29T22:34:00.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ= 2023-09-29T22:34:00.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0865f95c9c4259e-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.719 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.719 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.719 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.719 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.719 DEBUG:boto:Method: GET 2023-09-29T22:34:00.719 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.719 DEBUG:boto:Data: 2023-09-29T22:34:00.720 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.720 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.720 DEBUG:boto:Params: {} 2023-09-29T22:34:00.720 DEBUG:boto:Token: None 2023-09-29T22:34:00.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:00.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ= 2023-09-29T22:34:00.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aSNkozjgwBRKDsrLmJlP/9NuaRQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008abdfb9452a9cea2-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.724 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.724 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.724 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.724 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.724 DEBUG:boto:Method: GET 2023-09-29T22:34:00.724 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.725 DEBUG:boto:Data: 2023-09-29T22:34:00.725 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.725 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.725 DEBUG:boto:Params: {} 2023-09-29T22:34:00.725 DEBUG:boto:Token: None 2023-09-29T22:34:00.725 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8= 2023-09-29T22:34:00.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8=', 'Content-Length': '0'} 2023-09-29T22:34:00.730 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa32e18f3c4eadb-00651750d8-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.730 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.731 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.731 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.731 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.731 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.731 DEBUG:boto:Method: GET 2023-09-29T22:34:00.731 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.732 DEBUG:boto:Data: 2023-09-29T22:34:00.732 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.732 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.732 DEBUG:boto:Params: {} 2023-09-29T22:34:00.732 DEBUG:boto:Token: None 2023-09-29T22:34:00.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8= 2023-09-29T22:34:00.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8=', 'Content-Length': '0'} 2023-09-29T22:34:00.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f08afe64285e7dc6-00651750d8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.737 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.737 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.737 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.738 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.738 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.738 DEBUG:boto:Data: 2023-09-29T22:34:00.738 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.738 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.738 DEBUG:boto:Params: {} 2023-09-29T22:34:00.738 DEBUG:boto:Token: None 2023-09-29T22:34:00.739 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HI3Mv2nfY3gnrsrH/4E0ufN6aP4= 2023-09-29T22:34:00.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HI3Mv2nfY3gnrsrH/4E0ufN6aP4=', 'Content-Length': '0'} 2023-09-29T22:34:00.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033845bb99f332842-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.741 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.742 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.742 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.742 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.742 DEBUG:boto:Data: 2023-09-29T22:34:00.742 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.742 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.742 DEBUG:boto:Params: {} 2023-09-29T22:34:00.743 DEBUG:boto:Token: None 2023-09-29T22:34:00.743 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.743 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HI3Mv2nfY3gnrsrH/4E0ufN6aP4= 2023-09-29T22:34:00.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HI3Mv2nfY3gnrsrH/4E0ufN6aP4=', 'Content-Length': '0'} 2023-09-29T22:34:00.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d923bbb80fee9e-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-29T22:34:00.746 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.746 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.746 DEBUG:boto:Method: GET 2023-09-29T22:34:00.746 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.746 DEBUG:boto:Data: 2023-09-29T22:34:00.746 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.746 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.747 DEBUG:boto:Params: {} 2023-09-29T22:34:00.747 DEBUG:boto:Token: None 2023-09-29T22:34:00.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.747 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:waawCSHGPcAnEvouaJ/Fz85Fdko= 2023-09-29T22:34:00.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:waawCSHGPcAnEvouaJ/Fz85Fdko=', 'Content-Length': '0'} 2023-09-29T22:34:00.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d599cd8172a21785-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.750 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.750 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.750 DEBUG:boto:Method: GET 2023-09-29T22:34:00.751 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.751 DEBUG:boto:Data: 2023-09-29T22:34:00.751 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.751 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.751 DEBUG:boto:Params: {} 2023-09-29T22:34:00.751 DEBUG:boto:Token: None 2023-09-29T22:34:00.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:00.752 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:waawCSHGPcAnEvouaJ/Fz85Fdko= 2023-09-29T22:34:00.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:waawCSHGPcAnEvouaJ/Fz85Fdko=', 'Content-Length': '0'} 2023-09-29T22:34:00.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058810a2f9d84a19b-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.798 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.799 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.799 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.799 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.799 DEBUG:boto:Method: GET 2023-09-29T22:34:00.800 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.800 DEBUG:boto:Data: 2023-09-29T22:34:00.800 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.800 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.801 DEBUG:boto:Params: {} 2023-09-29T22:34:00.801 DEBUG:boto:Token: None 2023-09-29T22:34:00.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8= 2023-09-29T22:34:00.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8=', 'Content-Length': '0'} 2023-09-29T22:34:00.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056f2c08bdab4ac40-00651750d8-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.807 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.807 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.807 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.808 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.808 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.808 DEBUG:boto:Method: GET 2023-09-29T22:34:00.808 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:00.808 DEBUG:boto:Data: 2023-09-29T22:34:00.809 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.809 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.809 DEBUG:boto:Params: {} 2023-09-29T22:34:00.809 DEBUG:boto:Token: None 2023-09-29T22:34:00.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8= 2023-09-29T22:34:00.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8=', 'Content-Length': '0'} 2023-09-29T22:34:00.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7146560dbf0798-00651750d8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.815 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:33:40.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-29T22:34:00.815 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.816 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.816 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.816 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.816 DEBUG:boto:Method: GET 2023-09-29T22:34:00.817 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.817 DEBUG:boto:Data: 2023-09-29T22:34:00.817 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.817 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.818 DEBUG:boto:Params: {} 2023-09-29T22:34:00.818 DEBUG:boto:Token: None 2023-09-29T22:34:00.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518= 2023-09-29T22:34:00.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518=', 'Content-Length': '0'} 2023-09-29T22:34:00.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391efeb0df4af00d-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.866 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.866 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.867 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.867 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.867 DEBUG:boto:Method: GET 2023-09-29T22:34:00.867 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.868 DEBUG:boto:Data: 2023-09-29T22:34:00.868 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.868 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.868 DEBUG:boto:Params: {} 2023-09-29T22:34:00.869 DEBUG:boto:Token: None 2023-09-29T22:34:00.869 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518= 2023-09-29T22:34:00.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518=', 'Content-Length': '0'} 2023-09-29T22:34:00.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa1059eea6c31eb9-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.873 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.873 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.873 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.874 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.874 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.874 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.874 DEBUG:boto:Data: 2023-09-29T22:34:00.874 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.874 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.875 DEBUG:boto:Params: {} 2023-09-29T22:34:00.875 DEBUG:boto:Token: None 2023-09-29T22:34:00.875 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.875 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NaQjIafqvYdTWSKhXoTxFdX6G5g= 2023-09-29T22:34:00.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NaQjIafqvYdTWSKhXoTxFdX6G5g=', 'Content-Length': '0'} 2023-09-29T22:34:00.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0bbc4edac103e37-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.878 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.878 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.878 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.878 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.879 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.879 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.879 DEBUG:boto:Data: 2023-09-29T22:34:00.879 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.879 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.879 DEBUG:boto:Params: {} 2023-09-29T22:34:00.879 DEBUG:boto:Token: None 2023-09-29T22:34:00.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NaQjIafqvYdTWSKhXoTxFdX6G5g= 2023-09-29T22:34:00.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NaQjIafqvYdTWSKhXoTxFdX6G5g=', 'Content-Length': '0'} 2023-09-29T22:34:00.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085cde4868977372b-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-29T22:34:00.882 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.883 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.883 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.883 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.883 DEBUG:boto:Method: GET 2023-09-29T22:34:00.883 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.883 DEBUG:boto:Data: 2023-09-29T22:34:00.883 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.884 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.884 DEBUG:boto:Params: {} 2023-09-29T22:34:00.884 DEBUG:boto:Token: None 2023-09-29T22:34:00.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518= 2023-09-29T22:34:00.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518=', 'Content-Length': '0'} 2023-09-29T22:34:00.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005617618ad9adb215-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.887 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.887 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.887 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.887 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.887 DEBUG:boto:Method: GET 2023-09-29T22:34:00.888 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.888 DEBUG:boto:Data: 2023-09-29T22:34:00.888 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.888 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.888 DEBUG:boto:Params: {} 2023-09-29T22:34:00.888 DEBUG:boto:Token: None 2023-09-29T22:34:00.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:00.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518= 2023-09-29T22:34:00.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dmtsD/XsTIusnwdaxwncRg6C518=', 'Content-Length': '0'} 2023-09-29T22:34:00.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000904cacbe7291ca1f-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.891 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.891 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.892 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:00.892 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:00.892 DEBUG:boto:Method: GET 2023-09-29T22:34:00.892 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:00.892 DEBUG:boto:Data: 2023-09-29T22:34:00.892 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.892 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.892 DEBUG:boto:Params: {} 2023-09-29T22:34:00.893 DEBUG:boto:Token: None 2023-09-29T22:34:00.893 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8= 2023-09-29T22:34:00.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8=', 'Content-Length': '0'} 2023-09-29T22:34:00.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000291e1126457897c-00651750d8-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.898 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.898 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:00.898 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:00.898 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:00.899 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:00.899 DEBUG:boto:Method: GET 2023-09-29T22:34:00.899 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:00.899 DEBUG:boto:Data: 2023-09-29T22:34:00.899 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.899 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.900 DEBUG:boto:Params: {} 2023-09-29T22:34:00.900 DEBUG:boto:Token: None 2023-09-29T22:34:00.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/ 2023-09-29T22:34:00.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8= 2023-09-29T22:34:00.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KydnpWu6PL+8B2OjfSHpySZ3gw8=', 'Content-Length': '0'} 2023-09-29T22:34:00.904 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002746f3fba95ce4c8-00651750d8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.946 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:00.947 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.947 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.947 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.947 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.947 DEBUG:boto:Data: 2023-09-29T22:34:00.947 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.948 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.948 DEBUG:boto:Params: {} 2023-09-29T22:34:00.948 DEBUG:boto:Token: None 2023-09-29T22:34:00.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1UOsPje/4DB/Lpur+rE3AGGVkSI= 2023-09-29T22:34:00.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1UOsPje/4DB/Lpur+rE3AGGVkSI=', 'Content-Length': '0'} 2023-09-29T22:34:00.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000458f4a566e770e73-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.951 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.952 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.952 DEBUG:boto:Method: HEAD 2023-09-29T22:34:00.952 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.952 DEBUG:boto:Data: 2023-09-29T22:34:00.952 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:00.952 DEBUG:boto:Port: 8000 2023-09-29T22:34:00.952 DEBUG:boto:Params: {} 2023-09-29T22:34:00.952 DEBUG:boto:Token: None 2023-09-29T22:34:00.953 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1UOsPje/4DB/Lpur+rE3AGGVkSI= 2023-09-29T22:34:00.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1UOsPje/4DB/Lpur+rE3AGGVkSI=', 'Content-Length': '0'} 2023-09-29T22:34:00.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001aff572e776c5c48-00651750d8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:00.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-29T22:34:00.956 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.956 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.956 DEBUG:boto:Method: GET 2023-09-29T22:34:00.956 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.956 DEBUG:boto:Data: 2023-09-29T22:34:00.956 DEBUG:boto:Headers: {} 2023-09-29T22:34:00.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:00.956 DEBUG:boto:Port: 8001 2023-09-29T22:34:00.957 DEBUG:boto:Params: {} 2023-09-29T22:34:00.957 DEBUG:boto:Token: None 2023-09-29T22:34:00.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:00 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:00.957 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JUzvo5bihXY8LwJqWahtCSQSoAQ= 2023-09-29T22:34:00.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:00 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JUzvo5bihXY8LwJqWahtCSQSoAQ=', 'Content-Length': '0'} 2023-09-29T22:34:00.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afcbec00aab93b27-00651750d8-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:00 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.006 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:01.007 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:01.007 DEBUG:boto:Method: GET 2023-09-29T22:34:01.007 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:01.007 DEBUG:boto:Data: 2023-09-29T22:34:01.007 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.007 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.007 DEBUG:boto:Params: {} 2023-09-29T22:34:01.008 DEBUG:boto:Token: None 2023-09-29T22:34:01.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:01.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zc/Y9odPfxO3h1Bq7W3ahSAP+/c= 2023-09-29T22:34:01.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zc/Y9odPfxO3h1Bq7W3ahSAP+/c=', 'Content-Length': '0'} 2023-09-29T22:34:01.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f890bf93ebe3ede1-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.011 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.011 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.011 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:01.011 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:01.012 DEBUG:boto:Method: GET 2023-09-29T22:34:01.012 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:01.012 DEBUG:boto:Data: 2023-09-29T22:34:01.012 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.012 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.012 DEBUG:boto:Params: {} 2023-09-29T22:34:01.012 DEBUG:boto:Token: None 2023-09-29T22:34:01.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d0a40592409f4d-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.017 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.018 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.018 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.018 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:01.019 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:01.019 DEBUG:boto:Method: GET 2023-09-29T22:34:01.019 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:01.019 DEBUG:boto:Data: 2023-09-29T22:34:01.019 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.020 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.020 DEBUG:boto:Params: {} 2023-09-29T22:34:01.020 DEBUG:boto:Token: None 2023-09-29T22:34:01.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.025 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000665214ca945f533b-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.026 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:33:40.259Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-29T22:34:01.027 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.027 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.027 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.027 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.028 DEBUG:boto:Method: GET 2023-09-29T22:34:01.028 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.028 DEBUG:boto:Data: 2023-09-29T22:34:01.028 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.028 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.029 DEBUG:boto:Params: {} 2023-09-29T22:34:01.029 DEBUG:boto:Token: None 2023-09-29T22:34:01.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc= 2023-09-29T22:34:01.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc=', 'Content-Length': '0'} 2023-09-29T22:34:01.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000520e9ecb337d2271-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.033 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.033 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.033 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.033 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.034 DEBUG:boto:Method: GET 2023-09-29T22:34:01.034 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.034 DEBUG:boto:Data: 2023-09-29T22:34:01.034 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.035 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.035 DEBUG:boto:Params: {} 2023-09-29T22:34:01.035 DEBUG:boto:Token: None 2023-09-29T22:34:01.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc= 2023-09-29T22:34:01.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc=', 'Content-Length': '0'} 2023-09-29T22:34:01.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe95893b9c77369e-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.086 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.087 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.087 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.087 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.087 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.087 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.088 DEBUG:boto:Data: 2023-09-29T22:34:01.088 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.088 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.088 DEBUG:boto:Params: {} 2023-09-29T22:34:01.089 DEBUG:boto:Token: None 2023-09-29T22:34:01.089 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VGtalj85yQESW1GT2Md8x9mF/B0= 2023-09-29T22:34:01.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VGtalj85yQESW1GT2Md8x9mF/B0=', 'Content-Length': '0'} 2023-09-29T22:34:01.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f55af00678d763-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.093 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.093 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.093 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.093 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.093 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.093 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.093 DEBUG:boto:Data: 2023-09-29T22:34:01.093 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.094 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.094 DEBUG:boto:Params: {} 2023-09-29T22:34:01.094 DEBUG:boto:Token: None 2023-09-29T22:34:01.094 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VGtalj85yQESW1GT2Md8x9mF/B0= 2023-09-29T22:34:01.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VGtalj85yQESW1GT2Md8x9mF/B0=', 'Content-Length': '0'} 2023-09-29T22:34:01.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8fb2bcea0112a3-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-29T22:34:01.097 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.098 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.098 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.098 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.098 DEBUG:boto:Method: GET 2023-09-29T22:34:01.098 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.098 DEBUG:boto:Data: 2023-09-29T22:34:01.098 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.098 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.098 DEBUG:boto:Params: {} 2023-09-29T22:34:01.099 DEBUG:boto:Token: None 2023-09-29T22:34:01.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.099 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc= 2023-09-29T22:34:01.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc=', 'Content-Length': '0'} 2023-09-29T22:34:01.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b0fd4f04c8cb78e-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.146 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.147 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.147 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.147 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.147 DEBUG:boto:Method: GET 2023-09-29T22:34:01.147 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.147 DEBUG:boto:Data: 2023-09-29T22:34:01.147 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.147 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.148 DEBUG:boto:Params: {} 2023-09-29T22:34:01.148 DEBUG:boto:Token: None 2023-09-29T22:34:01.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:01.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc= 2023-09-29T22:34:01.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:etqf6QCcA0FVoj+AaqSGQZhqcdc=', 'Content-Length': '0'} 2023-09-29T22:34:01.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e985319ea8f069b-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.151 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.151 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.151 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.152 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.152 DEBUG:boto:Method: GET 2023-09-29T22:34:01.152 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.152 DEBUG:boto:Data: 2023-09-29T22:34:01.152 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.152 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.152 DEBUG:boto:Params: {} 2023-09-29T22:34:01.152 DEBUG:boto:Token: None 2023-09-29T22:34:01.153 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.157 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ebbedb30cf5f190-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.158 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.158 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.158 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.158 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.159 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.159 DEBUG:boto:Method: GET 2023-09-29T22:34:01.159 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.159 DEBUG:boto:Data: 2023-09-29T22:34:01.159 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.159 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.160 DEBUG:boto:Params: {} 2023-09-29T22:34:01.160 DEBUG:boto:Token: None 2023-09-29T22:34:01.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.160 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5590598d99b5e5f-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.167 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.168 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.168 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.168 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.168 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.169 DEBUG:boto:Data: 2023-09-29T22:34:01.169 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.169 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.169 DEBUG:boto:Params: {} 2023-09-29T22:34:01.169 DEBUG:boto:Token: None 2023-09-29T22:34:01.170 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.170 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vCBIIgOFwAoscp1ovy+Int7sYH0= 2023-09-29T22:34:01.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vCBIIgOFwAoscp1ovy+Int7sYH0=', 'Content-Length': '0'} 2023-09-29T22:34:01.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003af5b4da4692d66f-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.172 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.173 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.173 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.173 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.173 DEBUG:boto:Data: 2023-09-29T22:34:01.173 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.173 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.174 DEBUG:boto:Params: {} 2023-09-29T22:34:01.174 DEBUG:boto:Token: None 2023-09-29T22:34:01.174 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vCBIIgOFwAoscp1ovy+Int7sYH0= 2023-09-29T22:34:01.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vCBIIgOFwAoscp1ovy+Int7sYH0=', 'Content-Length': '0'} 2023-09-29T22:34:01.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d3d865a6f8a8be5-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-29T22:34:01.177 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.177 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.177 DEBUG:boto:Method: GET 2023-09-29T22:34:01.177 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.177 DEBUG:boto:Data: 2023-09-29T22:34:01.178 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.178 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.178 DEBUG:boto:Params: {} 2023-09-29T22:34:01.178 DEBUG:boto:Token: None 2023-09-29T22:34:01.178 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.179 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z/pJFtWteRb3ABfa1bCFtz1JsYo= 2023-09-29T22:34:01.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z/pJFtWteRb3ABfa1bCFtz1JsYo=', 'Content-Length': '0'} 2023-09-29T22:34:01.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8d3cc0faecb08d-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.181 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.181 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.181 DEBUG:boto:Method: GET 2023-09-29T22:34:01.181 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.182 DEBUG:boto:Data: 2023-09-29T22:34:01.182 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.182 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.182 DEBUG:boto:Params: {} 2023-09-29T22:34:01.182 DEBUG:boto:Token: None 2023-09-29T22:34:01.182 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:01.182 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z/pJFtWteRb3ABfa1bCFtz1JsYo= 2023-09-29T22:34:01.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z/pJFtWteRb3ABfa1bCFtz1JsYo=', 'Content-Length': '0'} 2023-09-29T22:34:01.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae4f641a032efd80-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.226 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.227 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.227 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.227 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.227 DEBUG:boto:Method: GET 2023-09-29T22:34:01.227 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.227 DEBUG:boto:Data: 2023-09-29T22:34:01.228 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.228 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.228 DEBUG:boto:Params: {} 2023-09-29T22:34:01.228 DEBUG:boto:Token: None 2023-09-29T22:34:01.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e2091f3e058ff4-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.234 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.235 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.235 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.235 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.235 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.235 DEBUG:boto:Method: GET 2023-09-29T22:34:01.235 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:01.235 DEBUG:boto:Data: 2023-09-29T22:34:01.235 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.236 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.236 DEBUG:boto:Params: {} 2023-09-29T22:34:01.236 DEBUG:boto:Token: None 2023-09-29T22:34:01.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.240 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004643c2f8fc5fa62-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.241 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:33:40.129Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-29T22:34:01.241 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.241 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.241 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.241 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.241 DEBUG:boto:Method: GET 2023-09-29T22:34:01.242 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.242 DEBUG:boto:Data: 2023-09-29T22:34:01.242 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.242 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.242 DEBUG:boto:Params: {} 2023-09-29T22:34:01.242 DEBUG:boto:Token: None 2023-09-29T22:34:01.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA= 2023-09-29T22:34:01.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA=', 'Content-Length': '0'} 2023-09-29T22:34:01.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024d2ae2074019ff6-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.286 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.287 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.287 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.287 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.287 DEBUG:boto:Method: GET 2023-09-29T22:34:01.287 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.287 DEBUG:boto:Data: 2023-09-29T22:34:01.287 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.288 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.288 DEBUG:boto:Params: {} 2023-09-29T22:34:01.288 DEBUG:boto:Token: None 2023-09-29T22:34:01.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA= 2023-09-29T22:34:01.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA=', 'Content-Length': '0'} 2023-09-29T22:34:01.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40d098d17aba5d0-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.292 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.292 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.292 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.292 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.292 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.293 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.293 DEBUG:boto:Data: 2023-09-29T22:34:01.293 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.293 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.293 DEBUG:boto:Params: {} 2023-09-29T22:34:01.293 DEBUG:boto:Token: None 2023-09-29T22:34:01.293 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GAFbRSoQ/eb4anpcBPB+h90BMMc= 2023-09-29T22:34:01.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GAFbRSoQ/eb4anpcBPB+h90BMMc=', 'Content-Length': '0'} 2023-09-29T22:34:01.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e844d6e45dfb775-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.296 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.297 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.297 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.297 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.297 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.297 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.297 DEBUG:boto:Data: 2023-09-29T22:34:01.297 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.297 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.297 DEBUG:boto:Params: {} 2023-09-29T22:34:01.298 DEBUG:boto:Token: None 2023-09-29T22:34:01.298 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GAFbRSoQ/eb4anpcBPB+h90BMMc= 2023-09-29T22:34:01.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GAFbRSoQ/eb4anpcBPB+h90BMMc=', 'Content-Length': '0'} 2023-09-29T22:34:01.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000645441e79e163292-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-29T22:34:01.301 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.301 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.301 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.301 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.301 DEBUG:boto:Method: GET 2023-09-29T22:34:01.302 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.302 DEBUG:boto:Data: 2023-09-29T22:34:01.302 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.302 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.302 DEBUG:boto:Params: {} 2023-09-29T22:34:01.302 DEBUG:boto:Token: None 2023-09-29T22:34:01.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA= 2023-09-29T22:34:01.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA=', 'Content-Length': '0'} 2023-09-29T22:34:01.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a8cbb4e9089a9d9-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.306 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.306 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.306 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.306 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.306 DEBUG:boto:Method: GET 2023-09-29T22:34:01.306 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.306 DEBUG:boto:Data: 2023-09-29T22:34:01.306 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.307 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.307 DEBUG:boto:Params: {} 2023-09-29T22:34:01.307 DEBUG:boto:Token: None 2023-09-29T22:34:01.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:01.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA= 2023-09-29T22:34:01.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vhjaJhbxq7LgN+fDGUWYKy9wlVA=', 'Content-Length': '0'} 2023-09-29T22:34:01.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007285e45b9e91994-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.310 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.310 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.311 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.311 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.311 DEBUG:boto:Method: GET 2023-09-29T22:34:01.311 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.311 DEBUG:boto:Data: 2023-09-29T22:34:01.311 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.311 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.311 DEBUG:boto:Params: {} 2023-09-29T22:34:01.311 DEBUG:boto:Token: None 2023-09-29T22:34:01.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037753321810ba85d-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.317 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.318 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.318 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.318 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.318 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.318 DEBUG:boto:Method: GET 2023-09-29T22:34:01.318 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.318 DEBUG:boto:Data: 2023-09-29T22:34:01.319 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.319 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.319 DEBUG:boto:Params: {} 2023-09-29T22:34:01.319 DEBUG:boto:Token: None 2023-09-29T22:34:01.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006cf1eb8cc978b41a-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.324 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.324 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.324 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.324 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.324 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.325 DEBUG:boto:Data: 2023-09-29T22:34:01.325 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.325 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.325 DEBUG:boto:Params: {} 2023-09-29T22:34:01.325 DEBUG:boto:Token: None 2023-09-29T22:34:01.325 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8pjkC0OtbAW0552zEUxfnQQpVdY= 2023-09-29T22:34:01.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8pjkC0OtbAW0552zEUxfnQQpVdY=', 'Content-Length': '0'} 2023-09-29T22:34:01.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d5ebf22f940a38-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.328 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.328 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.328 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.328 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.329 DEBUG:boto:Data: 2023-09-29T22:34:01.329 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.329 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.329 DEBUG:boto:Params: {} 2023-09-29T22:34:01.329 DEBUG:boto:Token: None 2023-09-29T22:34:01.329 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8pjkC0OtbAW0552zEUxfnQQpVdY= 2023-09-29T22:34:01.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8pjkC0OtbAW0552zEUxfnQQpVdY=', 'Content-Length': '0'} 2023-09-29T22:34:01.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e3dc2cef8abc29-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-29T22:34:01.332 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.332 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.332 DEBUG:boto:Method: GET 2023-09-29T22:34:01.332 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.333 DEBUG:boto:Data: 2023-09-29T22:34:01.333 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.333 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.333 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.333 DEBUG:boto:Params: {} 2023-09-29T22:34:01.333 DEBUG:boto:Token: None 2023-09-29T22:34:01.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Di9338pTB2Rf8RP5qQLkR6zCN+g= 2023-09-29T22:34:01.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Di9338pTB2Rf8RP5qQLkR6zCN+g=', 'Content-Length': '0'} 2023-09-29T22:34:01.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3cddb8170d81144-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.382 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.383 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.383 DEBUG:boto:Method: GET 2023-09-29T22:34:01.383 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.383 DEBUG:boto:Data: 2023-09-29T22:34:01.383 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.383 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.383 DEBUG:boto:Params: {} 2023-09-29T22:34:01.383 DEBUG:boto:Token: None 2023-09-29T22:34:01.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:01.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Di9338pTB2Rf8RP5qQLkR6zCN+g= 2023-09-29T22:34:01.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Di9338pTB2Rf8RP5qQLkR6zCN+g=', 'Content-Length': '0'} 2023-09-29T22:34:01.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e831fbc2b24b0e07-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.387 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.387 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.387 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.387 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.387 DEBUG:boto:Method: GET 2023-09-29T22:34:01.387 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.387 DEBUG:boto:Data: 2023-09-29T22:34:01.388 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.388 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.388 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.388 DEBUG:boto:Params: {} 2023-09-29T22:34:01.388 DEBUG:boto:Token: None 2023-09-29T22:34:01.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec100e84f41605b8-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.393 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.394 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.394 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.394 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.394 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.394 DEBUG:boto:Method: GET 2023-09-29T22:34:01.394 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:01.394 DEBUG:boto:Data: 2023-09-29T22:34:01.394 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.395 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.395 DEBUG:boto:Params: {} 2023-09-29T22:34:01.395 DEBUG:boto:Token: None 2023-09-29T22:34:01.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025d59ec2bffe15d3-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.402 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:33:40.182Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-29T22:34:01.403 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.403 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.403 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.403 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.403 DEBUG:boto:Method: GET 2023-09-29T22:34:01.404 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.404 DEBUG:boto:Data: 2023-09-29T22:34:01.404 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.404 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.404 DEBUG:boto:Params: {} 2023-09-29T22:34:01.404 DEBUG:boto:Token: None 2023-09-29T22:34:01.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A= 2023-09-29T22:34:01.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A=', 'Content-Length': '0'} 2023-09-29T22:34:01.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dcea4725be3876a-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.408 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.408 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.408 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.408 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.408 DEBUG:boto:Method: GET 2023-09-29T22:34:01.408 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.409 DEBUG:boto:Data: 2023-09-29T22:34:01.409 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.409 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.409 DEBUG:boto:Params: {} 2023-09-29T22:34:01.410 DEBUG:boto:Token: None 2023-09-29T22:34:01.410 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A= 2023-09-29T22:34:01.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A=', 'Content-Length': '0'} 2023-09-29T22:34:01.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000223b6d864e39e124-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.413 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.413 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.414 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.414 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.414 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.414 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.414 DEBUG:boto:Data: 2023-09-29T22:34:01.414 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.414 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.415 DEBUG:boto:Params: {} 2023-09-29T22:34:01.415 DEBUG:boto:Token: None 2023-09-29T22:34:01.415 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Or8oMFmRRl6gKr1OoeaoFNtRxc= 2023-09-29T22:34:01.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Or8oMFmRRl6gKr1OoeaoFNtRxc=', 'Content-Length': '0'} 2023-09-29T22:34:01.418 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65b2f13df75a005-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.418 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.418 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.418 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.419 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.419 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.419 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.419 DEBUG:boto:Data: 2023-09-29T22:34:01.419 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.419 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.419 DEBUG:boto:Params: {} 2023-09-29T22:34:01.420 DEBUG:boto:Token: None 2023-09-29T22:34:01.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Or8oMFmRRl6gKr1OoeaoFNtRxc= 2023-09-29T22:34:01.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Or8oMFmRRl6gKr1OoeaoFNtRxc=', 'Content-Length': '0'} 2023-09-29T22:34:01.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7640094cf37856f-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-29T22:34:01.423 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.423 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.423 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.423 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.423 DEBUG:boto:Method: GET 2023-09-29T22:34:01.424 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.424 DEBUG:boto:Data: 2023-09-29T22:34:01.424 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.424 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.424 DEBUG:boto:Params: {} 2023-09-29T22:34:01.424 DEBUG:boto:Token: None 2023-09-29T22:34:01.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A= 2023-09-29T22:34:01.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A=', 'Content-Length': '0'} 2023-09-29T22:34:01.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001117841a79661257-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.470 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.471 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.471 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.471 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.471 DEBUG:boto:Method: GET 2023-09-29T22:34:01.471 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.471 DEBUG:boto:Data: 2023-09-29T22:34:01.471 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.471 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.472 DEBUG:boto:Params: {} 2023-09-29T22:34:01.472 DEBUG:boto:Token: None 2023-09-29T22:34:01.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:01.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A= 2023-09-29T22:34:01.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6IW/kVkxGD85XGMNf5vLxgnag6A=', 'Content-Length': '0'} 2023-09-29T22:34:01.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29eda39cb9a3bb5-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.477 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.477 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.477 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.477 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.477 DEBUG:boto:Method: GET 2023-09-29T22:34:01.477 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.477 DEBUG:boto:Data: 2023-09-29T22:34:01.477 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.478 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.478 DEBUG:boto:Params: {} 2023-09-29T22:34:01.478 DEBUG:boto:Token: None 2023-09-29T22:34:01.478 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edde9f202dcb2560-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.483 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.484 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.484 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.484 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.484 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.484 DEBUG:boto:Method: GET 2023-09-29T22:34:01.484 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.485 DEBUG:boto:Data: 2023-09-29T22:34:01.485 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.485 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.485 DEBUG:boto:Params: {} 2023-09-29T22:34:01.485 DEBUG:boto:Token: None 2023-09-29T22:34:01.485 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b13847a606f16af4-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.530 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.531 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.531 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.531 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.531 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.532 DEBUG:boto:Data: 2023-09-29T22:34:01.532 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.532 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.532 DEBUG:boto:Params: {} 2023-09-29T22:34:01.532 DEBUG:boto:Token: None 2023-09-29T22:34:01.532 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GpaLU+5e5hwZfn6o07GVvTv2oTY= 2023-09-29T22:34:01.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GpaLU+5e5hwZfn6o07GVvTv2oTY=', 'Content-Length': '0'} 2023-09-29T22:34:01.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5bcdd678de7b434-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.536 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.536 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.536 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.536 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.536 DEBUG:boto:Data: 2023-09-29T22:34:01.537 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.537 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.537 DEBUG:boto:Params: {} 2023-09-29T22:34:01.537 DEBUG:boto:Token: None 2023-09-29T22:34:01.537 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GpaLU+5e5hwZfn6o07GVvTv2oTY= 2023-09-29T22:34:01.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GpaLU+5e5hwZfn6o07GVvTv2oTY=', 'Content-Length': '0'} 2023-09-29T22:34:01.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007030f3317a9c695-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-29T22:34:01.540 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.540 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.541 DEBUG:boto:Method: GET 2023-09-29T22:34:01.541 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.541 DEBUG:boto:Data: 2023-09-29T22:34:01.541 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.541 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.541 DEBUG:boto:Params: {} 2023-09-29T22:34:01.542 DEBUG:boto:Token: None 2023-09-29T22:34:01.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+zqgmsSSv2upNifVQi5D8JhxXuY= 2023-09-29T22:34:01.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+zqgmsSSv2upNifVQi5D8JhxXuY=', 'Content-Length': '0'} 2023-09-29T22:34:01.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab33cbc8fe5b9ad-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.545 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.545 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.546 DEBUG:boto:Method: GET 2023-09-29T22:34:01.546 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.546 DEBUG:boto:Data: 2023-09-29T22:34:01.546 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.546 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.546 DEBUG:boto:Params: {} 2023-09-29T22:34:01.547 DEBUG:boto:Token: None 2023-09-29T22:34:01.547 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:01.547 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+zqgmsSSv2upNifVQi5D8JhxXuY= 2023-09-29T22:34:01.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+zqgmsSSv2upNifVQi5D8JhxXuY=', 'Content-Length': '0'} 2023-09-29T22:34:01.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e834a6fc90af3ace-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.550 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.551 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.551 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.551 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.551 DEBUG:boto:Method: GET 2023-09-29T22:34:01.551 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.551 DEBUG:boto:Data: 2023-09-29T22:34:01.551 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.552 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.552 DEBUG:boto:Params: {} 2023-09-29T22:34:01.552 DEBUG:boto:Token: None 2023-09-29T22:34:01.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000030a55e762f24654d-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.557 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.558 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.558 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.558 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.558 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.559 DEBUG:boto:Method: GET 2023-09-29T22:34:01.559 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:01.559 DEBUG:boto:Data: 2023-09-29T22:34:01.559 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.559 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.559 DEBUG:boto:Params: {} 2023-09-29T22:34:01.559 DEBUG:boto:Token: None 2023-09-29T22:34:01.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.564 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e62ac24e69d6492-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.564 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:33:39.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-29T22:34:01.565 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.565 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.565 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.565 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.565 DEBUG:boto:Method: GET 2023-09-29T22:34:01.565 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.566 DEBUG:boto:Data: 2023-09-29T22:34:01.566 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.566 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.566 DEBUG:boto:Params: {} 2023-09-29T22:34:01.566 DEBUG:boto:Token: None 2023-09-29T22:34:01.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg= 2023-09-29T22:34:01.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg=', 'Content-Length': '0'} 2023-09-29T22:34:01.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000691aa006019707c9-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.610 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.611 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.611 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.611 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.611 DEBUG:boto:Method: GET 2023-09-29T22:34:01.611 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.611 DEBUG:boto:Data: 2023-09-29T22:34:01.611 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.612 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.612 DEBUG:boto:Params: {} 2023-09-29T22:34:01.612 DEBUG:boto:Token: None 2023-09-29T22:34:01.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg= 2023-09-29T22:34:01.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg=', 'Content-Length': '0'} 2023-09-29T22:34:01.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d1d56364d9a24f3-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.616 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.616 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.616 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.616 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.616 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.616 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.616 DEBUG:boto:Data: 2023-09-29T22:34:01.616 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.617 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.617 DEBUG:boto:Params: {} 2023-09-29T22:34:01.617 DEBUG:boto:Token: None 2023-09-29T22:34:01.617 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mVTaZIzO0T3fKiK6LDCbOYLUPPc= 2023-09-29T22:34:01.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mVTaZIzO0T3fKiK6LDCbOYLUPPc=', 'Content-Length': '0'} 2023-09-29T22:34:01.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000937a11eb98771363-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.620 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.620 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.620 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.621 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.621 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.621 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.621 DEBUG:boto:Data: 2023-09-29T22:34:01.621 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.621 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.621 DEBUG:boto:Params: {} 2023-09-29T22:34:01.621 DEBUG:boto:Token: None 2023-09-29T22:34:01.622 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mVTaZIzO0T3fKiK6LDCbOYLUPPc= 2023-09-29T22:34:01.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mVTaZIzO0T3fKiK6LDCbOYLUPPc=', 'Content-Length': '0'} 2023-09-29T22:34:01.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15d576248b61137-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-29T22:34:01.625 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.625 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.625 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.625 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.625 DEBUG:boto:Method: GET 2023-09-29T22:34:01.625 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.625 DEBUG:boto:Data: 2023-09-29T22:34:01.625 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.626 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.626 DEBUG:boto:Params: {} 2023-09-29T22:34:01.626 DEBUG:boto:Token: None 2023-09-29T22:34:01.626 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.626 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg= 2023-09-29T22:34:01.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg=', 'Content-Length': '0'} 2023-09-29T22:34:01.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009262bdfe57c688c3-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.629 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.630 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.630 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.630 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.630 DEBUG:boto:Method: GET 2023-09-29T22:34:01.630 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.630 DEBUG:boto:Data: 2023-09-29T22:34:01.630 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.630 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.631 DEBUG:boto:Params: {} 2023-09-29T22:34:01.631 DEBUG:boto:Token: None 2023-09-29T22:34:01.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:01.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg= 2023-09-29T22:34:01.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvINLcCvupDqZ9aV+M3UeX9c8Lg=', 'Content-Length': '0'} 2023-09-29T22:34:01.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009366002fcd97009c-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.678 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.679 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.679 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.679 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.679 DEBUG:boto:Method: GET 2023-09-29T22:34:01.679 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.679 DEBUG:boto:Data: 2023-09-29T22:34:01.679 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.679 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.680 DEBUG:boto:Params: {} 2023-09-29T22:34:01.680 DEBUG:boto:Token: None 2023-09-29T22:34:01.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000081107aa33d872c0b-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.685 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.686 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.686 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.686 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.686 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.686 DEBUG:boto:Method: GET 2023-09-29T22:34:01.686 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.686 DEBUG:boto:Data: 2023-09-29T22:34:01.687 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.687 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.687 DEBUG:boto:Params: {} 2023-09-29T22:34:01.687 DEBUG:boto:Token: None 2023-09-29T22:34:01.687 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.687 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb23e16bf4dd4f0-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.692 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.693 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.693 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.693 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.693 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.693 DEBUG:boto:Data: 2023-09-29T22:34:01.693 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.693 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.693 DEBUG:boto:Params: {} 2023-09-29T22:34:01.694 DEBUG:boto:Token: None 2023-09-29T22:34:01.694 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/e+1teNVPScgwN4Cgfhl7223NB4= 2023-09-29T22:34:01.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/e+1teNVPScgwN4Cgfhl7223NB4=', 'Content-Length': '0'} 2023-09-29T22:34:01.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e11442c26e89cb-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.697 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.697 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.697 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.697 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.697 DEBUG:boto:Data: 2023-09-29T22:34:01.697 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.697 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.698 DEBUG:boto:Params: {} 2023-09-29T22:34:01.698 DEBUG:boto:Token: None 2023-09-29T22:34:01.698 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/e+1teNVPScgwN4Cgfhl7223NB4= 2023-09-29T22:34:01.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/e+1teNVPScgwN4Cgfhl7223NB4=', 'Content-Length': '0'} 2023-09-29T22:34:01.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8dcf567e176e90e-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-29T22:34:01.701 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.701 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.701 DEBUG:boto:Method: GET 2023-09-29T22:34:01.701 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.701 DEBUG:boto:Data: 2023-09-29T22:34:01.701 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.701 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.702 DEBUG:boto:Params: {} 2023-09-29T22:34:01.702 DEBUG:boto:Token: None 2023-09-29T22:34:01.702 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nC8YyoSgq8H9a7TTgm/vC//mgMI= 2023-09-29T22:34:01.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nC8YyoSgq8H9a7TTgm/vC//mgMI=', 'Content-Length': '0'} 2023-09-29T22:34:01.705 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3c785c1d12259cc-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.746 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.747 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.747 DEBUG:boto:Method: GET 2023-09-29T22:34:01.747 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.747 DEBUG:boto:Data: 2023-09-29T22:34:01.747 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.747 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.747 DEBUG:boto:Params: {} 2023-09-29T22:34:01.747 DEBUG:boto:Token: None 2023-09-29T22:34:01.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:01.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nC8YyoSgq8H9a7TTgm/vC//mgMI= 2023-09-29T22:34:01.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nC8YyoSgq8H9a7TTgm/vC//mgMI=', 'Content-Length': '0'} 2023-09-29T22:34:01.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8eafaaf1271b92-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.751 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.751 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.751 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.752 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.752 DEBUG:boto:Method: GET 2023-09-29T22:34:01.752 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.752 DEBUG:boto:Data: 2023-09-29T22:34:01.752 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.752 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.752 DEBUG:boto:Params: {} 2023-09-29T22:34:01.752 DEBUG:boto:Token: None 2023-09-29T22:34:01.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.757 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db10f53b8067c6e4-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.757 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.758 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.758 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.758 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.759 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.759 DEBUG:boto:Method: GET 2023-09-29T22:34:01.759 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:01.759 DEBUG:boto:Data: 2023-09-29T22:34:01.759 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.759 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.759 DEBUG:boto:Params: {} 2023-09-29T22:34:01.760 DEBUG:boto:Token: None 2023-09-29T22:34:01.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000178e1cc787cec1fa-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.810 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:33:40.067Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-29T22:34:01.811 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.811 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.811 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.811 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.812 DEBUG:boto:Method: GET 2023-09-29T22:34:01.812 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.812 DEBUG:boto:Data: 2023-09-29T22:34:01.812 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.812 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.812 DEBUG:boto:Params: {} 2023-09-29T22:34:01.812 DEBUG:boto:Token: None 2023-09-29T22:34:01.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ= 2023-09-29T22:34:01.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ=', 'Content-Length': '0'} 2023-09-29T22:34:01.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000795a7e9962f2e997-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.818 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.818 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.818 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.819 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.819 DEBUG:boto:Method: GET 2023-09-29T22:34:01.819 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.819 DEBUG:boto:Data: 2023-09-29T22:34:01.819 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.819 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.820 DEBUG:boto:Params: {} 2023-09-29T22:34:01.820 DEBUG:boto:Token: None 2023-09-29T22:34:01.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ= 2023-09-29T22:34:01.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ=', 'Content-Length': '0'} 2023-09-29T22:34:01.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbe02ee226ce94f0-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.824 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.824 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.824 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.824 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.824 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.824 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.824 DEBUG:boto:Data: 2023-09-29T22:34:01.824 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.825 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.825 DEBUG:boto:Params: {} 2023-09-29T22:34:01.825 DEBUG:boto:Token: None 2023-09-29T22:34:01.825 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1J+1e3xZ9RAt7DQrsJq+Fku/Vo= 2023-09-29T22:34:01.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1J+1e3xZ9RAt7DQrsJq+Fku/Vo=', 'Content-Length': '0'} 2023-09-29T22:34:01.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2294fc583c6d9b5-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.828 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.828 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.829 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.829 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.829 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.829 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.829 DEBUG:boto:Data: 2023-09-29T22:34:01.829 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.829 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.829 DEBUG:boto:Params: {} 2023-09-29T22:34:01.829 DEBUG:boto:Token: None 2023-09-29T22:34:01.830 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g1J+1e3xZ9RAt7DQrsJq+Fku/Vo= 2023-09-29T22:34:01.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g1J+1e3xZ9RAt7DQrsJq+Fku/Vo=', 'Content-Length': '0'} 2023-09-29T22:34:01.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000942520525247f2d9-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-29T22:34:01.833 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.833 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.833 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.833 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.833 DEBUG:boto:Method: GET 2023-09-29T22:34:01.834 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.834 DEBUG:boto:Data: 2023-09-29T22:34:01.834 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.834 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.834 DEBUG:boto:Params: {} 2023-09-29T22:34:01.834 DEBUG:boto:Token: None 2023-09-29T22:34:01.834 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.835 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ= 2023-09-29T22:34:01.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ=', 'Content-Length': '0'} 2023-09-29T22:34:01.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040de86ef9f60d8dd-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.878 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.879 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.879 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.879 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.879 DEBUG:boto:Method: GET 2023-09-29T22:34:01.879 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.879 DEBUG:boto:Data: 2023-09-29T22:34:01.879 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.880 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.880 DEBUG:boto:Params: {} 2023-09-29T22:34:01.880 DEBUG:boto:Token: None 2023-09-29T22:34:01.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:01.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ= 2023-09-29T22:34:01.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cvnB3ygpHupr9fAwf5IdXQbc/UQ=', 'Content-Length': '0'} 2023-09-29T22:34:01.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037cc7f0d46215eef-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.883 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.883 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.884 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.884 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.884 DEBUG:boto:Method: GET 2023-09-29T22:34:01.884 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.884 DEBUG:boto:Data: 2023-09-29T22:34:01.884 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.884 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.884 DEBUG:boto:Params: {} 2023-09-29T22:34:01.884 DEBUG:boto:Token: None 2023-09-29T22:34:01.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ac8576bbaa5902d9-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.890 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.890 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.891 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.891 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.891 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.891 DEBUG:boto:Method: GET 2023-09-29T22:34:01.891 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.891 DEBUG:boto:Data: 2023-09-29T22:34:01.891 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.891 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.892 DEBUG:boto:Params: {} 2023-09-29T22:34:01.892 DEBUG:boto:Token: None 2023-09-29T22:34:01.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.897 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ab24d0e54f9c342-00651750d9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.938 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:01.939 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.939 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.939 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.939 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.939 DEBUG:boto:Data: 2023-09-29T22:34:01.939 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.940 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.940 DEBUG:boto:Params: {} 2023-09-29T22:34:01.940 DEBUG:boto:Token: None 2023-09-29T22:34:01.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iUpnqcan/Kdu31l4CYzjosdMISE= 2023-09-29T22:34:01.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iUpnqcan/Kdu31l4CYzjosdMISE=', 'Content-Length': '0'} 2023-09-29T22:34:01.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff11e3cde2dc01c-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.943 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.943 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.944 DEBUG:boto:Method: HEAD 2023-09-29T22:34:01.944 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.944 DEBUG:boto:Data: 2023-09-29T22:34:01.944 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.944 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.944 DEBUG:boto:Params: {} 2023-09-29T22:34:01.944 DEBUG:boto:Token: None 2023-09-29T22:34:01.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iUpnqcan/Kdu31l4CYzjosdMISE= 2023-09-29T22:34:01.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iUpnqcan/Kdu31l4CYzjosdMISE=', 'Content-Length': '0'} 2023-09-29T22:34:01.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b39a8aedffcad29-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-29T22:34:01.948 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.948 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.948 DEBUG:boto:Method: GET 2023-09-29T22:34:01.948 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.948 DEBUG:boto:Data: 2023-09-29T22:34:01.949 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.949 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.949 DEBUG:boto:Params: {} 2023-09-29T22:34:01.949 DEBUG:boto:Token: None 2023-09-29T22:34:01.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KAH4TYvrlX4Pm6WzBZsYtgulewk= 2023-09-29T22:34:01.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KAH4TYvrlX4Pm6WzBZsYtgulewk=', 'Content-Length': '0'} 2023-09-29T22:34:01.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1979fb7ec931d22-00651750d9-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.952 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.953 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.953 DEBUG:boto:Method: GET 2023-09-29T22:34:01.953 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.953 DEBUG:boto:Data: 2023-09-29T22:34:01.953 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:01.953 DEBUG:boto:Port: 8000 2023-09-29T22:34:01.953 DEBUG:boto:Params: {} 2023-09-29T22:34:01.954 DEBUG:boto:Token: None 2023-09-29T22:34:01.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:01.954 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KAH4TYvrlX4Pm6WzBZsYtgulewk= 2023-09-29T22:34:01.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KAH4TYvrlX4Pm6WzBZsYtgulewk=', 'Content-Length': '0'} 2023-09-29T22:34:01.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000853f7efa0256f67c-00651750d9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:01.957 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:01.957 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:01.957 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.957 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.958 DEBUG:boto:Method: GET 2023-09-29T22:34:01.958 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:01.958 DEBUG:boto:Data: 2023-09-29T22:34:01.958 DEBUG:boto:Headers: {} 2023-09-29T22:34:01.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:01.958 DEBUG:boto:Port: 8001 2023-09-29T22:34:01.958 DEBUG:boto:Params: {} 2023-09-29T22:34:01.958 DEBUG:boto:Token: None 2023-09-29T22:34:01.958 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:01 GMT /hcnphz-58/ 2023-09-29T22:34:01.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY= 2023-09-29T22:34:01.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:01 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:odbpwfAV3x9GoHrFCWxOclAGvwY=', 'Content-Length': '0'} 2023-09-29T22:34:01.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baccb354bef3225e-00651750d9-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:01 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.010 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.011 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.011 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.011 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:02.011 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:02.011 DEBUG:boto:Method: GET 2023-09-29T22:34:02.011 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:02.012 DEBUG:boto:Data: 2023-09-29T22:34:02.012 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.012 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.012 DEBUG:boto:Params: {} 2023-09-29T22:34:02.012 DEBUG:boto:Token: None 2023-09-29T22:34:02.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005b56a85f3f36ee64-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.016 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:33:40.116Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-29T22:34:02.016 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.016 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.017 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.017 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.017 DEBUG:boto:Method: GET 2023-09-29T22:34:02.017 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.017 DEBUG:boto:Data: 2023-09-29T22:34:02.017 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.017 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.017 DEBUG:boto:Params: {} 2023-09-29T22:34:02.017 DEBUG:boto:Token: None 2023-09-29T22:34:02.018 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.018 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ= 2023-09-29T22:34:02.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ=', 'Content-Length': '0'} 2023-09-29T22:34:02.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7967f015481e88f-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.020 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.020 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.021 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.021 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.021 DEBUG:boto:Method: GET 2023-09-29T22:34:02.021 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.021 DEBUG:boto:Data: 2023-09-29T22:34:02.021 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.022 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.022 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.022 DEBUG:boto:Params: {} 2023-09-29T22:34:02.023 DEBUG:boto:Token: None 2023-09-29T22:34:02.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ= 2023-09-29T22:34:02.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ=', 'Content-Length': '0'} 2023-09-29T22:34:02.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08bba39a8b2971c-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.070 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.071 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.071 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.071 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.071 DEBUG:boto:Data: 2023-09-29T22:34:02.071 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.072 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.072 DEBUG:boto:Params: {} 2023-09-29T22:34:02.072 DEBUG:boto:Token: None 2023-09-29T22:34:02.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7WNcYKC9CQOHzOfPmsF+x5c88w= 2023-09-29T22:34:02.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7WNcYKC9CQOHzOfPmsF+x5c88w=', 'Content-Length': '0'} 2023-09-29T22:34:02.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c40af83ec8371724-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.076 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.076 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.076 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.076 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.076 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.076 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.076 DEBUG:boto:Data: 2023-09-29T22:34:02.076 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.077 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.077 DEBUG:boto:Params: {} 2023-09-29T22:34:02.077 DEBUG:boto:Token: None 2023-09-29T22:34:02.077 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.077 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7WNcYKC9CQOHzOfPmsF+x5c88w= 2023-09-29T22:34:02.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7WNcYKC9CQOHzOfPmsF+x5c88w=', 'Content-Length': '0'} 2023-09-29T22:34:02.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000897c1a31bf493f5f-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-29T22:34:02.080 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.081 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.081 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.081 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.081 DEBUG:boto:Method: GET 2023-09-29T22:34:02.081 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.081 DEBUG:boto:Data: 2023-09-29T22:34:02.081 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.081 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.081 DEBUG:boto:Params: {} 2023-09-29T22:34:02.082 DEBUG:boto:Token: None 2023-09-29T22:34:02.082 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.083 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ= 2023-09-29T22:34:02.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ=', 'Content-Length': '0'} 2023-09-29T22:34:02.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005259eaaa51146fda-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.086 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.086 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.086 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.086 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.087 DEBUG:boto:Method: GET 2023-09-29T22:34:02.087 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.087 DEBUG:boto:Data: 2023-09-29T22:34:02.087 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.087 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.087 DEBUG:boto:Params: {} 2023-09-29T22:34:02.087 DEBUG:boto:Token: None 2023-09-29T22:34:02.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:02.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ= 2023-09-29T22:34:02.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uEc87No7S30AYCEIgMOKKs1SnnQ=', 'Content-Length': '0'} 2023-09-29T22:34:02.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001059b95e23cb0197-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.091 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.091 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.091 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.091 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.091 DEBUG:boto:Method: GET 2023-09-29T22:34:02.091 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.092 DEBUG:boto:Data: 2023-09-29T22:34:02.092 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.092 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.092 DEBUG:boto:Params: {} 2023-09-29T22:34:02.093 DEBUG:boto:Token: None 2023-09-29T22:34:02.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3f893daf9873838-00651750da-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.100 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.100 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.101 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.101 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.101 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.101 DEBUG:boto:Method: GET 2023-09-29T22:34:02.101 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.101 DEBUG:boto:Data: 2023-09-29T22:34:02.102 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.102 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.102 DEBUG:boto:Params: {} 2023-09-29T22:34:02.102 DEBUG:boto:Token: None 2023-09-29T22:34:02.103 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.103 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.108 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ec443536b2f609bc-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.150 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.151 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.151 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.151 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.152 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.152 DEBUG:boto:Data: 2023-09-29T22:34:02.152 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.152 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.152 DEBUG:boto:Params: {} 2023-09-29T22:34:02.153 DEBUG:boto:Token: None 2023-09-29T22:34:02.153 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wqMd9h2zDB0K9ee6PXoao9Zg1sU= 2023-09-29T22:34:02.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wqMd9h2zDB0K9ee6PXoao9Zg1sU=', 'Content-Length': '0'} 2023-09-29T22:34:02.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf51596e6e4a36df-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.157 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.157 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.157 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.158 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.158 DEBUG:boto:Data: 2023-09-29T22:34:02.158 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.158 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.158 DEBUG:boto:Params: {} 2023-09-29T22:34:02.158 DEBUG:boto:Token: None 2023-09-29T22:34:02.158 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wqMd9h2zDB0K9ee6PXoao9Zg1sU= 2023-09-29T22:34:02.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wqMd9h2zDB0K9ee6PXoao9Zg1sU=', 'Content-Length': '0'} 2023-09-29T22:34:02.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000182f26ad1e6c0f76-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-29T22:34:02.162 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.162 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.162 DEBUG:boto:Method: GET 2023-09-29T22:34:02.162 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.162 DEBUG:boto:Data: 2023-09-29T22:34:02.162 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.163 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.163 DEBUG:boto:Params: {} 2023-09-29T22:34:02.163 DEBUG:boto:Token: None 2023-09-29T22:34:02.163 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.163 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4+MH21Pjgv6QqAjNlz+rsn8ZW9s= 2023-09-29T22:34:02.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4+MH21Pjgv6QqAjNlz+rsn8ZW9s=', 'Content-Length': '0'} 2023-09-29T22:34:02.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaeb03ef2af6332f-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.210 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.211 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.211 DEBUG:boto:Method: GET 2023-09-29T22:34:02.212 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.212 DEBUG:boto:Data: 2023-09-29T22:34:02.212 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.213 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.213 DEBUG:boto:Params: {} 2023-09-29T22:34:02.213 DEBUG:boto:Token: None 2023-09-29T22:34:02.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:02.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4+MH21Pjgv6QqAjNlz+rsn8ZW9s= 2023-09-29T22:34:02.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4+MH21Pjgv6QqAjNlz+rsn8ZW9s=', 'Content-Length': '0'} 2023-09-29T22:34:02.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:40 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9dd04abb381bed-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.217 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.217 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.217 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.218 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.218 DEBUG:boto:Method: GET 2023-09-29T22:34:02.218 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.218 DEBUG:boto:Data: 2023-09-29T22:34:02.219 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.219 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.220 DEBUG:boto:Params: {} 2023-09-29T22:34:02.220 DEBUG:boto:Token: None 2023-09-29T22:34:02.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038897caf328c9685-00651750da-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.225 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.225 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.226 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.226 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.226 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.226 DEBUG:boto:Method: GET 2023-09-29T22:34:02.226 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:02.226 DEBUG:boto:Data: 2023-09-29T22:34:02.226 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.227 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.227 DEBUG:boto:Params: {} 2023-09-29T22:34:02.227 DEBUG:boto:Token: None 2023-09-29T22:34:02.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.230 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000386e365d35ae4cb5-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.233 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:33:40.201Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-29T22:34:02.234 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.234 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.234 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.234 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.235 DEBUG:boto:Method: GET 2023-09-29T22:34:02.235 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.235 DEBUG:boto:Data: 2023-09-29T22:34:02.235 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.235 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.235 DEBUG:boto:Params: {} 2023-09-29T22:34:02.235 DEBUG:boto:Token: None 2023-09-29T22:34:02.235 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo= 2023-09-29T22:34:02.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo=', 'Content-Length': '0'} 2023-09-29T22:34:02.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7077180973e4929-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.239 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.239 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.239 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.239 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.239 DEBUG:boto:Method: GET 2023-09-29T22:34:02.240 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.240 DEBUG:boto:Data: 2023-09-29T22:34:02.240 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.240 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.241 DEBUG:boto:Params: {} 2023-09-29T22:34:02.241 DEBUG:boto:Token: None 2023-09-29T22:34:02.241 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo= 2023-09-29T22:34:02.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo=', 'Content-Length': '0'} 2023-09-29T22:34:02.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab97b5963be3a7f-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.244 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.244 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.245 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.245 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.245 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.245 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.245 DEBUG:boto:Data: 2023-09-29T22:34:02.246 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.246 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.246 DEBUG:boto:Params: {} 2023-09-29T22:34:02.247 DEBUG:boto:Token: None 2023-09-29T22:34:02.247 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZRWt2Novg44FjRSzWtZry10Bw2g= 2023-09-29T22:34:02.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZRWt2Novg44FjRSzWtZry10Bw2g=', 'Content-Length': '0'} 2023-09-29T22:34:02.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035bdd6622d951a65-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.251 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.251 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.251 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.251 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.251 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.252 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.252 DEBUG:boto:Data: 2023-09-29T22:34:02.252 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.252 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.253 DEBUG:boto:Params: {} 2023-09-29T22:34:02.253 DEBUG:boto:Token: None 2023-09-29T22:34:02.253 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.254 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZRWt2Novg44FjRSzWtZry10Bw2g= 2023-09-29T22:34:02.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZRWt2Novg44FjRSzWtZry10Bw2g=', 'Content-Length': '0'} 2023-09-29T22:34:02.257 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6833a3c48bf47ec-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-29T22:34:02.258 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.258 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.258 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.258 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.258 DEBUG:boto:Method: GET 2023-09-29T22:34:02.259 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.259 DEBUG:boto:Data: 2023-09-29T22:34:02.259 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.259 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.259 DEBUG:boto:Params: {} 2023-09-29T22:34:02.259 DEBUG:boto:Token: None 2023-09-29T22:34:02.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.259 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo= 2023-09-29T22:34:02.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo=', 'Content-Length': '0'} 2023-09-29T22:34:02.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa080db8659abf0-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.306 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.307 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.307 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.307 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.307 DEBUG:boto:Method: GET 2023-09-29T22:34:02.307 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.307 DEBUG:boto:Data: 2023-09-29T22:34:02.307 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.307 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.308 DEBUG:boto:Params: {} 2023-09-29T22:34:02.308 DEBUG:boto:Token: None 2023-09-29T22:34:02.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:02.308 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo= 2023-09-29T22:34:02.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKuEGFDbR0eneeQWsk2jnMb8Tmo=', 'Content-Length': '0'} 2023-09-29T22:34:02.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006466ab5a0f5e453e-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.311 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.311 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.312 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.312 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.312 DEBUG:boto:Method: GET 2023-09-29T22:34:02.312 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.312 DEBUG:boto:Data: 2023-09-29T22:34:02.312 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.312 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.312 DEBUG:boto:Params: {} 2023-09-29T22:34:02.312 DEBUG:boto:Token: None 2023-09-29T22:34:02.313 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b5ceb68f5532511-00651750da-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.317 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.318 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.318 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.318 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.318 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.318 DEBUG:boto:Method: GET 2023-09-29T22:34:02.319 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.319 DEBUG:boto:Data: 2023-09-29T22:34:02.319 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.319 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.319 DEBUG:boto:Params: {} 2023-09-29T22:34:02.319 DEBUG:boto:Token: None 2023-09-29T22:34:02.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.324 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a30bfb6643320f3-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.370 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.541 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.541 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.541 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.541 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.542 DEBUG:boto:Data: 2023-09-29T22:34:02.542 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.542 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.542 DEBUG:boto:Params: {} 2023-09-29T22:34:02.543 DEBUG:boto:Token: None 2023-09-29T22:34:02.543 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.543 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QuMc5bllm4oFedPkr4fyqoKLhnI= 2023-09-29T22:34:02.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QuMc5bllm4oFedPkr4fyqoKLhnI=', 'Content-Length': '0'} 2023-09-29T22:34:02.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a39f645bdc96bf9-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.548 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.548 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.549 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.549 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.550 DEBUG:boto:Data: 2023-09-29T22:34:02.550 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.550 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.550 DEBUG:boto:Params: {} 2023-09-29T22:34:02.550 DEBUG:boto:Token: None 2023-09-29T22:34:02.551 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.551 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QuMc5bllm4oFedPkr4fyqoKLhnI= 2023-09-29T22:34:02.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QuMc5bllm4oFedPkr4fyqoKLhnI=', 'Content-Length': '0'} 2023-09-29T22:34:02.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b640ec1fe79b6962-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-29T22:34:02.555 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.555 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.556 DEBUG:boto:Method: GET 2023-09-29T22:34:02.557 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.557 DEBUG:boto:Data: 2023-09-29T22:34:02.557 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.557 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.558 DEBUG:boto:Params: {} 2023-09-29T22:34:02.558 DEBUG:boto:Token: None 2023-09-29T22:34:02.558 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.558 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rM2c9fh6vSRd5wfA/RVz7WBIbA0= 2023-09-29T22:34:02.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rM2c9fh6vSRd5wfA/RVz7WBIbA0=', 'Content-Length': '0'} 2023-09-29T22:34:02.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e19e4da539a53f6-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.561 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.562 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.562 DEBUG:boto:Method: GET 2023-09-29T22:34:02.562 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.562 DEBUG:boto:Data: 2023-09-29T22:34:02.563 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.563 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.563 DEBUG:boto:Params: {} 2023-09-29T22:34:02.563 DEBUG:boto:Token: None 2023-09-29T22:34:02.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:02.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rM2c9fh6vSRd5wfA/RVz7WBIbA0= 2023-09-29T22:34:02.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rM2c9fh6vSRd5wfA/RVz7WBIbA0=', 'Content-Length': '0'} 2023-09-29T22:34:02.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000022714899ece901a-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.568 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.568 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.569 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.569 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.569 DEBUG:boto:Method: GET 2023-09-29T22:34:02.569 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.570 DEBUG:boto:Data: 2023-09-29T22:34:02.570 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.570 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.570 DEBUG:boto:Params: {} 2023-09-29T22:34:02.571 DEBUG:boto:Token: None 2023-09-29T22:34:02.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000961b4deddcc7b64f-00651750da-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.576 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.576 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.576 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.577 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.577 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.577 DEBUG:boto:Method: GET 2023-09-29T22:34:02.577 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:02.577 DEBUG:boto:Data: 2023-09-29T22:34:02.578 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.578 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.578 DEBUG:boto:Params: {} 2023-09-29T22:34:02.579 DEBUG:boto:Token: None 2023-09-29T22:34:02.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004276b0e7f2d6e42f-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.584 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:33:39.974Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-29T22:34:02.584 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.584 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.584 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.584 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.585 DEBUG:boto:Method: GET 2023-09-29T22:34:02.585 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.585 DEBUG:boto:Data: 2023-09-29T22:34:02.585 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.585 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.585 DEBUG:boto:Params: {} 2023-09-29T22:34:02.585 DEBUG:boto:Token: None 2023-09-29T22:34:02.585 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08= 2023-09-29T22:34:02.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08=', 'Content-Length': '0'} 2023-09-29T22:34:02.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc6ed2632dc534e-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.630 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.631 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.631 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.631 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.631 DEBUG:boto:Method: GET 2023-09-29T22:34:02.631 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.631 DEBUG:boto:Data: 2023-09-29T22:34:02.632 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.632 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.632 DEBUG:boto:Params: {} 2023-09-29T22:34:02.632 DEBUG:boto:Token: None 2023-09-29T22:34:02.633 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08= 2023-09-29T22:34:02.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08=', 'Content-Length': '0'} 2023-09-29T22:34:02.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d78162e381046dcd-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.636 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.637 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.637 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.637 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.637 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.637 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.637 DEBUG:boto:Data: 2023-09-29T22:34:02.637 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.638 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.638 DEBUG:boto:Params: {} 2023-09-29T22:34:02.638 DEBUG:boto:Token: None 2023-09-29T22:34:02.638 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W73UOUdD+7iD9unH9OxAs4jebfw= 2023-09-29T22:34:02.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W73UOUdD+7iD9unH9OxAs4jebfw=', 'Content-Length': '0'} 2023-09-29T22:34:02.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d7392f0c79a3cef-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.641 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.642 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.642 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.642 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.642 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.642 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.642 DEBUG:boto:Data: 2023-09-29T22:34:02.642 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.643 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.643 DEBUG:boto:Params: {} 2023-09-29T22:34:02.643 DEBUG:boto:Token: None 2023-09-29T22:34:02.643 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.643 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W73UOUdD+7iD9unH9OxAs4jebfw= 2023-09-29T22:34:02.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W73UOUdD+7iD9unH9OxAs4jebfw=', 'Content-Length': '0'} 2023-09-29T22:34:02.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a80a70676a8c2df9-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-29T22:34:02.646 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.646 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.646 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.646 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.646 DEBUG:boto:Method: GET 2023-09-29T22:34:02.647 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.647 DEBUG:boto:Data: 2023-09-29T22:34:02.647 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.647 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.647 DEBUG:boto:Params: {} 2023-09-29T22:34:02.647 DEBUG:boto:Token: None 2023-09-29T22:34:02.647 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08= 2023-09-29T22:34:02.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08=', 'Content-Length': '0'} 2023-09-29T22:34:02.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000125ede0adb0c24ac-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.651 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.651 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.651 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.651 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.651 DEBUG:boto:Method: GET 2023-09-29T22:34:02.651 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.651 DEBUG:boto:Data: 2023-09-29T22:34:02.652 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.652 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.652 DEBUG:boto:Params: {} 2023-09-29T22:34:02.652 DEBUG:boto:Token: None 2023-09-29T22:34:02.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:02.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08= 2023-09-29T22:34:02.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s+rW9YyDwjDEjh78tmQfRKuAu08=', 'Content-Length': '0'} 2023-09-29T22:34:02.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2007b08131b5bac-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.655 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.655 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.656 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.656 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.656 DEBUG:boto:Method: GET 2023-09-29T22:34:02.656 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.656 DEBUG:boto:Data: 2023-09-29T22:34:02.656 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.656 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.656 DEBUG:boto:Params: {} 2023-09-29T22:34:02.656 DEBUG:boto:Token: None 2023-09-29T22:34:02.657 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d8689eea17af516-00651750da-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.663 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.663 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.663 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.663 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.663 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.663 DEBUG:boto:Method: GET 2023-09-29T22:34:02.664 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.664 DEBUG:boto:Data: 2023-09-29T22:34:02.664 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.664 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.664 DEBUG:boto:Params: {} 2023-09-29T22:34:02.664 DEBUG:boto:Token: None 2023-09-29T22:34:02.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639e18fa327cd4b0-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.710 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.711 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.711 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.711 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.712 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.712 DEBUG:boto:Data: 2023-09-29T22:34:02.712 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.712 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.712 DEBUG:boto:Params: {} 2023-09-29T22:34:02.713 DEBUG:boto:Token: None 2023-09-29T22:34:02.713 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n3xO/fxsR/TCEkgjLLHvAgv+WuU= 2023-09-29T22:34:02.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n3xO/fxsR/TCEkgjLLHvAgv+WuU=', 'Content-Length': '0'} 2023-09-29T22:34:02.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000085573a4a2e0e53f-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.717 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.717 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.717 DEBUG:boto:Method: HEAD 2023-09-29T22:34:02.717 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.717 DEBUG:boto:Data: 2023-09-29T22:34:02.717 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.717 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.718 DEBUG:boto:Params: {} 2023-09-29T22:34:02.718 DEBUG:boto:Token: None 2023-09-29T22:34:02.718 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n3xO/fxsR/TCEkgjLLHvAgv+WuU= 2023-09-29T22:34:02.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n3xO/fxsR/TCEkgjLLHvAgv+WuU=', 'Content-Length': '0'} 2023-09-29T22:34:02.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fecdebf307526b9-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-29T22:34:02.721 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.722 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.722 DEBUG:boto:Method: GET 2023-09-29T22:34:02.722 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.722 DEBUG:boto:Data: 2023-09-29T22:34:02.722 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.722 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.722 DEBUG:boto:Params: {} 2023-09-29T22:34:02.722 DEBUG:boto:Token: None 2023-09-29T22:34:02.723 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rnn/tUv2LRbkpngPxCpPQrXOcek= 2023-09-29T22:34:02.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rnn/tUv2LRbkpngPxCpPQrXOcek=', 'Content-Length': '0'} 2023-09-29T22:34:02.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f862f10c6ea337-00651750da-1a82-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.770 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.771 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.771 DEBUG:boto:Method: GET 2023-09-29T22:34:02.771 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.771 DEBUG:boto:Data: 2023-09-29T22:34:02.771 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.771 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.771 DEBUG:boto:Params: {} 2023-09-29T22:34:02.772 DEBUG:boto:Token: None 2023-09-29T22:34:02.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:02.772 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rnn/tUv2LRbkpngPxCpPQrXOcek= 2023-09-29T22:34:02.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rnn/tUv2LRbkpngPxCpPQrXOcek=', 'Content-Length': '0'} 2023-09-29T22:34:02.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:33:39 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3d35b9802f9f72b-00651750da-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.775 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.776 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.776 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.776 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.776 DEBUG:boto:Method: GET 2023-09-29T22:34:02.776 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.776 DEBUG:boto:Data: 2023-09-29T22:34:02.776 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:02.777 DEBUG:boto:Port: 8001 2023-09-29T22:34:02.777 DEBUG:boto:Params: {} 2023-09-29T22:34:02.777 DEBUG:boto:Token: None 2023-09-29T22:34:02.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000855da198f6c45591-00651750da-1a82-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.782 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.783 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:02.783 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:02.784 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.784 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.785 DEBUG:boto:Method: GET 2023-09-29T22:34:02.785 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:02.785 DEBUG:boto:Data: 2023-09-29T22:34:02.785 DEBUG:boto:Headers: {} 2023-09-29T22:34:02.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:02.786 DEBUG:boto:Port: 8000 2023-09-29T22:34:02.786 DEBUG:boto:Params: {} 2023-09-29T22:34:02.787 DEBUG:boto:Token: None 2023-09-29T22:34:02.787 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:02 GMT /hcnphz-58/ 2023-09-29T22:34:02.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk= 2023-09-29T22:34:02.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:02 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXzk2gncRMQtejCw3QHvh6u5jFk=', 'Content-Length': '0'} 2023-09-29T22:34:02.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d08dcd0cd91239f-00651750da-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:02 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:02.793 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:33:39.917Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:02.794 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-58 zones={a2, a1} 2023-09-29T22:34:02.794 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T22:34:02.794 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:34:02.795 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:34:02.792+0000 7f19d47be700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 63865) UID: 0 2023-09-29T22:34:02.796 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:34:02.792+0000 7f19d801df80 -1 shutting down 2023-09-29T22:34:04.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T22:34:08.894 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T22:34:08.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:08.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:08.894 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:08.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:08.942 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:09.023 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:34:09.023 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-58 2023-09-29T22:34:09.032 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-09-29T22:34:09.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:09.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:09.109 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:09.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:09.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000024", 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000020", 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000028", 2023-09-29T22:34:09.231 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000032", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-09-29T22:34:09.232 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000026", 2023-09-29T22:34:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000033", 2023-09-29T22:34:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-09-29T22:34:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000076" 2023-09-29T22:34:09.233 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:09.233 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:09.234 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.234 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.234 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.235 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.235 DEBUG:boto:Data: 2023-09-29T22:34:09.235 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.235 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.235 DEBUG:boto:Params: {} 2023-09-29T22:34:09.235 DEBUG:boto:Token: None 2023-09-29T22:34:09.235 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb3cb673d0ed38d7-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.240 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:09.240 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:09.241 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.241 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:09.241 DEBUG:boto:Data: 2023-09-29T22:34:09.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.241 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.241 DEBUG:boto:Params: {} 2023-09-29T22:34:09.241 DEBUG:boto:Token: None 2023-09-29T22:34:09.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:09.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g9VtEj4lpcEvOnGxNoW56WarYxY= 2023-09-29T22:34:09.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g9VtEj4lpcEvOnGxNoW56WarYxY='} 2023-09-29T22:34:09.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a18d2254eb5fdcd2-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.291 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.291 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.291 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.291 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.291 DEBUG:boto:Data: 2023-09-29T22:34:09.291 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.292 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.292 DEBUG:boto:Params: {} 2023-09-29T22:34:09.292 DEBUG:boto:Token: None 2023-09-29T22:34:09.292 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.297 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b10171ef67bb407b-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.298 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:09.298 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:09.298 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.298 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:09.298 DEBUG:boto:Data: 2023-09-29T22:34:09.298 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.299 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.299 DEBUG:boto:Params: {} 2023-09-29T22:34:09.299 DEBUG:boto:Token: None 2023-09-29T22:34:09.299 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:09.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pXOkO8BMBDDa7VZKpQJyiqGNRc0= 2023-09-29T22:34:09.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXOkO8BMBDDa7VZKpQJyiqGNRc0='} 2023-09-29T22:34:09.314 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005fbecb84cf4d4c40-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.314 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.314 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.314 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.315 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.315 DEBUG:boto:Data: 2023-09-29T22:34:09.315 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.315 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.315 DEBUG:boto:Params: {} 2023-09-29T22:34:09.315 DEBUG:boto:Token: None 2023-09-29T22:34:09.315 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.320 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067b5b1ab7471d0b2-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.320 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:09.320 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:09.320 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.320 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:09.320 DEBUG:boto:Data: 2023-09-29T22:34:09.321 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.321 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.321 DEBUG:boto:Params: {} 2023-09-29T22:34:09.321 DEBUG:boto:Token: None 2023-09-29T22:34:09.321 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:09.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2369d3q0Ehe5erwEHlZTv/qXYAk= 2023-09-29T22:34:09.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2369d3q0Ehe5erwEHlZTv/qXYAk='} 2023-09-29T22:34:09.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbcbbd2c2ab5943f-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.371 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.371 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.371 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.372 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.372 DEBUG:boto:Data: 2023-09-29T22:34:09.372 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.372 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.372 DEBUG:boto:Params: {} 2023-09-29T22:34:09.373 DEBUG:boto:Token: None 2023-09-29T22:34:09.373 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ed68c0ddbafcde2e-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.378 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:09.378 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:09.378 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.379 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:09.379 DEBUG:boto:Data: 2023-09-29T22:34:09.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.379 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.379 DEBUG:boto:Params: {} 2023-09-29T22:34:09.380 DEBUG:boto:Token: None 2023-09-29T22:34:09.380 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:09.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PB5OlUATXPOVOTkpFMtNN/a6CeY= 2023-09-29T22:34:09.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PB5OlUATXPOVOTkpFMtNN/a6CeY='} 2023-09-29T22:34:09.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff3de12df221793e-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.392 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.393 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.393 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.393 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.393 DEBUG:boto:Data: 2023-09-29T22:34:09.393 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.393 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.393 DEBUG:boto:Params: {} 2023-09-29T22:34:09.393 DEBUG:boto:Token: None 2023-09-29T22:34:09.394 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.398 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003322928c78fcf9dc-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.398 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:09.398 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:09.398 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.399 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:09.399 DEBUG:boto:Data: 2023-09-29T22:34:09.399 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.399 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.400 DEBUG:boto:Params: {} 2023-09-29T22:34:09.400 DEBUG:boto:Token: None 2023-09-29T22:34:09.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:09.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lknLDEd5PLOVENnreOKDv2Cgw5k= 2023-09-29T22:34:09.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lknLDEd5PLOVENnreOKDv2Cgw5k='} 2023-09-29T22:34:09.446 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006032fb45e552dd81-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.447 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.447 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.447 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.448 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.448 DEBUG:boto:Data: 2023-09-29T22:34:09.448 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.448 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.448 DEBUG:boto:Params: {} 2023-09-29T22:34:09.449 DEBUG:boto:Token: None 2023-09-29T22:34:09.449 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.449 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.455 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6f5f6854043628b-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.455 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:09.456 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:09.456 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.456 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:09.456 DEBUG:boto:Data: 2023-09-29T22:34:09.457 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.457 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.457 DEBUG:boto:Params: {} 2023-09-29T22:34:09.457 DEBUG:boto:Token: None 2023-09-29T22:34:09.458 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:09.458 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GutO8Qj/xuFOAZeF4pp+9uApJTo= 2023-09-29T22:34:09.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GutO8Qj/xuFOAZeF4pp+9uApJTo='} 2023-09-29T22:34:09.470 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ef0d7a547f826144-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.470 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.470 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.470 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.471 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.471 DEBUG:boto:Data: 2023-09-29T22:34:09.471 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.471 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.471 DEBUG:boto:Params: {} 2023-09-29T22:34:09.471 DEBUG:boto:Token: None 2023-09-29T22:34:09.471 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007963e2b733e5344d-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.476 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:09.476 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:09.476 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.477 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:09.477 DEBUG:boto:Data: 2023-09-29T22:34:09.477 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.477 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.477 DEBUG:boto:Params: {} 2023-09-29T22:34:09.477 DEBUG:boto:Token: None 2023-09-29T22:34:09.477 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:09.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wYip472wFI1/t/EFLrW6b1Olq8I= 2023-09-29T22:34:09.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wYip472wFI1/t/EFLrW6b1Olq8I='} 2023-09-29T22:34:09.522 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e356376555486d84-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.523 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.523 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.523 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.523 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.523 DEBUG:boto:Data: 2023-09-29T22:34:09.523 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.524 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.524 DEBUG:boto:Params: {} 2023-09-29T22:34:09.524 DEBUG:boto:Token: None 2023-09-29T22:34:09.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.531 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000104f2eba3bcd0224-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.532 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:09.532 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:09.532 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.532 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:09.532 DEBUG:boto:Data: 2023-09-29T22:34:09.533 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.533 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.533 DEBUG:boto:Params: {} 2023-09-29T22:34:09.533 DEBUG:boto:Token: None 2023-09-29T22:34:09.533 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:09.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bv6gF7+Dc/sC1O85E9Ccuk5/eqM= 2023-09-29T22:34:09.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bv6gF7+Dc/sC1O85E9Ccuk5/eqM='} 2023-09-29T22:34:09.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f99e891247fbe907-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.541 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.541 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.542 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.542 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.542 DEBUG:boto:Data: 2023-09-29T22:34:09.542 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.542 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.542 DEBUG:boto:Params: {} 2023-09-29T22:34:09.542 DEBUG:boto:Token: None 2023-09-29T22:34:09.543 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.543 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.546 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab1435534b239fa9-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.546 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:09.546 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:09.546 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.546 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:09.546 DEBUG:boto:Data: 2023-09-29T22:34:09.546 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.547 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.547 DEBUG:boto:Params: {} 2023-09-29T22:34:09.547 DEBUG:boto:Token: None 2023-09-29T22:34:09.547 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:09.547 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EQGwQOHBXCtZK8PRc1URuIVLl8U= 2023-09-29T22:34:09.547 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EQGwQOHBXCtZK8PRc1URuIVLl8U='} 2023-09-29T22:34:09.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d7912813977afdbd-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.591 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.591 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.591 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.591 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.591 DEBUG:boto:Data: 2023-09-29T22:34:09.591 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.592 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.592 DEBUG:boto:Params: {} 2023-09-29T22:34:09.592 DEBUG:boto:Token: None 2023-09-29T22:34:09.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.598 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9636a77363b5c84-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.598 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:09.599 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:09.599 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.599 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:09.599 DEBUG:boto:Data: 2023-09-29T22:34:09.599 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.600 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.600 DEBUG:boto:Params: {} 2023-09-29T22:34:09.600 DEBUG:boto:Token: None 2023-09-29T22:34:09.600 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:09.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VatmZEw+vI81j0l4o9iJ23iaUfM= 2023-09-29T22:34:09.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VatmZEw+vI81j0l4o9iJ23iaUfM='} 2023-09-29T22:34:09.612 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e44db594c26536bd-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.612 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.612 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.612 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.612 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.612 DEBUG:boto:Data: 2023-09-29T22:34:09.613 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.613 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.613 DEBUG:boto:Params: {} 2023-09-29T22:34:09.613 DEBUG:boto:Token: None 2023-09-29T22:34:09.613 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.613 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.618 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd30683ff559b1bf-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.618 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:09.618 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:09.618 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.618 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:09.618 DEBUG:boto:Data: 2023-09-29T22:34:09.619 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.619 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.619 DEBUG:boto:Params: {} 2023-09-29T22:34:09.619 DEBUG:boto:Token: None 2023-09-29T22:34:09.619 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:09.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ACaqHjUzTvuF1c7ZD3Z49XHHHHE= 2023-09-29T22:34:09.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ACaqHjUzTvuF1c7ZD3Z49XHHHHE='} 2023-09-29T22:34:09.666 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010eba76a7aca374a-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.667 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.667 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.667 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.667 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.667 DEBUG:boto:Data: 2023-09-29T22:34:09.667 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.667 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.668 DEBUG:boto:Params: {} 2023-09-29T22:34:09.668 DEBUG:boto:Token: None 2023-09-29T22:34:09.668 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.673 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f8fddfb2e2df1447-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.674 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:09.674 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:09.674 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.674 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:09.674 DEBUG:boto:Data: 2023-09-29T22:34:09.674 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.675 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.675 DEBUG:boto:Params: {} 2023-09-29T22:34:09.675 DEBUG:boto:Token: None 2023-09-29T22:34:09.675 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:09.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+JuBOxgb6nO/p3VziuAhOPI2PP0= 2023-09-29T22:34:09.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+JuBOxgb6nO/p3VziuAhOPI2PP0='} 2023-09-29T22:34:09.684 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006c97b2e104648fab-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.685 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.685 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.685 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.685 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.685 DEBUG:boto:Data: 2023-09-29T22:34:09.685 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.686 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.686 DEBUG:boto:Params: {} 2023-09-29T22:34:09.686 DEBUG:boto:Token: None 2023-09-29T22:34:09.686 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.690 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d4e679ef7b1a6da-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.690 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:09.690 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:09.690 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.690 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:09.690 DEBUG:boto:Data: 2023-09-29T22:34:09.690 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.691 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.691 DEBUG:boto:Params: {} 2023-09-29T22:34:09.691 DEBUG:boto:Token: None 2023-09-29T22:34:09.691 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:09.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VlBnra2xsKw5ZouUZ3cJ3DTPCfo= 2023-09-29T22:34:09.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VlBnra2xsKw5ZouUZ3cJ3DTPCfo='} 2023-09-29T22:34:09.739 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000575c9ea7a5f734c5-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.739 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.739 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.739 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.739 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.739 DEBUG:boto:Data: 2023-09-29T22:34:09.739 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.740 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.740 DEBUG:boto:Params: {} 2023-09-29T22:34:09.740 DEBUG:boto:Token: None 2023-09-29T22:34:09.740 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f0de4070db3c3cde-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.745 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:09.746 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:09.746 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.746 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:09.746 DEBUG:boto:Data: 2023-09-29T22:34:09.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.746 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.747 DEBUG:boto:Params: {} 2023-09-29T22:34:09.747 DEBUG:boto:Token: None 2023-09-29T22:34:09.747 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:09.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y49izTU5SsMivx6q/V5Ji6r8cZQ= 2023-09-29T22:34:09.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y49izTU5SsMivx6q/V5Ji6r8cZQ='} 2023-09-29T22:34:09.758 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ee06f6545e18529-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.758 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.758 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.759 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.759 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.759 DEBUG:boto:Data: 2023-09-29T22:34:09.759 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.759 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.759 DEBUG:boto:Params: {} 2023-09-29T22:34:09.759 DEBUG:boto:Token: None 2023-09-29T22:34:09.759 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.763 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc6dcc0e272c3cc1-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.763 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:09.764 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:09.764 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.764 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:09.764 DEBUG:boto:Data: 2023-09-29T22:34:09.764 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.764 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.764 DEBUG:boto:Params: {} 2023-09-29T22:34:09.765 DEBUG:boto:Token: None 2023-09-29T22:34:09.765 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:09.765 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2PoK5rSlCIXokZ9Fr4bRQMURM0Q= 2023-09-29T22:34:09.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2PoK5rSlCIXokZ9Fr4bRQMURM0Q='} 2023-09-29T22:34:09.810 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002a87257f7341c446-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.811 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:09.811 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:09.811 DEBUG:boto:Method: HEAD 2023-09-29T22:34:09.811 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:09.812 DEBUG:boto:Data: 2023-09-29T22:34:09.812 DEBUG:boto:Headers: {} 2023-09-29T22:34:09.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.812 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.812 DEBUG:boto:Params: {} 2023-09-29T22:34:09.812 DEBUG:boto:Token: None 2023-09-29T22:34:09.812 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/ 2023-09-29T22:34:09.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE= 2023-09-29T22:34:09.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f7PP435ujgdkexZmsV7FXWGvKyE=', 'Content-Length': '0'} 2023-09-29T22:34:09.818 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009706796af8034062-00651750e1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.819 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:09.819 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:09.819 DEBUG:boto:Method: PUT 2023-09-29T22:34:09.820 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:09.820 DEBUG:boto:Data: 2023-09-29T22:34:09.821 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:09.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:09.822 DEBUG:boto:Port: 8000 2023-09-29T22:34:09.822 DEBUG:boto:Params: {} 2023-09-29T22:34:09.823 DEBUG:boto:Token: None 2023-09-29T22:34:09.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:09 GMT /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:09.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIFxn3+HFJzsMisKTFRQdR9v7e8= 2023-09-29T22:34:09.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIFxn3+HFJzsMisKTFRQdR9v7e8='} 2023-09-29T22:34:09.834 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d24d8e62fa4b39d2-00651750e1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:09.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:09.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:09.835 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:09.883 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:09.883 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:09.967 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:34:09.967 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-58 2023-09-29T22:34:09.982 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 22 2023-09-29T22:34:10.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:10.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:10.068 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:10.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:10.118 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:10.188 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:34:10.188 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000025", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000015", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-09-29T22:34:10.189 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007" 2023-09-29T22:34:10.190 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:10.191 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:10.191 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.192 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.192 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.192 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.192 DEBUG:boto:Data: 2023-09-29T22:34:10.192 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.192 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.192 DEBUG:boto:Params: {} 2023-09-29T22:34:10.193 DEBUG:boto:Token: None 2023-09-29T22:34:10.193 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.197 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000843b2fe082a018c-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.198 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:10.198 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:10.198 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.198 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:10.198 DEBUG:boto:Data: 2023-09-29T22:34:10.199 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.199 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.199 DEBUG:boto:Params: {} 2023-09-29T22:34:10.199 DEBUG:boto:Token: None 2023-09-29T22:34:10.199 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:10.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:luDckl5eMOe+2GDv/jCUkO8i8o8= 2023-09-29T22:34:10.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:luDckl5eMOe+2GDv/jCUkO8i8o8='} 2023-09-29T22:34:10.243 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000dd4f17a771fccbf-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.243 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.243 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.243 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.243 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.243 DEBUG:boto:Data: 2023-09-29T22:34:10.243 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.244 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.244 DEBUG:boto:Params: {} 2023-09-29T22:34:10.244 DEBUG:boto:Token: None 2023-09-29T22:34:10.244 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.249 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002a28345c426ae521-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.250 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:10.250 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:10.250 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.250 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:10.250 DEBUG:boto:Data: 2023-09-29T22:34:10.250 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.251 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.251 DEBUG:boto:Params: {} 2023-09-29T22:34:10.251 DEBUG:boto:Token: None 2023-09-29T22:34:10.251 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:10.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6j0gvZfJ5SDgUqhHPJvng244G3U= 2023-09-29T22:34:10.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6j0gvZfJ5SDgUqhHPJvng244G3U='} 2023-09-29T22:34:10.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c4a9dd994582b370-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.267 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.267 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.267 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.267 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.267 DEBUG:boto:Data: 2023-09-29T22:34:10.268 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.268 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.269 DEBUG:boto:Params: {} 2023-09-29T22:34:10.269 DEBUG:boto:Token: None 2023-09-29T22:34:10.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.274 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fe3087084f2438d3-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.274 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:10.275 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:10.275 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.275 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:10.275 DEBUG:boto:Data: 2023-09-29T22:34:10.275 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.275 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.275 DEBUG:boto:Params: {} 2023-09-29T22:34:10.276 DEBUG:boto:Token: None 2023-09-29T22:34:10.276 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:10.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K8XpZ/FkqFKfaKoNV0IGsSvirzs= 2023-09-29T22:34:10.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K8XpZ/FkqFKfaKoNV0IGsSvirzs='} 2023-09-29T22:34:10.318 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063734ee240b42194-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.319 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.319 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.319 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.320 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.320 DEBUG:boto:Data: 2023-09-29T22:34:10.320 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.321 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.321 DEBUG:boto:Params: {} 2023-09-29T22:34:10.321 DEBUG:boto:Token: None 2023-09-29T22:34:10.321 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.322 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.327 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000055cce304a5664b67-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.328 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:10.328 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:10.328 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.328 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:10.329 DEBUG:boto:Data: 2023-09-29T22:34:10.329 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.329 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.329 DEBUG:boto:Params: {} 2023-09-29T22:34:10.330 DEBUG:boto:Token: None 2023-09-29T22:34:10.330 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:10.331 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FccIizb6Dhbn6twMfWE34ggFjHc= 2023-09-29T22:34:10.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FccIizb6Dhbn6twMfWE34ggFjHc='} 2023-09-29T22:34:10.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T22:34:10.342 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005705416cd801105d-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.342 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.343 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.343 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.343 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.343 DEBUG:boto:Data: 2023-09-29T22:34:10.344 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.344 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.344 DEBUG:boto:Params: {} 2023-09-29T22:34:10.345 DEBUG:boto:Token: None 2023-09-29T22:34:10.345 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.350 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000075737ff40ddf5e3-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.351 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:10.351 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:10.351 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.352 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:10.352 DEBUG:boto:Data: 2023-09-29T22:34:10.352 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.352 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.353 DEBUG:boto:Params: {} 2023-09-29T22:34:10.353 DEBUG:boto:Token: None 2023-09-29T22:34:10.353 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:10.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T9IwEKw2DLGCN9YgqPVYYchaOLw= 2023-09-29T22:34:10.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T9IwEKw2DLGCN9YgqPVYYchaOLw='} 2023-09-29T22:34:10.398 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b45a8f7c79a7fc8-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.399 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.399 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.399 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.399 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.400 DEBUG:boto:Data: 2023-09-29T22:34:10.400 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.400 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.400 DEBUG:boto:Params: {} 2023-09-29T22:34:10.401 DEBUG:boto:Token: None 2023-09-29T22:34:10.401 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000faa8bfa54edce996-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.407 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:10.407 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:10.408 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.408 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:10.408 DEBUG:boto:Data: 2023-09-29T22:34:10.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.408 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.408 DEBUG:boto:Params: {} 2023-09-29T22:34:10.408 DEBUG:boto:Token: None 2023-09-29T22:34:10.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:10.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QFSXgEz8+MKmZonRvOuAgvNzxAg= 2023-09-29T22:34:10.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QFSXgEz8+MKmZonRvOuAgvNzxAg='} 2023-09-29T22:34:10.420 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b3466528d81ffdee-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.420 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.421 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.421 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.421 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.421 DEBUG:boto:Data: 2023-09-29T22:34:10.421 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.421 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.421 DEBUG:boto:Params: {} 2023-09-29T22:34:10.422 DEBUG:boto:Token: None 2023-09-29T22:34:10.422 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006afd3331fe651870-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.426 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:10.426 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:10.426 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.427 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:10.427 DEBUG:boto:Data: 2023-09-29T22:34:10.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.427 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.427 DEBUG:boto:Params: {} 2023-09-29T22:34:10.427 DEBUG:boto:Token: None 2023-09-29T22:34:10.427 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:10.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kDUOIBwBgWks5qTPWMQ6N2L+yyk= 2023-09-29T22:34:10.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kDUOIBwBgWks5qTPWMQ6N2L+yyk='} 2023-09-29T22:34:10.471 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020884ac125a70374-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.471 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.471 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.472 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.472 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.472 DEBUG:boto:Data: 2023-09-29T22:34:10.472 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.473 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.473 DEBUG:boto:Params: {} 2023-09-29T22:34:10.473 DEBUG:boto:Token: None 2023-09-29T22:34:10.473 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000338c13359452a622-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.481 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:10.481 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:10.481 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.482 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:10.482 DEBUG:boto:Data: 2023-09-29T22:34:10.482 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.482 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.483 DEBUG:boto:Params: {} 2023-09-29T22:34:10.483 DEBUG:boto:Token: None 2023-09-29T22:34:10.483 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:10.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NiK+lrHgUdrCqyC1GCEV0t3sYf8= 2023-09-29T22:34:10.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NiK+lrHgUdrCqyC1GCEV0t3sYf8='} 2023-09-29T22:34:10.497 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000674880a774cc991a-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.497 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.497 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.497 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.498 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.498 DEBUG:boto:Data: 2023-09-29T22:34:10.498 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.498 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.498 DEBUG:boto:Params: {} 2023-09-29T22:34:10.498 DEBUG:boto:Token: None 2023-09-29T22:34:10.498 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.499 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ef4eb28df2220ea6-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.504 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:10.504 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:10.504 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.504 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:10.504 DEBUG:boto:Data: 2023-09-29T22:34:10.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.504 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.505 DEBUG:boto:Params: {} 2023-09-29T22:34:10.505 DEBUG:boto:Token: None 2023-09-29T22:34:10.505 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:10.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QQVlqxkzq2Z43+qwhgZ8+mqEJxg= 2023-09-29T22:34:10.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QQVlqxkzq2Z43+qwhgZ8+mqEJxg='} 2023-09-29T22:34:10.551 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004db01b4992dfd607-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.551 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.551 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.551 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.551 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.551 DEBUG:boto:Data: 2023-09-29T22:34:10.551 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.552 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.552 DEBUG:boto:Params: {} 2023-09-29T22:34:10.552 DEBUG:boto:Token: None 2023-09-29T22:34:10.552 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.557 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f82d4a856a3dabca-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.558 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:10.558 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:10.558 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.558 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:10.559 DEBUG:boto:Data: 2023-09-29T22:34:10.559 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.559 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.559 DEBUG:boto:Params: {} 2023-09-29T22:34:10.560 DEBUG:boto:Token: None 2023-09-29T22:34:10.560 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:10.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VeOGAmZHOtG4fyltOVnLOdoPSe4= 2023-09-29T22:34:10.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VeOGAmZHOtG4fyltOVnLOdoPSe4='} 2023-09-29T22:34:10.572 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000980a4ad23745bfd8-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.572 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.572 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.572 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.572 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.572 DEBUG:boto:Data: 2023-09-29T22:34:10.572 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.573 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.573 DEBUG:boto:Params: {} 2023-09-29T22:34:10.573 DEBUG:boto:Token: None 2023-09-29T22:34:10.573 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.578 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000fa6d562e56ba952-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.579 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:10.579 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:10.579 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.579 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:10.579 DEBUG:boto:Data: 2023-09-29T22:34:10.580 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.580 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.580 DEBUG:boto:Params: {} 2023-09-29T22:34:10.580 DEBUG:boto:Token: None 2023-09-29T22:34:10.581 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:10.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wPf7blghJLvu8lkWjXzQdVjQ7Ls= 2023-09-29T22:34:10.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wPf7blghJLvu8lkWjXzQdVjQ7Ls='} 2023-09-29T22:34:10.627 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f733711947c28d92-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.627 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.627 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.628 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.628 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.628 DEBUG:boto:Data: 2023-09-29T22:34:10.628 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.628 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.628 DEBUG:boto:Params: {} 2023-09-29T22:34:10.628 DEBUG:boto:Token: None 2023-09-29T22:34:10.629 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.635 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047421f0cef01f072-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.635 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:10.635 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:10.635 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.635 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:10.635 DEBUG:boto:Data: 2023-09-29T22:34:10.636 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.636 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.636 DEBUG:boto:Params: {} 2023-09-29T22:34:10.636 DEBUG:boto:Token: None 2023-09-29T22:34:10.636 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:10.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AbKRaAn7xnixY/AElF+6uZUjj58= 2023-09-29T22:34:10.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AbKRaAn7xnixY/AElF+6uZUjj58='} 2023-09-29T22:34:10.647 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000036d6b64db95b5818-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.647 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.647 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.648 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.648 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.648 DEBUG:boto:Data: 2023-09-29T22:34:10.648 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.648 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.648 DEBUG:boto:Params: {} 2023-09-29T22:34:10.649 DEBUG:boto:Token: None 2023-09-29T22:34:10.649 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.653 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000871ed3a5d539347a-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.654 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:10.654 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:10.654 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.654 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:10.654 DEBUG:boto:Data: 2023-09-29T22:34:10.654 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.654 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.655 DEBUG:boto:Params: {} 2023-09-29T22:34:10.655 DEBUG:boto:Token: None 2023-09-29T22:34:10.655 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:10.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XTycu10MIcTA3e94fzrAlA+bOhg= 2023-09-29T22:34:10.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XTycu10MIcTA3e94fzrAlA+bOhg='} 2023-09-29T22:34:10.699 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a4aa51ecc13fb2c-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.699 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.699 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.699 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.699 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.699 DEBUG:boto:Data: 2023-09-29T22:34:10.700 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.700 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.700 DEBUG:boto:Params: {} 2023-09-29T22:34:10.700 DEBUG:boto:Token: None 2023-09-29T22:34:10.700 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.706 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041cd3f558cbe594e-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.706 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:10.706 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:10.707 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.707 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:10.707 DEBUG:boto:Data: 2023-09-29T22:34:10.707 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.707 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.707 DEBUG:boto:Params: {} 2023-09-29T22:34:10.707 DEBUG:boto:Token: None 2023-09-29T22:34:10.707 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:10.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pAB2NRzkqZcDuVBGcOjxB/dZRP8= 2023-09-29T22:34:10.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pAB2NRzkqZcDuVBGcOjxB/dZRP8='} 2023-09-29T22:34:10.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f89c85bef5b3375b-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.721 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.721 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.721 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.722 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.722 DEBUG:boto:Data: 2023-09-29T22:34:10.722 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.722 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.723 DEBUG:boto:Params: {} 2023-09-29T22:34:10.723 DEBUG:boto:Token: None 2023-09-29T22:34:10.723 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8643592d46523f1-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.728 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:10.728 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:10.728 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.728 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:10.729 DEBUG:boto:Data: 2023-09-29T22:34:10.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.729 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.729 DEBUG:boto:Params: {} 2023-09-29T22:34:10.729 DEBUG:boto:Token: None 2023-09-29T22:34:10.729 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:10.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t9tKXDsBBPNWSR4Ls6dqI+Zl+wk= 2023-09-29T22:34:10.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t9tKXDsBBPNWSR4Ls6dqI+Zl+wk='} 2023-09-29T22:34:10.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000157e0b2436233dbf-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.775 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.775 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.775 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.775 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.775 DEBUG:boto:Data: 2023-09-29T22:34:10.775 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.776 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.776 DEBUG:boto:Params: {} 2023-09-29T22:34:10.776 DEBUG:boto:Token: None 2023-09-29T22:34:10.776 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.782 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d81a33a3e003478f-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.782 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:10.783 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:10.783 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.784 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:10.784 DEBUG:boto:Data: 2023-09-29T22:34:10.784 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.784 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.785 DEBUG:boto:Params: {} 2023-09-29T22:34:10.785 DEBUG:boto:Token: None 2023-09-29T22:34:10.785 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:10.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bYAvGSbpRywiSeMVDrmLpblKTaA= 2023-09-29T22:34:10.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bYAvGSbpRywiSeMVDrmLpblKTaA='} 2023-09-29T22:34:10.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f4e2095a743a620-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.799 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.799 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.799 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.800 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.800 DEBUG:boto:Data: 2023-09-29T22:34:10.800 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.800 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.800 DEBUG:boto:Params: {} 2023-09-29T22:34:10.800 DEBUG:boto:Token: None 2023-09-29T22:34:10.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.806 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008cc2954642a45428-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.806 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:10.806 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:10.806 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.806 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:10.806 DEBUG:boto:Data: 2023-09-29T22:34:10.807 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.807 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.807 DEBUG:boto:Params: {} 2023-09-29T22:34:10.807 DEBUG:boto:Token: None 2023-09-29T22:34:10.807 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:10.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:68URKJco7aUZ06j1auwhuwyISiM= 2023-09-29T22:34:10.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:68URKJco7aUZ06j1auwhuwyISiM='} 2023-09-29T22:34:10.850 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe7e67e2bfb707cc-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.851 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:10.851 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:10.851 DEBUG:boto:Method: HEAD 2023-09-29T22:34:10.851 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:10.851 DEBUG:boto:Data: 2023-09-29T22:34:10.852 DEBUG:boto:Headers: {} 2023-09-29T22:34:10.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.852 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.852 DEBUG:boto:Params: {} 2023-09-29T22:34:10.852 DEBUG:boto:Token: None 2023-09-29T22:34:10.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/ 2023-09-29T22:34:10.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4= 2023-09-29T22:34:10.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylPMdNIRiuoUvXS+2uimeRC67T4=', 'Content-Length': '0'} 2023-09-29T22:34:10.858 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000257fb22e2f17b50e-00651750e2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.858 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:10.858 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:10.858 DEBUG:boto:Method: PUT 2023-09-29T22:34:10.859 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:10.859 DEBUG:boto:Data: 2023-09-29T22:34:10.859 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:10.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:10.859 DEBUG:boto:Port: 8000 2023-09-29T22:34:10.859 DEBUG:boto:Params: {} 2023-09-29T22:34:10.859 DEBUG:boto:Token: None 2023-09-29T22:34:10.859 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:10 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:10.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GzwCzrRWM1B0zluk/rDPNCvmNH0= 2023-09-29T22:34:10.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:10 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GzwCzrRWM1B0zluk/rDPNCvmNH0='} 2023-09-29T22:34:10.873 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f8962071ae038813-00651750e2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:10 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:10.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:10.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:10.874 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:10.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:10.921 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:11.002 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:34:11.003 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-58 2023-09-29T22:34:11.017 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 27 2023-09-29T22:34:11.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:11.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:11.108 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:11.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:11.237 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:11.237 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:34:11.237 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000058", 2023-09-29T22:34:11.237 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000026", 2023-09-29T22:34:11.237 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-09-29T22:34:11.237 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000046", 2023-09-29T22:34:11.238 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000050", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000032", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000039", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-09-29T22:34:11.239 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000040", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000045", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000033", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:34:11.240 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-09-29T22:34:11.241 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-09-29T22:34:11.241 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000055", 2023-09-29T22:34:11.241 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008" 2023-09-29T22:34:11.241 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:11.241 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:11.242 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.242 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.242 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.242 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.243 DEBUG:boto:Data: 2023-09-29T22:34:11.243 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.243 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.243 DEBUG:boto:Params: {} 2023-09-29T22:34:11.243 DEBUG:boto:Token: None 2023-09-29T22:34:11.243 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.250 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf04adec3c5d6f0d-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.250 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:11.250 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:11.250 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.250 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:11.250 DEBUG:boto:Data: 2023-09-29T22:34:11.251 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.251 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.251 DEBUG:boto:Params: {} 2023-09-29T22:34:11.251 DEBUG:boto:Token: None 2023-09-29T22:34:11.251 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:11.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Smo8JiJO0nPFBHRa7vgR8nlsf0o= 2023-09-29T22:34:11.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Smo8JiJO0nPFBHRa7vgR8nlsf0o='} 2023-09-29T22:34:11.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a7cf65fd2326ab73-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.295 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.296 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.296 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.296 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.296 DEBUG:boto:Data: 2023-09-29T22:34:11.296 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.297 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.297 DEBUG:boto:Params: {} 2023-09-29T22:34:11.297 DEBUG:boto:Token: None 2023-09-29T22:34:11.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0e432a8e7e5e27c-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.306 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:11.306 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:11.306 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.306 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:11.306 DEBUG:boto:Data: 2023-09-29T22:34:11.306 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.306 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.307 DEBUG:boto:Params: {} 2023-09-29T22:34:11.307 DEBUG:boto:Token: None 2023-09-29T22:34:11.307 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:11.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gqnmrtFpLn1yX1wQbTZT71PngfY= 2023-09-29T22:34:11.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gqnmrtFpLn1yX1wQbTZT71PngfY='} 2023-09-29T22:34:11.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008863249da8678577-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.318 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.318 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.318 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.318 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.318 DEBUG:boto:Data: 2023-09-29T22:34:11.319 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.319 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.319 DEBUG:boto:Params: {} 2023-09-29T22:34:11.319 DEBUG:boto:Token: None 2023-09-29T22:34:11.319 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.324 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000703ec65111a281ca-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.324 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:11.324 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:11.324 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.324 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:11.325 DEBUG:boto:Data: 2023-09-29T22:34:11.325 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.325 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.325 DEBUG:boto:Params: {} 2023-09-29T22:34:11.325 DEBUG:boto:Token: None 2023-09-29T22:34:11.325 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:11.326 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3JLMGwkw8+RYEbyi7o9aVj6xOKg= 2023-09-29T22:34:11.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3JLMGwkw8+RYEbyi7o9aVj6xOKg='} 2023-09-29T22:34:11.370 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000088a9757db81d867f-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.371 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.371 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.371 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.371 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.371 DEBUG:boto:Data: 2023-09-29T22:34:11.371 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.372 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.372 DEBUG:boto:Params: {} 2023-09-29T22:34:11.372 DEBUG:boto:Token: None 2023-09-29T22:34:11.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.378 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a2580894f9d96dc8-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.378 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:11.378 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:11.378 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.379 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:11.379 DEBUG:boto:Data: 2023-09-29T22:34:11.379 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.379 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.379 DEBUG:boto:Params: {} 2023-09-29T22:34:11.379 DEBUG:boto:Token: None 2023-09-29T22:34:11.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:11.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UHXbthwocvr5cu++FjpsqEH6xfw= 2023-09-29T22:34:11.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UHXbthwocvr5cu++FjpsqEH6xfw='} 2023-09-29T22:34:11.392 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6fb6549e983e474-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.393 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.393 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.393 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.393 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.393 DEBUG:boto:Data: 2023-09-29T22:34:11.393 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.393 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.394 DEBUG:boto:Params: {} 2023-09-29T22:34:11.394 DEBUG:boto:Token: None 2023-09-29T22:34:11.394 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.399 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7c2ed45bdc5dfdb-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.399 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:11.399 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:11.399 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.399 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:11.399 DEBUG:boto:Data: 2023-09-29T22:34:11.400 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.400 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.400 DEBUG:boto:Params: {} 2023-09-29T22:34:11.400 DEBUG:boto:Token: None 2023-09-29T22:34:11.400 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:11.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/64MvJrLU/VSwe0eJXDvBXG66Kg= 2023-09-29T22:34:11.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/64MvJrLU/VSwe0eJXDvBXG66Kg='} 2023-09-29T22:34:11.442 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000521ddf7505fcd1a5-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.443 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.443 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.443 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.443 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.444 DEBUG:boto:Data: 2023-09-29T22:34:11.444 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.444 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.444 DEBUG:boto:Params: {} 2023-09-29T22:34:11.444 DEBUG:boto:Token: None 2023-09-29T22:34:11.444 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c51a097acc63c792-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.451 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:11.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:11.451 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.451 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:11.452 DEBUG:boto:Data: 2023-09-29T22:34:11.452 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.452 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.452 DEBUG:boto:Params: {} 2023-09-29T22:34:11.452 DEBUG:boto:Token: None 2023-09-29T22:34:11.452 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:11.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aHOYrVP0/+sPZPA6EnGcHuz6Fq8= 2023-09-29T22:34:11.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aHOYrVP0/+sPZPA6EnGcHuz6Fq8='} 2023-09-29T22:34:11.463 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cbe7e54b9b614bf7-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.463 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.463 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.463 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.464 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.464 DEBUG:boto:Data: 2023-09-29T22:34:11.464 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.464 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.464 DEBUG:boto:Params: {} 2023-09-29T22:34:11.465 DEBUG:boto:Token: None 2023-09-29T22:34:11.465 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa6da954e15c2d93-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.469 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:11.470 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:11.470 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.470 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:11.470 DEBUG:boto:Data: 2023-09-29T22:34:11.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.470 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.471 DEBUG:boto:Params: {} 2023-09-29T22:34:11.471 DEBUG:boto:Token: None 2023-09-29T22:34:11.471 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:11.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l1/a4Cf8L1btLvbCcs6uNTjoEX8= 2023-09-29T22:34:11.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l1/a4Cf8L1btLvbCcs6uNTjoEX8='} 2023-09-29T22:34:11.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f6f6f7f9c472a15-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.519 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.519 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.519 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.519 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.519 DEBUG:boto:Data: 2023-09-29T22:34:11.519 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.520 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.520 DEBUG:boto:Params: {} 2023-09-29T22:34:11.520 DEBUG:boto:Token: None 2023-09-29T22:34:11.520 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f064951bf56436cf-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.528 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:11.528 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:11.528 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.528 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:11.528 DEBUG:boto:Data: 2023-09-29T22:34:11.528 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.529 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.529 DEBUG:boto:Params: {} 2023-09-29T22:34:11.529 DEBUG:boto:Token: None 2023-09-29T22:34:11.530 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:11.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G3YtQOH8wMPEiWfKOvKmAlkp9sw= 2023-09-29T22:34:11.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G3YtQOH8wMPEiWfKOvKmAlkp9sw='} 2023-09-29T22:34:11.541 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b5f7ea63769d731a-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.541 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.541 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.541 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.541 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.541 DEBUG:boto:Data: 2023-09-29T22:34:11.542 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.542 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.542 DEBUG:boto:Params: {} 2023-09-29T22:34:11.542 DEBUG:boto:Token: None 2023-09-29T22:34:11.542 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.548 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000df8f7f290d90744d-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.548 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:11.548 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:11.549 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.549 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:11.549 DEBUG:boto:Data: 2023-09-29T22:34:11.549 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.549 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.549 DEBUG:boto:Params: {} 2023-09-29T22:34:11.549 DEBUG:boto:Token: None 2023-09-29T22:34:11.549 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:11.550 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FNlTzayjYjBzofRAJ7Lqvzl+93s= 2023-09-29T22:34:11.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FNlTzayjYjBzofRAJ7Lqvzl+93s='} 2023-09-29T22:34:11.594 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000058fff622b4f27101-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.595 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.595 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.595 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.595 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.595 DEBUG:boto:Data: 2023-09-29T22:34:11.595 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.596 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.596 DEBUG:boto:Params: {} 2023-09-29T22:34:11.596 DEBUG:boto:Token: None 2023-09-29T22:34:11.596 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.602 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000098079070d11bfb26-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.603 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:11.603 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:11.603 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.603 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:11.603 DEBUG:boto:Data: 2023-09-29T22:34:11.603 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.603 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.604 DEBUG:boto:Params: {} 2023-09-29T22:34:11.604 DEBUG:boto:Token: None 2023-09-29T22:34:11.604 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:11.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xIsh7Ir9zSvw4+wloRxbdM79G28= 2023-09-29T22:34:11.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xIsh7Ir9zSvw4+wloRxbdM79G28='} 2023-09-29T22:34:11.617 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000493188f9016a54a6-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.617 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.617 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.618 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.618 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.618 DEBUG:boto:Data: 2023-09-29T22:34:11.618 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.618 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.618 DEBUG:boto:Params: {} 2023-09-29T22:34:11.618 DEBUG:boto:Token: None 2023-09-29T22:34:11.619 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.623 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d1a73a8bad4fb65c-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.624 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:11.624 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:11.624 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.624 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:11.624 DEBUG:boto:Data: 2023-09-29T22:34:11.624 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.624 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.624 DEBUG:boto:Params: {} 2023-09-29T22:34:11.625 DEBUG:boto:Token: None 2023-09-29T22:34:11.625 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:11.625 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z+Mtm7D4Uz7GTp0ghGO03Yusev4= 2023-09-29T22:34:11.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z+Mtm7D4Uz7GTp0ghGO03Yusev4='} 2023-09-29T22:34:11.671 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000432e4b2c1bdb90d-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.671 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.671 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.671 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.671 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.671 DEBUG:boto:Data: 2023-09-29T22:34:11.671 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.672 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.672 DEBUG:boto:Params: {} 2023-09-29T22:34:11.672 DEBUG:boto:Token: None 2023-09-29T22:34:11.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.678 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000030c4d644d28bf444-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.679 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:11.679 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:11.679 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.679 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:11.679 DEBUG:boto:Data: 2023-09-29T22:34:11.679 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.679 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.679 DEBUG:boto:Params: {} 2023-09-29T22:34:11.680 DEBUG:boto:Token: None 2023-09-29T22:34:11.680 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:11.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:95Shk+teoz/Gw/eJTJjkh/3GjdI= 2023-09-29T22:34:11.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:95Shk+teoz/Gw/eJTJjkh/3GjdI='} 2023-09-29T22:34:11.693 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007d9a00ba9ce5b022-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.694 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.694 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.694 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.695 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.695 DEBUG:boto:Data: 2023-09-29T22:34:11.695 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.695 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.695 DEBUG:boto:Params: {} 2023-09-29T22:34:11.695 DEBUG:boto:Token: None 2023-09-29T22:34:11.696 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c2786dd3d726e2f-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.701 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:11.701 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:11.701 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.701 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:11.701 DEBUG:boto:Data: 2023-09-29T22:34:11.702 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.702 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.702 DEBUG:boto:Params: {} 2023-09-29T22:34:11.702 DEBUG:boto:Token: None 2023-09-29T22:34:11.702 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:11.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a3JkfRtsWsFzFC31t/8GeYBitYk= 2023-09-29T22:34:11.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a3JkfRtsWsFzFC31t/8GeYBitYk='} 2023-09-29T22:34:11.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fae63acc56cb5ee-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.747 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.747 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.747 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.747 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.747 DEBUG:boto:Data: 2023-09-29T22:34:11.747 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.748 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.748 DEBUG:boto:Params: {} 2023-09-29T22:34:11.748 DEBUG:boto:Token: None 2023-09-29T22:34:11.748 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.754 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fceb484b02051c3e-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.754 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:11.754 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:11.755 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.755 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:11.755 DEBUG:boto:Data: 2023-09-29T22:34:11.755 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.755 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.755 DEBUG:boto:Params: {} 2023-09-29T22:34:11.755 DEBUG:boto:Token: None 2023-09-29T22:34:11.756 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:11.756 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn5+luuVszJb+EXGn+5X3Fplt/E= 2023-09-29T22:34:11.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn5+luuVszJb+EXGn+5X3Fplt/E='} 2023-09-29T22:34:11.767 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004986d981c24975f2-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.767 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.767 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.767 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.767 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.767 DEBUG:boto:Data: 2023-09-29T22:34:11.767 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.768 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.768 DEBUG:boto:Params: {} 2023-09-29T22:34:11.768 DEBUG:boto:Token: None 2023-09-29T22:34:11.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.773 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f326902d04f07f8d-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.773 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:11.773 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:11.774 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.774 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:11.774 DEBUG:boto:Data: 2023-09-29T22:34:11.774 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.774 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.774 DEBUG:boto:Params: {} 2023-09-29T22:34:11.774 DEBUG:boto:Token: None 2023-09-29T22:34:11.774 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:11.775 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lrQBEXUezEHmSDXp8p0ZsTCooc0= 2023-09-29T22:34:11.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lrQBEXUezEHmSDXp8p0ZsTCooc0='} 2023-09-29T22:34:11.819 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002eb2efaf106088c8-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.820 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.820 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.820 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.820 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.820 DEBUG:boto:Data: 2023-09-29T22:34:11.820 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.820 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.821 DEBUG:boto:Params: {} 2023-09-29T22:34:11.821 DEBUG:boto:Token: None 2023-09-29T22:34:11.821 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000149aae139bcaa2eb-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.827 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:11.828 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:11.828 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.828 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:11.828 DEBUG:boto:Data: 2023-09-29T22:34:11.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.828 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.828 DEBUG:boto:Params: {} 2023-09-29T22:34:11.828 DEBUG:boto:Token: None 2023-09-29T22:34:11.829 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:11.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VyNQ9Lo/xedYZhS2qyLbrZ4MkKw= 2023-09-29T22:34:11.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VyNQ9Lo/xedYZhS2qyLbrZ4MkKw='} 2023-09-29T22:34:11.841 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000184ccc2a52ad1e5c-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.842 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.842 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.842 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.842 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.842 DEBUG:boto:Data: 2023-09-29T22:34:11.842 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.843 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.843 DEBUG:boto:Params: {} 2023-09-29T22:34:11.843 DEBUG:boto:Token: None 2023-09-29T22:34:11.843 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.848 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000052dbcb51ec95174a-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.849 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:11.849 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:11.849 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.849 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:11.849 DEBUG:boto:Data: 2023-09-29T22:34:11.849 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.850 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.850 DEBUG:boto:Params: {} 2023-09-29T22:34:11.850 DEBUG:boto:Token: None 2023-09-29T22:34:11.850 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:11.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QgZdyzWwAlr2qWIyYLOM+lXsOPo= 2023-09-29T22:34:11.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QgZdyzWwAlr2qWIyYLOM+lXsOPo='} 2023-09-29T22:34:11.899 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ad209397b6fd6c0-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.899 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.899 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.899 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.900 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.900 DEBUG:boto:Data: 2023-09-29T22:34:11.900 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.900 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.901 DEBUG:boto:Params: {} 2023-09-29T22:34:11.901 DEBUG:boto:Token: None 2023-09-29T22:34:11.901 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.907 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb3f6b23942c3842-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.908 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:11.908 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:11.908 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.908 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:11.908 DEBUG:boto:Data: 2023-09-29T22:34:11.908 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.909 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.909 DEBUG:boto:Params: {} 2023-09-29T22:34:11.909 DEBUG:boto:Token: None 2023-09-29T22:34:11.909 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:11.909 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+ty+JMAI5YS/znVxLFaTcMX9wOg= 2023-09-29T22:34:11.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+ty+JMAI5YS/znVxLFaTcMX9wOg='} 2023-09-29T22:34:11.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035fd9289dd49b491-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.923 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.923 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.923 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.923 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.923 DEBUG:boto:Data: 2023-09-29T22:34:11.923 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.924 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.924 DEBUG:boto:Params: {} 2023-09-29T22:34:11.924 DEBUG:boto:Token: None 2023-09-29T22:34:11.924 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d5210931834467e9-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.930 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:11.930 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:11.930 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.930 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:11.930 DEBUG:boto:Data: 2023-09-29T22:34:11.930 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.930 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.931 DEBUG:boto:Params: {} 2023-09-29T22:34:11.931 DEBUG:boto:Token: None 2023-09-29T22:34:11.931 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:11.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZK1LnZRy/9Av/Iakj4dhRlPqIg= 2023-09-29T22:34:11.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZK1LnZRy/9Av/Iakj4dhRlPqIg='} 2023-09-29T22:34:11.978 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002dfea8891bdb8513-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.979 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:11.979 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:11.979 DEBUG:boto:Method: HEAD 2023-09-29T22:34:11.979 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:11.979 DEBUG:boto:Data: 2023-09-29T22:34:11.979 DEBUG:boto:Headers: {} 2023-09-29T22:34:11.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.980 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.980 DEBUG:boto:Params: {} 2023-09-29T22:34:11.980 DEBUG:boto:Token: None 2023-09-29T22:34:11.980 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/ 2023-09-29T22:34:11.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4= 2023-09-29T22:34:11.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhciuH2TxOGqwZB54ZKoWyki9l4=', 'Content-Length': '0'} 2023-09-29T22:34:11.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e80d5b8ce9b95deb-00651750e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:11.987 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:11.987 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:11.987 DEBUG:boto:Method: PUT 2023-09-29T22:34:11.987 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:11.987 DEBUG:boto:Data: 2023-09-29T22:34:11.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:11.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:11.987 DEBUG:boto:Port: 8000 2023-09-29T22:34:11.987 DEBUG:boto:Params: {} 2023-09-29T22:34:11.988 DEBUG:boto:Token: None 2023-09-29T22:34:11.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:11 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:11.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1tzAD8pEbIhCbA/ADZuBSlM2xzI= 2023-09-29T22:34:11.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1tzAD8pEbIhCbA/ADZuBSlM2xzI='} 2023-09-29T22:34:12.000 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b932127127ec3476-00651750e3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.000 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.000 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.001 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.001 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.001 DEBUG:boto:Data: 2023-09-29T22:34:12.001 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.001 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.001 DEBUG:boto:Params: {} 2023-09-29T22:34:12.001 DEBUG:boto:Token: None 2023-09-29T22:34:12.002 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d9d4cbc2dbd146d-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.007 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:12.007 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:12.007 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.007 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:12.007 DEBUG:boto:Data: 2023-09-29T22:34:12.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.008 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.008 DEBUG:boto:Params: {} 2023-09-29T22:34:12.008 DEBUG:boto:Token: None 2023-09-29T22:34:12.008 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:12.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9W9LV86haFxQdpqPpMXlVK3JQ7Y= 2023-09-29T22:34:12.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9W9LV86haFxQdpqPpMXlVK3JQ7Y='} 2023-09-29T22:34:12.050 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a0bae56c197a8fc-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:12.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:12.051 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:12.101 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:12.102 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:12.180 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:34:12.180 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-58 2023-09-29T22:34:12.196 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 34 2023-09-29T22:34:12.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:12.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:12.300 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 29 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:12.348 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:12.427 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:34:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-09-29T22:34:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000059", 2023-09-29T22:34:12.427 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000027", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000049", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000017", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000015", 2023-09-29T22:34:12.428 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000031", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000022", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000020", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000037", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000042", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-09-29T22:34:12.429 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000025", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021" 2023-09-29T22:34:12.430 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:12.431 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:12.431 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.432 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.432 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.432 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.432 DEBUG:boto:Data: 2023-09-29T22:34:12.432 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.432 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.432 DEBUG:boto:Params: {} 2023-09-29T22:34:12.433 DEBUG:boto:Token: None 2023-09-29T22:34:12.433 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.438 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000202f3c9979aa7388-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.439 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:12.439 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:12.439 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.439 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:12.439 DEBUG:boto:Data: 2023-09-29T22:34:12.439 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.440 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.440 DEBUG:boto:Params: {} 2023-09-29T22:34:12.440 DEBUG:boto:Token: None 2023-09-29T22:34:12.440 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:12.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDOPVoBNNXSXS2hli1Emj1r8eQI= 2023-09-29T22:34:12.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDOPVoBNNXSXS2hli1Emj1r8eQI='} 2023-09-29T22:34:12.450 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f97ea3906c520b52-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.451 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.451 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.451 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.451 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.451 DEBUG:boto:Data: 2023-09-29T22:34:12.451 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.451 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.451 DEBUG:boto:Params: {} 2023-09-29T22:34:12.452 DEBUG:boto:Token: None 2023-09-29T22:34:12.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.457 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000049ae46b87a922d01-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.457 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:12.457 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:12.457 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.457 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:12.458 DEBUG:boto:Data: 2023-09-29T22:34:12.458 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.458 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.458 DEBUG:boto:Params: {} 2023-09-29T22:34:12.458 DEBUG:boto:Token: None 2023-09-29T22:34:12.458 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:12.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hwipGQ1quuvAQk3qKuCu0pyq0x8= 2023-09-29T22:34:12.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hwipGQ1quuvAQk3qKuCu0pyq0x8='} 2023-09-29T22:34:12.503 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d3491fdafe73af84-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.503 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.503 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.503 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.503 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.503 DEBUG:boto:Data: 2023-09-29T22:34:12.503 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.504 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.504 DEBUG:boto:Params: {} 2023-09-29T22:34:12.504 DEBUG:boto:Token: None 2023-09-29T22:34:12.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.510 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e7ea3a13e576286-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.511 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:12.511 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:12.511 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.511 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:12.511 DEBUG:boto:Data: 2023-09-29T22:34:12.511 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.511 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.512 DEBUG:boto:Params: {} 2023-09-29T22:34:12.512 DEBUG:boto:Token: None 2023-09-29T22:34:12.512 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:12.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ClQN5HviT3GkSIT4aY/re8AVa/4= 2023-09-29T22:34:12.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ClQN5HviT3GkSIT4aY/re8AVa/4='} 2023-09-29T22:34:12.524 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082c86683da653e33-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.525 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.525 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.525 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.525 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.525 DEBUG:boto:Data: 2023-09-29T22:34:12.525 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.526 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.526 DEBUG:boto:Params: {} 2023-09-29T22:34:12.526 DEBUG:boto:Token: None 2023-09-29T22:34:12.527 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.532 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b22fb6df6c61ff11-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.533 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:12.533 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:12.533 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.534 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:12.534 DEBUG:boto:Data: 2023-09-29T22:34:12.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.534 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.535 DEBUG:boto:Params: {} 2023-09-29T22:34:12.535 DEBUG:boto:Token: None 2023-09-29T22:34:12.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:12.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B974fQvPWMg7E3+h7cXsZQ/jCsY= 2023-09-29T22:34:12.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B974fQvPWMg7E3+h7cXsZQ/jCsY='} 2023-09-29T22:34:12.583 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000089ba39cc9b6ab36d-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.583 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.583 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.584 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.584 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.584 DEBUG:boto:Data: 2023-09-29T22:34:12.584 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.585 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.585 DEBUG:boto:Params: {} 2023-09-29T22:34:12.585 DEBUG:boto:Token: None 2023-09-29T22:34:12.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.593 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000044077f978076b4ab-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.593 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:12.593 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:12.593 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.594 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:12.594 DEBUG:boto:Data: 2023-09-29T22:34:12.594 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.594 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.594 DEBUG:boto:Params: {} 2023-09-29T22:34:12.594 DEBUG:boto:Token: None 2023-09-29T22:34:12.594 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:12.594 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ATELPFac9vbhuXuHEfdEb/xpNzw= 2023-09-29T22:34:12.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ATELPFac9vbhuXuHEfdEb/xpNzw='} 2023-09-29T22:34:12.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000096b85281e382c5b8-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.607 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.608 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.608 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.608 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.608 DEBUG:boto:Data: 2023-09-29T22:34:12.608 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.609 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.609 DEBUG:boto:Params: {} 2023-09-29T22:34:12.609 DEBUG:boto:Token: None 2023-09-29T22:34:12.610 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.610 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.616 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000bba7e3f52834a89-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.616 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:12.616 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:12.616 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.616 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:12.616 DEBUG:boto:Data: 2023-09-29T22:34:12.617 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.617 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.617 DEBUG:boto:Params: {} 2023-09-29T22:34:12.617 DEBUG:boto:Token: None 2023-09-29T22:34:12.617 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:12.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/IiDhxuKx541ZkXeD6VVlhQqpto= 2023-09-29T22:34:12.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/IiDhxuKx541ZkXeD6VVlhQqpto='} 2023-09-29T22:34:12.667 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004161bbb2694da11c-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.667 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.667 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.667 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.668 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.668 DEBUG:boto:Data: 2023-09-29T22:34:12.668 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.668 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.669 DEBUG:boto:Params: {} 2023-09-29T22:34:12.669 DEBUG:boto:Token: None 2023-09-29T22:34:12.669 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.677 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004191d951165e8e44-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.677 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:12.678 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:12.678 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.678 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:12.678 DEBUG:boto:Data: 2023-09-29T22:34:12.678 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.678 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.678 DEBUG:boto:Params: {} 2023-09-29T22:34:12.678 DEBUG:boto:Token: None 2023-09-29T22:34:12.679 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:12.679 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WOgTSne9FQwuVHsmq3dmcUZtVig= 2023-09-29T22:34:12.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WOgTSne9FQwuVHsmq3dmcUZtVig='} 2023-09-29T22:34:12.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004d6f0818b33e369a-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.690 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.690 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.690 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.690 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.691 DEBUG:boto:Data: 2023-09-29T22:34:12.691 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.691 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.691 DEBUG:boto:Params: {} 2023-09-29T22:34:12.691 DEBUG:boto:Token: None 2023-09-29T22:34:12.691 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.696 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2e239bc98f37e60-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.696 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:12.697 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:12.697 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.697 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:12.697 DEBUG:boto:Data: 2023-09-29T22:34:12.697 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.697 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.697 DEBUG:boto:Params: {} 2023-09-29T22:34:12.697 DEBUG:boto:Token: None 2023-09-29T22:34:12.698 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:12.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kvNPlzxJ+1wkPTBswdsZuSoMkj4= 2023-09-29T22:34:12.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kvNPlzxJ+1wkPTBswdsZuSoMkj4='} 2023-09-29T22:34:12.747 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c6a44c717ea56f7-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.747 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.747 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.748 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.748 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.748 DEBUG:boto:Data: 2023-09-29T22:34:12.748 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.749 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.749 DEBUG:boto:Params: {} 2023-09-29T22:34:12.749 DEBUG:boto:Token: None 2023-09-29T22:34:12.749 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.750 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e394d278f0bc3be7-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.757 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:12.757 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:12.757 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.757 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:12.757 DEBUG:boto:Data: 2023-09-29T22:34:12.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.757 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.758 DEBUG:boto:Params: {} 2023-09-29T22:34:12.758 DEBUG:boto:Token: None 2023-09-29T22:34:12.758 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:12.758 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:estlSdUT/9bmupuF+WDEW4Z9DTk= 2023-09-29T22:34:12.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:estlSdUT/9bmupuF+WDEW4Z9DTk='} 2023-09-29T22:34:12.768 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cf49baae25da6568-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.769 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.769 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.769 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.769 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.769 DEBUG:boto:Data: 2023-09-29T22:34:12.769 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.769 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.770 DEBUG:boto:Params: {} 2023-09-29T22:34:12.770 DEBUG:boto:Token: None 2023-09-29T22:34:12.770 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.770 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.775 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045586faa4cb592dd-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.776 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:12.776 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:12.776 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.776 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:12.776 DEBUG:boto:Data: 2023-09-29T22:34:12.776 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.776 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.777 DEBUG:boto:Params: {} 2023-09-29T22:34:12.777 DEBUG:boto:Token: None 2023-09-29T22:34:12.777 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:12.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4TQLFAbb0jgVhb7W1djiPUh19QQ= 2023-09-29T22:34:12.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4TQLFAbb0jgVhb7W1djiPUh19QQ='} 2023-09-29T22:34:12.823 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aecb49d80d8fa0eb-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.823 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.823 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.823 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.824 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.824 DEBUG:boto:Data: 2023-09-29T22:34:12.824 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.824 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.824 DEBUG:boto:Params: {} 2023-09-29T22:34:12.824 DEBUG:boto:Token: None 2023-09-29T22:34:12.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.830 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b052a7d0c3c73777-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.831 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:12.831 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:12.831 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.831 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:12.832 DEBUG:boto:Data: 2023-09-29T22:34:12.832 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.832 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.832 DEBUG:boto:Params: {} 2023-09-29T22:34:12.832 DEBUG:boto:Token: None 2023-09-29T22:34:12.833 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:12.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EkAoZTq8jOuEMOjgnVpzQDuDY0k= 2023-09-29T22:34:12.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EkAoZTq8jOuEMOjgnVpzQDuDY0k='} 2023-09-29T22:34:12.845 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b8bcc432fef99a6c-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.846 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.846 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.846 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.846 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.846 DEBUG:boto:Data: 2023-09-29T22:34:12.847 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.847 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.847 DEBUG:boto:Params: {} 2023-09-29T22:34:12.847 DEBUG:boto:Token: None 2023-09-29T22:34:12.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '36'), ('X-RGW-Bytes-Used', '108'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000af57c62719c7fd51-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.854 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:12.854 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:12.855 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.855 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:12.855 DEBUG:boto:Data: 2023-09-29T22:34:12.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.856 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.856 DEBUG:boto:Params: {} 2023-09-29T22:34:12.856 DEBUG:boto:Token: None 2023-09-29T22:34:12.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:12.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fcSoxc2VcGWJu9/CvCD9y2VvYJ4= 2023-09-29T22:34:12.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fcSoxc2VcGWJu9/CvCD9y2VvYJ4='} 2023-09-29T22:34:12.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2cc4a92993a409e-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.903 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.903 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.904 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.904 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.904 DEBUG:boto:Data: 2023-09-29T22:34:12.904 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.905 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.905 DEBUG:boto:Params: {} 2023-09-29T22:34:12.905 DEBUG:boto:Token: None 2023-09-29T22:34:12.905 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.913 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '37'), ('X-RGW-Bytes-Used', '111'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000307e18ca8b237f12-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.913 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:12.914 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:12.914 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.914 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:12.914 DEBUG:boto:Data: 2023-09-29T22:34:12.914 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.914 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.914 DEBUG:boto:Params: {} 2023-09-29T22:34:12.914 DEBUG:boto:Token: None 2023-09-29T22:34:12.915 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:12.915 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jVzGefxRo4bJY4i50j9lkVTD5ZA= 2023-09-29T22:34:12.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jVzGefxRo4bJY4i50j9lkVTD5ZA='} 2023-09-29T22:34:12.924 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e7eb14b85591e535-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.925 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.925 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.925 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.925 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.926 DEBUG:boto:Data: 2023-09-29T22:34:12.926 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.926 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.927 DEBUG:boto:Params: {} 2023-09-29T22:34:12.927 DEBUG:boto:Token: None 2023-09-29T22:34:12.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.934 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000040794c10b535e0-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.935 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:12.935 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:12.935 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.935 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:12.935 DEBUG:boto:Data: 2023-09-29T22:34:12.935 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.936 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.936 DEBUG:boto:Params: {} 2023-09-29T22:34:12.936 DEBUG:boto:Token: None 2023-09-29T22:34:12.936 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:12.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:crzMiyTi7zjgT2ll1luQmhDlNe8= 2023-09-29T22:34:12.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:crzMiyTi7zjgT2ll1luQmhDlNe8='} 2023-09-29T22:34:12.983 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000048cca26ec246d0b5-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.983 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:12.984 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:12.984 DEBUG:boto:Method: HEAD 2023-09-29T22:34:12.984 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:12.984 DEBUG:boto:Data: 2023-09-29T22:34:12.984 DEBUG:boto:Headers: {} 2023-09-29T22:34:12.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.984 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.984 DEBUG:boto:Params: {} 2023-09-29T22:34:12.985 DEBUG:boto:Token: None 2023-09-29T22:34:12.985 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/ 2023-09-29T22:34:12.985 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk= 2023-09-29T22:34:12.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHclt5JAhVHto9eZAPIT9iY01qk=', 'Content-Length': '0'} 2023-09-29T22:34:12.992 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa308182844df2fb-00651750e4-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:12.992 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:12.992 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:12.992 DEBUG:boto:Method: PUT 2023-09-29T22:34:12.992 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:12.992 DEBUG:boto:Data: 2023-09-29T22:34:12.993 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:12.993 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:12.993 DEBUG:boto:Port: 8000 2023-09-29T22:34:12.993 DEBUG:boto:Params: {} 2023-09-29T22:34:12.993 DEBUG:boto:Token: None 2023-09-29T22:34:12.993 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:12 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:12.993 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YX02y1j2LsZHqM79NSsFQVCCPpc= 2023-09-29T22:34:12.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YX02y1j2LsZHqM79NSsFQVCCPpc='} 2023-09-29T22:34:13.005 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000631e8554450e3431-00651750e4-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.005 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.005 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.006 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.006 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.006 DEBUG:boto:Data: 2023-09-29T22:34:13.006 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.006 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.006 DEBUG:boto:Params: {} 2023-09-29T22:34:13.006 DEBUG:boto:Token: None 2023-09-29T22:34:13.007 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004a7b0fb0009aebde-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.013 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:13.013 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:13.013 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.013 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:13.013 DEBUG:boto:Data: 2023-09-29T22:34:13.013 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.013 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.014 DEBUG:boto:Params: {} 2023-09-29T22:34:13.014 DEBUG:boto:Token: None 2023-09-29T22:34:13.014 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:13.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VuM9Xh/y4C/9FsPsVA/vyQ9rIpg= 2023-09-29T22:34:13.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VuM9Xh/y4C/9FsPsVA/vyQ9rIpg='} 2023-09-29T22:34:13.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002e0e9577c6b40d94-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.063 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.063 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.063 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.063 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.063 DEBUG:boto:Data: 2023-09-29T22:34:13.064 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.064 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.064 DEBUG:boto:Params: {} 2023-09-29T22:34:13.064 DEBUG:boto:Token: None 2023-09-29T22:34:13.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.071 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '38'), ('X-RGW-Bytes-Used', '114'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088941ca14579012c-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.071 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.072 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.072 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.072 DEBUG:boto:Data: 2023-09-29T22:34:13.072 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.072 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.072 DEBUG:boto:Params: {} 2023-09-29T22:34:13.072 DEBUG:boto:Token: None 2023-09-29T22:34:13.073 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QTAtQQ/uEYFPzdW8RwjwyTvND4A= 2023-09-29T22:34:13.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QTAtQQ/uEYFPzdW8RwjwyTvND4A='} 2023-09-29T22:34:13.083 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fb45422912e3eeb9-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.083 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.083 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.083 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.083 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.083 DEBUG:boto:Data: 2023-09-29T22:34:13.083 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.084 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.084 DEBUG:boto:Params: {} 2023-09-29T22:34:13.084 DEBUG:boto:Token: None 2023-09-29T22:34:13.084 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.090 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '39'), ('X-RGW-Bytes-Used', '117'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009885a85dfbcd2d29-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.090 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:13.091 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:13.091 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.091 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:13.091 DEBUG:boto:Data: 2023-09-29T22:34:13.091 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.091 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.091 DEBUG:boto:Params: {} 2023-09-29T22:34:13.092 DEBUG:boto:Token: None 2023-09-29T22:34:13.092 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:13.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kkZ2x69fmcnT3HMc4IiMt+j9Rrs= 2023-09-29T22:34:13.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kkZ2x69fmcnT3HMc4IiMt+j9Rrs='} 2023-09-29T22:34:13.134 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000092e7a935028f6a8c-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.135 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.135 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.135 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.135 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.135 DEBUG:boto:Data: 2023-09-29T22:34:13.135 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.136 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.136 DEBUG:boto:Params: {} 2023-09-29T22:34:13.136 DEBUG:boto:Token: None 2023-09-29T22:34:13.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000356dcba6a0f40d6-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.143 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:13.143 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:13.143 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.144 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:13.144 DEBUG:boto:Data: 2023-09-29T22:34:13.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.144 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.144 DEBUG:boto:Params: {} 2023-09-29T22:34:13.144 DEBUG:boto:Token: None 2023-09-29T22:34:13.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:13.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Js1WSbLbFnV/MyDxIoYT8r2hJEA= 2023-09-29T22:34:13.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Js1WSbLbFnV/MyDxIoYT8r2hJEA='} 2023-09-29T22:34:13.156 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000453ea6d0937a3764-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.156 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.156 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.156 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.156 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.157 DEBUG:boto:Data: 2023-09-29T22:34:13.157 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.157 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.157 DEBUG:boto:Params: {} 2023-09-29T22:34:13.157 DEBUG:boto:Token: None 2023-09-29T22:34:13.157 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '40'), ('X-RGW-Bytes-Used', '120'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048bd3574d90d6261-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.163 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:13.163 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:13.163 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.163 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:13.163 DEBUG:boto:Data: 2023-09-29T22:34:13.163 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.163 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.164 DEBUG:boto:Params: {} 2023-09-29T22:34:13.164 DEBUG:boto:Token: None 2023-09-29T22:34:13.164 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:13.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fKfmEsUXN2lhpQronRVfORL8ODw= 2023-09-29T22:34:13.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fKfmEsUXN2lhpQronRVfORL8ODw='} 2023-09-29T22:34:13.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d03d683b9e19b82-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.211 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.211 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.211 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.211 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.211 DEBUG:boto:Data: 2023-09-29T22:34:13.211 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.212 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.212 DEBUG:boto:Params: {} 2023-09-29T22:34:13.212 DEBUG:boto:Token: None 2023-09-29T22:34:13.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.219 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000014e2e2b7239984f8-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.219 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:13.219 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:13.219 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.219 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:13.220 DEBUG:boto:Data: 2023-09-29T22:34:13.220 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.220 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.220 DEBUG:boto:Params: {} 2023-09-29T22:34:13.220 DEBUG:boto:Token: None 2023-09-29T22:34:13.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:13.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lqJ1BToF00myDJrk++Nb/Afsc3Q= 2023-09-29T22:34:13.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lqJ1BToF00myDJrk++Nb/Afsc3Q='} 2023-09-29T22:34:13.233 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034e0dcb7d9f2aeb9-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.233 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.234 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.234 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.234 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.234 DEBUG:boto:Data: 2023-09-29T22:34:13.234 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.234 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.234 DEBUG:boto:Params: {} 2023-09-29T22:34:13.234 DEBUG:boto:Token: None 2023-09-29T22:34:13.235 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.235 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.240 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '41'), ('X-RGW-Bytes-Used', '123'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d34b9de6c5ac42ff-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.240 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:13.241 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:13.241 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.241 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:13.241 DEBUG:boto:Data: 2023-09-29T22:34:13.241 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.241 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.241 DEBUG:boto:Params: {} 2023-09-29T22:34:13.242 DEBUG:boto:Token: None 2023-09-29T22:34:13.242 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:13.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JXvXn0PM/3VosyDGHYrXYFS0Xdk= 2023-09-29T22:34:13.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JXvXn0PM/3VosyDGHYrXYFS0Xdk='} 2023-09-29T22:34:13.286 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005b0c8ceb3dbea290-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.287 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.287 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.287 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.287 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.287 DEBUG:boto:Data: 2023-09-29T22:34:13.287 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.288 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.288 DEBUG:boto:Params: {} 2023-09-29T22:34:13.288 DEBUG:boto:Token: None 2023-09-29T22:34:13.288 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.295 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7a8b1d549c72653-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.295 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:13.296 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:13.296 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.296 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:13.296 DEBUG:boto:Data: 2023-09-29T22:34:13.296 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.296 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.296 DEBUG:boto:Params: {} 2023-09-29T22:34:13.297 DEBUG:boto:Token: None 2023-09-29T22:34:13.297 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:13.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fRCaDHBwDi9fmX17N1JlR7qSh5Y= 2023-09-29T22:34:13.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fRCaDHBwDi9fmX17N1JlR7qSh5Y='} 2023-09-29T22:34:13.309 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000063586179716ec61e-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.309 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.309 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.309 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.309 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.309 DEBUG:boto:Data: 2023-09-29T22:34:13.309 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.310 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.310 DEBUG:boto:Params: {} 2023-09-29T22:34:13.310 DEBUG:boto:Token: None 2023-09-29T22:34:13.310 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.310 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.315 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b5191455aeaaa9cf-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.316 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:13.316 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:13.316 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.316 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:13.316 DEBUG:boto:Data: 2023-09-29T22:34:13.316 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.317 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.317 DEBUG:boto:Params: {} 2023-09-29T22:34:13.317 DEBUG:boto:Token: None 2023-09-29T22:34:13.317 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:13.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lg9BfHSQ6MM+Ssi0WvS83ZVbcZg= 2023-09-29T22:34:13.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lg9BfHSQ6MM+Ssi0WvS83ZVbcZg='} 2023-09-29T22:34:13.363 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000086431b19a6463721-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.363 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.363 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.363 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.363 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.363 DEBUG:boto:Data: 2023-09-29T22:34:13.363 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.364 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.364 DEBUG:boto:Params: {} 2023-09-29T22:34:13.364 DEBUG:boto:Token: None 2023-09-29T22:34:13.364 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.371 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0b5058b162c6d36-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.371 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:13.371 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:13.371 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.371 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:13.372 DEBUG:boto:Data: 2023-09-29T22:34:13.372 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.372 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.372 DEBUG:boto:Params: {} 2023-09-29T22:34:13.372 DEBUG:boto:Token: None 2023-09-29T22:34:13.372 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:13.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIpKu1H5GaEWlnY3/g/QPaKGK84= 2023-09-29T22:34:13.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIpKu1H5GaEWlnY3/g/QPaKGK84='} 2023-09-29T22:34:13.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac765853e96b425f-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.385 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.385 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.385 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.385 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.385 DEBUG:boto:Data: 2023-09-29T22:34:13.385 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.386 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.386 DEBUG:boto:Params: {} 2023-09-29T22:34:13.386 DEBUG:boto:Token: None 2023-09-29T22:34:13.386 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.391 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000393d71e8fd0de65c-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.392 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:13.392 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:13.392 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.392 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:13.392 DEBUG:boto:Data: 2023-09-29T22:34:13.392 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.392 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.393 DEBUG:boto:Params: {} 2023-09-29T22:34:13.393 DEBUG:boto:Token: None 2023-09-29T22:34:13.393 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:13.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QV7RO17QT9HQQA6M1Fw7hjw6MWk= 2023-09-29T22:34:13.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QV7RO17QT9HQQA6M1Fw7hjw6MWk='} 2023-09-29T22:34:13.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fd4dcb56078d9ed1-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.439 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.439 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.439 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.439 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.439 DEBUG:boto:Data: 2023-09-29T22:34:13.440 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.440 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.440 DEBUG:boto:Params: {} 2023-09-29T22:34:13.440 DEBUG:boto:Token: None 2023-09-29T22:34:13.440 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003da1b65f91129a44-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:13.447 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:13.447 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.447 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:13.448 DEBUG:boto:Data: 2023-09-29T22:34:13.448 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.448 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.448 DEBUG:boto:Params: {} 2023-09-29T22:34:13.448 DEBUG:boto:Token: None 2023-09-29T22:34:13.448 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:13.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IRNYw3lIdR6mTVfEc8JvEMSepOI= 2023-09-29T22:34:13.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IRNYw3lIdR6mTVfEc8JvEMSepOI='} 2023-09-29T22:34:13.460 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da62f3d6c364f1a5-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:13.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:13.461 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:13.507 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:13.508 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:13.588 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:34:13.589 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-58 2023-09-29T22:34:13.611 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 42 2023-09-29T22:34:13.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:13.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:13.728 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 31 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:13.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:13.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:13.847 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:13.847 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:34:13.847 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-09-29T22:34:13.847 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000467", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000056", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000023", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000317", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000022", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:34:13.848 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000271", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000018", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000017", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000162", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-09-29T22:34:13.849 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000423", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000283", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000061", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000050", 2023-09-29T22:34:13.850 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000029", 2023-09-29T22:34:13.851 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000133" 2023-09-29T22:34:13.851 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:13.851 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:13.852 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.852 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.852 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.852 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.852 DEBUG:boto:Data: 2023-09-29T22:34:13.853 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.853 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.853 DEBUG:boto:Params: {} 2023-09-29T22:34:13.853 DEBUG:boto:Token: None 2023-09-29T22:34:13.853 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.853 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '42'), ('X-RGW-Bytes-Used', '126'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037ba019bca98078d-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.859 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:13.859 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:13.859 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.859 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:13.860 DEBUG:boto:Data: 2023-09-29T22:34:13.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.860 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.860 DEBUG:boto:Params: {} 2023-09-29T22:34:13.860 DEBUG:boto:Token: None 2023-09-29T22:34:13.860 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:13.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F3pIDP5TzNIc5+pD2iF0UaSYENo= 2023-09-29T22:34:13.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F3pIDP5TzNIc5+pD2iF0UaSYENo='} 2023-09-29T22:34:13.903 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df822eca39969bc9-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.903 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.903 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.903 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.903 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.904 DEBUG:boto:Data: 2023-09-29T22:34:13.904 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.904 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.904 DEBUG:boto:Params: {} 2023-09-29T22:34:13.904 DEBUG:boto:Token: None 2023-09-29T22:34:13.904 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.911 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b9910b5edcc1b1fe-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.911 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:13.911 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:13.912 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.912 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:13.912 DEBUG:boto:Data: 2023-09-29T22:34:13.912 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.912 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.912 DEBUG:boto:Params: {} 2023-09-29T22:34:13.913 DEBUG:boto:Token: None 2023-09-29T22:34:13.913 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:13.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wbtpn+MsFhMpt452zkyqc8VbxZA= 2023-09-29T22:34:13.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wbtpn+MsFhMpt452zkyqc8VbxZA='} 2023-09-29T22:34:13.925 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e5da376b1b99362-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.926 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.926 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.926 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.926 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.926 DEBUG:boto:Data: 2023-09-29T22:34:13.926 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.926 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.927 DEBUG:boto:Params: {} 2023-09-29T22:34:13.927 DEBUG:boto:Token: None 2023-09-29T22:34:13.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.932 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dcbc457225f91d81-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.933 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.933 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.933 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.933 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.933 DEBUG:boto:Data: 2023-09-29T22:34:13.933 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.933 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.934 DEBUG:boto:Params: {} 2023-09-29T22:34:13.934 DEBUG:boto:Token: None 2023-09-29T22:34:13.934 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:13.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QTAtQQ/uEYFPzdW8RwjwyTvND4A= 2023-09-29T22:34:13.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QTAtQQ/uEYFPzdW8RwjwyTvND4A='} 2023-09-29T22:34:13.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ec56a74728a97414-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.979 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.979 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.979 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.979 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.979 DEBUG:boto:Data: 2023-09-29T22:34:13.980 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.980 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.980 DEBUG:boto:Params: {} 2023-09-29T22:34:13.980 DEBUG:boto:Token: None 2023-09-29T22:34:13.980 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/ 2023-09-29T22:34:13.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY= 2023-09-29T22:34:13.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V7R2FrMYrRh6ylsyq+MI4Kz8nuY=', 'Content-Length': '0'} 2023-09-29T22:34:13.987 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '43'), ('X-RGW-Bytes-Used', '129'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b70ff5549529f0e3-00651750e5-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.987 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:13.987 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:13.987 DEBUG:boto:Method: PUT 2023-09-29T22:34:13.987 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:13.987 DEBUG:boto:Data: 2023-09-29T22:34:13.988 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:13.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:13.988 DEBUG:boto:Port: 8000 2023-09-29T22:34:13.988 DEBUG:boto:Params: {} 2023-09-29T22:34:13.988 DEBUG:boto:Token: None 2023-09-29T22:34:13.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:13 GMT /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:13.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2YVWbZWwpgd6qmfuCgIzunbEHGk= 2023-09-29T22:34:13.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2YVWbZWwpgd6qmfuCgIzunbEHGk='} 2023-09-29T22:34:13.998 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005139d25d915e0170-00651750e5-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:13.999 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:13.999 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:13.999 DEBUG:boto:Method: HEAD 2023-09-29T22:34:13.999 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:13.999 DEBUG:boto:Data: 2023-09-29T22:34:13.999 DEBUG:boto:Headers: {} 2023-09-29T22:34:13.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.000 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.000 DEBUG:boto:Params: {} 2023-09-29T22:34:14.000 DEBUG:boto:Token: None 2023-09-29T22:34:14.000 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.006 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000441356ac12f25e24-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.006 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:14.006 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:14.006 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.007 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:14.007 DEBUG:boto:Data: 2023-09-29T22:34:14.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.007 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.007 DEBUG:boto:Params: {} 2023-09-29T22:34:14.007 DEBUG:boto:Token: None 2023-09-29T22:34:14.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:14.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3/4s10CWhPyTZve+RaaY4trqK1o= 2023-09-29T22:34:14.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3/4s10CWhPyTZve+RaaY4trqK1o='} 2023-09-29T22:34:14.051 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070427e6379e72ea7-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.051 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.051 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.051 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.051 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.051 DEBUG:boto:Data: 2023-09-29T22:34:14.051 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.052 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.052 DEBUG:boto:Params: {} 2023-09-29T22:34:14.052 DEBUG:boto:Token: None 2023-09-29T22:34:14.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003af41d8375ab9247-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.059 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:14.060 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:14.060 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.060 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:14.060 DEBUG:boto:Data: 2023-09-29T22:34:14.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.060 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.060 DEBUG:boto:Params: {} 2023-09-29T22:34:14.060 DEBUG:boto:Token: None 2023-09-29T22:34:14.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:14.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eZLxTklIgYGW/E3zgW5/SfJkxto= 2023-09-29T22:34:14.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eZLxTklIgYGW/E3zgW5/SfJkxto='} 2023-09-29T22:34:14.072 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c0462016f6b41fb6-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.072 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.072 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.072 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.072 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.073 DEBUG:boto:Data: 2023-09-29T22:34:14.073 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.073 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.073 DEBUG:boto:Params: {} 2023-09-29T22:34:14.073 DEBUG:boto:Token: None 2023-09-29T22:34:14.073 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2bf767824fa6b39-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.078 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:14.078 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:14.079 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.079 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:14.079 DEBUG:boto:Data: 2023-09-29T22:34:14.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.079 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.079 DEBUG:boto:Params: {} 2023-09-29T22:34:14.079 DEBUG:boto:Token: None 2023-09-29T22:34:14.080 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:14.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GNGYTCWu0BPD6PQfGMgmVCYxbkw= 2023-09-29T22:34:14.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GNGYTCWu0BPD6PQfGMgmVCYxbkw='} 2023-09-29T22:34:14.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cef6816ef7809bdd-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.123 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.123 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.123 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.123 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.123 DEBUG:boto:Data: 2023-09-29T22:34:14.123 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.124 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.124 DEBUG:boto:Params: {} 2023-09-29T22:34:14.124 DEBUG:boto:Token: None 2023-09-29T22:34:14.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '44'), ('X-RGW-Bytes-Used', '132'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002e38096589025a8b-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.128 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:14.128 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:14.128 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.128 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:14.129 DEBUG:boto:Data: 2023-09-29T22:34:14.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.129 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.129 DEBUG:boto:Params: {} 2023-09-29T22:34:14.129 DEBUG:boto:Token: None 2023-09-29T22:34:14.129 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:14.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ce7ROrHL0o7zA/bjO4Dun/az6pY= 2023-09-29T22:34:14.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ce7ROrHL0o7zA/bjO4Dun/az6pY='} 2023-09-29T22:34:14.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000043643585425bd32a-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.137 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.137 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.137 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.138 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.138 DEBUG:boto:Data: 2023-09-29T22:34:14.138 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.138 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.138 DEBUG:boto:Params: {} 2023-09-29T22:34:14.138 DEBUG:boto:Token: None 2023-09-29T22:34:14.138 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '45'), ('X-RGW-Bytes-Used', '135'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d43a3b45252cdb1f-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.143 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:14.143 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:14.143 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.143 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:14.143 DEBUG:boto:Data: 2023-09-29T22:34:14.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.144 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.144 DEBUG:boto:Params: {} 2023-09-29T22:34:14.144 DEBUG:boto:Token: None 2023-09-29T22:34:14.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:14.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XfjFZzr0E0IwlyNOdKBzF1iusP8= 2023-09-29T22:34:14.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XfjFZzr0E0IwlyNOdKBzF1iusP8='} 2023-09-29T22:34:14.187 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006326db418b6f9361-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.187 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.187 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.187 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.187 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.187 DEBUG:boto:Data: 2023-09-29T22:34:14.187 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.188 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.188 DEBUG:boto:Params: {} 2023-09-29T22:34:14.188 DEBUG:boto:Token: None 2023-09-29T22:34:14.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.195 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b9dcfd23cf93583-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.195 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:14.196 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:14.196 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.196 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:14.196 DEBUG:boto:Data: 2023-09-29T22:34:14.196 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.196 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.196 DEBUG:boto:Params: {} 2023-09-29T22:34:14.196 DEBUG:boto:Token: None 2023-09-29T22:34:14.197 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:14.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xEUlrvJHVh0GgM9B2C2ntWiGaQo= 2023-09-29T22:34:14.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xEUlrvJHVh0GgM9B2C2ntWiGaQo='} 2023-09-29T22:34:14.207 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe857561dda8eb89-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.208 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.208 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.208 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.208 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.208 DEBUG:boto:Data: 2023-09-29T22:34:14.208 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.209 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.209 DEBUG:boto:Params: {} 2023-09-29T22:34:14.209 DEBUG:boto:Token: None 2023-09-29T22:34:14.209 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.210 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006dafa375156bd792-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.216 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:14.216 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:14.216 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.216 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:14.216 DEBUG:boto:Data: 2023-09-29T22:34:14.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.216 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.217 DEBUG:boto:Params: {} 2023-09-29T22:34:14.217 DEBUG:boto:Token: None 2023-09-29T22:34:14.217 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:14.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tzB5OHw1JJQoF6/UCjBQhblJRMo= 2023-09-29T22:34:14.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tzB5OHw1JJQoF6/UCjBQhblJRMo='} 2023-09-29T22:34:14.263 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db6890a3ee412cf3-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.263 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.263 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.263 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.263 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.263 DEBUG:boto:Data: 2023-09-29T22:34:14.264 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.264 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.264 DEBUG:boto:Params: {} 2023-09-29T22:34:14.264 DEBUG:boto:Token: None 2023-09-29T22:34:14.264 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.271 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebccabc3e512b6a9-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.271 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:14.271 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:14.271 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.271 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:14.271 DEBUG:boto:Data: 2023-09-29T22:34:14.272 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.272 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.272 DEBUG:boto:Params: {} 2023-09-29T22:34:14.272 DEBUG:boto:Token: None 2023-09-29T22:34:14.272 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:14.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cn5ezAlI5CULsLYXw0BUAzi2Vvs= 2023-09-29T22:34:14.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cn5ezAlI5CULsLYXw0BUAzi2Vvs='} 2023-09-29T22:34:14.285 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2a2264f7c7d7c15-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.285 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.285 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.285 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.285 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.285 DEBUG:boto:Data: 2023-09-29T22:34:14.286 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.286 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.286 DEBUG:boto:Params: {} 2023-09-29T22:34:14.286 DEBUG:boto:Token: None 2023-09-29T22:34:14.286 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.286 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.292 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000048109c5e54d29daf-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.293 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:14.293 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:14.293 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.293 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:14.293 DEBUG:boto:Data: 2023-09-29T22:34:14.293 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.293 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.293 DEBUG:boto:Params: {} 2023-09-29T22:34:14.294 DEBUG:boto:Token: None 2023-09-29T22:34:14.294 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:14.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SeQ4E0pMsIikg9Zl1yTlIbtBb0U= 2023-09-29T22:34:14.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SeQ4E0pMsIikg9Zl1yTlIbtBb0U='} 2023-09-29T22:34:14.338 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049a0c3248c55682b-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.339 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.339 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.339 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.339 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.339 DEBUG:boto:Data: 2023-09-29T22:34:14.340 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.340 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.340 DEBUG:boto:Params: {} 2023-09-29T22:34:14.340 DEBUG:boto:Token: None 2023-09-29T22:34:14.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.347 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '46'), ('X-RGW-Bytes-Used', '138'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a5157144b12b81d2-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.348 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:14.348 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:14.348 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.348 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:14.348 DEBUG:boto:Data: 2023-09-29T22:34:14.348 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.348 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.349 DEBUG:boto:Params: {} 2023-09-29T22:34:14.349 DEBUG:boto:Token: None 2023-09-29T22:34:14.349 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:14.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o2ys6Kkr0LOuEyXFYxq++0X5fCk= 2023-09-29T22:34:14.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o2ys6Kkr0LOuEyXFYxq++0X5fCk='} 2023-09-29T22:34:14.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009060dec7ce031d73-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.359 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.360 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.360 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.360 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.360 DEBUG:boto:Data: 2023-09-29T22:34:14.360 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.360 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.360 DEBUG:boto:Params: {} 2023-09-29T22:34:14.361 DEBUG:boto:Token: None 2023-09-29T22:34:14.361 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.361 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d6669074f5950d3a-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.367 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:14.367 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:14.367 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.367 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:14.367 DEBUG:boto:Data: 2023-09-29T22:34:14.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.368 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.368 DEBUG:boto:Params: {} 2023-09-29T22:34:14.368 DEBUG:boto:Token: None 2023-09-29T22:34:14.368 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:14.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:33HFCv0P/Rx71fOib/tFJZ2imG8= 2023-09-29T22:34:14.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:33HFCv0P/Rx71fOib/tFJZ2imG8='} 2023-09-29T22:34:14.415 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c5180658c43479bc-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.415 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.415 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.416 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.416 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.416 DEBUG:boto:Data: 2023-09-29T22:34:14.416 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.416 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.417 DEBUG:boto:Params: {} 2023-09-29T22:34:14.417 DEBUG:boto:Token: None 2023-09-29T22:34:14.417 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.424 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003af78964b6dd25fa-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.424 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:14.424 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:14.425 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.425 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:14.425 DEBUG:boto:Data: 2023-09-29T22:34:14.425 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.426 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.426 DEBUG:boto:Params: {} 2023-09-29T22:34:14.426 DEBUG:boto:Token: None 2023-09-29T22:34:14.427 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:14.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H6OTlFmFiP7Y5xC+//pe0AcWM1I= 2023-09-29T22:34:14.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H6OTlFmFiP7Y5xC+//pe0AcWM1I='} 2023-09-29T22:34:14.439 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007e4b38307f80d94e-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.439 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.439 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.439 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.440 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.440 DEBUG:boto:Data: 2023-09-29T22:34:14.440 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.440 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.440 DEBUG:boto:Params: {} 2023-09-29T22:34:14.440 DEBUG:boto:Token: None 2023-09-29T22:34:14.441 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.441 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.447 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '47'), ('X-RGW-Bytes-Used', '141'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c1a866d527500fb3-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.449 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:14.449 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:14.449 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.450 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:14.450 DEBUG:boto:Data: 2023-09-29T22:34:14.450 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.450 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.451 DEBUG:boto:Params: {} 2023-09-29T22:34:14.451 DEBUG:boto:Token: None 2023-09-29T22:34:14.451 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:14.451 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H0Ik/mf7nhpMK9oLm2AB+xpnS0E= 2023-09-29T22:34:14.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H0Ik/mf7nhpMK9oLm2AB+xpnS0E='} 2023-09-29T22:34:14.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000099e6a9a3857e1d2a-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.495 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.495 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.495 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.495 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.495 DEBUG:boto:Data: 2023-09-29T22:34:14.495 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.496 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.496 DEBUG:boto:Params: {} 2023-09-29T22:34:14.496 DEBUG:boto:Token: None 2023-09-29T22:34:14.496 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.503 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fc01679f1072673c-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.504 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:14.504 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:14.504 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.504 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:14.504 DEBUG:boto:Data: 2023-09-29T22:34:14.504 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.504 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.504 DEBUG:boto:Params: {} 2023-09-29T22:34:14.505 DEBUG:boto:Token: None 2023-09-29T22:34:14.505 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:14.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8L3P7eyQ6MfI54eo6XrLg/FPDkw= 2023-09-29T22:34:14.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8L3P7eyQ6MfI54eo6XrLg/FPDkw='} 2023-09-29T22:34:14.516 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000078a34b296eabba7b-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.517 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.517 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.517 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.517 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.517 DEBUG:boto:Data: 2023-09-29T22:34:14.517 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.518 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.518 DEBUG:boto:Params: {} 2023-09-29T22:34:14.518 DEBUG:boto:Token: None 2023-09-29T22:34:14.518 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.518 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.526 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cab8cda42aa9bdd5-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.526 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:14.526 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:14.526 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.526 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:14.526 DEBUG:boto:Data: 2023-09-29T22:34:14.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.527 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.527 DEBUG:boto:Params: {} 2023-09-29T22:34:14.527 DEBUG:boto:Token: None 2023-09-29T22:34:14.527 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:14.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:48vyIAETCt/Vs0bKKjDN90N6+Q8= 2023-09-29T22:34:14.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:48vyIAETCt/Vs0bKKjDN90N6+Q8='} 2023-09-29T22:34:14.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df54f85f09ac1fad-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.571 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.571 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.571 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.571 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.571 DEBUG:boto:Data: 2023-09-29T22:34:14.571 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.572 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.572 DEBUG:boto:Params: {} 2023-09-29T22:34:14.572 DEBUG:boto:Token: None 2023-09-29T22:34:14.572 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.579 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '48'), ('X-RGW-Bytes-Used', '144'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d3fefd1ec35916b-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.579 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:14.579 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:14.579 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.579 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:14.579 DEBUG:boto:Data: 2023-09-29T22:34:14.580 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.580 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.580 DEBUG:boto:Params: {} 2023-09-29T22:34:14.580 DEBUG:boto:Token: None 2023-09-29T22:34:14.580 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:14.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:igiivKJP5OeTzu2XRjSK+NXBjug= 2023-09-29T22:34:14.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:igiivKJP5OeTzu2XRjSK+NXBjug='} 2023-09-29T22:34:14.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000075ec4fb28319224c-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.590 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.590 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.590 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.590 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.590 DEBUG:boto:Data: 2023-09-29T22:34:14.590 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.591 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.591 DEBUG:boto:Params: {} 2023-09-29T22:34:14.591 DEBUG:boto:Token: None 2023-09-29T22:34:14.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.596 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '49'), ('X-RGW-Bytes-Used', '147'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000396d579b8d70a3a6-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.596 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:14.596 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:14.596 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.597 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:14.597 DEBUG:boto:Data: 2023-09-29T22:34:14.597 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.597 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.597 DEBUG:boto:Params: {} 2023-09-29T22:34:14.597 DEBUG:boto:Token: None 2023-09-29T22:34:14.597 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:14.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OUOmBPQCADBCU8iL82Rn3xRlOzw= 2023-09-29T22:34:14.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OUOmBPQCADBCU8iL82Rn3xRlOzw='} 2023-09-29T22:34:14.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a243f142ac1b785a-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.643 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.643 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.643 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.643 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.643 DEBUG:boto:Data: 2023-09-29T22:34:14.643 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.644 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.644 DEBUG:boto:Params: {} 2023-09-29T22:34:14.644 DEBUG:boto:Token: None 2023-09-29T22:34:14.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '50'), ('X-RGW-Bytes-Used', '150'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b896653dd14ac873-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.651 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:14.651 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:14.651 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.651 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:14.652 DEBUG:boto:Data: 2023-09-29T22:34:14.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.652 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.652 DEBUG:boto:Params: {} 2023-09-29T22:34:14.652 DEBUG:boto:Token: None 2023-09-29T22:34:14.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:14.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ipt3G2ELdfth4DgFkCkjeYncObI= 2023-09-29T22:34:14.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ipt3G2ELdfth4DgFkCkjeYncObI='} 2023-09-29T22:34:14.661 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d0bdfdc90fcc9db4-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.662 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.662 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.662 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.662 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.662 DEBUG:boto:Data: 2023-09-29T22:34:14.662 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.663 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.663 DEBUG:boto:Params: {} 2023-09-29T22:34:14.663 DEBUG:boto:Token: None 2023-09-29T22:34:14.663 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.668 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '51'), ('X-RGW-Bytes-Used', '153'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003cf1c323d4be9637-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.668 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:14.668 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:14.669 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.669 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:14.669 DEBUG:boto:Data: 2023-09-29T22:34:14.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.669 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.669 DEBUG:boto:Params: {} 2023-09-29T22:34:14.669 DEBUG:boto:Token: None 2023-09-29T22:34:14.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:14.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0ih9Vu4ePUQpGnydFlbYMU3RX84= 2023-09-29T22:34:14.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0ih9Vu4ePUQpGnydFlbYMU3RX84='} 2023-09-29T22:34:14.718 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e757b750c24485d-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.719 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.719 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.719 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.719 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.719 DEBUG:boto:Data: 2023-09-29T22:34:14.719 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.720 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.720 DEBUG:boto:Params: {} 2023-09-29T22:34:14.720 DEBUG:boto:Token: None 2023-09-29T22:34:14.720 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d249659dd83aa1dc-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.728 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:14.728 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:14.728 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.728 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:14.728 DEBUG:boto:Data: 2023-09-29T22:34:14.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.729 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.729 DEBUG:boto:Params: {} 2023-09-29T22:34:14.729 DEBUG:boto:Token: None 2023-09-29T22:34:14.729 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:14.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/rG3t9TzIOGinoGJXOOM6CV1Zxo= 2023-09-29T22:34:14.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/rG3t9TzIOGinoGJXOOM6CV1Zxo='} 2023-09-29T22:34:14.740 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a10a72b0e5b82ae4-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.741 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.741 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.741 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.741 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.742 DEBUG:boto:Data: 2023-09-29T22:34:14.742 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.742 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.742 DEBUG:boto:Params: {} 2023-09-29T22:34:14.742 DEBUG:boto:Token: None 2023-09-29T22:34:14.742 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.742 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.748 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e7c3c03c29f80535-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.748 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:14.748 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:14.748 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.748 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:14.748 DEBUG:boto:Data: 2023-09-29T22:34:14.749 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.749 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.749 DEBUG:boto:Params: {} 2023-09-29T22:34:14.749 DEBUG:boto:Token: None 2023-09-29T22:34:14.749 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:14.749 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FhozoVTjH9EruBbcoidzdZZVvMM= 2023-09-29T22:34:14.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FhozoVTjH9EruBbcoidzdZZVvMM='} 2023-09-29T22:34:14.799 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094dc24042e131b09-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.799 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.799 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.800 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.800 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.800 DEBUG:boto:Data: 2023-09-29T22:34:14.800 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.800 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.800 DEBUG:boto:Params: {} 2023-09-29T22:34:14.801 DEBUG:boto:Token: None 2023-09-29T22:34:14.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '52'), ('X-RGW-Bytes-Used', '156'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000051065147276c2d3a-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.808 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:14.808 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:14.808 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.808 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:14.808 DEBUG:boto:Data: 2023-09-29T22:34:14.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.808 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.809 DEBUG:boto:Params: {} 2023-09-29T22:34:14.809 DEBUG:boto:Token: None 2023-09-29T22:34:14.809 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:14.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L7QPlI9ZwsWWOVnyvHjcw6nM8hc= 2023-09-29T22:34:14.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L7QPlI9ZwsWWOVnyvHjcw6nM8hc='} 2023-09-29T22:34:14.818 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000413b3d4bab7ea3d4-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.819 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.819 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.819 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.819 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.820 DEBUG:boto:Data: 2023-09-29T22:34:14.820 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.820 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.820 DEBUG:boto:Params: {} 2023-09-29T22:34:14.821 DEBUG:boto:Token: None 2023-09-29T22:34:14.821 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.827 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '53'), ('X-RGW-Bytes-Used', '159'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000090a27c00746e798b-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.827 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:14.827 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:14.828 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.828 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:14.828 DEBUG:boto:Data: 2023-09-29T22:34:14.828 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.828 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.828 DEBUG:boto:Params: {} 2023-09-29T22:34:14.828 DEBUG:boto:Token: None 2023-09-29T22:34:14.829 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:14.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YSUZUGZ+mfmoZaKqn0iH8gZsSmw= 2023-09-29T22:34:14.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YSUZUGZ+mfmoZaKqn0iH8gZsSmw='} 2023-09-29T22:34:14.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005536452f6718ac3b-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.875 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:14.875 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:14.875 DEBUG:boto:Method: HEAD 2023-09-29T22:34:14.875 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:14.875 DEBUG:boto:Data: 2023-09-29T22:34:14.875 DEBUG:boto:Headers: {} 2023-09-29T22:34:14.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.876 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.876 DEBUG:boto:Params: {} 2023-09-29T22:34:14.876 DEBUG:boto:Token: None 2023-09-29T22:34:14.876 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/ 2023-09-29T22:34:14.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4= 2023-09-29T22:34:14.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/C1fdY1NXRA5+43Jn1OA+gXFY4=', 'Content-Length': '0'} 2023-09-29T22:34:14.882 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000212501de84a80b1c-00651750e6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.882 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:14.883 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:14.883 DEBUG:boto:Method: PUT 2023-09-29T22:34:14.883 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:14.883 DEBUG:boto:Data: 2023-09-29T22:34:14.883 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:14.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:14.883 DEBUG:boto:Port: 8000 2023-09-29T22:34:14.883 DEBUG:boto:Params: {} 2023-09-29T22:34:14.883 DEBUG:boto:Token: None 2023-09-29T22:34:14.884 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:14 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:14.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vzgEjwtR4rZWSiZ/gfk8mgMb+Oo= 2023-09-29T22:34:14.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vzgEjwtR4rZWSiZ/gfk8mgMb+Oo='} 2023-09-29T22:34:14.896 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000354c61eafe3048e5-00651750e6-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:14.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:14.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:14.897 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:14.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:14.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:15.030 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:34:15.031 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-58 2023-09-29T22:34:15.055 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 54 2023-09-29T22:34:15.176 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:15.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'shard', 'objects', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--num-shards', '37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:15.177 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket shard objects --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --num-shards 37 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:15.269 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:15.270 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:15.351 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:15.351 INFO:teuthology.orchestra.run.smithi112.stdout: "objs": [ 2023-09-29T22:34:15.351 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000060", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000000", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000011", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000001", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000033", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000065", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000005", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000016", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000048", 2023-09-29T22:34:15.352 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000015", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000026", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000030", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000062", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000009", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000040", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000024", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000012", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000002", 2023-09-29T22:34:15.353 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000013", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000045", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000006", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000017", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000007", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000039", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000027", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000010", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000021", 2023-09-29T22:34:15.354 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000053", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000020", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000004", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000036", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000003", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000014", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000019", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000057", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000018", 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: "obj00000000000000000008" 2023-09-29T22:34:15.355 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:15.356 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:15.357 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.357 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.357 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.357 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.357 DEBUG:boto:Data: 2023-09-29T22:34:15.357 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.358 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.358 DEBUG:boto:Params: {} 2023-09-29T22:34:15.358 DEBUG:boto:Token: None 2023-09-29T22:34:15.358 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.364 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '54'), ('X-RGW-Bytes-Used', '162'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000089c8e40920c4de79-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.365 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:15.365 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:15.365 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.365 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:15.366 DEBUG:boto:Data: 2023-09-29T22:34:15.366 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.366 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.366 DEBUG:boto:Params: {} 2023-09-29T22:34:15.366 DEBUG:boto:Token: None 2023-09-29T22:34:15.366 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:15.366 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZTEg0M1EQtbEeNuqOZ+Uc9RcEI= 2023-09-29T22:34:15.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZTEg0M1EQtbEeNuqOZ+Uc9RcEI='} 2023-09-29T22:34:15.411 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035a00390e214f537-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.411 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.411 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.411 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.411 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.411 DEBUG:boto:Data: 2023-09-29T22:34:15.412 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.412 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.412 DEBUG:boto:Params: {} 2023-09-29T22:34:15.412 DEBUG:boto:Token: None 2023-09-29T22:34:15.412 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003eb1bf219430696f-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.419 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:15.420 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:15.420 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.420 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:15.420 DEBUG:boto:Data: 2023-09-29T22:34:15.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.420 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.420 DEBUG:boto:Params: {} 2023-09-29T22:34:15.421 DEBUG:boto:Token: None 2023-09-29T22:34:15.421 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:15.421 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TDSjHFRrSqdlJv4Zxx/Kxi/r6Eo= 2023-09-29T22:34:15.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TDSjHFRrSqdlJv4Zxx/Kxi/r6Eo='} 2023-09-29T22:34:15.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000af808f34847403e2-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.434 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.434 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.434 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.434 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.434 DEBUG:boto:Data: 2023-09-29T22:34:15.435 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.435 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.435 DEBUG:boto:Params: {} 2023-09-29T22:34:15.435 DEBUG:boto:Token: None 2023-09-29T22:34:15.435 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.435 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.441 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000085e77777270cc24a-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.441 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:15.441 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:15.441 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.441 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:15.441 DEBUG:boto:Data: 2023-09-29T22:34:15.442 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.442 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.442 DEBUG:boto:Params: {} 2023-09-29T22:34:15.442 DEBUG:boto:Token: None 2023-09-29T22:34:15.442 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:15.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a97Onu0Pjyw4sN7BTu+QVIk3eis= 2023-09-29T22:34:15.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a97Onu0Pjyw4sN7BTu+QVIk3eis='} 2023-09-29T22:34:15.491 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bbbd746b37597756-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.491 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.491 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.491 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.491 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.491 DEBUG:boto:Data: 2023-09-29T22:34:15.491 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.492 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.492 DEBUG:boto:Params: {} 2023-09-29T22:34:15.492 DEBUG:boto:Token: None 2023-09-29T22:34:15.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '55'), ('X-RGW-Bytes-Used', '165'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000084b2d6a85f8c5837-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.500 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:15.500 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:15.500 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.500 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:15.501 DEBUG:boto:Data: 2023-09-29T22:34:15.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.501 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.501 DEBUG:boto:Params: {} 2023-09-29T22:34:15.501 DEBUG:boto:Token: None 2023-09-29T22:34:15.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:15.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AGR2vTNjeH+VPHcGKEKayiNyBbo= 2023-09-29T22:34:15.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AGR2vTNjeH+VPHcGKEKayiNyBbo='} 2023-09-29T22:34:15.510 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000073f88a07b47bbd83-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.510 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.511 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.511 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.511 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.511 DEBUG:boto:Data: 2023-09-29T22:34:15.511 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.511 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.511 DEBUG:boto:Params: {} 2023-09-29T22:34:15.511 DEBUG:boto:Token: None 2023-09-29T22:34:15.511 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033053a10ec3a338b-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.517 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:15.517 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:15.517 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.518 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:15.518 DEBUG:boto:Data: 2023-09-29T22:34:15.518 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.518 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.518 DEBUG:boto:Params: {} 2023-09-29T22:34:15.518 DEBUG:boto:Token: None 2023-09-29T22:34:15.518 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:15.519 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TvdZWYYBVvl2qRzjoOq75Oby7o4= 2023-09-29T22:34:15.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TvdZWYYBVvl2qRzjoOq75Oby7o4='} 2023-09-29T22:34:15.567 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009b393edda9ba30ef-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.567 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.567 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.567 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.567 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.567 DEBUG:boto:Data: 2023-09-29T22:34:15.567 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.568 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.568 DEBUG:boto:Params: {} 2023-09-29T22:34:15.568 DEBUG:boto:Token: None 2023-09-29T22:34:15.568 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.575 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2b6e617c09f8f71-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.576 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:15.576 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:15.576 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.576 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:15.576 DEBUG:boto:Data: 2023-09-29T22:34:15.576 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.577 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.577 DEBUG:boto:Params: {} 2023-09-29T22:34:15.577 DEBUG:boto:Token: None 2023-09-29T22:34:15.577 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:15.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q+cwKh6OlwDGrBioryuPBA6CW9M= 2023-09-29T22:34:15.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q+cwKh6OlwDGrBioryuPBA6CW9M='} 2023-09-29T22:34:15.589 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000219ad714f2d24f30-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.590 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.590 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.590 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.590 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.590 DEBUG:boto:Data: 2023-09-29T22:34:15.590 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.590 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.590 DEBUG:boto:Params: {} 2023-09-29T22:34:15.591 DEBUG:boto:Token: None 2023-09-29T22:34:15.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000246b6fdc014c923f-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.597 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:15.597 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:15.597 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.597 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:15.597 DEBUG:boto:Data: 2023-09-29T22:34:15.598 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.598 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.598 DEBUG:boto:Params: {} 2023-09-29T22:34:15.598 DEBUG:boto:Token: None 2023-09-29T22:34:15.598 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:15.598 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggdrZ3VvzRejveGbNf26m0n/v8g= 2023-09-29T22:34:15.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggdrZ3VvzRejveGbNf26m0n/v8g='} 2023-09-29T22:34:15.642 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000db20631f6a1936dc-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.643 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.643 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.643 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.643 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.643 DEBUG:boto:Data: 2023-09-29T22:34:15.643 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.644 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.644 DEBUG:boto:Params: {} 2023-09-29T22:34:15.644 DEBUG:boto:Token: None 2023-09-29T22:34:15.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.651 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a0491d90bf462ede-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.651 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:15.651 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:15.651 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.651 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:15.651 DEBUG:boto:Data: 2023-09-29T22:34:15.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.652 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.652 DEBUG:boto:Params: {} 2023-09-29T22:34:15.652 DEBUG:boto:Token: None 2023-09-29T22:34:15.652 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:15.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7A03ATdpTKfMHx872SOm/ycD7Ds= 2023-09-29T22:34:15.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7A03ATdpTKfMHx872SOm/ycD7Ds='} 2023-09-29T22:34:15.663 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a717afe86bdf8b4a-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.664 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.664 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.664 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.664 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.664 DEBUG:boto:Data: 2023-09-29T22:34:15.664 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.664 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.664 DEBUG:boto:Params: {} 2023-09-29T22:34:15.665 DEBUG:boto:Token: None 2023-09-29T22:34:15.665 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.671 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2423e800d24e8bf-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.671 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:15.671 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:15.672 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.672 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:15.672 DEBUG:boto:Data: 2023-09-29T22:34:15.672 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.672 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.672 DEBUG:boto:Params: {} 2023-09-29T22:34:15.672 DEBUG:boto:Token: None 2023-09-29T22:34:15.672 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:15.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mNPyNkVIH2mxb0YHQ9scFaRgLRI= 2023-09-29T22:34:15.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mNPyNkVIH2mxb0YHQ9scFaRgLRI='} 2023-09-29T22:34:15.714 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a6a15ce2bd3b868-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.715 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.715 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.715 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.715 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.715 DEBUG:boto:Data: 2023-09-29T22:34:15.715 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.716 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.716 DEBUG:boto:Params: {} 2023-09-29T22:34:15.716 DEBUG:boto:Token: None 2023-09-29T22:34:15.716 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.723 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003242e71743988a25-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.724 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:15.724 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:15.724 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.724 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:15.724 DEBUG:boto:Data: 2023-09-29T22:34:15.724 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.725 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.725 DEBUG:boto:Params: {} 2023-09-29T22:34:15.725 DEBUG:boto:Token: None 2023-09-29T22:34:15.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:15.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5I4XE0Tlcpswu0bA2ue95lpEHE0= 2023-09-29T22:34:15.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5I4XE0Tlcpswu0bA2ue95lpEHE0='} 2023-09-29T22:34:15.736 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b79b9ab9b32ab14a-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.737 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.737 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.737 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.737 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.737 DEBUG:boto:Data: 2023-09-29T22:34:15.737 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.738 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.738 DEBUG:boto:Params: {} 2023-09-29T22:34:15.738 DEBUG:boto:Token: None 2023-09-29T22:34:15.738 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.745 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ecdb005f08827d8c-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.745 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:15.745 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:15.745 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.746 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:15.746 DEBUG:boto:Data: 2023-09-29T22:34:15.746 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.746 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.746 DEBUG:boto:Params: {} 2023-09-29T22:34:15.746 DEBUG:boto:Token: None 2023-09-29T22:34:15.746 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:15.747 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UIu6hDEIC9g6v8QWkvsxLJizmrs= 2023-09-29T22:34:15.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UIu6hDEIC9g6v8QWkvsxLJizmrs='} 2023-09-29T22:34:15.794 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008807cad64f0e7aee-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.795 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.795 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.795 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.795 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.795 DEBUG:boto:Data: 2023-09-29T22:34:15.796 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.796 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.796 DEBUG:boto:Params: {} 2023-09-29T22:34:15.796 DEBUG:boto:Token: None 2023-09-29T22:34:15.796 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '56'), ('X-RGW-Bytes-Used', '168'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005fd11a98b30f641b-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.805 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:15.805 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:15.805 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.805 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:15.805 DEBUG:boto:Data: 2023-09-29T22:34:15.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.805 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.805 DEBUG:boto:Params: {} 2023-09-29T22:34:15.806 DEBUG:boto:Token: None 2023-09-29T22:34:15.806 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:15.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CJBGAN5PjZ1JAg02EP83KTbTFXo= 2023-09-29T22:34:15.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CJBGAN5PjZ1JAg02EP83KTbTFXo='} 2023-09-29T22:34:15.814 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000240a991bd76bc84-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.814 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.814 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.815 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.815 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.815 DEBUG:boto:Data: 2023-09-29T22:34:15.815 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.815 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.815 DEBUG:boto:Params: {} 2023-09-29T22:34:15.815 DEBUG:boto:Token: None 2023-09-29T22:34:15.815 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.821 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c7d2d764a72e0e01-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.821 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:15.822 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:15.822 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.822 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:15.822 DEBUG:boto:Data: 2023-09-29T22:34:15.822 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.822 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.823 DEBUG:boto:Params: {} 2023-09-29T22:34:15.823 DEBUG:boto:Token: None 2023-09-29T22:34:15.823 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:15.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQt/b20kfnnhH5WT8pmzq3wVmI0= 2023-09-29T22:34:15.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQt/b20kfnnhH5WT8pmzq3wVmI0='} 2023-09-29T22:34:15.867 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006d3ff8d6d0d5ab74-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.867 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.867 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.867 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.867 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.868 DEBUG:boto:Data: 2023-09-29T22:34:15.868 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.868 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.868 DEBUG:boto:Params: {} 2023-09-29T22:34:15.868 DEBUG:boto:Token: None 2023-09-29T22:34:15.868 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.876 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004270c9206b86cdfe-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.876 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:15.876 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:15.877 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.877 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:15.877 DEBUG:boto:Data: 2023-09-29T22:34:15.877 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.877 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.877 DEBUG:boto:Params: {} 2023-09-29T22:34:15.877 DEBUG:boto:Token: None 2023-09-29T22:34:15.878 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:15.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6WZFSriTM27oi8A2XKloS6rHRYs= 2023-09-29T22:34:15.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6WZFSriTM27oi8A2XKloS6rHRYs='} 2023-09-29T22:34:15.890 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039b842177915dfcb-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.890 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.890 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.891 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.891 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.891 DEBUG:boto:Data: 2023-09-29T22:34:15.891 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.891 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.891 DEBUG:boto:Params: {} 2023-09-29T22:34:15.892 DEBUG:boto:Token: None 2023-09-29T22:34:15.892 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.898 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068b1e1f6ca0e2862-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.899 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:15.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:15.899 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.899 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:15.899 DEBUG:boto:Data: 2023-09-29T22:34:15.899 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.899 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.899 DEBUG:boto:Params: {} 2023-09-29T22:34:15.900 DEBUG:boto:Token: None 2023-09-29T22:34:15.900 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:15.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FoBnA6VBL6ye3uEsKJElHd4X52Q= 2023-09-29T22:34:15.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FoBnA6VBL6ye3uEsKJElHd4X52Q='} 2023-09-29T22:34:15.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c1e4fe823a716ef3-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.943 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.943 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.943 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.943 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.943 DEBUG:boto:Data: 2023-09-29T22:34:15.943 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.944 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.944 DEBUG:boto:Params: {} 2023-09-29T22:34:15.944 DEBUG:boto:Token: None 2023-09-29T22:34:15.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.951 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1c3f61f899afd51-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.952 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:15.952 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:15.952 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.952 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:15.952 DEBUG:boto:Data: 2023-09-29T22:34:15.952 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.952 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.953 DEBUG:boto:Params: {} 2023-09-29T22:34:15.953 DEBUG:boto:Token: None 2023-09-29T22:34:15.953 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:15.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h8pmk4/N95HXvUPAWVOZISgspBg= 2023-09-29T22:34:15.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h8pmk4/N95HXvUPAWVOZISgspBg='} 2023-09-29T22:34:15.964 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1dbbe86bc069997-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.964 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:15.964 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:15.964 DEBUG:boto:Method: HEAD 2023-09-29T22:34:15.964 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:15.964 DEBUG:boto:Data: 2023-09-29T22:34:15.965 DEBUG:boto:Headers: {} 2023-09-29T22:34:15.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.965 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.965 DEBUG:boto:Params: {} 2023-09-29T22:34:15.965 DEBUG:boto:Token: None 2023-09-29T22:34:15.965 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/ 2023-09-29T22:34:15.965 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M= 2023-09-29T22:34:15.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pn4EvERHpRSwFT/UKCNCES2Fm7M=', 'Content-Length': '0'} 2023-09-29T22:34:15.971 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004dbc20c8bd2969c9-00651750e7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:15.972 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:15.972 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:15.972 DEBUG:boto:Method: PUT 2023-09-29T22:34:15.972 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:15.972 DEBUG:boto:Data: 2023-09-29T22:34:15.972 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:15.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:15.973 DEBUG:boto:Port: 8000 2023-09-29T22:34:15.973 DEBUG:boto:Params: {} 2023-09-29T22:34:15.973 DEBUG:boto:Token: None 2023-09-29T22:34:15.973 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:15 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:15.973 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kvvh9SHWtnYSk7jIskvdzfyQJ5w= 2023-09-29T22:34:15.973 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kvvh9SHWtnYSk7jIskvdzfyQJ5w='} 2023-09-29T22:34:16.023 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008a5810e38e1ba5b5-00651750e7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.023 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.023 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.023 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.023 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.023 DEBUG:boto:Data: 2023-09-29T22:34:16.023 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.024 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.024 DEBUG:boto:Params: {} 2023-09-29T22:34:16.024 DEBUG:boto:Token: None 2023-09-29T22:34:16.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.032 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004fb8469357633981-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.032 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:16.033 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:16.033 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.033 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:16.033 DEBUG:boto:Data: 2023-09-29T22:34:16.033 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.033 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.033 DEBUG:boto:Params: {} 2023-09-29T22:34:16.034 DEBUG:boto:Token: None 2023-09-29T22:34:16.034 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:16.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0efVdQ392wC5Gg+M6ukjcI9eI/c= 2023-09-29T22:34:16.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0efVdQ392wC5Gg+M6ukjcI9eI/c='} 2023-09-29T22:34:16.045 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000d336a86a05fed91-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.046 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.046 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.046 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.046 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.046 DEBUG:boto:Data: 2023-09-29T22:34:16.046 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.047 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.047 DEBUG:boto:Params: {} 2023-09-29T22:34:16.047 DEBUG:boto:Token: None 2023-09-29T22:34:16.047 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.054 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '57'), ('X-RGW-Bytes-Used', '171'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000196b5cd1c988a66d-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.054 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:16.054 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:16.054 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.055 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:16.055 DEBUG:boto:Data: 2023-09-29T22:34:16.055 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.055 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.055 DEBUG:boto:Params: {} 2023-09-29T22:34:16.055 DEBUG:boto:Token: None 2023-09-29T22:34:16.055 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:16.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tkQzFYnfAgRvJH5IRPMOMo4NT5M= 2023-09-29T22:34:16.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tkQzFYnfAgRvJH5IRPMOMo4NT5M='} 2023-09-29T22:34:16.102 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dfe0fb02577e4c5a-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.103 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.103 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.103 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.103 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.103 DEBUG:boto:Data: 2023-09-29T22:34:16.103 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.104 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.104 DEBUG:boto:Params: {} 2023-09-29T22:34:16.104 DEBUG:boto:Token: None 2023-09-29T22:34:16.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.112 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b6ea78475b5623fb-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.112 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:16.112 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:16.113 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.113 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:16.113 DEBUG:boto:Data: 2023-09-29T22:34:16.113 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.113 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.113 DEBUG:boto:Params: {} 2023-09-29T22:34:16.113 DEBUG:boto:Token: None 2023-09-29T22:34:16.113 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:16.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wxr8xRARU/uq8ma9DcGi2huzSdA= 2023-09-29T22:34:16.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wxr8xRARU/uq8ma9DcGi2huzSdA='} 2023-09-29T22:34:16.122 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003ba3ba7105bc7822-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.122 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.122 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.122 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.122 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.123 DEBUG:boto:Data: 2023-09-29T22:34:16.123 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.123 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.123 DEBUG:boto:Params: {} 2023-09-29T22:34:16.123 DEBUG:boto:Token: None 2023-09-29T22:34:16.123 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.128 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e50f02d964cf9ad0-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.128 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:16.129 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:16.129 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.129 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:16.129 DEBUG:boto:Data: 2023-09-29T22:34:16.129 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.130 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.130 DEBUG:boto:Params: {} 2023-09-29T22:34:16.130 DEBUG:boto:Token: None 2023-09-29T22:34:16.130 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:16.130 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w38+4AdXZ2PeszZpADkq6bIAubo= 2023-09-29T22:34:16.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w38+4AdXZ2PeszZpADkq6bIAubo='} 2023-09-29T22:34:16.179 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac5ab5ed3d539b7c-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.180 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.180 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.180 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.180 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.180 DEBUG:boto:Data: 2023-09-29T22:34:16.180 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.181 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.181 DEBUG:boto:Params: {} 2023-09-29T22:34:16.181 DEBUG:boto:Token: None 2023-09-29T22:34:16.181 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.189 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000765b04cdc9c68f31-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.190 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:16.190 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:16.190 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.190 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:16.190 DEBUG:boto:Data: 2023-09-29T22:34:16.190 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.191 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.191 DEBUG:boto:Params: {} 2023-09-29T22:34:16.191 DEBUG:boto:Token: None 2023-09-29T22:34:16.191 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:16.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fjW0T5FNwtrz34NrAoYwz+Zt/6c= 2023-09-29T22:34:16.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fjW0T5FNwtrz34NrAoYwz+Zt/6c='} 2023-09-29T22:34:16.202 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006f932fa791f0379b-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.202 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.202 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.202 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.203 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.203 DEBUG:boto:Data: 2023-09-29T22:34:16.203 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.203 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.203 DEBUG:boto:Params: {} 2023-09-29T22:34:16.203 DEBUG:boto:Token: None 2023-09-29T22:34:16.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.209 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c2bcdf51f0e29a4b-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.210 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:16.210 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:16.210 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.210 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:16.210 DEBUG:boto:Data: 2023-09-29T22:34:16.210 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.211 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.211 DEBUG:boto:Params: {} 2023-09-29T22:34:16.211 DEBUG:boto:Token: None 2023-09-29T22:34:16.211 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:16.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y8J8ORFn4nLCWqCcrmaPUsH1rB4= 2023-09-29T22:34:16.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y8J8ORFn4nLCWqCcrmaPUsH1rB4='} 2023-09-29T22:34:16.258 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d4866a7e375cee8d-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.259 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.259 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.259 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.259 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.259 DEBUG:boto:Data: 2023-09-29T22:34:16.260 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.260 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.260 DEBUG:boto:Params: {} 2023-09-29T22:34:16.260 DEBUG:boto:Token: None 2023-09-29T22:34:16.260 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.268 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000ac44e3e90c8e642-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.268 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:16.268 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:16.268 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.269 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:16.269 DEBUG:boto:Data: 2023-09-29T22:34:16.269 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.269 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.269 DEBUG:boto:Params: {} 2023-09-29T22:34:16.269 DEBUG:boto:Token: None 2023-09-29T22:34:16.270 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:16.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y6iqv8/YwgrXhTNDuuhm5L1Y1NA= 2023-09-29T22:34:16.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y6iqv8/YwgrXhTNDuuhm5L1Y1NA='} 2023-09-29T22:34:16.281 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e29559ecec33be69-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.281 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.281 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.282 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.282 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.282 DEBUG:boto:Data: 2023-09-29T22:34:16.282 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.282 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.282 DEBUG:boto:Params: {} 2023-09-29T22:34:16.282 DEBUG:boto:Token: None 2023-09-29T22:34:16.283 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.283 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.289 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf0cd6076b3f1989-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.289 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:16.290 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:16.290 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.290 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:16.290 DEBUG:boto:Data: 2023-09-29T22:34:16.290 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.290 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.290 DEBUG:boto:Params: {} 2023-09-29T22:34:16.291 DEBUG:boto:Token: None 2023-09-29T22:34:16.291 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:16.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:chD8hVcy10mW7RG0I/RabFfqp0o= 2023-09-29T22:34:16.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:chD8hVcy10mW7RG0I/RabFfqp0o='} 2023-09-29T22:34:16.335 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c7392115d2892fc-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.335 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.335 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.335 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.335 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.335 DEBUG:boto:Data: 2023-09-29T22:34:16.335 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.336 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.336 DEBUG:boto:Params: {} 2023-09-29T22:34:16.336 DEBUG:boto:Token: None 2023-09-29T22:34:16.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.343 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '58'), ('X-RGW-Bytes-Used', '174'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000804058bc80046b15-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.343 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:16.343 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:16.344 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.344 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:16.344 DEBUG:boto:Data: 2023-09-29T22:34:16.344 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.344 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.344 DEBUG:boto:Params: {} 2023-09-29T22:34:16.344 DEBUG:boto:Token: None 2023-09-29T22:34:16.344 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:16.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R1q6o4tbmA1bhh0M6aLGZd2CQSI= 2023-09-29T22:34:16.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R1q6o4tbmA1bhh0M6aLGZd2CQSI='} 2023-09-29T22:34:16.352 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000507326d82e2b1b8d-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.352 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.352 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.353 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.353 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.353 DEBUG:boto:Data: 2023-09-29T22:34:16.353 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.353 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.353 DEBUG:boto:Params: {} 2023-09-29T22:34:16.353 DEBUG:boto:Token: None 2023-09-29T22:34:16.354 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c8f1a51dbd7ed6c-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.359 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:16.359 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:16.359 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.359 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:16.359 DEBUG:boto:Data: 2023-09-29T22:34:16.360 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.360 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.360 DEBUG:boto:Params: {} 2023-09-29T22:34:16.360 DEBUG:boto:Token: None 2023-09-29T22:34:16.360 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:16.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vcNFc4WOzgVkyI0obkzdmeUipDc= 2023-09-29T22:34:16.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vcNFc4WOzgVkyI0obkzdmeUipDc='} 2023-09-29T22:34:16.403 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000599d6aa57514e6db-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.403 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.403 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.403 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.403 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.403 DEBUG:boto:Data: 2023-09-29T22:34:16.403 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.404 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.404 DEBUG:boto:Params: {} 2023-09-29T22:34:16.404 DEBUG:boto:Token: None 2023-09-29T22:34:16.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.413 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '59'), ('X-RGW-Bytes-Used', '177'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005d84b93c7328e782-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.413 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:16.413 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:16.414 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.414 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:16.414 DEBUG:boto:Data: 2023-09-29T22:34:16.414 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.414 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.414 DEBUG:boto:Params: {} 2023-09-29T22:34:16.414 DEBUG:boto:Token: None 2023-09-29T22:34:16.415 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:16.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OqNWcnLRGi/LdLlgR9C//fJY4Lo= 2023-09-29T22:34:16.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OqNWcnLRGi/LdLlgR9C//fJY4Lo='} 2023-09-29T22:34:16.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000066326169526ce34-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.423 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.424 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.424 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.424 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.424 DEBUG:boto:Data: 2023-09-29T22:34:16.424 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.424 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.424 DEBUG:boto:Params: {} 2023-09-29T22:34:16.424 DEBUG:boto:Token: None 2023-09-29T22:34:16.425 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b320f12ca2f2414a-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.431 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:16.431 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:16.431 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.431 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:16.431 DEBUG:boto:Data: 2023-09-29T22:34:16.431 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.432 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.432 DEBUG:boto:Params: {} 2023-09-29T22:34:16.432 DEBUG:boto:Token: None 2023-09-29T22:34:16.432 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:16.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v9KkmBlaD61qqQ6qL0tFiNfpB9M= 2023-09-29T22:34:16.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v9KkmBlaD61qqQ6qL0tFiNfpB9M='} 2023-09-29T22:34:16.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c2bd16f10d90477-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.479 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.479 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.479 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.479 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.479 DEBUG:boto:Data: 2023-09-29T22:34:16.480 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.480 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.480 DEBUG:boto:Params: {} 2023-09-29T22:34:16.480 DEBUG:boto:Token: None 2023-09-29T22:34:16.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.488 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '60'), ('X-RGW-Bytes-Used', '180'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ea05a7de0a8ce25a-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.488 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:16.488 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:16.488 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.488 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:16.489 DEBUG:boto:Data: 2023-09-29T22:34:16.489 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.489 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.489 DEBUG:boto:Params: {} 2023-09-29T22:34:16.489 DEBUG:boto:Token: None 2023-09-29T22:34:16.489 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:16.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5Ro6sYSk8+u3JwbI8LRvTJOjX/4= 2023-09-29T22:34:16.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5Ro6sYSk8+u3JwbI8LRvTJOjX/4='} 2023-09-29T22:34:16.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000bb94714b2ee5719-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.499 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.499 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.499 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.499 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.499 DEBUG:boto:Data: 2023-09-29T22:34:16.499 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.499 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.499 DEBUG:boto:Params: {} 2023-09-29T22:34:16.500 DEBUG:boto:Token: None 2023-09-29T22:34:16.500 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.505 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095c2a9c9672995e7-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.506 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:16.506 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:16.506 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.506 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:16.506 DEBUG:boto:Data: 2023-09-29T22:34:16.506 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.507 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.507 DEBUG:boto:Params: {} 2023-09-29T22:34:16.507 DEBUG:boto:Token: None 2023-09-29T22:34:16.507 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:16.507 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fKd03Xknm0DMsySnAFj8AKMSn+c= 2023-09-29T22:34:16.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fKd03Xknm0DMsySnAFj8AKMSn+c='} 2023-09-29T22:34:16.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000705b24c69e2bf05e-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.555 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.555 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.555 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.555 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.555 DEBUG:boto:Data: 2023-09-29T22:34:16.556 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.556 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.556 DEBUG:boto:Params: {} 2023-09-29T22:34:16.556 DEBUG:boto:Token: None 2023-09-29T22:34:16.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.564 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001c04994f73c85dde-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.564 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:16.564 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:16.564 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.564 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:16.565 DEBUG:boto:Data: 2023-09-29T22:34:16.565 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.565 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.565 DEBUG:boto:Params: {} 2023-09-29T22:34:16.565 DEBUG:boto:Token: None 2023-09-29T22:34:16.565 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:16.566 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sOAxurOflEJVezwKQxm53eSKrr0= 2023-09-29T22:34:16.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sOAxurOflEJVezwKQxm53eSKrr0='} 2023-09-29T22:34:16.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000014da7dded591d680-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.578 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.578 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.578 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.579 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.579 DEBUG:boto:Data: 2023-09-29T22:34:16.579 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.579 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.579 DEBUG:boto:Params: {} 2023-09-29T22:34:16.579 DEBUG:boto:Token: None 2023-09-29T22:34:16.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.586 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000388584ea299d288f-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.586 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:16.586 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:16.587 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.587 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:16.587 DEBUG:boto:Data: 2023-09-29T22:34:16.587 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.587 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.587 DEBUG:boto:Params: {} 2023-09-29T22:34:16.587 DEBUG:boto:Token: None 2023-09-29T22:34:16.587 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:16.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:45q3HzuqEKag1tt3r+H5VWlVN3U= 2023-09-29T22:34:16.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:45q3HzuqEKag1tt3r+H5VWlVN3U='} 2023-09-29T22:34:16.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000040ba21f6752ffc18-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.631 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:16.631 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:16.631 DEBUG:boto:Method: HEAD 2023-09-29T22:34:16.631 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:16.631 DEBUG:boto:Data: 2023-09-29T22:34:16.631 DEBUG:boto:Headers: {} 2023-09-29T22:34:16.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.632 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.632 DEBUG:boto:Params: {} 2023-09-29T22:34:16.632 DEBUG:boto:Token: None 2023-09-29T22:34:16.632 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/ 2023-09-29T22:34:16.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU= 2023-09-29T22:34:16.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BIX15D0+r8r7W6xW/TKNsZiwEoU=', 'Content-Length': '0'} 2023-09-29T22:34:16.640 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e357e077af59b7c-00651750e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.640 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:16.641 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:16.641 DEBUG:boto:Method: PUT 2023-09-29T22:34:16.641 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:16.641 DEBUG:boto:Data: 2023-09-29T22:34:16.641 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:34:16.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:16.641 DEBUG:boto:Port: 8000 2023-09-29T22:34:16.641 DEBUG:boto:Params: {} 2023-09-29T22:34:16.642 DEBUG:boto:Token: None 2023-09-29T22:34:16.642 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:34:16 GMT /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:16.642 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uFqtl0WsdzG5MMjAECbb6lXHwc4= 2023-09-29T22:34:16.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:34:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uFqtl0WsdzG5MMjAECbb6lXHwc4='} 2023-09-29T22:34:16.653 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000aaaa0fabdb3047f7-00651750e8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:16.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:16.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:34:16.654 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:34:16.700 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:16.700 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:16.840 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.841 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 2 2023-09-29T22:34:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 2 2023-09-29T22:34:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000001.136.6 2023-09-29T22:34:16.842 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 2 2023-09-29T22:34:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position=00000000001.45.6 2023-09-29T22:34:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 2 2023-09-29T22:34:16.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000001.55.6 2023-09-29T22:34:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 2 2023-09-29T22:34:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000001.55.6 2023-09-29T22:34:16.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 2 2023-09-29T22:34:16.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000001.57.6 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 2 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000001.53.6 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 2 2023-09-29T22:34:16.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000001.69.6 2023-09-29T22:34:16.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 2 2023-09-29T22:34:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000001.60.6 2023-09-29T22:34:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 2 2023-09-29T22:34:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000001.23.6 2023-09-29T22:34:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 2 2023-09-29T22:34:16.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.848+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000001.25.6 2023-09-29T22:34:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:16.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:16.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-29T22:34:16.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000007 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: entry->timestamp=2023-09-29T22:34:09.679460+0000 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000007]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000007[0] 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.888+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position=00000000001.31.6 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 0 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: parsed entry: id=00000000001.28.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000024 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: entry->timestamp=2023-09-29T22:34:09.768604+0000 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.28.6 op=0 op_state=1 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: : injecting data sync error on key=obj00000000000000000024 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000001.28.6 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000001.57.6 iter->object=obj00000000000000000028 iter->instance= name=obj00000000000000000028 instance= ns= 2023-09-29T22:34:16.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000028 2023-09-29T22:34:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:09.550964+0000 2023-09-29T22:34:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000028]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.57.6 op=0 op_state=1 2023-09-29T22:34:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000024 2023-09-29T22:34:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000028]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000028[0] 2023-09-29T22:34:16.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000001.57.6 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 0 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000016 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:09.325400+0000 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000016]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000016[0] 2023-09-29T22:34:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-29T22:34:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 0 2023-09-29T22:34:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.892+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket shard: 0 2023-09-29T22:34:16.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket shard: 0 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket shard: 0 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.896+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10/obj00000000000000000013 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: entry->timestamp=2023-09-29T22:34:09.404119+0000 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000020 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: entry->timestamp=2023-09-29T22:34:09.462964+0000 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-29T22:34:16.904 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: a sync operation returned error 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000013]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000013[0] 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000002.26.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000000 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:09.695191+0000 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.26.6 op=0 op_state=1 2023-09-29T22:34:16.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000020]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000020[0] 2023-09-29T22:34:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-29T22:34:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000000]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000000[0] 2023-09-29T22:34:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000002.26.6 2023-09-29T22:34:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: backing out with sync_status=-5 2023-09-29T22:34:16.906 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.900+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000001.29.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-29T22:34:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000019 2023-09-29T22:34:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:09.604589+0000 2023-09-29T22:34:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.29.6 op=0 op_state=1 2023-09-29T22:34:16.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000001.30.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000021 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:09.384286+0000 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.30.6 op=0 op_state=1 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: parsed entry: id=00000000001.39.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13/obj00000000000000000026 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: entry->timestamp=2023-09-29T22:34:09.752119+0000 2023-09-29T22:34:16.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.39.6 op=0 op_state=1 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: parsed entry: id=00000000002.55.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15/obj00000000000000000004 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: entry->timestamp=2023-09-29T22:34:09.246302+0000 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.55.6 op=0 op_state=1 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000001.29.6 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000001.64.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000076 2023-09-29T22:34:16.911 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:09.827103+0000 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.64.6 op=0 op_state=1 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000021]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000021[0] 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000001.30.6 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000026]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000026[0] 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position=00000000001.39.6 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000004]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000004[0] 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position=00000000002.55.6 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:16.912 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000019 2023-09-29T22:34:16.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: : injecting data sync error on key=obj00000000000000000076 2023-09-29T22:34:16.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000001.64.6 2023-09-29T22:34:16.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:16.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.904+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000076 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: a sync operation returned error 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: backing out with sync_status=-5 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: a sync operation returned error 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: backing out with sync_status=-5 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:16.914 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.908+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:16.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000007]: success 2023-09-29T22:34:16.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.912+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0:1 marker=00000000001.31.6 timestamp=2023-09-29T22:34:09.679460+0000 2023-09-29T22:34:16.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: parsed entry: id=00000000001.28.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-09-29T22:34:16.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000024 2023-09-29T22:34:16.917 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: entry->timestamp=2023-09-29T22:34:09.768604+0000 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.28.6 op=0 op_state=1 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000024[0] 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000001.28.6 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000001.29.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000019 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:09.604589+0000 2023-09-29T22:34:16.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.29.6 op=0 op_state=1 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: : injecting data sync error on key=obj00000000000000000019 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000001.29.6 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000001.64.6 iter->object=obj00000000000000000076 iter->instance= name=obj00000000000000000076 instance= ns= 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000076 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:09.827103+0000 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.64.6 op=0 op_state=1 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000019 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000076[0] 2023-09-29T22:34:16.919 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000001.64.6 2023-09-29T22:34:16.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000016]: success 2023-09-29T22:34:16.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.916+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1:1 marker=00000000001.54.6 timestamp=2023-09-29T22:34:09.325400+0000 2023-09-29T22:34:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000000]: success 2023-09-29T22:34:16.923 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5:1 marker=00000000002.26.6 timestamp=2023-09-29T22:34:09.695191+0000 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000028]: success 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4:1 marker=00000000001.57.6 timestamp=2023-09-29T22:34:09.550964+0000 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000013]: success 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10:1 marker=00000000002.61.6 timestamp=2023-09-29T22:34:09.404119+0000 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: a sync operation returned error 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: backing out with sync_status=-5 2023-09-29T22:34:16.924 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:16.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.920+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:16.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.924+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.924+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:16.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.924+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.926 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.924+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000020]: success 2023-09-29T22:34:16.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.924+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3:1 marker=00000000001.54.6 timestamp=2023-09-29T22:34:09.462964+0000 2023-09-29T22:34:16.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000021]: success 2023-09-29T22:34:16.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11:1 marker=00000000001.30.6 timestamp=2023-09-29T22:34:09.384286+0000 2023-09-29T22:34:16.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000001.29.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-29T22:34:16.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000019 2023-09-29T22:34:16.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:09.604589+0000 2023-09-29T22:34:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.29.6 op=0 op_state=1 2023-09-29T22:34:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000019[0] 2023-09-29T22:34:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000001.29.6 2023-09-29T22:34:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000024]: success 2023-09-29T22:34:16.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.928+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2:1 marker=00000000001.28.6 timestamp=2023-09-29T22:34:09.768604+0000 2023-09-29T22:34:16.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.932+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000026]: success 2023-09-29T22:34:16.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.932+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13:1 marker=00000000001.39.6 timestamp=2023-09-29T22:34:09.752119+0000 2023-09-29T22:34:16.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.932+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000076]: success 2023-09-29T22:34:16.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.932+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16:1 marker=00000000001.64.6 timestamp=2023-09-29T22:34:09.827103+0000 2023-09-29T22:34:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-29T22:34:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000010 2023-09-29T22:34:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:09.303924+0000 2023-09-29T22:34:16.940 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: : injecting data sync error on key=obj00000000000000000010 2023-09-29T22:34:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:16.941 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000010 2023-09-29T22:34:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: a sync operation returned error 2023-09-29T22:34:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: backing out with sync_status=-5 2023-09-29T22:34:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:16.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:16.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.940+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.943 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:16.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: parsed entry: id=00000000001.36.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6/obj00000000000000000032 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: entry->timestamp=2023-09-29T22:34:09.481640+0000 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.36.6 op=0 op_state=1 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000030 2023-09-29T22:34:16.945 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:09.623650+0000 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: parsed entry: id=00000000001.21.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14/obj00000000000000000033 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: entry->timestamp=2023-09-29T22:34:09.537440+0000 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.21.6 op=0 op_state=1 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000032[0] 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000001.36.6 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000030]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000030[0] 2023-09-29T22:34:16.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000001.34.6 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000033]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000033[0] 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position=00000000001.21.6 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000010 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:09.303924+0000 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:16.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000010[0] 2023-09-29T22:34:16.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.944+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:16.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.952+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000030]: success 2023-09-29T22:34:16.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.952+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12:1 marker=00000000001.34.6 timestamp=2023-09-29T22:34:09.623650+0000 2023-09-29T22:34:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.956+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000032]: success 2023-09-29T22:34:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.956+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6:1 marker=00000000001.36.6 timestamp=2023-09-29T22:34:09.481640+0000 2023-09-29T22:34:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.956+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000010]: success 2023-09-29T22:34:16.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.956+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8:1 marker=00000000002.34.6 timestamp=2023-09-29T22:34:09.303924+0000 2023-09-29T22:34:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.960+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000033]: success 2023-09-29T22:34:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.960+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14:1 marker=00000000001.21.6 timestamp=2023-09-29T22:34:09.537440+0000 2023-09-29T22:34:16.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.960+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000004]: success 2023-09-29T22:34:16.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.960+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15:1 marker=00000000002.55.6 timestamp=2023-09-29T22:34:09.246302+0000 2023-09-29T22:34:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.980+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000019]: success 2023-09-29T22:34:16.983 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.980+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9:1 marker=00000000001.29.6 timestamp=2023-09-29T22:34:09.604589+0000 2023-09-29T22:34:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.990 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.991 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:16.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.988+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.992 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:16.993 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 0 2023-09-29T22:34:16.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position= 2023-09-29T22:34:16.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.996 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: parsed entry: id=00000000002.21.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000016 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: entry->timestamp=2023-09-29T22:34:10.508803+0000 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.21.6 op=0 op_state=1 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000025 2023-09-29T22:34:16.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:10.564777+0000 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000016]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000016[0] 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000002.21.6 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000005 2023-09-29T22:34:16.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: entry->timestamp=2023-09-29T22:34:10.811182+0000 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.992+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000025]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000025[0] 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000004 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:10.279726+0000 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-09-29T22:34:16.999 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000005]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000005[0] 2023-09-29T22:34:17.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position=00000000002.54.6 2023-09-29T22:34:17.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000004]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000004[0] 2023-09-29T22:34:17.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000002.39.6 2023-09-29T22:34:17.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:17.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 0 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 0 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket shard: 0 2023-09-29T22:34:17.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6/obj00000000000000000030 2023-09-29T22:34:17.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: entry->timestamp=2023-09-29T22:34:10.659114+0000 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000030]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000030[0] 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:16.996+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000002.38.6 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.000+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:17.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.000+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket shard: 0 2023-09-29T22:34:17.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket shard: 0 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket shard: 0 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket shard: 0 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000016]: success 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2:2 marker=00000000002.21.6 timestamp=2023-09-29T22:34:10.508803+0000 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.008 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000003 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:10.789797+0000 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: parsed entry: id=00000000002.54.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7/obj00000000000000000015 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: entry->timestamp=2023-09-29T22:34:10.412947+0000 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.54.6 op=0 op_state=1 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.009 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000003]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000003[0] 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000002.74.6 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000015]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000015[0] 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position=00000000002.54.6 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000004]: success 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4:2 marker=00000000002.39.6 timestamp=2023-09-29T22:34:10.279726+0000 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000019 2023-09-29T22:34:17.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:10.431404+0000 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: parsed entry: id=00000000001.147.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10/obj00000000000000000009 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: entry->timestamp=2023-09-29T22:34:10.334973+0000 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.147.6 op=0 op_state=1 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000014 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:10.640569+0000 2023-09-29T22:34:17.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000019]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000019[0] 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000009]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000009[0] 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000001.147.6 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14/obj00000000000000000000 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: entry->timestamp=2023-09-29T22:34:10.863867+0000 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-09-29T22:34:17.012 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000012 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:10.712117+0000 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000014]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000014[0] 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000001.56.6 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000000]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000000[0] 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position=00000000002.45.6 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000012]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000012[0] 2023-09-29T22:34:17.013 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000002.50.6 2023-09-29T22:34:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000030]: success 2023-09-29T22:34:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.004+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6:2 marker=00000000002.38.6 timestamp=2023-09-29T22:34:10.659114+0000 2023-09-29T22:34:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: parsed entry: id=00000000002.43.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13/obj00000000000000000002 2023-09-29T22:34:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: entry->timestamp=2023-09-29T22:34:10.203077+0000 2023-09-29T22:34:17.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.43.6 op=0 op_state=1 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000011 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:10.358150+0000 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: parsed entry: id=00000000001.66.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17/obj00000000000000000001 2023-09-29T22:34:17.015 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: entry->timestamp=2023-09-29T22:34:10.584797+0000 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.66.6 op=0 op_state=1 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000002]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000002[0] 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position=00000000002.43.6 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000011]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000011[0] 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000001[0] 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position=00000000001.66.6 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000025]: success 2023-09-29T22:34:17.016 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5:2 marker=00000000001.54.6 timestamp=2023-09-29T22:34:10.564777+0000 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000007 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: entry->timestamp=2023-09-29T22:34:10.733540+0000 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000007]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000007[0] 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.008+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-29T22:34:17.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.016+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000000]: success 2023-09-29T22:34:17.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.016+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14:2 marker=00000000002.45.6 timestamp=2023-09-29T22:34:10.863867+0000 2023-09-29T22:34:17.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.016+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000005]: success 2023-09-29T22:34:17.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.016+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0:2 marker=00000000002.54.6 timestamp=2023-09-29T22:34:10.811182+0000 2023-09-29T22:34:17.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.016+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000019]: success 2023-09-29T22:34:17.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.016+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8:2 marker=00000000002.64.6 timestamp=2023-09-29T22:34:10.431404+0000 2023-09-29T22:34:17.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000007]: success 2023-09-29T22:34:17.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18:2 marker=00000000002.52.6 timestamp=2023-09-29T22:34:10.733540+0000 2023-09-29T22:34:17.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000003]: success 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9:2 marker=00000000002.74.6 timestamp=2023-09-29T22:34:10.789797+0000 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000015]: success 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7:2 marker=00000000002.54.6 timestamp=2023-09-29T22:34:10.412947+0000 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000014]: success 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.020+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11:2 marker=00000000001.56.6 timestamp=2023-09-29T22:34:10.640569+0000 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.024+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000009]: success 2023-09-29T22:34:17.026 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.024+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10:2 marker=00000000001.147.6 timestamp=2023-09-29T22:34:10.334973+0000 2023-09-29T22:34:17.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.024+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000012]: success 2023-09-29T22:34:17.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.024+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12:2 marker=00000000002.50.6 timestamp=2023-09-29T22:34:10.712117+0000 2023-09-29T22:34:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.028+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000011]: success 2023-09-29T22:34:17.028 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.028+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16:2 marker=00000000002.52.6 timestamp=2023-09-29T22:34:10.358150+0000 2023-09-29T22:34:17.048 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.048+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: parsed entry: id=00000000002.145.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-29T22:34:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.048+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15/obj00000000000000000006 2023-09-29T22:34:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.048+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: entry->timestamp=2023-09-29T22:34:10.255589+0000 2023-09-29T22:34:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.048+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.145.6 op=0 op_state=1 2023-09-29T22:34:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.048+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000006]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000006[0] 2023-09-29T22:34:17.049 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.048+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position=00000000002.145.6 2023-09-29T22:34:17.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-29T22:34:17.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000010 2023-09-29T22:34:17.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:10.488155+0000 2023-09-29T22:34:17.052 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-29T22:34:17.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000010]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000010[0] 2023-09-29T22:34:17.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-29T22:34:17.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000010]: success 2023-09-29T22:34:17.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.052+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1:2 marker=00000000002.52.6 timestamp=2023-09-29T22:34:10.488155+0000 2023-09-29T22:34:17.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.068+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000001]: success 2023-09-29T22:34:17.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.068+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17:2 marker=00000000001.66.6 timestamp=2023-09-29T22:34:10.584797+0000 2023-09-29T22:34:17.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.068+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000002]: success 2023-09-29T22:34:17.069 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.068+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13:2 marker=00000000002.43.6 timestamp=2023-09-29T22:34:10.203077+0000 2023-09-29T22:34:17.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.096+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000006]: success 2023-09-29T22:34:17.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.096+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15:2 marker=00000000002.145.6 timestamp=2023-09-29T22:34:10.255589+0000 2023-09-29T22:34:17.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.104+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.104+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.104+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.104+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 0 2023-09-29T22:34:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 0 2023-09-29T22:34:17.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.108+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:17.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000002.51.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-09-29T22:34:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000026 2023-09-29T22:34:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:11.311010+0000 2023-09-29T22:34:17.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.51.6 op=0 op_state=1 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000026]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000026[0] 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000002.51.6 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000058 iter->instance= name=obj00000000000000000058 instance= ns= 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000058 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: entry->timestamp=2023-09-29T22:34:11.934857+0000 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000058]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-29T22:34:17.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000058]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000058[0] 2023-09-29T22:34:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-29T22:34:17.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.112+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 0 2023-09-29T22:34:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-29T22:34:17.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000014 2023-09-29T22:34:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:11.384012+0000 2023-09-29T22:34:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-09-29T22:34:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7/obj00000000000000000002 2023-09-29T22:34:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: entry->timestamp=2023-09-29T22:34:11.329397+0000 2023-09-29T22:34:17.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-29T22:34:17.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000032 iter->instance= name=obj00000000000000000032 instance= ns= 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6/obj00000000000000000032 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: entry->timestamp=2023-09-29T22:34:11.833061+0000 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000032]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000014]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000014[0] 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000002.66.6 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000002]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000002[0] 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000032]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000032[0] 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000001.70.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000050 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:11.759872+0000 2023-09-29T22:34:17.121 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.70.6 op=0 op_state=1 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000050]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000050[0] 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000001.70.6 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:17.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.116+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket shard: 0 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000007 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:11.778835+0000 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000001.73.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000039 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:11.553475+0000 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.73.6 op=0 op_state=1 2023-09-29T22:34:17.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000007]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000007[0] 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket shard: 0 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000039]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000039[0] 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000001.73.6 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000026]: success 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1:3 marker=00000000002.51.6 timestamp=2023-09-29T22:34:11.311010+0000 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket shard: 0 2023-09-29T22:34:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket shard: 0 2023-09-29T22:34:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.120+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket shard: 0 2023-09-29T22:34:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000002]: success 2023-09-29T22:34:17.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7:3 marker=00000000002.52.6 timestamp=2023-09-29T22:34:11.329397+0000 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket shard: 0 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket shard: 0 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000014]: success 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4:3 marker=00000000002.66.6 timestamp=2023-09-29T22:34:11.384012+0000 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000032]: success 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6:3 marker=00000000002.46.6 timestamp=2023-09-29T22:34:11.833061+0000 2023-09-29T22:34:17.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000040 iter->instance= name=obj00000000000000000040 instance= ns= 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000040 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:11.404418+0000 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000040]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000009 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:11.853847+0000 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: parsed entry: id=00000000001.44.6 iter->object=obj00000000000000000046 iter->instance= name=obj00000000000000000046 instance= ns= 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000046 2023-09-29T22:34:17.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: entry->timestamp=2023-09-29T22:34:11.534435+0000 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000046]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.44.6 op=0 op_state=1 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000040]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000040[0] 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000001.58.6 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket shard: 0 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000009]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000009[0] 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000002.79.6 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000046]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000046[0] 2023-09-29T22:34:17.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.124+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000001.44.6 2023-09-29T22:34:17.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.128+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.128+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: parsed entry: id=00000000001.152.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-09-29T22:34:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13/obj00000000000000000045 2023-09-29T22:34:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: entry->timestamp=2023-09-29T22:34:11.456769+0000 2023-09-29T22:34:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.152.6 op=0 op_state=1 2023-09-29T22:34:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-29T22:34:17.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17/obj00000000000000000001 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: entry->timestamp=2023-09-29T22:34:11.913293+0000 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000000 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: entry->timestamp=2023-09-29T22:34:11.991963+0000 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000033 2023-09-29T22:34:17.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:12.012051+0000 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: parsed entry: id=00000000002.15.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15/obj00000000000000000012 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: entry->timestamp=2023-09-29T22:34:11.608131+0000 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.15.6 op=0 op_state=1 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000007]: success 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9:3 marker=00000000002.52.6 timestamp=2023-09-29T22:34:11.778835+0000 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000045]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000045[0] 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position=00000000001.152.6 2023-09-29T22:34:17.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000001]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000001[0] 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position=00000000002.59.6 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000000]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000000[0] 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000033]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000033[0] 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000012]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000012[0] 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position=00000000002.15.6 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: parsed entry: id=00000000002.47.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-29T22:34:17.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19/obj00000000000000000021 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: entry->timestamp=2023-09-29T22:34:11.628891+0000 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.47.6 op=0 op_state=1 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000021]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000021[0] 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position=00000000002.47.6 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: parsed entry: id=00000000002.75.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20/obj00000000000000000003 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: entry->timestamp=2023-09-29T22:34:11.684151+0000 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.75.6 op=0 op_state=1 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000058]: success 2023-09-29T22:34:17.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0:3 marker=00000000001.54.6 timestamp=2023-09-29T22:34:11.934857+0000 2023-09-29T22:34:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000003]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000003[0] 2023-09-29T22:34:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.132+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position=00000000002.75.6 2023-09-29T22:34:17.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000009]: success 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12:3 marker=00000000002.79.6 timestamp=2023-09-29T22:34:11.853847+0000 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000001.50.6 iter->object=obj00000000000000000055 iter->instance= name=obj00000000000000000055 instance= ns= 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000055 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:11.475030+0000 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000055]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.50.6 op=0 op_state=1 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000055]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000055[0] 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000001.50.6 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000039]: success 2023-09-29T22:34:17.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.136+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8:3 marker=00000000001.73.6 timestamp=2023-09-29T22:34:11.553475+0000 2023-09-29T22:34:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000001]: success 2023-09-29T22:34:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17:3 marker=00000000002.59.6 timestamp=2023-09-29T22:34:11.913293+0000 2023-09-29T22:34:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000021]: success 2023-09-29T22:34:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19:3 marker=00000000002.47.6 timestamp=2023-09-29T22:34:11.628891+0000 2023-09-29T22:34:17.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000003]: success 2023-09-29T22:34:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20:3 marker=00000000002.75.6 timestamp=2023-09-29T22:34:11.684151+0000 2023-09-29T22:34:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000000]: success 2023-09-29T22:34:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.140+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18:3 marker=00000000002.62.6 timestamp=2023-09-29T22:34:11.991963+0000 2023-09-29T22:34:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000033]: success 2023-09-29T22:34:17.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16:3 marker=00000000002.49.6 timestamp=2023-09-29T22:34:12.012051+0000 2023-09-29T22:34:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000012]: success 2023-09-29T22:34:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15:3 marker=00000000002.15.6 timestamp=2023-09-29T22:34:11.608131+0000 2023-09-29T22:34:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000040]: success 2023-09-29T22:34:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11:3 marker=00000000001.58.6 timestamp=2023-09-29T22:34:11.404418+0000 2023-09-29T22:34:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000046]: success 2023-09-29T22:34:17.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.144+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3:3 marker=00000000001.44.6 timestamp=2023-09-29T22:34:11.534435+0000 2023-09-29T22:34:17.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.148+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000055]: success 2023-09-29T22:34:17.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.148+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21:3 marker=00000000001.50.6 timestamp=2023-09-29T22:34:11.475030+0000 2023-09-29T22:34:17.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-29T22:34:17.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10/obj00000000000000000006 2023-09-29T22:34:17.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: entry->timestamp=2023-09-29T22:34:11.256042+0000 2023-09-29T22:34:17.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-09-29T22:34:17.160 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000006]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000006[0] 2023-09-29T22:34:17.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000002.66.6 2023-09-29T22:34:17.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000006]: success 2023-09-29T22:34:17.162 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.160+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10:3 marker=00000000002.66.6 timestamp=2023-09-29T22:34:11.256042+0000 2023-09-29T22:34:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-29T22:34:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14/obj00000000000000000013 2023-09-29T22:34:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: entry->timestamp=2023-09-29T22:34:11.706294+0000 2023-09-29T22:34:17.172 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-09-29T22:34:17.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000013]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000013[0] 2023-09-29T22:34:17.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position=00000000002.68.6 2023-09-29T22:34:17.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000013]: success 2023-09-29T22:34:17.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.172+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14:3 marker=00000000002.68.6 timestamp=2023-09-29T22:34:11.706294+0000 2023-09-29T22:34:17.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.176+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000050]: success 2023-09-29T22:34:17.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.176+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5:3 marker=00000000001.70.6 timestamp=2023-09-29T22:34:11.759872+0000 2023-09-29T22:34:17.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.184+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000045]: success 2023-09-29T22:34:17.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.184+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13:3 marker=00000000001.152.6 timestamp=2023-09-29T22:34:11.456769+0000 2023-09-29T22:34:17.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.256 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:17.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:17.257 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:17.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.252+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:17.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000027 2023-09-29T22:34:17.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: entry->timestamp=2023-09-29T22:34:12.444351+0000 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000027]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000027[0] 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000001.56.6 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000006 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: entry->timestamp=2023-09-29T22:34:13.300922+0000 2023-09-29T22:34:17.262 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: : injecting data sync error on key=obj00000000000000000006 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position=00000000002.77.6 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000001.54.6 iter->object=obj00000000000000000059 iter->instance= name=obj00000000000000000059 instance= ns= 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000059 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:13.245600+0000 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000059]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.54.6 op=0 op_state=1 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000006 2023-09-29T22:34:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000059]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000059[0] 2023-09-29T22:34:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000001.54.6 2023-09-29T22:34:17.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.256+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 0 2023-09-29T22:34:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 0 2023-09-29T22:34:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 0 2023-09-29T22:34:17.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket shard: 0 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000002.84.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000011 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:12.621092+0000 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.84.6 op=0 op_state=1 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000002 2023-09-29T22:34:17.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:13.148591+0000 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.269 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000011]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000011[0] 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000002.84.6 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.270 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000002 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket shard: 0 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket shard: 0 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.260+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:17.271 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.264+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.272 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.264+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.264+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.264+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.264+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.264+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket shard: 0 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket shard: 0 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket shard: 0 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket shard: 0 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket shard: 0 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.274 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket shard: 0 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000017 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:13.076763+0000 2023-09-29T22:34:17.275 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: sync status for source bucket shard: 0 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: sync status for source bucket shard: 0 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000001.156.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000031 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:12.919016+0000 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.156.6 op=0 op_state=1 2023-09-29T22:34:17.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: parsed entry: id=00000000002.52.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10/obj00000000000000000015 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: entry->timestamp=2023-09-29T22:34:12.462263+0000 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.52.6 op=0 op_state=1 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000022 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:12.761935+0000 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-29T22:34:17.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13/obj00000000000000000013 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: entry->timestamp=2023-09-29T22:34:13.017772+0000 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6/obj00000000000000000019 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: entry->timestamp=2023-09-29T22:34:13.224636+0000 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000017]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000017[0] 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000001.83.6 2023-09-29T22:34:17.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: a sync operation returned error 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: : injecting data sync error on key=obj00000000000000000031 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000001.156.6 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000015]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000015[0] 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000002.52.6 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000022]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000022[0] 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000002.39.6 2023-09-29T22:34:17.279 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000013]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000013[0] 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position=00000000002.32.6 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000019]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000019[0] 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000001.47.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000008 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:12.861089+0000 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.47.6 op=0 op_state=1 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000031 2023-09-29T22:34:17.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000008]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000008[0] 2023-09-29T22:34:17.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000001.47.6 2023-09-29T22:34:17.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.268+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: a sync operation returned error 2023-09-29T22:34:17.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: backing out with sync_status=-5 2023-09-29T22:34:17.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: sync status for source bucket shard: 0 2023-09-29T22:34:17.281 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: sync status for source bucket shard: 0 2023-09-29T22:34:17.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: sync status for source bucket shard: 0 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.272+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: backing out with sync_status=-5 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000011]: success 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4:4 marker=00000000002.84.6 timestamp=2023-09-29T22:34:12.621092+0000 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: parsed entry: id=00000000001.85.6 iter->object=obj00000000000000000037 iter->instance= name=obj00000000000000000037 instance= ns= 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15/obj00000000000000000037 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: entry->timestamp=2023-09-29T22:34:13.095565+0000 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000037]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.85.6 op=0 op_state=1 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000005 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:13.321035+0000 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-09-29T22:34:17.284 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000037]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000037[0] 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position=00000000001.85.6 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14/obj00000000000000000020 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: entry->timestamp=2023-09-29T22:34:12.997366+0000 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.28.6 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000027]: success 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2:4 marker=00000000001.56.6 timestamp=2023-09-29T22:34:12.444351+0000 2023-09-29T22:34:17.285 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000020]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000020[0] 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000004 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: entry->timestamp=2023-09-29T22:34:12.598453+0000 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: parsed entry: id=00000000002.44.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22/obj00000000000000000009 2023-09-29T22:34:17.286 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: entry->timestamp=2023-09-29T22:34:13.452426+0000 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.44.6 op=0 op_state=1 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000059]: success 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1:4 marker=00000000001.54.6 timestamp=2023-09-29T22:34:13.245600+0000 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000003 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: entry->timestamp=2023-09-29T22:34:13.376465+0000 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: parsed entry: id=00000000001.34.6 iter->object=obj00000000000000000042 iter->instance= name=obj00000000000000000042 instance= ns= 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19/obj00000000000000000042 2023-09-29T22:34:17.287 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: entry->timestamp=2023-09-29T22:34:13.167841+0000 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000042]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.34.6 op=0 op_state=1 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: parsed entry: id=00000000002.45.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20/obj00000000000000000010 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: entry->timestamp=2023-09-29T22:34:12.682804+0000 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.45.6 op=0 op_state=1 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000001 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:12.939908+0000 2023-09-29T22:34:17.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000005 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000004]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000004[0] 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000002.30.6 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000009]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000009[0] 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position=00000000002.44.6 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000042]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000042[0] 2023-09-29T22:34:17.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position=00000000001.34.6 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000010]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000010[0] 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position=00000000002.45.6 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000001]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000001[0] 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000002.69.6 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000025 iter->instance= name=obj00000000000000000025 instance= ns= 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23/obj00000000000000000025 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: entry->timestamp=2023-09-29T22:34:13.396957+0000 2023-09-29T22:34:17.290 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:entry[obj00000000000000000025]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-09-29T22:34:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.276+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000003 2023-09-29T22:34:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.280+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:entry[obj00000000000000000025]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000025[0] 2023-09-29T22:34:17.291 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.280+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: listing bilog for incremental sync; position=00000000002.85.6 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: parsed entry: id=00000000002.24.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28/obj00000000000000000021 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: entry->timestamp=2023-09-29T22:34:12.781007+0000 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.24.6 op=0 op_state=1 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000019]: success 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6:4 marker=00000000002.34.6 timestamp=2023-09-29T22:34:13.224636+0000 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000013]: success 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13:4 marker=00000000002.32.6 timestamp=2023-09-29T22:34:13.017772+0000 2023-09-29T22:34:17.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000015]: success 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10:4 marker=00000000002.52.6 timestamp=2023-09-29T22:34:12.462263+0000 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: parsed entry: id=00000000002.32.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27/obj00000000000000000030 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: entry->timestamp=2023-09-29T22:34:12.837079+0000 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.32.6 op=0 op_state=1 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000004]: success 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3:4 marker=00000000002.30.6 timestamp=2023-09-29T22:34:12.598453+0000 2023-09-29T22:34:17.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000021]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000021[0] 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: listing bilog for incremental sync; position=00000000002.24.6 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.284+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000009]: success 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22:4 marker=00000000002.44.6 timestamp=2023-09-29T22:34:13.452426+0000 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: a sync operation returned error 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000030]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000030[0] 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: listing bilog for incremental sync; position=00000000002.32.6 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: a sync operation returned error 2023-09-29T22:34:17.295 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000001]: success 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21:4 marker=00000000002.69.6 timestamp=2023-09-29T22:34:12.939908+0000 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000020]: success 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14:4 marker=00000000002.34.6 timestamp=2023-09-29T22:34:12.997366+0000 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: backing out with sync_status=-5 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: backing out with sync_status=-5 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.296 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.288+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:entry[obj00000000000000000025]: success 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23:4 marker=00000000002.85.6 timestamp=2023-09-29T22:34:13.396957+0000 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000010]: success 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20:4 marker=00000000002.45.6 timestamp=2023-09-29T22:34:12.682804+0000 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000021]: success 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28:4 marker=00000000002.24.6 timestamp=2023-09-29T22:34:12.781007+0000 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000022]: success 2023-09-29T22:34:17.297 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12:4 marker=00000000002.39.6 timestamp=2023-09-29T22:34:12.761935+0000 2023-09-29T22:34:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000030]: success 2023-09-29T22:34:17.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27:4 marker=00000000002.32.6 timestamp=2023-09-29T22:34:12.837079+0000 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000008]: success 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9:4 marker=00000000001.47.6 timestamp=2023-09-29T22:34:12.861089+0000 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000037]: success 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.292+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15:4 marker=00000000001.85.6 timestamp=2023-09-29T22:34:13.095565+0000 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.296+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.296+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.296+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:17.299 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.296+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.296+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.296+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.300 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: parsed entry: id=00000000002.77.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000006 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: entry->timestamp=2023-09-29T22:34:13.300922+0000 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.77.6 op=0 op_state=1 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000006[0] 2023-09-29T22:34:17.301 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position=00000000002.77.6 2023-09-29T22:34:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket shard: 0 2023-09-29T22:34:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.302 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000006]: success 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0:4 marker=00000000002.77.6 timestamp=2023-09-29T22:34:13.300922+0000 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.28.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000005 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:13.321035+0000 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.28.6 op=0 op_state=1 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000005[0] 2023-09-29T22:34:17.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.28.6 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: parsed entry: id=00000000002.62.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000003 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: entry->timestamp=2023-09-29T22:34:13.376465+0000 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.62.6 op=0 op_state=1 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000003[0] 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position=00000000002.62.6 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000002 2023-09-29T22:34:17.305 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:13.148591+0000 2023-09-29T22:34:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000002[0] 2023-09-29T22:34:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.300+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.304+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000005]: success 2023-09-29T22:34:17.306 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.304+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16:4 marker=00000000002.28.6 timestamp=2023-09-29T22:34:13.321035+0000 2023-09-29T22:34:17.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.304+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000002]: success 2023-09-29T22:34:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.304+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5:4 marker=00000000002.34.6 timestamp=2023-09-29T22:34:13.148591+0000 2023-09-29T22:34:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.304+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000003]: success 2023-09-29T22:34:17.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.304+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18:4 marker=00000000002.62.6 timestamp=2023-09-29T22:34:13.376465+0000 2023-09-29T22:34:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-29T22:34:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17/obj00000000000000000012 2023-09-29T22:34:17.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: entry->timestamp=2023-09-29T22:34:12.516071+0000 2023-09-29T22:34:17.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-09-29T22:34:17.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000012]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000012[0] 2023-09-29T22:34:17.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position=00000000002.30.6 2023-09-29T22:34:17.318 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000012]: success 2023-09-29T22:34:17.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.316+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17:4 marker=00000000002.30.6 timestamp=2023-09-29T22:34:12.516071+0000 2023-09-29T22:34:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.320+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: a sync operation returned error 2023-09-29T22:34:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.320+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: backing out with sync_status=-5 2023-09-29T22:34:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.320+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.320+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.320+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000017]: success 2023-09-29T22:34:17.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.320+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8:4 marker=00000000001.83.6 timestamp=2023-09-29T22:34:13.076763+0000 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: parsed entry: id=00000000002.22.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26/obj00000000000000000014 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: entry->timestamp=2023-09-29T22:34:12.701591+0000 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.22.6 op=0 op_state=1 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: parsed entry: id=00000000002.83.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25/obj00000000000000000000 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: entry->timestamp=2023-09-29T22:34:12.539695+0000 2023-09-29T22:34:17.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.83.6 op=0 op_state=1 2023-09-29T22:34:17.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:entry[obj00000000000000000014]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000014[0] 2023-09-29T22:34:17.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: listing bilog for incremental sync; position=00000000002.22.6 2023-09-29T22:34:17.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000000]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000000[0] 2023-09-29T22:34:17.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: listing bilog for incremental sync; position=00000000002.83.6 2023-09-29T22:34:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:entry[obj00000000000000000014]: success 2023-09-29T22:34:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26:4 marker=00000000002.22.6 timestamp=2023-09-29T22:34:12.701591+0000 2023-09-29T22:34:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000000]: success 2023-09-29T22:34:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25:4 marker=00000000002.83.6 timestamp=2023-09-29T22:34:12.539695+0000 2023-09-29T22:34:17.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.324+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000001.156.6 iter->object=obj00000000000000000031 iter->instance= name=obj00000000000000000031 instance= ns= 2023-09-29T22:34:17.330 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000031 2023-09-29T22:34:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:12.919016+0000 2023-09-29T22:34:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.156.6 op=0 op_state=1 2023-09-29T22:34:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000031[0] 2023-09-29T22:34:17.331 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.328+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000001.156.6 2023-09-29T22:34:17.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.336+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000042]: success 2023-09-29T22:34:17.336 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.336+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19:4 marker=00000000001.34.6 timestamp=2023-09-29T22:34:13.167841+0000 2023-09-29T22:34:17.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.336+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000031]: success 2023-09-29T22:34:17.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.336+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11:4 marker=00000000001.156.6 timestamp=2023-09-29T22:34:12.919016+0000 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.484 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.480+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.485 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 0 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.486 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.487 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 0 2023-09-29T22:34:17.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.488 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.484+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:17.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:17.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket shard: 0 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 0 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.491 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: parsed entry: id=00000000002.41.6 iter->object=obj00000000000000000023 iter->instance= name=obj00000000000000000023 instance= ns= 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6/obj00000000000000000023 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: entry->timestamp=2023-09-29T22:34:13.864080+0000 2023-09-29T22:34:17.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000023]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.41.6 op=0 op_state=1 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000023]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000023[0] 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000002.41.6 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: parsed entry: id=00000000001.19.6 iter->object=obj00000000000000000467 iter->instance= name=obj00000000000000000467 instance= ns= 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000467 2023-09-29T22:34:17.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: entry->timestamp=2023-09-29T22:34:14.584478+0000 2023-09-29T22:34:17.494 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000467]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.19.6 op=0 op_state=1 2023-09-29T22:34:17.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000467]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000467[0] 2023-09-29T22:34:17.495 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000001.19.6 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000002.162.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000004 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:14.372069+0000 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.162.6 op=0 op_state=1 2023-09-29T22:34:17.496 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000005 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: entry->timestamp=2023-09-29T22:34:14.508820+0000 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.488+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket shard: 0 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000004]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000004[0] 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000002.162.6 2023-09-29T22:34:17.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: : injecting data sync error on key=obj00000000000000000005 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000002.71.6 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket shard: 0 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.498 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000005 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: parsed entry: id=00000000001.29.6 iter->object=obj00000000000000000317 iter->instance= name=obj00000000000000000317 instance= ns= 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7/obj00000000000000000317 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: entry->timestamp=2023-09-29T22:34:13.992396+0000 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000317]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.29.6 op=0 op_state=1 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000317]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000317[0] 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position=00000000001.29.6 2023-09-29T22:34:17.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000022 iter->instance= name=obj00000000000000000022 instance= ns= 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000022 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:14.297905+0000 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000022]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13/obj00000000000000000008 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: entry->timestamp=2023-09-29T22:34:13.917400+0000 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000022]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000022[0] 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000002.63.6 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000008]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000008[0] 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: parsed entry: id=00000000002.63.6 iter->object=obj00000000000000000021 iter->instance= name=obj00000000000000000021 instance= ns= 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10/obj00000000000000000021 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: entry->timestamp=2023-09-29T22:34:14.064728+0000 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000021]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.63.6 op=0 op_state=1 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000271 2023-09-29T22:34:17.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:14.133098+0000 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket shard: 0 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000021]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000021[0] 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000002.63.6 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000009 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:14.888053+0000 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-09-29T22:34:17.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: : injecting data sync error on key=obj00000000000000000271 2023-09-29T22:34:17.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000001.83.6 2023-09-29T22:34:17.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.492+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: : injecting data sync error on key=obj00000000000000000009 2023-09-29T22:34:17.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000002.85.6 2023-09-29T22:34:17.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000002.49.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000000 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:14.276270+0000 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.49.6 op=0 op_state=1 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000271 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000009 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000000]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000000[0] 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.496+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000002.49.6 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000056 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:14.147844+0000 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket shard: 0 2023-09-29T22:34:17.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket shard: 0 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: : injecting data sync error on key=obj00000000000000000056 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000001.31.6 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket shard: 0 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000056 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket shard: 0 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: sync status for source bucket shard: 0 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: sync status for source bucket shard: 0 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: parsed entry: id=00000000001.88.6 iter->object=obj00000000000000000162 iter->instance= name=obj00000000000000000162 instance= ns= 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17/obj00000000000000000162 2023-09-29T22:34:17.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: entry->timestamp=2023-09-29T22:34:14.673477+0000 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000162]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.88.6 op=0 op_state=1 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.66.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000017 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:13.937939+0000 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.66.6 op=0 op_state=1 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: sync status for source bucket shard: 0 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000004]: success 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5:5 marker=00000000002.162.6 timestamp=2023-09-29T22:34:14.372069+0000 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: a sync operation returned error 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000162]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000162[0] 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position=00000000001.88.6 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000017]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000017[0] 2023-09-29T22:34:17.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.66.6 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000006 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:14.431088+0000 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.500+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: : injecting data sync error on key=obj00000000000000000006 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-29T22:34:17.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000022]: success 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8:5 marker=00000000002.63.6 timestamp=2023-09-29T22:34:14.297905+0000 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000023]: success 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6:5 marker=00000000002.41.6 timestamp=2023-09-29T22:34:13.864080+0000 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000008]: success 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13:5 marker=00000000002.61.6 timestamp=2023-09-29T22:34:13.917400+0000 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: backing out with sync_status=-5 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.513 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000006 2023-09-29T22:34:17.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: sync status for source bucket shard: 0 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket shard: 0 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.504+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: sync status for source bucket shard: 0 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: sync status for source bucket shard: 0 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: sync status for source bucket shard: 0 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.508+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14/obj00000000000000000018 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: entry->timestamp=2023-09-29T22:34:14.601425+0000 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000021]: success 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10:5 marker=00000000002.63.6 timestamp=2023-09-29T22:34:14.064728+0000 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: parsed entry: id=00000000001.52.6 iter->object=obj00000000000000000423 iter->instance= name=obj00000000000000000423 instance= ns= 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20/obj00000000000000000423 2023-09-29T22:34:17.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: entry->timestamp=2023-09-29T22:34:14.812873+0000 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000423]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.52.6 op=0 op_state=1 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19/obj00000000000000000003 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: entry->timestamp=2023-09-29T22:34:14.530970+0000 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: parsed entry: id=00000000002.57.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000016 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: entry->timestamp=2023-09-29T22:34:14.200629+0000 2023-09-29T22:34:17.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.57.6 op=0 op_state=1 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: a sync operation returned error 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: a sync operation returned error 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000018]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000018[0] 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000423]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000423[0] 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position=00000000001.52.6 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: : injecting data sync error on key=obj00000000000000000003 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position=00000000002.31.6 2023-09-29T22:34:17.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:14.011338+0000 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000016]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000016[0] 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position=00000000002.57.6 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000317]: success 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7:5 marker=00000000001.29.6 timestamp=2023-09-29T22:34:13.992396+0000 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000000]: success 2023-09-29T22:34:17.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9:5 marker=00000000002.49.6 timestamp=2023-09-29T22:34:14.276270+0000 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19/obj00000000000000000003 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: backing out with sync_status=-5 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: parsed entry: id=00000000001.40.6 iter->object=obj00000000000000000283 iter->instance= name=obj00000000000000000283 instance= ns= 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25/obj00000000000000000283 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: entry->timestamp=2023-09-29T22:34:14.455477+0000 2023-09-29T22:34:17.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000283]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.40.6 op=0 op_state=1 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: a sync operation returned error 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000467]: success 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2:5 marker=00000000001.19.6 timestamp=2023-09-29T22:34:14.584478+0000 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000017]: success 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16:5 marker=00000000002.66.6 timestamp=2023-09-29T22:34:13.937939+0000 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: backing out with sync_status=-5 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000283]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000283[0] 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: listing bilog for incremental sync; position=00000000001.40.6 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: backing out with sync_status=-5 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.512+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000016]: success 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18:5 marker=00000000002.57.6 timestamp=2023-09-29T22:34:14.200629+0000 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24/obj00000000000000000011 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: entry->timestamp=2023-09-29T22:34:14.220614+0000 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: : injecting data sync error on key=obj00000000000000000011 2023-09-29T22:34:17.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position=00000000002.69.6 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.516+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24/obj00000000000000000011 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000050 iter->instance= name=obj00000000000000000050 instance= ns= 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28/obj00000000000000000050 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: entry->timestamp=2023-09-29T22:34:14.753202+0000 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000050]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: parsed entry: id=00000000001.164.6 iter->object=obj00000000000000000133 iter->instance= name=obj00000000000000000133 instance= ns= 2023-09-29T22:34:17.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30/obj00000000000000000133 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: entry->timestamp=2023-09-29T22:34:14.832671+0000 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:entry[obj00000000000000000133]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.164.6 op=0 op_state=1 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: parsed entry: id=00000000001.56.6 iter->object=obj00000000000000000029 iter->instance= name=obj00000000000000000029 instance= ns= 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29/obj00000000000000000029 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: entry->timestamp=2023-09-29T22:34:14.353418+0000 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:entry[obj00000000000000000029]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.56.6 op=0 op_state=1 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000050]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000050[0] 2023-09-29T22:34:17.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: listing bilog for incremental sync; position=00000000002.68.6 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:entry[obj00000000000000000133]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000133[0] 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: listing bilog for incremental sync; position=00000000001.164.6 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: a sync operation returned error 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:entry[obj00000000000000000029]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000029[0] 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: listing bilog for incremental sync; position=00000000001.56.6 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: backing out with sync_status=-5 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: a sync operation returned error 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: backing out with sync_status=-5 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.520+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:17.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000050]: success 2023-09-29T22:34:17.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28:5 marker=00000000002.68.6 timestamp=2023-09-29T22:34:14.753202+0000 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:entry[obj00000000000000000423]: success 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20:5 marker=00000000001.52.6 timestamp=2023-09-29T22:34:14.812873+0000 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.524+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000001.83.6 iter->object=obj00000000000000000271 iter->instance= name=obj00000000000000000271 instance= ns= 2023-09-29T22:34:17.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000271 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:14.133098+0000 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.83.6 op=0 op_state=1 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000002.85.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000009 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:14.888053+0000 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.85.6 op=0 op_state=1 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000271[0] 2023-09-29T22:34:17.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000001.83.6 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000009[0] 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000002.85.6 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket shard: 0 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket shard: 0 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:entry[obj00000000000000000029]: success 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29:5 marker=00000000001.56.6 timestamp=2023-09-29T22:34:14.353418+0000 2023-09-29T22:34:17.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000001.31.6 iter->object=obj00000000000000000056 iter->instance= name=obj00000000000000000056 instance= ns= 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000056 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:14.147844+0000 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.31.6 op=0 op_state=1 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000056[0] 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000001.31.6 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000009]: success 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11:5 marker=00000000002.85.6 timestamp=2023-09-29T22:34:14.888053+0000 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: parsed entry: id=00000000002.31.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19/obj00000000000000000003 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: entry->timestamp=2023-09-29T22:34:14.530970+0000 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.31.6 op=0 op_state=1 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000003[0] 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position=00000000002.31.6 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24/obj00000000000000000011 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: entry->timestamp=2023-09-29T22:34:14.220614+0000 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.528+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-09-29T22:34:17.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.532+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000011[0] 2023-09-29T22:34:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.532+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position=00000000002.69.6 2023-09-29T22:34:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.532+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:entry[obj00000000000000000133]: success 2023-09-29T22:34:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.532+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30:5 marker=00000000001.164.6 timestamp=2023-09-29T22:34:14.832671+0000 2023-09-29T22:34:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.536+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000011]: success 2023-09-29T22:34:17.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.536+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24:5 marker=00000000002.69.6 timestamp=2023-09-29T22:34:14.220614+0000 2023-09-29T22:34:17.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.536+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000003]: success 2023-09-29T22:34:17.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.536+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19:5 marker=00000000002.31.6 timestamp=2023-09-29T22:34:14.530970+0000 2023-09-29T22:34:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.540+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000056]: success 2023-09-29T22:34:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.540+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4:5 marker=00000000001.31.6 timestamp=2023-09-29T22:34:14.147844+0000 2023-09-29T22:34:17.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.540+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000271]: success 2023-09-29T22:34:17.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.540+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12:5 marker=00000000001.83.6 timestamp=2023-09-29T22:34:14.133098+0000 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22/obj00000000000000000012 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: entry->timestamp=2023-09-29T22:34:14.733253+0000 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000012]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000012[0] 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position=00000000002.93.6 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: parsed entry: id=00000000001.59.6 iter->object=obj00000000000000000061 iter->instance= name=obj00000000000000000061 instance= ns= 2023-09-29T22:34:17.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27/obj00000000000000000061 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: entry->timestamp=2023-09-29T22:34:14.656225+0000 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000061]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.59.6 op=0 op_state=1 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: parsed entry: id=00000000002.74.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26/obj00000000000000000010 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: entry->timestamp=2023-09-29T22:34:14.083719+0000 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.74.6 op=0 op_state=1 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000061]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000061[0] 2023-09-29T22:34:17.554 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: listing bilog for incremental sync; position=00000000001.59.6 2023-09-29T22:34:17.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:entry[obj00000000000000000010]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000010[0] 2023-09-29T22:34:17.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: listing bilog for incremental sync; position=00000000002.74.6 2023-09-29T22:34:17.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: a sync operation returned error 2023-09-29T22:34:17.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: backing out with sync_status=-5 2023-09-29T22:34:17.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.552+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.556+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000012]: success 2023-09-29T22:34:17.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.556+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22:5 marker=00000000002.93.6 timestamp=2023-09-29T22:34:14.733253+0000 2023-09-29T22:34:17.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.556+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:entry[obj00000000000000000010]: success 2023-09-29T22:34:17.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.556+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26:5 marker=00000000002.74.6 timestamp=2023-09-29T22:34:14.083719+0000 2023-09-29T22:34:17.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.556+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000162]: success 2023-09-29T22:34:17.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.556+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17:5 marker=00000000001.88.6 timestamp=2023-09-29T22:34:14.673477+0000 2023-09-29T22:34:17.561 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.560+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.564+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: a sync operation returned error 2023-09-29T22:34:17.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.564+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: backing out with sync_status=-5 2023-09-29T22:34:17.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.564+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.564+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-29T22:34:17.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000005 2023-09-29T22:34:17.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: entry->timestamp=2023-09-29T22:34:14.508820+0000 2023-09-29T22:34:17.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-09-29T22:34:17.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000005[0] 2023-09-29T22:34:17.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000002.71.6 2023-09-29T22:34:17.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:entry[obj00000000000000000018]: success 2023-09-29T22:34:17.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14:5 marker=00000000002.61.6 timestamp=2023-09-29T22:34:14.601425+0000 2023-09-29T22:34:17.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000005]: success 2023-09-29T22:34:17.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3:5 marker=00000000002.71.6 timestamp=2023-09-29T22:34:14.508820+0000 2023-09-29T22:34:17.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:17.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.571 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000283]: success 2023-09-29T22:34:17.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.568+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25:5 marker=00000000001.40.6 timestamp=2023-09-29T22:34:14.455477+0000 2023-09-29T22:34:17.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000002.64.6 iter->object=obj00000000000000000006 iter->instance= name=obj00000000000000000006 instance= ns= 2023-09-29T22:34:17.573 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000006 2023-09-29T22:34:17.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:14.431088+0000 2023-09-29T22:34:17.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.64.6 op=0 op_state=1 2023-09-29T22:34:17.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000006[0] 2023-09-29T22:34:17.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000002.64.6 2023-09-29T22:34:17.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.575 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000006]: success 2023-09-29T22:34:17.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.572+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1:5 marker=00000000002.64.6 timestamp=2023-09-29T22:34:14.431088+0000 2023-09-29T22:34:17.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:14.011338+0000 2023-09-29T22:34:17.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-09-29T22:34:17.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.577 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: a sync operation returned error 2023-09-29T22:34:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: backing out with sync_status=-5 2023-09-29T22:34:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.576+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.584+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.586 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.584+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.584+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:14.011338+0000 2023-09-29T22:34:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: : injecting data sync error on key=obj00000000000000000002 2023-09-29T22:34:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.589 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: a sync operation returned error 2023-09-29T22:34:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: backing out with sync_status=-5 2023-09-29T22:34:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.588+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.596+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.596+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000002 2023-09-29T22:34:17.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:14.011338+0000 2023-09-29T22:34:17.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000002[0] 2023-09-29T22:34:17.603 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.600+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.604+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000002]: success 2023-09-29T22:34:17.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.604+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21:5 marker=00000000002.46.6 timestamp=2023-09-29T22:34:14.011338+0000 2023-09-29T22:34:17.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.604+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000061]: success 2023-09-29T22:34:17.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.604+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27:5 marker=00000000001.59.6 timestamp=2023-09-29T22:34:14.656225+0000 2023-09-29T22:34:17.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.753 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: sync status for source bucket shard: 0 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: sync status for source bucket shard: 0 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: sync status for source bucket shard: 0 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: sync status for source bucket shard: 0 2023-09-29T22:34:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: sync status for source bucket shard: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: sync status for source bucket shard: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.752+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: sync status for source bucket shard: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: sync status for source bucket shard: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: sync status for source bucket shard: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: sync status for source bucket shard: 0 2023-09-29T22:34:17.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: parsed entry: id=00000000002.69.6 iter->object=obj00000000000000000011 iter->instance= name=obj00000000000000000011 instance= ns= 2023-09-29T22:34:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2/obj00000000000000000011 2023-09-29T22:34:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: entry->timestamp=2023-09-29T22:34:15.976923+0000 2023-09-29T22:34:17.762 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000011]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.69.6 op=0 op_state=1 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: parsed entry: id=00000000002.37.6 iter->object=obj00000000000000000033 iter->instance= name=obj00000000000000000033 instance= ns= 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4/obj00000000000000000033 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: entry->timestamp=2023-09-29T22:34:15.581480+0000 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000033]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.37.6 op=0 op_state=1 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: parsed entry: id=00000000002.34.6 iter->object=obj00000000000000000000 iter->instance= name=obj00000000000000000000 instance= ns= 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1/obj00000000000000000000 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: entry->timestamp=2023-09-29T22:34:15.882165+0000 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000000]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.34.6 op=0 op_state=1 2023-09-29T22:34:17.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: parsed entry: id=00000000001.68.6 iter->object=obj00000000000000000060 iter->instance= name=obj00000000000000000060 instance= ns= 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0/obj00000000000000000060 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: entry->timestamp=2023-09-29T22:34:15.370719+0000 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000060]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.68.6 op=0 op_state=1 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: parsed entry: id=00000000002.61.6 iter->object=obj00000000000000000001 iter->instance= name=obj00000000000000000001 instance= ns= 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3/obj00000000000000000001 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: entry->timestamp=2023-09-29T22:34:16.117191+0000 2023-09-29T22:34:17.764 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000001]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.61.6 op=0 op_state=1 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000011]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000011[0] 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: listing bilog for incremental sync; position=00000000002.69.6 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000033]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000033[0] 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: listing bilog for incremental sync; position=00000000002.37.6 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000000]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000000[0] 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: listing bilog for incremental sync; position=00000000002.34.6 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000060]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000060[0] 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: listing bilog for incremental sync; position=00000000001.68.6 2023-09-29T22:34:17.765 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000001]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000001[0] 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: listing bilog for incremental sync; position=00000000002.61.6 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: sync status for source bucket shard: 0 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: sync status for source bucket shard: 0 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: sync status for source bucket shard: 0 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: sync status for source bucket shard: 0 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.756+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: parsed entry: id=00000000001.58.6 iter->object=obj00000000000000000065 iter->instance= name=obj00000000000000000065 instance= ns= 2023-09-29T22:34:17.766 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5/obj00000000000000000065 2023-09-29T22:34:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: entry->timestamp=2023-09-29T22:34:15.810152+0000 2023-09-29T22:34:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000065]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.58.6 op=0 op_state=1 2023-09-29T22:34:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.767 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.771 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000065]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000065[0] 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: listing bilog for incremental sync; position=00000000001.58.6 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: parsed entry: id=00000000002.107.6 iter->object=obj00000000000000000005 iter->instance= name=obj00000000000000000005 instance= ns= 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6/obj00000000000000000005 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: entry->timestamp=2023-09-29T22:34:16.215141+0000 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000005]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.107.6 op=0 op_state=1 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: sync status for source bucket shard: 0 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: sync status for source bucket shard: 0 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000005]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000005[0] 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.760+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: listing bilog for incremental sync; position=00000000002.107.6 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: parsed entry: id=00000000002.30.6 iter->object=obj00000000000000000016 iter->instance= name=obj00000000000000000016 instance= ns= 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7/obj00000000000000000016 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: entry->timestamp=2023-09-29T22:34:15.903767+0000 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000016]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.30.6 op=0 op_state=1 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.773 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: parsed entry: id=00000000001.64.6 iter->object=obj00000000000000000048 iter->instance= name=obj00000000000000000048 instance= ns= 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8/obj00000000000000000048 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: entry->timestamp=2023-09-29T22:34:16.493505+0000 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000048]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.64.6 op=0 op_state=1 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:14]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000016]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000016[0] 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: listing bilog for incremental sync; position=00000000002.30.6 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.774 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: sync status for source bucket shard: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000048]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000048[0] 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: listing bilog for incremental sync; position=00000000001.64.6 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: sync status for source bucket shard: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.764+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:entry[obj00000000000000000033]: success 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:4:6 marker=00000000002.37.6 timestamp=2023-09-29T22:34:15.581480+0000 2023-09-29T22:34:17.776 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:entry[obj00000000000000000000]: success 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:1:6 marker=00000000002.34.6 timestamp=2023-09-29T22:34:15.882165+0000 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: parsed entry: id=00000000002.79.6 iter->object=obj00000000000000000015 iter->instance= name=obj00000000000000000015 instance= ns= 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9/obj00000000000000000015 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: entry->timestamp=2023-09-29T22:34:15.602423+0000 2023-09-29T22:34:17.777 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000015]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.79.6 op=0 op_state=1 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:entry[obj00000000000000000011]: success 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:2:6 marker=00000000002.69.6 timestamp=2023-09-29T22:34:15.976923+0000 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:entry[obj00000000000000000001]: success 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:3:6 marker=00000000002.61.6 timestamp=2023-09-29T22:34:16.117191+0000 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: parsed entry: id=00000000002.105.6 iter->object=obj00000000000000000030 iter->instance= name=obj00000000000000000030 instance= ns= 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11/obj00000000000000000030 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: entry->timestamp=2023-09-29T22:34:16.037992+0000 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000030]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.105.6 op=0 op_state=1 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: parsed entry: id=00000000002.39.6 iter->object=obj00000000000000000026 iter->instance= name=obj00000000000000000026 instance= ns= 2023-09-29T22:34:17.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10/obj00000000000000000026 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: entry->timestamp=2023-09-29T22:34:16.436274+0000 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000026]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.39.6 op=0 op_state=1 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: sync status for source bucket shard: 0 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: sync status for source bucket shard: 0 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000015]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000015[0] 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: listing bilog for incremental sync; position=00000000002.79.6 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: sync status for source bucket shard: 0 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000030]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000030[0] 2023-09-29T22:34:17.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: listing bilog for incremental sync; position=00000000002.105.6 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000026]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000026[0] 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: listing bilog for incremental sync; position=00000000002.39.6 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: sync status for source bucket shard: 0 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:20]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.780 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.768+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: parsed entry: id=00000000001.67.6 iter->object=obj00000000000000000062 iter->instance= name=obj00000000000000000062 instance= ns= 2023-09-29T22:34:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12/obj00000000000000000062 2023-09-29T22:34:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: entry->timestamp=2023-09-29T22:34:16.059492+0000 2023-09-29T22:34:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000062]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.67.6 op=0 op_state=1 2023-09-29T22:34:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000062]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000062[0] 2023-09-29T22:34:17.785 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: listing bilog for incremental sync; position=00000000001.67.6 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: sync status for source bucket shard: 0 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: sync status for source bucket shard: 0 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: sync status for source bucket shard: 0 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: sync status for source bucket shard: 0 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: sync status for source bucket shard: 0 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.772+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: sync status for source bucket shard: 0 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.786 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:26]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: sync status for source bucket shard: 0 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: sync status for source bucket shard: 0 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: parsed entry: id=00000000002.71.6 iter->object=obj00000000000000000024 iter->instance= name=obj00000000000000000024 instance= ns= 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15/obj00000000000000000024 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: entry->timestamp=2023-09-29T22:34:15.957248+0000 2023-09-29T22:34:17.787 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000024]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.71.6 op=0 op_state=1 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:entry[obj00000000000000000005]: success 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:6:6 marker=00000000002.107.6 timestamp=2023-09-29T22:34:16.215141+0000 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: parsed entry: id=00000000002.170.6 iter->object=obj00000000000000000013 iter->instance= name=obj00000000000000000013 instance= ns= 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18/obj00000000000000000013 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: entry->timestamp=2023-09-29T22:34:15.656305+0000 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000013]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.170.6 op=0 op_state=1 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: sync status for source bucket shard: 0 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: sync status for source bucket shard: 0 2023-09-29T22:34:17.788 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: sync status for source bucket shard: 0 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: sync status for source bucket shard: 0 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: sync status for source bucket shard: 0 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: sync status for source bucket shard: 0 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000012 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:15.446261+0000 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: parsed entry: id=00000000002.111.6 iter->object=obj00000000000000000002 iter->instance= name=obj00000000000000000002 instance= ns= 2023-09-29T22:34:17.789 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17/obj00000000000000000002 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: entry->timestamp=2023-09-29T22:34:16.569814+0000 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000002]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.111.6 op=0 op_state=1 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000024]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000024[0] 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.776+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: listing bilog for incremental sync; position=00000000002.71.6 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000013]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000013[0] 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: listing bilog for incremental sync; position=00000000002.170.6 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.790 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:entry[obj00000000000000000016]: success 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:7:6 marker=00000000002.30.6 timestamp=2023-09-29T22:34:15.903767+0000 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: : injecting data sync error on key=obj00000000000000000012 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.68.6 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000002]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000002[0] 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: listing bilog for incremental sync; position=00000000002.111.6 2023-09-29T22:34:17.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000012 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:entry[obj00000000000000000015]: success 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:9:6 marker=00000000002.79.6 timestamp=2023-09-29T22:34:15.602423+0000 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000017 iter->instance= name=obj00000000000000000017 instance= ns= 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21/obj00000000000000000017 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: entry->timestamp=2023-09-29T22:34:15.750403+0000 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000017]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: parsed entry: id=00000000002.24.6 iter->object=obj00000000000000000045 iter->instance= name=obj00000000000000000045 instance= ns= 2023-09-29T22:34:17.792 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19/obj00000000000000000045 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: entry->timestamp=2023-09-29T22:34:15.728963+0000 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000045]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.24.6 op=0 op_state=1 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:entry[obj00000000000000000030]: success 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:11:6 marker=00000000002.105.6 timestamp=2023-09-29T22:34:16.037992+0000 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000017]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000017[0] 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: listing bilog for incremental sync; position=00000000002.59.6 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:entry[obj00000000000000000065]: success 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:5:6 marker=00000000001.58.6 timestamp=2023-09-29T22:34:15.810152+0000 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:entry[obj00000000000000000026]: success 2023-09-29T22:34:17.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:10:6 marker=00000000002.39.6 timestamp=2023-09-29T22:34:16.436274+0000 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000045]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000045[0] 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.780+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: listing bilog for incremental sync; position=00000000002.24.6 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:entry[obj00000000000000000060]: success 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:0:6 marker=00000000001.68.6 timestamp=2023-09-29T22:34:15.370719+0000 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:entry[obj00000000000000000048]: success 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:8:6 marker=00000000001.64.6 timestamp=2023-09-29T22:34:16.493505+0000 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000020 iter->instance= name=obj00000000000000000020 instance= ns= 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28/obj00000000000000000020 2023-09-29T22:34:17.794 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: entry->timestamp=2023-09-29T22:34:16.591245+0000 2023-09-29T22:34:17.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000020]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-09-29T22:34:17.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: parsed entry: id=00000000002.73.6 iter->object=obj00000000000000000003 iter->instance= name=obj00000000000000000003 instance= ns= 2023-09-29T22:34:17.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31/obj00000000000000000003 2023-09-29T22:34:17.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: entry->timestamp=2023-09-29T22:34:16.511366+0000 2023-09-29T22:34:17.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:entry[obj00000000000000000003]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.73.6 op=0 op_state=1 2023-09-29T22:34:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000020]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000020[0] 2023-09-29T22:34:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: listing bilog for incremental sync; position=00000000002.93.6 2023-09-29T22:34:17.798 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:entry[obj00000000000000000003]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000003[0] 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: listing bilog for incremental sync; position=00000000002.73.6 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: parsed entry: id=00000000002.40.6 iter->object=obj00000000000000000004 iter->instance= name=obj00000000000000000004 instance= ns= 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29/obj00000000000000000004 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: entry->timestamp=2023-09-29T22:34:16.195556+0000 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:entry[obj00000000000000000004]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.40.6 op=0 op_state=1 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000014 iter->instance= name=obj00000000000000000014 instance= ns= 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32/obj00000000000000000014 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: entry->timestamp=2023-09-29T22:34:15.522529+0000 2023-09-29T22:34:17.799 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.784+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:entry[obj00000000000000000014]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:entry[obj00000000000000000004]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000004[0] 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: listing bilog for incremental sync; position=00000000002.40.6 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:entry[obj00000000000000000014]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000014[0] 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35/obj00000000000000000018 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: entry->timestamp=2023-09-29T22:34:16.646366+0000 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: parsed entry: id=00000000002.93.6 iter->object=obj00000000000000000053 iter->instance= name=obj00000000000000000053 instance= ns= 2023-09-29T22:34:17.800 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27/obj00000000000000000053 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: entry->timestamp=2023-09-29T22:34:16.418633+0000 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000053]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.93.6 op=0 op_state=1 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: parsed entry: id=00000000002.46.6 iter->object=obj00000000000000000010 iter->instance= name=obj00000000000000000010 instance= ns= 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25/obj00000000000000000010 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: entry->timestamp=2023-09-29T22:34:15.676466+0000 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000010]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.46.6 op=0 op_state=1 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: parsed entry: id=00000000001.35.6 iter->object=obj00000000000000000036 iter->instance= name=obj00000000000000000036 instance= ns= 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30/obj00000000000000000036 2023-09-29T22:34:17.801 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: entry->timestamp=2023-09-29T22:34:16.348566+0000 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:entry[obj00000000000000000036]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.35.6 op=0 op_state=1 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: parsed entry: id=00000000001.69.6 iter->object=obj00000000000000000057 iter->instance= name=obj00000000000000000057 instance= ns= 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34/obj00000000000000000057 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.788+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: entry->timestamp=2023-09-29T22:34:15.505043+0000 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:entry[obj00000000000000000057]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000001.69.6 op=0 op_state=1 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: parsed entry: id=00000000002.76.6 iter->object=obj00000000000000000008 iter->instance= name=obj00000000000000000008 instance= ns= 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36/obj00000000000000000008 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: entry->timestamp=2023-09-29T22:34:16.294909+0000 2023-09-29T22:34:17.802 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:entry[obj00000000000000000008]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.76.6 op=0 op_state=1 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: : injecting data sync error on key=obj00000000000000000018 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: listing bilog for incremental sync; position=00000000002.78.6 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000053]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000053[0] 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: listing bilog for incremental sync; position=00000000002.93.6 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000010]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000010[0] 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: listing bilog for incremental sync; position=00000000002.46.6 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:entry[obj00000000000000000024]: success 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:15:6 marker=00000000002.71.6 timestamp=2023-09-29T22:34:15.957248+0000 2023-09-29T22:34:17.803 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:entry[obj00000000000000000036]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000036[0] 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: listing bilog for incremental sync; position=00000000001.35.6 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:entry[obj00000000000000000057]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000057[0] 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: listing bilog for incremental sync; position=00000000001.69.6 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:entry[obj00000000000000000008]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000008[0] 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: listing bilog for incremental sync; position=00000000002.76.6 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 0 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: ERROR: failed to sync object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35/obj00000000000000000018 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:entry[obj00000000000000000017]: success 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:21:6 marker=00000000002.59.6 timestamp=2023-09-29T22:34:15.750403+0000 2023-09-29T22:34:17.804 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: parsed entry: id=00000000002.109.6 iter->object=obj00000000000000000007 iter->instance= name=obj00000000000000000007 instance= ns= 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22/obj00000000000000000007 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: entry->timestamp=2023-09-29T22:34:16.364164+0000 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000007]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.109.6 op=0 op_state=1 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: parsed entry: id=00000000002.50.6 iter->object=obj00000000000000000019 iter->instance= name=obj00000000000000000019 instance= ns= 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33/obj00000000000000000019 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: entry->timestamp=2023-09-29T22:34:16.273751+0000 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:entry[obj00000000000000000019]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.50.6 op=0 op_state=1 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:entry[obj00000000000000000013]: success 2023-09-29T22:34:17.805 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:18:6 marker=00000000002.170.6 timestamp=2023-09-29T22:34:15.656305+0000 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000007]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000007[0] 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: listing bilog for incremental sync; position=00000000002.109.6 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:entry[obj00000000000000000045]: success 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:19:6 marker=00000000002.24.6 timestamp=2023-09-29T22:34:15.728963+0000 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:entry[obj00000000000000000002]: success 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:17:6 marker=00000000002.111.6 timestamp=2023-09-29T22:34:16.569814+0000 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:entry[obj00000000000000000019]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000019[0] 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: listing bilog for incremental sync; position=00000000002.50.6 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: a sync operation returned error 2023-09-29T22:34:17.806 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: backing out with sync_status=-5 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.792+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:entry[obj00000000000000000003]: success 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:31:6 marker=00000000002.73.6 timestamp=2023-09-29T22:34:16.511366+0000 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:entry[obj00000000000000000004]: success 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:29:6 marker=00000000002.40.6 timestamp=2023-09-29T22:34:16.195556+0000 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:entry[obj00000000000000000020]: success 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:28:6 marker=00000000002.93.6 timestamp=2023-09-29T22:34:16.591245+0000 2023-09-29T22:34:17.807 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:entry[obj00000000000000000014]: success 2023-09-29T22:34:17.808 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:32:6 marker=00000000002.46.6 timestamp=2023-09-29T22:34:15.522529+0000 2023-09-29T22:34:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: a sync operation returned error 2023-09-29T22:34:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: backing out with sync_status=-5 2023-09-29T22:34:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: incremental sync on bucket failed, retcode=-5 2023-09-29T22:34:17.809 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.796+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: ERROR: incremental sync failed. error: -5 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: sync status for source bucket shard: 0 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: sync status for source bucket shard: 0 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: listing bilog for incremental sync; position= 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: parsed entry: id=00000000002.78.6 iter->object=obj00000000000000000018 iter->instance= name=obj00000000000000000018 instance= ns= 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35/obj00000000000000000018 2023-09-29T22:34:17.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: entry->timestamp=2023-09-29T22:34:16.646366+0000 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.78.6 op=0 op_state=1 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000018[0] 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.800+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: listing bilog for incremental sync; position=00000000002.78.6 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.808+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: parsed entry: id=00000000002.38.6 iter->object=obj00000000000000000009 iter->instance= name=obj00000000000000000009 instance= ns= 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.808+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13/obj00000000000000000009 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.808+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: entry->timestamp=2023-09-29T22:34:15.425177+0000 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.808+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000009]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.38.6 op=0 op_state=1 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.808+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000009]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000009[0] 2023-09-29T22:34:17.811 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.808+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: listing bilog for incremental sync; position=00000000002.38.6 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: parsed entry: id=00000000002.48.6 iter->object=obj00000000000000000039 iter->instance= name=obj00000000000000000039 instance= ns= 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23/obj00000000000000000039 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: entry->timestamp=2023-09-29T22:34:16.133920+0000 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:entry[obj00000000000000000039]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.48.6 op=0 op_state=1 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:entry[obj00000000000000000039]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000039[0] 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: listing bilog for incremental sync; position=00000000002.48.6 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: parsed entry: id=00000000002.59.6 iter->object=obj00000000000000000027 iter->instance= name=obj00000000000000000027 instance= ns= 2023-09-29T22:34:17.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24/obj00000000000000000027 2023-09-29T22:34:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: entry->timestamp=2023-09-29T22:34:15.827069+0000 2023-09-29T22:34:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000027]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.59.6 op=0 op_state=1 2023-09-29T22:34:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000027]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000027[0] 2023-09-29T22:34:17.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.824+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: listing bilog for incremental sync; position=00000000002.59.6 2023-09-29T22:34:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.840+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:entry[obj00000000000000000019]: success 2023-09-29T22:34:17.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:33:6 marker=00000000002.50.6 timestamp=2023-09-29T22:34:16.273751+0000 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.840+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:entry[obj00000000000000000007]: success 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:22:6 marker=00000000002.109.6 timestamp=2023-09-29T22:34:16.364164+0000 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.840+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:entry[obj00000000000000000010]: success 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.840+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:25:6 marker=00000000002.46.6 timestamp=2023-09-29T22:34:15.676466+0000 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:entry[obj00000000000000000008]: success 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:36:6 marker=00000000002.76.6 timestamp=2023-09-29T22:34:16.294909+0000 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: parsed entry: id=00000000002.68.6 iter->object=obj00000000000000000012 iter->instance= name=obj00000000000000000012 instance= ns= 2023-09-29T22:34:17.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: syncing object: hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16/obj00000000000000000012 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: entry->timestamp=2023-09-29T22:34:15.446261+0000 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=00000000002.68.6 op=0 op_state=1 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:entry[obj00000000000000000009]: success 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:13:6 marker=00000000002.38.6 timestamp=2023-09-29T22:34:15.425177+0000 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:entry[obj00000000000000000039]: success 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:23:6 marker=00000000002.48.6 timestamp=2023-09-29T22:34:16.133920+0000 2023-09-29T22:34:17.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:entry[obj00000000000000000027]: success 2023-09-29T22:34:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:24:6 marker=00000000002.59.6 timestamp=2023-09-29T22:34:15.827069+0000 2023-09-29T22:34:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 5 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-58[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23])/obj00000000000000000012[0] 2023-09-29T22:34:17.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: listing bilog for incremental sync; position=00000000002.68.6 2023-09-29T22:34:17.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:entry[obj00000000000000000018]: success 2023-09-29T22:34:17.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.844+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:35:6 marker=00000000002.78.6 timestamp=2023-09-29T22:34:16.646366+0000 2023-09-29T22:34:17.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:entry[obj00000000000000000053]: success 2023-09-29T22:34:17.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:27:6 marker=00000000002.93.6 timestamp=2023-09-29T22:34:16.418633+0000 2023-09-29T22:34:17.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:entry[obj00000000000000000012]: success 2023-09-29T22:34:17.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:16:6 marker=00000000002.68.6 timestamp=2023-09-29T22:34:15.446261+0000 2023-09-29T22:34:17.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:entry[obj00000000000000000062]: success 2023-09-29T22:34:17.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:12:6 marker=00000000001.67.6 timestamp=2023-09-29T22:34:16.059492+0000 2023-09-29T22:34:17.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:entry[obj00000000000000000036]: success 2023-09-29T22:34:17.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:30:6 marker=00000000001.35.6 timestamp=2023-09-29T22:34:16.348566+0000 2023-09-29T22:34:17.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.852+0000 7f859b6bdf40 10 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:entry[obj00000000000000000057]: success 2023-09-29T22:34:17.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:17.856+0000 7f859b6bdf40 20 RGW-SYNC:bucket[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23<-hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]:inc_sync[hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34]: updating marker marker_oid=bucket.sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23:34:6 marker=00000000001.69.6 timestamp=2023-09-29T22:34:15.505043+0000 2023-09-29T22:34:17.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:17.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:17.886 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:17.976 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:17.976 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:18.039 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:34:18.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.039 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000000", 2023-09-29T22:34:18.039 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.039 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.882165Z", 2023-09-29T22:34:18.040 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4159146924459810763", 2023-09-29T22:34:18.041 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000001", 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:34:18.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.117191Z", 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.043 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.044 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4297483464031893538", 2023-09-29T22:34:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.045 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000002", 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.046 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.569814Z", 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.047 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.048 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.1502651821700798080", 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.049 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000003", 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.050 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.511366Z", 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.051 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.8096105190604402782", 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000004", 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.053 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.195556Z", 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.054 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.8039822156540950427", 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.055 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000005", 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.056 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.215141Z", 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.057 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15314044673247211149", 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.058 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000006", 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20 2023-09-29T22:34:18.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.431088Z", 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.060 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.061 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.9100429252980955470", 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000007", 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.063 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.364164Z", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.064 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.6457434699585611483", 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.065 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000008", 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 23 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.066 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.294909Z", 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.067 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.6661828877086135036", 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000009", 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.069 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.425177Z", 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.070 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.12646265209379619810", 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000010", 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 22 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.072 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.676466Z", 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.073 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4209200776973891688", 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.074 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000011", 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.075 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.976923Z", 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.076 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.9968736344712390069", 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000012", 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-09-29T22:34:18.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.446261Z", 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.079 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.13528096859529770838", 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.080 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.081 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.081 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000013", 2023-09-29T22:34:18.081 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.081 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.082 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.656305Z", 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.083 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.12040285541575396170", 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.084 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000014", 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 19 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.085 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.522529Z", 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.086 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11185040271307583727", 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.087 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000015", 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.088 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.602423Z", 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.089 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15789729280136132316", 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.090 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000016", 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.091 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.903767Z", 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.092 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.13971571779289837299", 2023-09-29T22:34:18.093 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000017", 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.094 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 16 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.750403Z", 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.095 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.9802026135793335022", 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000018", 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.097 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.646366Z", 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.098 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.12297658963279759351", 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000019", 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.100 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.273751Z", 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.101 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16327054898211569257", 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000020", 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.103 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.591245Z", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.104 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4664077706507320344", 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000021", 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.105 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.064728Z", 2023-09-29T22:34:18.106 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.13854796586652278710", 2023-09-29T22:34:18.107 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000022", 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.108 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.297905Z", 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.109 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5305454922783090731", 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.110 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000023", 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 16 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.111 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.864080Z", 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.112 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16105486663514758089", 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000024", 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.114 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.957248Z", 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.115 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11663125145895475607", 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.116 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000025", 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.117 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.396957Z", 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.118 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.18252468435156836049", 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000026", 2023-09-29T22:34:18.119 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.120 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.436274Z", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.121 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11253318368869549175", 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000027", 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.122 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.123 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.827069Z", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.7872284275177270132", 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000028", 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.125 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:09.550964Z", 2023-09-29T22:34:18.126 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15533240654388460989", 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.127 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000029", 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.128 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.353418Z", 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.129 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10403560088962014579", 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000030", 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.131 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.037992Z", 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.132 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.951221072754109841", 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000031", 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.134 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:12.919016Z", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.135 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16711473624230520117", 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000032", 2023-09-29T22:34:18.136 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 35 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.137 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.833061Z", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.1750998837288050268", 2023-09-29T22:34:18.138 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000033", 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.581480Z", 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.140 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.2421484234638249776", 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.141 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000036", 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.142 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.348566Z", 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.143 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5797019855285328781", 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.144 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000037", 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.145 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.095565Z", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.146 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10585615494298954380", 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000039", 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.147 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.133920Z", 2023-09-29T22:34:18.148 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.12419438952968330108", 2023-09-29T22:34:18.149 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000040", 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.150 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.404418Z", 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.151 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5917131179139322277", 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000042", 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.153 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.167841Z", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.154 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.937829008670825346", 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000045", 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.155 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.728963Z", 2023-09-29T22:34:18.156 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.13230338452696510794", 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.157 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000046", 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.158 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.534435Z", 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.159 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.13112206553025180442", 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.160 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000048", 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.493505Z", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.844784559335560985", 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000050", 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.753202Z", 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10726488012861152009", 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000053", 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.418633Z", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.460253465355406900", 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000055", 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.475030Z", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.170 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11488523768318863893", 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000056", 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.171 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.147844Z", 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.172 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.7144638933428573025", 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.173 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000057", 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.174 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.505043Z", 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.175 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.8356580874283367811", 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000058", 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.177 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.934857Z", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.178 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.3314271682622620947", 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000059", 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.179 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.245600Z", 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.180 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.6560773699153076880", 2023-09-29T22:34:18.181 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000060", 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.182 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.370719Z", 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.183 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.3864092401087083831", 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000061", 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.185 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.656225Z", 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.186 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15041457370482646452", 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000062", 2023-09-29T22:34:18.187 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.188 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.059492Z", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16132169852717517914", 2023-09-29T22:34:18.189 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000065", 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.190 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.810152Z", 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.191 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.162316029999365252", 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.192 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000076", 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.193 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:09.827103Z", 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.194 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15153924876928760274", 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.195 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000133", 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 17 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.196 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.832671Z", 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.197 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.6140171211861699643", 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.198 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000162", 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 17 2023-09-29T22:34:18.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.673477Z", 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.200 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.7959403659092314205", 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.201 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000271", 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:34:18.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.133098Z", 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.203 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4856065144672473898", 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.204 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000283", 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.206 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.455477Z", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.207 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11089737652230823210", 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000317", 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.208 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.992396Z", 2023-09-29T22:34:18.209 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5852940490057318768", 2023-09-29T22:34:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.211 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000423", 2023-09-29T22:34:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.212 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.213 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.812873Z", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4700418031300289492", 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000467", 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.584478Z", 2023-09-29T22:34:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.8497254225063518796", 2023-09-29T22:34:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:18.218 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:34:18.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:18.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-58', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:18.219 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-58 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:18.269 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:18.269 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:18.333 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:34:18.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.333 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000000", 2023-09-29T22:34:18.333 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.333 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.334 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.882165Z", 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.335 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_yvwIzV-NKOssFFhbL409QNBnCItTDDG", 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000001", 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.337 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.117191Z", 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.338 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_EHcOg3OOJtz-P62V928n0aUBYFTgXSJ", 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.339 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000002", 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.569814Z", 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.341 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.342 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_1FjV_rgAmR2xuUzytiAa9wJts3sCuPq", 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000003", 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.343 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.344 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.511366Z", 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.345 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Rgg7CyMUew6gpuDdjpq1AQmYElRj0HX", 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000004", 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.347 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.195556Z", 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.348 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CeqRuRrg3LXyNc6va1bNPKQQHjMhfmf", 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.349 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000005", 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.215141Z", 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.351 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Tiro-UzVGxFvqm4GkUxzyNJi1khp5TF", 2023-09-29T22:34:18.352 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000006", 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.354 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.431088Z", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_7pqEndsiOqBxQuDlzK3GkBGUHWCuO3Y", 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000007", 2023-09-29T22:34:18.356 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.357 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.364164Z", 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.358 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_-wXxOqSjxsVR0_5FHxx6t3AEVG4Znwe", 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000008", 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.360 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.294909Z", 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.361 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_UfK7EIvpvh-vxMf27VavlJxD0cU6tJU", 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.362 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000009", 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.363 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.425177Z", 2023-09-29T22:34:18.364 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.365 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_gUuQ2Y-sV__cvM5HmQHNbHpwQW8jeRJ", 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000010", 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.366 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.367 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.676466Z", 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.368 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_LdQ3SiZl16mSVSE9yOaiottM6pG2IbU", 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.369 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000011", 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.370 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.976923Z", 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.371 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_IMCrrJsh-DrY2kVlYofc0XZQp1myhcY", 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.372 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000012", 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.373 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.446261Z", 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.374 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_9xy8h0ZAhdAoXDigITf69uqAqKhWf_d", 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.375 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000013", 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.376 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.656305Z", 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.377 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_USi-wluJLPjGPPtEOQn06MXf6MatsAf", 2023-09-29T22:34:18.378 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000014", 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.379 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.522529Z", 2023-09-29T22:34:18.380 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.381 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ZnAXAopaFLeQUENIBN_1gTjmoBsWMeP", 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000015", 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.382 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.383 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.602423Z", 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.384 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Zab9YCB-htLxCLt9goK9ThR21Xe_8Hj", 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000016", 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.386 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.903767Z", 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.387 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fx9w9OQ7gNTP58b_KWULzbDVL7s9CGa", 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.388 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000017", 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.389 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.750403Z", 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.390 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fy3hHqJ14BBfdea5VcWht0Y_VxJSiqi", 2023-09-29T22:34:18.391 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000018", 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.392 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.646366Z", 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.393 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.394 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Dd0zGSgX-Oq1LiW5szHOKdavdsRZuyv", 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000019", 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.395 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.396 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.273751Z", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.397 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_kAfLeOjdpd4rd_nZ7qCxgOEQuyxV1I-", 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.398 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000020", 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.399 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.591245Z", 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.400 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_B3nACzqNKEk7hzgJTiOQtQKh7Wr1rT8", 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.401 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.402 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000021", 2023-09-29T22:34:18.402 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.402 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.064728Z", 2023-09-29T22:34:18.403 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_HBxMJncuUgQt2DdFg4Zodo9ztyM7OOd", 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000022", 2023-09-29T22:34:18.405 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.406 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.297905Z", 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.407 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_3GmSnr0drd_OuuX6EdR5zhhWwo9uQf0", 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000023", 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.409 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.864080Z", 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.410 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Wagw2Mdy6YqyrJzTLNDvelGhk7ZGrvA", 2023-09-29T22:34:18.411 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000024", 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.412 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.413 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.957248Z", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.414 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_51eyMSwsjo8CZ5AXLPXAdsHsCPEUxlh", 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.415 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000025", 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.416 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.396957Z", 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.417 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_oyrjda7ods0Uq82M-kcX6J9txx0PRsW", 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000026", 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.419 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.436274Z", 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.420 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.421 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.421 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.421 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.421 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.421 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_9hGbEFx5hhHT6kDSwBgdZERGJfR1Kvu", 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.422 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000027", 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.423 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.827069Z", 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.424 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_aRDYE5dnapxXJPB8ci8HePhUXythRsq", 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.425 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000028", 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.426 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:09.550964Z", 2023-09-29T22:34:18.427 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Pva5ictFB9oMywFBt1KYRpSfUP3CkdR", 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000029", 2023-09-29T22:34:18.429 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.430 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.353418Z", 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.431 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Y7aSWiytNi_MN7o0dbyJ06dlkWKrrLp", 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000030", 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.433 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.037992Z", 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.434 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.435 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_j1JDxJk4zCGJToO-L9zRZQ_Cr-PhRVj", 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000031", 2023-09-29T22:34:18.436 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.437 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:12.919016Z", 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.438 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_G9FJW6HcSdiZIdlGuZemyMaszKRLJGs", 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000032", 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 33 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.440 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.833061Z", 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.441 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_q4g3eHDfet4YO_So3d8Mh8Zj5qfjLRg", 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.442 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000033", 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.443 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.581480Z", 2023-09-29T22:34:18.444 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_3gSqzRfg7nhIg1CTJIju1YkkA2x-8c_", 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.446 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000036", 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.447 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.348566Z", 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.448 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_mSJnn0AMDwdMi_9qz68-XyKnIKwMW5O", 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000037", 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.450 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.095565Z", 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.451 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_2LX8fKIUB_WzRIq4yy86cGhTGo3X5mm", 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.452 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000039", 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.453 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.133920Z", 2023-09-29T22:34:18.454 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.455 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ex_cjq45cuhGlrcOABIYFxat6NVFry5", 2023-09-29T22:34:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000040", 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.458 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.404418Z", 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.459 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_390LMnNnrltuncFutSQjRoyETApTuTO", 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000042", 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.461 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.167841Z", 2023-09-29T22:34:18.462 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_O7MdEyW2nPOtYhHC8VuOBFSZqw49MR-", 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000045", 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.464 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.465 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.728963Z", 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.466 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_YU__W0sE92MgZ_P9-YwDODLZCN6KdTz", 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000046", 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 17 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.468 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.534435Z", 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.469 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_P9Ni8hX0Pt2RFrNim6XLIYqVBXB8idv", 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000048", 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.493505Z", 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.472 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_TNsklRJwuEgkJGsrSpkgQ6hRzIH2KXH", 2023-09-29T22:34:18.473 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000050", 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.753202Z", 2023-09-29T22:34:18.475 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_5LqrYM78Ai88KA0viCEbUYcJ1NSoO0Y", 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000053", 2023-09-29T22:34:18.477 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.478 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.418633Z", 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.479 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_JtDQd4UHDBlRaUB9PH-zx77yt0mSAXJ", 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000055", 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.481 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.475030Z", 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.482 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_hH6FB04UfptRqCugjQaSGyosAqHdN3E", 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.483 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000056", 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.484 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.147844Z", 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.485 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_wPpb9ht_I4LiONs8d2vC_0rGbBKCTEY", 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000057", 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.487 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.505043Z", 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.488 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.489 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fUikYr7c-COLQDnzY2AMB8Kf0L9ynUN", 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000058", 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.490 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.491 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:11.934857Z", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.492 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_JzXNFOZ9Xiy9LdOSySZg8XXK6_tOUV-", 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000059", 2023-09-29T22:34:18.493 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.494 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.245600Z", 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.495 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CzFUxKtg3NMwLEB6WbmiJ5cA0ihPSTx", 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000060", 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.497 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.370719Z", 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.498 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_gK7vSUznXVpw6rAvSDTPY7Dy_ig-yLq", 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.499 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000061", 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.500 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.656225Z", 2023-09-29T22:34:18.501 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.502 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_omWBYIXz3hdzHb-9F3Eio-e_bfy1Z_O", 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000062", 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.503 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.504 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:16.059492Z", 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.505 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_XK4KwpZcuKOy99c_-EgjEb4SiO5jhNJ", 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000065", 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.507 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:15.810152Z", 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.508 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_FR55qH_sAkX8GggbrqfesaC6xiITOOz", 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.509 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000076", 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.510 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:09.827103Z", 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.511 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.512 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_lBrtdoM5jLPW6Cp2iWdE5cPCNMCArA4", 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000133", 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:34:18.514 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.832671Z", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.515 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_9Hy3-fm38TDvsRgluXwlFF0KL3AXq6l", 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000162", 2023-09-29T22:34:18.516 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.517 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.673477Z", 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.518 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_nUaphgE58XbWjgCmMtl5cRF0UK24y_C", 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.519 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000271", 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.521 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.133098Z", 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.522 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_ipAaRE2WoZrHpAHVc6nM0wHvKQI3KOc", 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000283", 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.455477Z", 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.525 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.526 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_baDjN0S7KCkNFN8DAf6IRe_SwnO_OUF", 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000317", 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.527 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.528 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:13.992396Z", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Qd48cQARdFjZvf-tmcbFzlaXaUUdpmB", 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.530 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000423", 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.531 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.812873Z", 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.532 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_H54kuokRBXe37dWkruv9m27wGBJ-9a7", 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj00000000000000000467", 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:34:18.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:34:14.584478Z", 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:34:18.535 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:18.536 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_2RIl0NGZYTqGQZeV3r3AlD7iQKx1uyV", 2023-09-29T22:34:18.537 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:34:18.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:34:18.537 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:34:18.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:18.537 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:34:18.538 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:34:18.538 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:34:18.541 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:34:18.541 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:18.559 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:34:18.559 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:34:18.559 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:34:18.559 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T22:34:18.562 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:34:19.563 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:19.581 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:34:19.581 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:34:19.584 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T22:34:19.585 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T22:34:19.585 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-58 2023-09-29T22:34:19.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:34:19.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:34:19.586 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-58 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:34:19.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:19.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:19.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:19.708+0000 7fe1668e2f40 1 bucket sync caught up with source: 2023-09-29T22:34:19.709 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, 00000000002.34.6, 00000000002.69.6, 00000000002.61.6, 00000000002.37.6, 00000000001.58.6, 00000000002.107.6, 00000000002.30.6, 00000000001.64.6, 00000000002.79.6, 00000000002.39.6, 00000000002.105.6, 00000000001.67.6, 00000000002.38.6, , 00000000002.71.6, 00000000002.68.6, 00000000002.111.6, 00000000002.170.6, 00000000002.24.6, , 00000000002.59.6, 00000000002.109.6, 00000000002.48.6, 00000000002.59.6, 00000000002.46.6, , 00000000002.93.6, 00000000002.93.6, 00000000002.40.6, 00000000001.35.6, 00000000002.73.6, 00000000002.46.6, 00000000002.50.6, 00000000001.69.6, 00000000002.78.6, 00000000002.76.6] 2023-09-29T22:34:19.709 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, 00000000002.34.6, 00000000002.69.6, 00000000002.61.6, 00000000002.37.6, 00000000001.58.6, 00000000002.107.6, 00000000002.30.6, 00000000001.64.6, 00000000002.79.6, 00000000002.39.6, 00000000002.105.6, 00000000001.67.6, 00000000002.38.6, , 00000000002.71.6, 00000000002.68.6, 00000000002.111.6, 00000000002.170.6, 00000000002.24.6, , 00000000002.59.6, 00000000002.109.6, 00000000002.48.6, 00000000002.59.6, 00000000002.46.6, , 00000000002.93.6, 00000000002.93.6, 00000000002.40.6, 00000000001.35.6, 00000000002.73.6, 00000000002.46.6, 00000000002.50.6, 00000000001.69.6, 00000000002.78.6, 00000000002.76.6] 2023-09-29T22:34:19.709 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:19.708+0000 7fe1668e2f40 0 bucket checkpoint complete 2023-09-29T22:34:19.733 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-58 2023-09-29T22:34:19.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:34:19.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-58', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:34:19.733 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-58 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:34:19.786 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:19.786 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:19.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:19.844+0000 7f610c16ff40 1 waiting for incremental sync to catch up: 2023-09-29T22:34:19.846 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.184.6, 00000000001.198.6, 00000000001.120.6, 00000000001.122.6, 00000000001.199.6, 00000000001.121.6, 00000000001.547.6, 00000000001.304.6, 00000000001.121.6, 00000000001.163.6] 2023-09-29T22:34:19.846 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.191.6, 00000000007.196.6, 00000000006.222.6, 00000000010.147.6, 00000000005.142.6, 00000000008.223.6, 00000000008.153.6, 00000000006.557.6, 00000000007.316.6, 00000000007.140.6, 00000000004.169.6] 2023-09-29T22:34:24.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:24.848+0000 7f610c16ff40 1 bucket sync caught up with source: 2023-09-29T22:34:24.848 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000003.191.6, 00000000007.196.6, 00000000006.222.6, 00000000010.147.6, 00000000005.142.6, 00000000008.223.6, 00000000008.153.6, 00000000006.557.6, 00000000007.316.6, 00000000007.140.6, 00000000004.169.6] 2023-09-29T22:34:24.848 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000003.191.6, 00000000007.196.6, 00000000006.222.6, 00000000010.147.6, 00000000005.142.6, 00000000008.223.6, 00000000008.153.6, 00000000006.557.6, 00000000007.316.6, 00000000007.140.6, 00000000004.169.6] 2023-09-29T22:34:24.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:24.848+0000 7f610c16ff40 0 bucket checkpoint complete 2023-09-29T22:34:24.878 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-58 zones={a2, a1} 2023-09-29T22:34:24.878 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:24.878 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:24.879 DEBUG:boto:Method: HEAD 2023-09-29T22:34:24.879 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:24.879 DEBUG:boto:Data: 2023-09-29T22:34:24.879 DEBUG:boto:Headers: {} 2023-09-29T22:34:24.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:24.879 DEBUG:boto:Port: 8001 2023-09-29T22:34:24.879 DEBUG:boto:Params: {} 2023-09-29T22:34:24.879 DEBUG:boto:Token: None 2023-09-29T22:34:24.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:24 GMT /hcnphz-58/ 2023-09-29T22:34:24.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BfpTBqYHfNW7GWVfu3nJp2FLLNU= 2023-09-29T22:34:24.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BfpTBqYHfNW7GWVfu3nJp2FLLNU=', 'Content-Length': '0'} 2023-09-29T22:34:24.880 DEBUG:boto:encountered RemoteDisconnected exception, reconnecting 2023-09-29T22:34:24.881 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:34:25.296 DEBUG:boto:Token: None 2023-09-29T22:34:25.296 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:24 GMT /hcnphz-58/ 2023-09-29T22:34:25.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BfpTBqYHfNW7GWVfu3nJp2FLLNU= 2023-09-29T22:34:25.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BfpTBqYHfNW7GWVfu3nJp2FLLNU=', 'Content-Length': '0'} 2023-09-29T22:34:25.306 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a70a6bf29e430420-00651750f1-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.307 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.307 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.307 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.307 DEBUG:boto:Path: /hcnphz-58/ 2023-09-29T22:34:25.307 DEBUG:boto:Data: 2023-09-29T22:34:25.308 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.308 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.308 DEBUG:boto:Params: {} 2023-09-29T22:34:25.308 DEBUG:boto:Token: None 2023-09-29T22:34:25.309 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.309 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sXRANtdGuAsXEd4X8JSknf0qhf4= 2023-09-29T22:34:25.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sXRANtdGuAsXEd4X8JSknf0qhf4=', 'Content-Length': '0'} 2023-09-29T22:34:25.316 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '61'), ('X-RGW-Bytes-Used', '183'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000035b6c2d28c86c2bd-00651750f1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.316 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:34:25.317 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.317 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.317 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.317 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.317 DEBUG:boto:Method: GET 2023-09-29T22:34:25.317 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.317 DEBUG:boto:Data: 2023-09-29T22:34:25.317 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.318 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.318 DEBUG:boto:Params: {} 2023-09-29T22:34:25.318 DEBUG:boto:Token: None 2023-09-29T22:34:25.318 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/?versions 2023-09-29T22:34:25.318 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w= 2023-09-29T22:34:25.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w=', 'Content-Length': '0'} 2023-09-29T22:34:25.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000730d9ebcba2c57e4-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.323 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-29T22:34:25.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-09-29T22:34:25.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-29T22:34:25.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-29T22:34:25.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-09-29T22:34:25.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-09-29T22:34:25.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-09-29T22:34:25.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-09-29T22:34:25.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-09-29T22:34:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-09-29T22:34:25.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-09-29T22:34:25.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-09-29T22:34:25.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-09-29T22:34:25.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-09-29T22:34:25.334 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:34:25.334 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.334 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.334 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.334 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.335 DEBUG:boto:Method: GET 2023-09-29T22:34:25.335 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.335 DEBUG:boto:Data: 2023-09-29T22:34:25.335 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.335 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.335 DEBUG:boto:Params: {} 2023-09-29T22:34:25.336 DEBUG:boto:Token: None 2023-09-29T22:34:25.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/?versions 2023-09-29T22:34:25.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w= 2023-09-29T22:34:25.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w=', 'Content-Length': '0'} 2023-09-29T22:34:25.344 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000029f13e009276dbf6-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.345 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000000 2023-09-29T22:34:25.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000001 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000002 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000003 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000004 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000005 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000006 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000007 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000008 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000009 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000010 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000011 2023-09-29T22:34:25.349 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000012 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000013 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000014 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000015 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000016 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000017 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000018 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000019 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000020 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000021 2023-09-29T22:34:25.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000022 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000023 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000024 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000025 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000026 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000027 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000028 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000029 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000030 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000031 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000032 2023-09-29T22:34:25.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000033 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000036 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000037 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000039 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000040 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000042 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000045 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000046 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000048 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000050 2023-09-29T22:34:25.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000053 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000055 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000056 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000057 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000058 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000059 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000060 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000061 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000062 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000065 2023-09-29T22:34:25.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000076 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000133 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000162 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000271 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000283 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000317 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000423 2023-09-29T22:34:25.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj00000000000000000467 2023-09-29T22:34:25.354 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.355 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.355 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.355 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.355 DEBUG:boto:Method: GET 2023-09-29T22:34:25.355 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.355 DEBUG:boto:Data: 2023-09-29T22:34:25.355 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.355 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.355 DEBUG:boto:Params: {} 2023-09-29T22:34:25.356 DEBUG:boto:Token: None 2023-09-29T22:34:25.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/?versions 2023-09-29T22:34:25.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w= 2023-09-29T22:34:25.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w=', 'Content-Length': '0'} 2023-09-29T22:34:25.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c77f0202a8c13957-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.361 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.364 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.365 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.365 DEBUG:boto:path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.365 DEBUG:boto:auth_path=/hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.365 DEBUG:boto:Method: GET 2023-09-29T22:34:25.365 DEBUG:boto:Path: /hcnphz-58/?versions&max-keys=999 2023-09-29T22:34:25.365 DEBUG:boto:Data: 2023-09-29T22:34:25.365 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.365 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.365 DEBUG:boto:Params: {} 2023-09-29T22:34:25.366 DEBUG:boto:Token: None 2023-09-29T22:34:25.366 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/?versions 2023-09-29T22:34:25.366 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w= 2023-09-29T22:34:25.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PN3yysbqzRkZYxMW+hMHSvTNr6w=', 'Content-Length': '0'} 2023-09-29T22:34:25.375 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25f98563a99b4c8-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.375 DEBUG:boto:b'hcnphz-58999falseobj00000000000000000000nulltrue2023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000001nulltrue2023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000002nulltrue2023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000003nulltrue2023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000004nulltrue2023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000005nulltrue2023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000006nulltrue2023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000007nulltrue2023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000008nulltrue2023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000009nulltrue2023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000010nulltrue2023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000011nulltrue2023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000012nulltrue2023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000013nulltrue2023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000014nulltrue2023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000015nulltrue2023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000016nulltrue2023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000017nulltrue2023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000018nulltrue2023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000019nulltrue2023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000020nulltrue2023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000021nulltrue2023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000022nulltrue2023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000023nulltrue2023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000024nulltrue2023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000025nulltrue2023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000026nulltrue2023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000027nulltrue2023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000028nulltrue2023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000029nulltrue2023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000030nulltrue2023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000031nulltrue2023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000032nulltrue2023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000033nulltrue2023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000036nulltrue2023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000037nulltrue2023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000039nulltrue2023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000040nulltrue2023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000042nulltrue2023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000045nulltrue2023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000046nulltrue2023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000048nulltrue2023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000050nulltrue2023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000053nulltrue2023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000055nulltrue2023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000056nulltrue2023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000057nulltrue2023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000058nulltrue2023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000059nulltrue2023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000060nulltrue2023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000061nulltrue2023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000062nulltrue2023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000065nulltrue2023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000076nulltrue2023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000133nulltrue2023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000162nulltrue2023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000271nulltrue2023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000283nulltrue2023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000317nulltrue2023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000423nulltrue2023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj00000000000000000467nulltrue2023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-29T22:34:25.380 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.380 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.380 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.380 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.381 DEBUG:boto:Method: GET 2023-09-29T22:34:25.381 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.381 DEBUG:boto:Data: 2023-09-29T22:34:25.381 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.381 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.381 DEBUG:boto:Params: {} 2023-09-29T22:34:25.381 DEBUG:boto:Token: None 2023-09-29T22:34:25.382 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.382 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM= 2023-09-29T22:34:25.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM=', 'Content-Length': '0'} 2023-09-29T22:34:25.386 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1af16dc9552c865-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.430 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.431 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.431 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.431 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.431 DEBUG:boto:Method: GET 2023-09-29T22:34:25.431 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.431 DEBUG:boto:Data: 2023-09-29T22:34:25.431 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.431 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.433 DEBUG:boto:Params: {} 2023-09-29T22:34:25.433 DEBUG:boto:Token: None 2023-09-29T22:34:25.433 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM= 2023-09-29T22:34:25.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM=', 'Content-Length': '0'} 2023-09-29T22:34:25.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f728c786cfe791ae-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.437 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.437 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.437 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.437 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.437 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.438 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.438 DEBUG:boto:Data: 2023-09-29T22:34:25.438 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.438 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.438 DEBUG:boto:Params: {} 2023-09-29T22:34:25.438 DEBUG:boto:Token: None 2023-09-29T22:34:25.438 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3+sQpjLlQbuiVMLAZqVL15jeicY= 2023-09-29T22:34:25.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3+sQpjLlQbuiVMLAZqVL15jeicY=', 'Content-Length': '0'} 2023-09-29T22:34:25.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6cefa6a41fbaafd-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.441 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.441 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.442 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.442 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.442 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.442 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.442 DEBUG:boto:Data: 2023-09-29T22:34:25.442 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.443 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.443 DEBUG:boto:Params: {} 2023-09-29T22:34:25.443 DEBUG:boto:Token: None 2023-09-29T22:34:25.443 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3+sQpjLlQbuiVMLAZqVL15jeicY= 2023-09-29T22:34:25.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3+sQpjLlQbuiVMLAZqVL15jeicY=', 'Content-Length': '0'} 2023-09-29T22:34:25.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d9212d54bdf1f62-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-29T22:34:25.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.447 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.447 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.447 DEBUG:boto:Method: GET 2023-09-29T22:34:25.447 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.447 DEBUG:boto:Data: 2023-09-29T22:34:25.447 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.448 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.448 DEBUG:boto:Params: {} 2023-09-29T22:34:25.448 DEBUG:boto:Token: None 2023-09-29T22:34:25.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM= 2023-09-29T22:34:25.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM=', 'Content-Length': '0'} 2023-09-29T22:34:25.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000653bfca2f5a02033-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.451 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.452 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.452 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.452 DEBUG:boto:Method: GET 2023-09-29T22:34:25.452 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.452 DEBUG:boto:Data: 2023-09-29T22:34:25.452 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.452 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.452 DEBUG:boto:Params: {} 2023-09-29T22:34:25.452 DEBUG:boto:Token: None 2023-09-29T22:34:25.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000?versionId=null 2023-09-29T22:34:25.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM= 2023-09-29T22:34:25.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pj93kySHd22yOIKJ4xfUc8nBBKM=', 'Content-Length': '0'} 2023-09-29T22:34:25.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000882f62f21d56c72f-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.456 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.456 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.456 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.456 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.456 DEBUG:boto:Method: GET 2023-09-29T22:34:25.457 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.457 DEBUG:boto:Data: 2023-09-29T22:34:25.457 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.457 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.457 DEBUG:boto:Params: {} 2023-09-29T22:34:25.457 DEBUG:boto:Token: None 2023-09-29T22:34:25.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006871fc2184f9b98f-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.462 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.463 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.463 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.463 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.463 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.463 DEBUG:boto:Method: GET 2023-09-29T22:34:25.463 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.463 DEBUG:boto:Data: 2023-09-29T22:34:25.464 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.464 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.464 DEBUG:boto:Params: {} 2023-09-29T22:34:25.464 DEBUG:boto:Token: None 2023-09-29T22:34:25.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087693dd5ff12b84c-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.472 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.472 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.472 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.472 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.473 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.473 DEBUG:boto:Data: 2023-09-29T22:34:25.473 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.473 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.473 DEBUG:boto:Params: {} 2023-09-29T22:34:25.473 DEBUG:boto:Token: None 2023-09-29T22:34:25.473 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.473 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qt2HSZ5HzkixOMnND4T2EC7vbt8= 2023-09-29T22:34:25.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qt2HSZ5HzkixOMnND4T2EC7vbt8=', 'Content-Length': '0'} 2023-09-29T22:34:25.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002718cfa27f6cd7df-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.476 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.476 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.476 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.476 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.476 DEBUG:boto:Data: 2023-09-29T22:34:25.476 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.476 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.476 DEBUG:boto:Params: {} 2023-09-29T22:34:25.477 DEBUG:boto:Token: None 2023-09-29T22:34:25.477 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qt2HSZ5HzkixOMnND4T2EC7vbt8= 2023-09-29T22:34:25.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qt2HSZ5HzkixOMnND4T2EC7vbt8=', 'Content-Length': '0'} 2023-09-29T22:34:25.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f83ca4b2c9d30581-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000000 2023-09-29T22:34:25.479 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.479 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.479 DEBUG:boto:Method: GET 2023-09-29T22:34:25.479 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.479 DEBUG:boto:Data: 2023-09-29T22:34:25.479 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.479 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.480 DEBUG:boto:Params: {} 2023-09-29T22:34:25.480 DEBUG:boto:Token: None 2023-09-29T22:34:25.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3+kNt8Zuhf+ImclDTFumsSVXjbw= 2023-09-29T22:34:25.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3+kNt8Zuhf+ImclDTFumsSVXjbw=', 'Content-Length': '0'} 2023-09-29T22:34:25.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddd653684ca609be-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.522 DEBUG:boto:path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.523 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.523 DEBUG:boto:Method: GET 2023-09-29T22:34:25.523 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.523 DEBUG:boto:Data: 2023-09-29T22:34:25.523 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.523 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.523 DEBUG:boto:Params: {} 2023-09-29T22:34:25.523 DEBUG:boto:Token: None 2023-09-29T22:34:25.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000000 2023-09-29T22:34:25.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3+kNt8Zuhf+ImclDTFumsSVXjbw= 2023-09-29T22:34:25.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3+kNt8Zuhf+ImclDTFumsSVXjbw=', 'Content-Length': '0'} 2023-09-29T22:34:25.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb39446f6c95a3f6-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.570 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.571 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.571 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.571 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.571 DEBUG:boto:Method: GET 2023-09-29T22:34:25.571 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.571 DEBUG:boto:Data: 2023-09-29T22:34:25.571 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.571 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.571 DEBUG:boto:Params: {} 2023-09-29T22:34:25.572 DEBUG:boto:Token: None 2023-09-29T22:34:25.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005075d8ae8ff34711-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.577 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.577 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.578 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.578 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.578 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.578 DEBUG:boto:Method: GET 2023-09-29T22:34:25.578 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000000 2023-09-29T22:34:25.578 DEBUG:boto:Data: 2023-09-29T22:34:25.578 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.579 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.579 DEBUG:boto:Params: {} 2023-09-29T22:34:25.579 DEBUG:boto:Token: None 2023-09-29T22:34:25.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab99ee575f924f7-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.587 DEBUG:boto:b'hcnphz-58obj000000000000000000001000falseobj000000000000000000002023-09-29T22:34:15.882Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-29T22:34:25.587 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.588 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.588 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.588 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.588 DEBUG:boto:Method: GET 2023-09-29T22:34:25.588 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.588 DEBUG:boto:Data: 2023-09-29T22:34:25.588 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.588 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.589 DEBUG:boto:Params: {} 2023-09-29T22:34:25.589 DEBUG:boto:Token: None 2023-09-29T22:34:25.589 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo= 2023-09-29T22:34:25.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo=', 'Content-Length': '0'} 2023-09-29T22:34:25.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004767183665323317-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.595 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.595 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.595 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.595 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.595 DEBUG:boto:Method: GET 2023-09-29T22:34:25.595 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.595 DEBUG:boto:Data: 2023-09-29T22:34:25.595 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.596 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.596 DEBUG:boto:Params: {} 2023-09-29T22:34:25.596 DEBUG:boto:Token: None 2023-09-29T22:34:25.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo= 2023-09-29T22:34:25.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo=', 'Content-Length': '0'} 2023-09-29T22:34:25.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df144e50dff58bb9-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.599 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.599 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.599 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.599 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.600 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.600 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.600 DEBUG:boto:Data: 2023-09-29T22:34:25.600 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.600 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.600 DEBUG:boto:Params: {} 2023-09-29T22:34:25.600 DEBUG:boto:Token: None 2023-09-29T22:34:25.600 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4CZWH+wbN6haXFcfIwIw6WsvK8w= 2023-09-29T22:34:25.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4CZWH+wbN6haXFcfIwIw6WsvK8w=', 'Content-Length': '0'} 2023-09-29T22:34:25.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fdd20fbddcde5c5-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.603 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.603 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.603 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.603 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.603 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.603 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.604 DEBUG:boto:Data: 2023-09-29T22:34:25.604 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.604 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.604 DEBUG:boto:Params: {} 2023-09-29T22:34:25.604 DEBUG:boto:Token: None 2023-09-29T22:34:25.604 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4CZWH+wbN6haXFcfIwIw6WsvK8w= 2023-09-29T22:34:25.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4CZWH+wbN6haXFcfIwIw6WsvK8w=', 'Content-Length': '0'} 2023-09-29T22:34:25.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de161973816cea2d-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-29T22:34:25.607 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.607 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.607 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.607 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.608 DEBUG:boto:Method: GET 2023-09-29T22:34:25.608 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.608 DEBUG:boto:Data: 2023-09-29T22:34:25.608 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.608 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.608 DEBUG:boto:Params: {} 2023-09-29T22:34:25.608 DEBUG:boto:Token: None 2023-09-29T22:34:25.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo= 2023-09-29T22:34:25.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo=', 'Content-Length': '0'} 2023-09-29T22:34:25.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd85b332d393ad1-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.658 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.659 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.659 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.659 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.659 DEBUG:boto:Method: GET 2023-09-29T22:34:25.659 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.659 DEBUG:boto:Data: 2023-09-29T22:34:25.660 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.660 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.660 DEBUG:boto:Params: {} 2023-09-29T22:34:25.660 DEBUG:boto:Token: None 2023-09-29T22:34:25.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001?versionId=null 2023-09-29T22:34:25.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo= 2023-09-29T22:34:25.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zPXK3JL813/T/xf1bhRknF5keo=', 'Content-Length': '0'} 2023-09-29T22:34:25.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000739c8be206d6d428-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.664 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.664 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.664 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.664 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.664 DEBUG:boto:Method: GET 2023-09-29T22:34:25.664 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.664 DEBUG:boto:Data: 2023-09-29T22:34:25.664 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.665 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.665 DEBUG:boto:Params: {} 2023-09-29T22:34:25.665 DEBUG:boto:Token: None 2023-09-29T22:34:25.665 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d6f267e62d43be8-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.671 DEBUG:boto:b'hcnphz-58obj000000000000000000011000falseobj000000000000000000012023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.671 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.671 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.671 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.672 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.672 DEBUG:boto:Method: GET 2023-09-29T22:34:25.672 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.672 DEBUG:boto:Data: 2023-09-29T22:34:25.672 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.672 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.672 DEBUG:boto:Params: {} 2023-09-29T22:34:25.672 DEBUG:boto:Token: None 2023-09-29T22:34:25.673 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f3108637b80608-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.682 DEBUG:boto:b'hcnphz-58obj000000000000000000011000falseobj000000000000000000012023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.682 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.682 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.682 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.683 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.683 DEBUG:boto:Data: 2023-09-29T22:34:25.683 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.683 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.683 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.683 DEBUG:boto:Params: {} 2023-09-29T22:34:25.684 DEBUG:boto:Token: None 2023-09-29T22:34:25.684 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.684 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5/vIJNRuhJ477QlBBpIvnn8xH2A= 2023-09-29T22:34:25.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5/vIJNRuhJ477QlBBpIvnn8xH2A=', 'Content-Length': '0'} 2023-09-29T22:34:25.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070a42abe115550b4-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.687 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.687 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.687 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.687 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.687 DEBUG:boto:Data: 2023-09-29T22:34:25.687 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.687 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.688 DEBUG:boto:Params: {} 2023-09-29T22:34:25.688 DEBUG:boto:Token: None 2023-09-29T22:34:25.688 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5/vIJNRuhJ477QlBBpIvnn8xH2A= 2023-09-29T22:34:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5/vIJNRuhJ477QlBBpIvnn8xH2A=', 'Content-Length': '0'} 2023-09-29T22:34:25.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8da78431ad55eb-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000001 2023-09-29T22:34:25.691 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.691 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.691 DEBUG:boto:Method: GET 2023-09-29T22:34:25.691 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.691 DEBUG:boto:Data: 2023-09-29T22:34:25.691 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.691 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.692 DEBUG:boto:Params: {} 2023-09-29T22:34:25.692 DEBUG:boto:Token: None 2023-09-29T22:34:25.692 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:21nl/LnVI9qSHSs7c7OKgO2tktY= 2023-09-29T22:34:25.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:21nl/LnVI9qSHSs7c7OKgO2tktY=', 'Content-Length': '0'} 2023-09-29T22:34:25.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019ea845cff86361d-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.694 DEBUG:boto:path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.694 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.695 DEBUG:boto:Method: GET 2023-09-29T22:34:25.695 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.695 DEBUG:boto:Data: 2023-09-29T22:34:25.695 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.695 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.695 DEBUG:boto:Params: {} 2023-09-29T22:34:25.695 DEBUG:boto:Token: None 2023-09-29T22:34:25.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000001 2023-09-29T22:34:25.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:21nl/LnVI9qSHSs7c7OKgO2tktY= 2023-09-29T22:34:25.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:21nl/LnVI9qSHSs7c7OKgO2tktY=', 'Content-Length': '0'} 2023-09-29T22:34:25.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069441d154893db49-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.699 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.699 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.699 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.699 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.700 DEBUG:boto:Method: GET 2023-09-29T22:34:25.700 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.700 DEBUG:boto:Data: 2023-09-29T22:34:25.700 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.700 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.701 DEBUG:boto:Params: {} 2023-09-29T22:34:25.701 DEBUG:boto:Token: None 2023-09-29T22:34:25.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000414cf633c23ccef6-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.706 DEBUG:boto:b'hcnphz-58obj000000000000000000011000falseobj000000000000000000012023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.706 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.706 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.706 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.706 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.707 DEBUG:boto:Method: GET 2023-09-29T22:34:25.707 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000001 2023-09-29T22:34:25.707 DEBUG:boto:Data: 2023-09-29T22:34:25.707 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.707 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.707 DEBUG:boto:Params: {} 2023-09-29T22:34:25.708 DEBUG:boto:Token: None 2023-09-29T22:34:25.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009d82fae58e42a598-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.758 DEBUG:boto:b'hcnphz-58obj000000000000000000011000falseobj000000000000000000012023-09-29T22:34:16.117Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-29T22:34:25.759 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.759 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.759 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.759 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.760 DEBUG:boto:Method: GET 2023-09-29T22:34:25.760 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.760 DEBUG:boto:Data: 2023-09-29T22:34:25.760 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.760 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.760 DEBUG:boto:Params: {} 2023-09-29T22:34:25.760 DEBUG:boto:Token: None 2023-09-29T22:34:25.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q= 2023-09-29T22:34:25.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q=', 'Content-Length': '0'} 2023-09-29T22:34:25.764 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f603f5c923fac30-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.764 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.764 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.764 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.764 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.765 DEBUG:boto:Method: GET 2023-09-29T22:34:25.765 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.765 DEBUG:boto:Data: 2023-09-29T22:34:25.765 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.765 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.765 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.765 DEBUG:boto:Params: {} 2023-09-29T22:34:25.765 DEBUG:boto:Token: None 2023-09-29T22:34:25.765 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.766 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q= 2023-09-29T22:34:25.766 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q=', 'Content-Length': '0'} 2023-09-29T22:34:25.768 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057d041aa73ce9b74-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.769 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.769 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.769 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.769 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.769 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.770 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.770 DEBUG:boto:Data: 2023-09-29T22:34:25.770 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.770 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.770 DEBUG:boto:Params: {} 2023-09-29T22:34:25.770 DEBUG:boto:Token: None 2023-09-29T22:34:25.770 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.770 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eHy4MLMejDD6pziMn3I20LISOgw= 2023-09-29T22:34:25.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eHy4MLMejDD6pziMn3I20LISOgw=', 'Content-Length': '0'} 2023-09-29T22:34:25.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b93eb2fca1d00d1-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.773 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.774 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.774 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.774 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.774 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.774 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.774 DEBUG:boto:Data: 2023-09-29T22:34:25.774 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.774 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.774 DEBUG:boto:Params: {} 2023-09-29T22:34:25.775 DEBUG:boto:Token: None 2023-09-29T22:34:25.775 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.775 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eHy4MLMejDD6pziMn3I20LISOgw= 2023-09-29T22:34:25.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eHy4MLMejDD6pziMn3I20LISOgw=', 'Content-Length': '0'} 2023-09-29T22:34:25.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d88faeeba00ec2d8-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-29T22:34:25.778 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.778 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.778 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.778 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.778 DEBUG:boto:Method: GET 2023-09-29T22:34:25.778 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.778 DEBUG:boto:Data: 2023-09-29T22:34:25.778 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.779 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.779 DEBUG:boto:Params: {} 2023-09-29T22:34:25.779 DEBUG:boto:Token: None 2023-09-29T22:34:25.779 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.779 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q= 2023-09-29T22:34:25.779 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q=', 'Content-Length': '0'} 2023-09-29T22:34:25.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4dd97272c696cfb-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.822 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.823 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.823 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.823 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.823 DEBUG:boto:Method: GET 2023-09-29T22:34:25.823 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.823 DEBUG:boto:Data: 2023-09-29T22:34:25.823 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.823 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.823 DEBUG:boto:Params: {} 2023-09-29T22:34:25.824 DEBUG:boto:Token: None 2023-09-29T22:34:25.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002?versionId=null 2023-09-29T22:34:25.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q= 2023-09-29T22:34:25.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4oG/418cz6qPcpu98WWuFjMdu0Q=', 'Content-Length': '0'} 2023-09-29T22:34:25.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068363d416107ec92-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.827 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.828 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.828 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.828 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.828 DEBUG:boto:Method: GET 2023-09-29T22:34:25.828 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.828 DEBUG:boto:Data: 2023-09-29T22:34:25.828 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.828 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.829 DEBUG:boto:Params: {} 2023-09-29T22:34:25.829 DEBUG:boto:Token: None 2023-09-29T22:34:25.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057d714dcf3da7f94-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.833 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.834 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.834 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.834 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.834 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.834 DEBUG:boto:Method: GET 2023-09-29T22:34:25.835 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.835 DEBUG:boto:Data: 2023-09-29T22:34:25.835 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.835 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.835 DEBUG:boto:Params: {} 2023-09-29T22:34:25.835 DEBUG:boto:Token: None 2023-09-29T22:34:25.835 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.835 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003927a869d0a5aa90-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.843 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.843 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.844 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.844 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.844 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.844 DEBUG:boto:Data: 2023-09-29T22:34:25.844 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.844 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.844 DEBUG:boto:Params: {} 2023-09-29T22:34:25.844 DEBUG:boto:Token: None 2023-09-29T22:34:25.845 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0Z0BhJwaOd4P2eITW9rb+z9kjUI= 2023-09-29T22:34:25.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0Z0BhJwaOd4P2eITW9rb+z9kjUI=', 'Content-Length': '0'} 2023-09-29T22:34:25.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056766c5497abed35-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.847 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.848 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.848 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.848 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.848 DEBUG:boto:Data: 2023-09-29T22:34:25.848 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.848 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.848 DEBUG:boto:Params: {} 2023-09-29T22:34:25.848 DEBUG:boto:Token: None 2023-09-29T22:34:25.849 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0Z0BhJwaOd4P2eITW9rb+z9kjUI= 2023-09-29T22:34:25.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0Z0BhJwaOd4P2eITW9rb+z9kjUI=', 'Content-Length': '0'} 2023-09-29T22:34:25.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbcc4776d1a15b38-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000002 2023-09-29T22:34:25.851 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.851 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.851 DEBUG:boto:Method: GET 2023-09-29T22:34:25.852 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.852 DEBUG:boto:Data: 2023-09-29T22:34:25.852 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.852 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.852 DEBUG:boto:Params: {} 2023-09-29T22:34:25.852 DEBUG:boto:Token: None 2023-09-29T22:34:25.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VkgNcQOb/NKgcUZ1LX0ABHkehfQ= 2023-09-29T22:34:25.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VkgNcQOb/NKgcUZ1LX0ABHkehfQ=', 'Content-Length': '0'} 2023-09-29T22:34:25.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaced55636f86fd8-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.898 DEBUG:boto:path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.899 DEBUG:boto:Method: GET 2023-09-29T22:34:25.899 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.899 DEBUG:boto:Data: 2023-09-29T22:34:25.899 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.899 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.899 DEBUG:boto:Params: {} 2023-09-29T22:34:25.900 DEBUG:boto:Token: None 2023-09-29T22:34:25.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000002 2023-09-29T22:34:25.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VkgNcQOb/NKgcUZ1LX0ABHkehfQ= 2023-09-29T22:34:25.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VkgNcQOb/NKgcUZ1LX0ABHkehfQ=', 'Content-Length': '0'} 2023-09-29T22:34:25.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae62c6fef477bcc-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.904 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.904 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.904 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.904 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.905 DEBUG:boto:Method: GET 2023-09-29T22:34:25.905 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.905 DEBUG:boto:Data: 2023-09-29T22:34:25.905 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.905 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.906 DEBUG:boto:Params: {} 2023-09-29T22:34:25.906 DEBUG:boto:Token: None 2023-09-29T22:34:25.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.906 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.911 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bd4eaefbfc680a3-00651750f1-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.911 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.912 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.912 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.912 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.912 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.912 DEBUG:boto:Method: GET 2023-09-29T22:34:25.912 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000002 2023-09-29T22:34:25.913 DEBUG:boto:Data: 2023-09-29T22:34:25.913 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.913 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.913 DEBUG:boto:Params: {} 2023-09-29T22:34:25.913 DEBUG:boto:Token: None 2023-09-29T22:34:25.913 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/ 2023-09-29T22:34:25.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA= 2023-09-29T22:34:25.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OOfRu8WPCESZ1xEWv1HOOadi/BA=', 'Content-Length': '0'} 2023-09-29T22:34:25.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038d6f2d6ec66f4bf-00651750f1-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.921 DEBUG:boto:b'hcnphz-58obj000000000000000000021000falseobj000000000000000000022023-09-29T22:34:16.569Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:25.922 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-29T22:34:25.922 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.922 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.922 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.923 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.923 DEBUG:boto:Method: GET 2023-09-29T22:34:25.923 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.923 DEBUG:boto:Data: 2023-09-29T22:34:25.923 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.923 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.924 DEBUG:boto:Params: {} 2023-09-29T22:34:25.924 DEBUG:boto:Token: None 2023-09-29T22:34:25.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg= 2023-09-29T22:34:25.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg=', 'Content-Length': '0'} 2023-09-29T22:34:25.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8c69b600aeedb10-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.930 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.930 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.930 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.930 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.930 DEBUG:boto:Method: GET 2023-09-29T22:34:25.930 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.931 DEBUG:boto:Data: 2023-09-29T22:34:25.931 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.931 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.931 DEBUG:boto:Params: {} 2023-09-29T22:34:25.931 DEBUG:boto:Token: None 2023-09-29T22:34:25.931 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg= 2023-09-29T22:34:25.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg=', 'Content-Length': '0'} 2023-09-29T22:34:25.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012df6da449b1340f-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.934 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.934 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.934 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.934 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.935 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.935 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.935 DEBUG:boto:Data: 2023-09-29T22:34:25.935 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.935 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.935 DEBUG:boto:Params: {} 2023-09-29T22:34:25.935 DEBUG:boto:Token: None 2023-09-29T22:34:25.935 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r1HjW1wo+fU2NAgw412XJ1RHybE= 2023-09-29T22:34:25.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r1HjW1wo+fU2NAgw412XJ1RHybE=', 'Content-Length': '0'} 2023-09-29T22:34:25.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e06d1bd734ccc97c-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.938 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.938 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.939 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.939 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.939 DEBUG:boto:Method: HEAD 2023-09-29T22:34:25.939 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.939 DEBUG:boto:Data: 2023-09-29T22:34:25.939 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.939 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.939 DEBUG:boto:Params: {} 2023-09-29T22:34:25.940 DEBUG:boto:Token: None 2023-09-29T22:34:25.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r1HjW1wo+fU2NAgw412XJ1RHybE= 2023-09-29T22:34:25.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r1HjW1wo+fU2NAgw412XJ1RHybE=', 'Content-Length': '0'} 2023-09-29T22:34:25.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f9383dcfd1f8d1a-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-29T22:34:25.943 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.944 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.944 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.944 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.944 DEBUG:boto:Method: GET 2023-09-29T22:34:25.945 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.945 DEBUG:boto:Data: 2023-09-29T22:34:25.945 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:25.945 DEBUG:boto:Port: 8001 2023-09-29T22:34:25.946 DEBUG:boto:Params: {} 2023-09-29T22:34:25.946 DEBUG:boto:Token: None 2023-09-29T22:34:25.946 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.946 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg= 2023-09-29T22:34:25.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg=', 'Content-Length': '0'} 2023-09-29T22:34:25.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb7303d62a38ff08-00651750f1-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.994 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.995 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:25.995 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.995 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.995 DEBUG:boto:Method: GET 2023-09-29T22:34:25.995 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.995 DEBUG:boto:Data: 2023-09-29T22:34:25.995 DEBUG:boto:Headers: {} 2023-09-29T22:34:25.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:25.995 DEBUG:boto:Port: 8000 2023-09-29T22:34:25.996 DEBUG:boto:Params: {} 2023-09-29T22:34:25.996 DEBUG:boto:Token: None 2023-09-29T22:34:25.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:25 GMT /hcnphz-58/obj00000000000000000003?versionId=null 2023-09-29T22:34:25.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg= 2023-09-29T22:34:25.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HK9g3fYE54AgVIXtCYwbn25Hipg=', 'Content-Length': '0'} 2023-09-29T22:34:25.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a56bfc2693414d7f-00651750f1-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:25.999 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:25.999 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:25.999 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:25.999 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.000 DEBUG:boto:Method: GET 2023-09-29T22:34:26.000 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.000 DEBUG:boto:Data: 2023-09-29T22:34:26.000 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.000 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.000 DEBUG:boto:Params: {} 2023-09-29T22:34:26.000 DEBUG:boto:Token: None 2023-09-29T22:34:26.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.001 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f3263118d44a3eb-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.006 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.006 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.006 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.006 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.007 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.007 DEBUG:boto:Method: GET 2023-09-29T22:34:26.007 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.007 DEBUG:boto:Data: 2023-09-29T22:34:26.007 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.007 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.007 DEBUG:boto:Params: {} 2023-09-29T22:34:26.008 DEBUG:boto:Token: None 2023-09-29T22:34:26.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000114e6b41a18dd8d5-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.062 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.063 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.063 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.063 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.063 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.064 DEBUG:boto:Data: 2023-09-29T22:34:26.064 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.064 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.064 DEBUG:boto:Params: {} 2023-09-29T22:34:26.064 DEBUG:boto:Token: None 2023-09-29T22:34:26.064 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O/0pCdZAbVU0fvpP4zB0PQQ/iI4= 2023-09-29T22:34:26.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O/0pCdZAbVU0fvpP4zB0PQQ/iI4=', 'Content-Length': '0'} 2023-09-29T22:34:26.067 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000435b43b02c05ec8f-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.068 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.068 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.068 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.068 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.068 DEBUG:boto:Data: 2023-09-29T22:34:26.068 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.068 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.068 DEBUG:boto:Params: {} 2023-09-29T22:34:26.068 DEBUG:boto:Token: None 2023-09-29T22:34:26.069 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O/0pCdZAbVU0fvpP4zB0PQQ/iI4= 2023-09-29T22:34:26.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O/0pCdZAbVU0fvpP4zB0PQQ/iI4=', 'Content-Length': '0'} 2023-09-29T22:34:26.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb06e9c0b9a073e2-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000003 2023-09-29T22:34:26.072 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.072 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.072 DEBUG:boto:Method: GET 2023-09-29T22:34:26.072 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.072 DEBUG:boto:Data: 2023-09-29T22:34:26.072 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.072 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.073 DEBUG:boto:Params: {} 2023-09-29T22:34:26.073 DEBUG:boto:Token: None 2023-09-29T22:34:26.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uoukj9oteGMI4WsfJD9TOt4XZ0w= 2023-09-29T22:34:26.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uoukj9oteGMI4WsfJD9TOt4XZ0w=', 'Content-Length': '0'} 2023-09-29T22:34:26.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326c407ff84cdc04-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.076 DEBUG:boto:path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.076 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.076 DEBUG:boto:Method: GET 2023-09-29T22:34:26.076 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.077 DEBUG:boto:Data: 2023-09-29T22:34:26.077 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.077 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.077 DEBUG:boto:Params: {} 2023-09-29T22:34:26.077 DEBUG:boto:Token: None 2023-09-29T22:34:26.077 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000003 2023-09-29T22:34:26.077 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uoukj9oteGMI4WsfJD9TOt4XZ0w= 2023-09-29T22:34:26.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uoukj9oteGMI4WsfJD9TOt4XZ0w=', 'Content-Length': '0'} 2023-09-29T22:34:26.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c3335e23b8b9c6-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.081 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.081 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.081 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.081 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.081 DEBUG:boto:Method: GET 2023-09-29T22:34:26.081 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.081 DEBUG:boto:Data: 2023-09-29T22:34:26.081 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.082 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.082 DEBUG:boto:Params: {} 2023-09-29T22:34:26.082 DEBUG:boto:Token: None 2023-09-29T22:34:26.082 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.082 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000242b606ba6e189f8-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.134 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.135 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.135 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.135 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.135 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.135 DEBUG:boto:Method: GET 2023-09-29T22:34:26.135 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000003 2023-09-29T22:34:26.135 DEBUG:boto:Data: 2023-09-29T22:34:26.135 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.136 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.136 DEBUG:boto:Params: {} 2023-09-29T22:34:26.136 DEBUG:boto:Token: None 2023-09-29T22:34:26.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a3f472477b4a7cf-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.145 DEBUG:boto:b'hcnphz-58obj000000000000000000031000falseobj000000000000000000032023-09-29T22:34:16.511Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-29T22:34:26.146 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.146 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.146 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.146 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.146 DEBUG:boto:Method: GET 2023-09-29T22:34:26.146 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.146 DEBUG:boto:Data: 2023-09-29T22:34:26.147 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.147 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.147 DEBUG:boto:Params: {} 2023-09-29T22:34:26.147 DEBUG:boto:Token: None 2023-09-29T22:34:26.147 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.147 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4= 2023-09-29T22:34:26.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4=', 'Content-Length': '0'} 2023-09-29T22:34:26.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d628376b7bff73bb-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.152 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.152 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.152 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.152 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.152 DEBUG:boto:Method: GET 2023-09-29T22:34:26.153 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.153 DEBUG:boto:Data: 2023-09-29T22:34:26.153 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.153 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.153 DEBUG:boto:Params: {} 2023-09-29T22:34:26.153 DEBUG:boto:Token: None 2023-09-29T22:34:26.153 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4= 2023-09-29T22:34:26.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4=', 'Content-Length': '0'} 2023-09-29T22:34:26.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539cdfca71deac65-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.202 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.202 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.203 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.203 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.203 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.203 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.203 DEBUG:boto:Data: 2023-09-29T22:34:26.203 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.203 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.203 DEBUG:boto:Params: {} 2023-09-29T22:34:26.203 DEBUG:boto:Token: None 2023-09-29T22:34:26.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NqyKwmlhEvGW5r4+Ku/4hid/k3w= 2023-09-29T22:34:26.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NqyKwmlhEvGW5r4+Ku/4hid/k3w=', 'Content-Length': '0'} 2023-09-29T22:34:26.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd86fef3dd7f5e20-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.207 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.207 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.207 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.207 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.208 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.208 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.208 DEBUG:boto:Data: 2023-09-29T22:34:26.208 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.208 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.208 DEBUG:boto:Params: {} 2023-09-29T22:34:26.208 DEBUG:boto:Token: None 2023-09-29T22:34:26.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.209 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NqyKwmlhEvGW5r4+Ku/4hid/k3w= 2023-09-29T22:34:26.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NqyKwmlhEvGW5r4+Ku/4hid/k3w=', 'Content-Length': '0'} 2023-09-29T22:34:26.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ebc2471a922e4f-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-29T22:34:26.212 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.212 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.212 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.212 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.212 DEBUG:boto:Method: GET 2023-09-29T22:34:26.212 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.212 DEBUG:boto:Data: 2023-09-29T22:34:26.212 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.213 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.213 DEBUG:boto:Params: {} 2023-09-29T22:34:26.213 DEBUG:boto:Token: None 2023-09-29T22:34:26.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4= 2023-09-29T22:34:26.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4=', 'Content-Length': '0'} 2023-09-29T22:34:26.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de9ca50781bdff7f-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.216 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.216 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.216 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.217 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.217 DEBUG:boto:Method: GET 2023-09-29T22:34:26.217 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.217 DEBUG:boto:Data: 2023-09-29T22:34:26.217 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.217 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.217 DEBUG:boto:Params: {} 2023-09-29T22:34:26.217 DEBUG:boto:Token: None 2023-09-29T22:34:26.217 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004?versionId=null 2023-09-29T22:34:26.218 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4= 2023-09-29T22:34:26.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MRew9C2YG1tAC0rjIKOlMDPhah4=', 'Content-Length': '0'} 2023-09-29T22:34:26.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1447d754a0ba47-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.221 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.222 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.222 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.222 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.222 DEBUG:boto:Method: GET 2023-09-29T22:34:26.222 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.223 DEBUG:boto:Data: 2023-09-29T22:34:26.223 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.223 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.223 DEBUG:boto:Params: {} 2023-09-29T22:34:26.224 DEBUG:boto:Token: None 2023-09-29T22:34:26.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.229 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000041aeddd77816063-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.274 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.275 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.275 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.275 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.275 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.275 DEBUG:boto:Method: GET 2023-09-29T22:34:26.275 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.276 DEBUG:boto:Data: 2023-09-29T22:34:26.276 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.276 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.276 DEBUG:boto:Params: {} 2023-09-29T22:34:26.276 DEBUG:boto:Token: None 2023-09-29T22:34:26.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d4d8bc6a6f2a2d3-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.286 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.286 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.286 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.286 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.287 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.287 DEBUG:boto:Data: 2023-09-29T22:34:26.287 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.287 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.287 DEBUG:boto:Params: {} 2023-09-29T22:34:26.287 DEBUG:boto:Token: None 2023-09-29T22:34:26.287 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T3zlNFYbIfSJY07y2ZyORwngN/0= 2023-09-29T22:34:26.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T3zlNFYbIfSJY07y2ZyORwngN/0=', 'Content-Length': '0'} 2023-09-29T22:34:26.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eae6dd1d09e58477-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.290 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.290 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.290 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.291 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.291 DEBUG:boto:Data: 2023-09-29T22:34:26.291 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.291 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.291 DEBUG:boto:Params: {} 2023-09-29T22:34:26.291 DEBUG:boto:Token: None 2023-09-29T22:34:26.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T3zlNFYbIfSJY07y2ZyORwngN/0= 2023-09-29T22:34:26.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T3zlNFYbIfSJY07y2ZyORwngN/0=', 'Content-Length': '0'} 2023-09-29T22:34:26.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004785bedc1d550f8d-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000004 2023-09-29T22:34:26.294 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.294 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.294 DEBUG:boto:Method: GET 2023-09-29T22:34:26.294 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.294 DEBUG:boto:Data: 2023-09-29T22:34:26.294 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.295 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.295 DEBUG:boto:Params: {} 2023-09-29T22:34:26.295 DEBUG:boto:Token: None 2023-09-29T22:34:26.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0pR8Srs9SF6kq9SfhRJTAGkYoPg= 2023-09-29T22:34:26.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0pR8Srs9SF6kq9SfhRJTAGkYoPg=', 'Content-Length': '0'} 2023-09-29T22:34:26.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ef7e7dc6f6c88f-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.297 DEBUG:boto:path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.298 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.298 DEBUG:boto:Method: GET 2023-09-29T22:34:26.298 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.298 DEBUG:boto:Data: 2023-09-29T22:34:26.298 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.298 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.298 DEBUG:boto:Params: {} 2023-09-29T22:34:26.299 DEBUG:boto:Token: None 2023-09-29T22:34:26.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000004 2023-09-29T22:34:26.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0pR8Srs9SF6kq9SfhRJTAGkYoPg= 2023-09-29T22:34:26.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0pR8Srs9SF6kq9SfhRJTAGkYoPg=', 'Content-Length': '0'} 2023-09-29T22:34:26.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dcaa380ae3e5470-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.342 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.342 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.343 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.343 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.343 DEBUG:boto:Method: GET 2023-09-29T22:34:26.343 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.343 DEBUG:boto:Data: 2023-09-29T22:34:26.343 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.343 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.343 DEBUG:boto:Params: {} 2023-09-29T22:34:26.344 DEBUG:boto:Token: None 2023-09-29T22:34:26.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000743bb7b6a8473db8-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.348 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.349 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.349 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.349 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.349 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.349 DEBUG:boto:Method: GET 2023-09-29T22:34:26.349 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000004 2023-09-29T22:34:26.349 DEBUG:boto:Data: 2023-09-29T22:34:26.349 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.350 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.350 DEBUG:boto:Params: {} 2023-09-29T22:34:26.350 DEBUG:boto:Token: None 2023-09-29T22:34:26.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.350 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.359 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060c839736f481fc1-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.359 DEBUG:boto:b'hcnphz-58obj000000000000000000041000falseobj000000000000000000042023-09-29T22:34:16.195Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-29T22:34:26.359 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.360 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.360 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.360 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.360 DEBUG:boto:Method: GET 2023-09-29T22:34:26.360 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.360 DEBUG:boto:Data: 2023-09-29T22:34:26.360 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.360 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.360 DEBUG:boto:Params: {} 2023-09-29T22:34:26.361 DEBUG:boto:Token: None 2023-09-29T22:34:26.361 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.361 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ= 2023-09-29T22:34:26.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ=', 'Content-Length': '0'} 2023-09-29T22:34:26.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfc150ca9a3cb1af-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.363 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.363 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.364 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.364 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.364 DEBUG:boto:Method: GET 2023-09-29T22:34:26.364 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.364 DEBUG:boto:Data: 2023-09-29T22:34:26.364 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.364 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.364 DEBUG:boto:Params: {} 2023-09-29T22:34:26.365 DEBUG:boto:Token: None 2023-09-29T22:34:26.365 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ= 2023-09-29T22:34:26.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ=', 'Content-Length': '0'} 2023-09-29T22:34:26.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095700226837cb774-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.367 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.367 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.368 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.368 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.368 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.368 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.368 DEBUG:boto:Data: 2023-09-29T22:34:26.368 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.368 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.368 DEBUG:boto:Params: {} 2023-09-29T22:34:26.369 DEBUG:boto:Token: None 2023-09-29T22:34:26.369 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.369 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wFKFGmbLD82s3+7+eV+TNpfgls4= 2023-09-29T22:34:26.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wFKFGmbLD82s3+7+eV+TNpfgls4=', 'Content-Length': '0'} 2023-09-29T22:34:26.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a3ee768469bc777-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.371 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.371 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.371 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.371 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.372 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.372 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.372 DEBUG:boto:Data: 2023-09-29T22:34:26.372 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.372 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.372 DEBUG:boto:Params: {} 2023-09-29T22:34:26.372 DEBUG:boto:Token: None 2023-09-29T22:34:26.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wFKFGmbLD82s3+7+eV+TNpfgls4= 2023-09-29T22:34:26.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wFKFGmbLD82s3+7+eV+TNpfgls4=', 'Content-Length': '0'} 2023-09-29T22:34:26.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023489a6c5a028eb4-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-29T22:34:26.376 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.376 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.376 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.376 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.376 DEBUG:boto:Method: GET 2023-09-29T22:34:26.376 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.376 DEBUG:boto:Data: 2023-09-29T22:34:26.377 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.377 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.377 DEBUG:boto:Params: {} 2023-09-29T22:34:26.377 DEBUG:boto:Token: None 2023-09-29T22:34:26.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ= 2023-09-29T22:34:26.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ=', 'Content-Length': '0'} 2023-09-29T22:34:26.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c75bc0cd943240e1-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.422 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.422 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.423 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.423 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.423 DEBUG:boto:Method: GET 2023-09-29T22:34:26.423 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.423 DEBUG:boto:Data: 2023-09-29T22:34:26.423 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.423 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.423 DEBUG:boto:Params: {} 2023-09-29T22:34:26.424 DEBUG:boto:Token: None 2023-09-29T22:34:26.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005?versionId=null 2023-09-29T22:34:26.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ= 2023-09-29T22:34:26.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YJGYUNAUhtpLnF9TDY9L0y62DgQ=', 'Content-Length': '0'} 2023-09-29T22:34:26.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e1387e630309ca1-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.427 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.427 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.427 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.427 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.427 DEBUG:boto:Method: GET 2023-09-29T22:34:26.428 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.428 DEBUG:boto:Data: 2023-09-29T22:34:26.428 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.428 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.428 DEBUG:boto:Params: {} 2023-09-29T22:34:26.428 DEBUG:boto:Token: None 2023-09-29T22:34:26.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0f18818be79eb82-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.433 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.434 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.434 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.434 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.434 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.434 DEBUG:boto:Method: GET 2023-09-29T22:34:26.434 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.435 DEBUG:boto:Data: 2023-09-29T22:34:26.435 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.435 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.435 DEBUG:boto:Params: {} 2023-09-29T22:34:26.435 DEBUG:boto:Token: None 2023-09-29T22:34:26.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.435 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a22c5b8493a9f556-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.444 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.444 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.444 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.444 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.444 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.444 DEBUG:boto:Data: 2023-09-29T22:34:26.444 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.445 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.445 DEBUG:boto:Params: {} 2023-09-29T22:34:26.445 DEBUG:boto:Token: None 2023-09-29T22:34:26.445 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oXeTLxQd5FAFyRguJqmeGU4Xg5w= 2023-09-29T22:34:26.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oXeTLxQd5FAFyRguJqmeGU4Xg5w=', 'Content-Length': '0'} 2023-09-29T22:34:26.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069945354eb864a1c-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.447 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.447 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.448 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.448 DEBUG:boto:Data: 2023-09-29T22:34:26.448 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.448 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.448 DEBUG:boto:Params: {} 2023-09-29T22:34:26.448 DEBUG:boto:Token: None 2023-09-29T22:34:26.448 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oXeTLxQd5FAFyRguJqmeGU4Xg5w= 2023-09-29T22:34:26.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oXeTLxQd5FAFyRguJqmeGU4Xg5w=', 'Content-Length': '0'} 2023-09-29T22:34:26.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057a9d4d73f032d4f-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000005 2023-09-29T22:34:26.451 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.451 DEBUG:boto:Method: GET 2023-09-29T22:34:26.451 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.451 DEBUG:boto:Data: 2023-09-29T22:34:26.451 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.453 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.453 DEBUG:boto:Params: {} 2023-09-29T22:34:26.453 DEBUG:boto:Token: None 2023-09-29T22:34:26.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.454 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:39vTkg0N2v+ZzuBlTeKmdOTU6iI= 2023-09-29T22:34:26.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:39vTkg0N2v+ZzuBlTeKmdOTU6iI=', 'Content-Length': '0'} 2023-09-29T22:34:26.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028fdede34a2299f5-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.502 DEBUG:boto:path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.503 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.503 DEBUG:boto:Method: GET 2023-09-29T22:34:26.503 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.503 DEBUG:boto:Data: 2023-09-29T22:34:26.504 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.504 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.504 DEBUG:boto:Params: {} 2023-09-29T22:34:26.504 DEBUG:boto:Token: None 2023-09-29T22:34:26.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000005 2023-09-29T22:34:26.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:39vTkg0N2v+ZzuBlTeKmdOTU6iI= 2023-09-29T22:34:26.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:39vTkg0N2v+ZzuBlTeKmdOTU6iI=', 'Content-Length': '0'} 2023-09-29T22:34:26.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b6077bf872a5c0e-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.508 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.508 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.508 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.508 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.508 DEBUG:boto:Method: GET 2023-09-29T22:34:26.508 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.508 DEBUG:boto:Data: 2023-09-29T22:34:26.509 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.509 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.509 DEBUG:boto:Params: {} 2023-09-29T22:34:26.509 DEBUG:boto:Token: None 2023-09-29T22:34:26.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000943051f50fea735d-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.514 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.515 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.515 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.515 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.515 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.515 DEBUG:boto:Method: GET 2023-09-29T22:34:26.515 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000005 2023-09-29T22:34:26.515 DEBUG:boto:Data: 2023-09-29T22:34:26.515 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.516 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.516 DEBUG:boto:Params: {} 2023-09-29T22:34:26.516 DEBUG:boto:Token: None 2023-09-29T22:34:26.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.516 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009acecdab9308a068-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.566 DEBUG:boto:b'hcnphz-58obj000000000000000000051000falseobj000000000000000000052023-09-29T22:34:16.215Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-29T22:34:26.567 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.567 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.567 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.567 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.568 DEBUG:boto:Method: GET 2023-09-29T22:34:26.568 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.568 DEBUG:boto:Data: 2023-09-29T22:34:26.568 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.568 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.568 DEBUG:boto:Params: {} 2023-09-29T22:34:26.568 DEBUG:boto:Token: None 2023-09-29T22:34:26.569 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks= 2023-09-29T22:34:26.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks=', 'Content-Length': '0'} 2023-09-29T22:34:26.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dab187cf0fd39fa5-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.572 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.572 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.572 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.573 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.573 DEBUG:boto:Method: GET 2023-09-29T22:34:26.573 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.573 DEBUG:boto:Data: 2023-09-29T22:34:26.573 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.573 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.573 DEBUG:boto:Params: {} 2023-09-29T22:34:26.574 DEBUG:boto:Token: None 2023-09-29T22:34:26.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks= 2023-09-29T22:34:26.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks=', 'Content-Length': '0'} 2023-09-29T22:34:26.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d486a9225793c8a-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.577 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.577 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.577 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.577 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.578 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.578 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.578 DEBUG:boto:Data: 2023-09-29T22:34:26.578 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.578 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.578 DEBUG:boto:Params: {} 2023-09-29T22:34:26.578 DEBUG:boto:Token: None 2023-09-29T22:34:26.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VVUbzGoayCUeTMZV9AeVaCECrHY= 2023-09-29T22:34:26.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VVUbzGoayCUeTMZV9AeVaCECrHY=', 'Content-Length': '0'} 2023-09-29T22:34:26.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007581db8b6ae39999-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.582 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.583 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.583 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.583 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.583 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.584 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.584 DEBUG:boto:Data: 2023-09-29T22:34:26.584 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.584 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.584 DEBUG:boto:Params: {} 2023-09-29T22:34:26.584 DEBUG:boto:Token: None 2023-09-29T22:34:26.585 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.585 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VVUbzGoayCUeTMZV9AeVaCECrHY= 2023-09-29T22:34:26.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VVUbzGoayCUeTMZV9AeVaCECrHY=', 'Content-Length': '0'} 2023-09-29T22:34:26.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf72b0fa07ba297a-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-29T22:34:26.588 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.588 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.588 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.589 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.589 DEBUG:boto:Method: GET 2023-09-29T22:34:26.589 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.589 DEBUG:boto:Data: 2023-09-29T22:34:26.589 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.589 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.589 DEBUG:boto:Params: {} 2023-09-29T22:34:26.589 DEBUG:boto:Token: None 2023-09-29T22:34:26.590 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.590 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks= 2023-09-29T22:34:26.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks=', 'Content-Length': '0'} 2023-09-29T22:34:26.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d6fb0bb69b9cb03-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.593 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.594 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.594 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.594 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.594 DEBUG:boto:Method: GET 2023-09-29T22:34:26.594 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.595 DEBUG:boto:Data: 2023-09-29T22:34:26.595 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.595 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.596 DEBUG:boto:Params: {} 2023-09-29T22:34:26.596 DEBUG:boto:Token: None 2023-09-29T22:34:26.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006?versionId=null 2023-09-29T22:34:26.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks= 2023-09-29T22:34:26.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lD034MHScxcjxdGRJEYoQHdgWks=', 'Content-Length': '0'} 2023-09-29T22:34:26.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051fb6af65a6a6971-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.600 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.600 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.600 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.600 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.600 DEBUG:boto:Method: GET 2023-09-29T22:34:26.600 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.600 DEBUG:boto:Data: 2023-09-29T22:34:26.601 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.601 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.601 DEBUG:boto:Params: {} 2023-09-29T22:34:26.601 DEBUG:boto:Token: None 2023-09-29T22:34:26.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2d1b5a7e095ed1d-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.606 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.606 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.606 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.607 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.607 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.607 DEBUG:boto:Method: GET 2023-09-29T22:34:26.607 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.607 DEBUG:boto:Data: 2023-09-29T22:34:26.607 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.607 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.607 DEBUG:boto:Params: {} 2023-09-29T22:34:26.608 DEBUG:boto:Token: None 2023-09-29T22:34:26.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bed512823212f228-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.662 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.663 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.663 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.663 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.663 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.664 DEBUG:boto:Data: 2023-09-29T22:34:26.664 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.664 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.664 DEBUG:boto:Params: {} 2023-09-29T22:34:26.664 DEBUG:boto:Token: None 2023-09-29T22:34:26.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tYOW5rYoa4QmKL6XjkdpBCkPt/o= 2023-09-29T22:34:26.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tYOW5rYoa4QmKL6XjkdpBCkPt/o=', 'Content-Length': '0'} 2023-09-29T22:34:26.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e55a62b78cba205-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.668 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.668 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.668 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.668 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.668 DEBUG:boto:Data: 2023-09-29T22:34:26.669 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.669 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.669 DEBUG:boto:Params: {} 2023-09-29T22:34:26.669 DEBUG:boto:Token: None 2023-09-29T22:34:26.669 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tYOW5rYoa4QmKL6XjkdpBCkPt/o= 2023-09-29T22:34:26.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tYOW5rYoa4QmKL6XjkdpBCkPt/o=', 'Content-Length': '0'} 2023-09-29T22:34:26.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001fd988d585e8372-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000006 2023-09-29T22:34:26.673 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.673 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.673 DEBUG:boto:Method: GET 2023-09-29T22:34:26.673 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.673 DEBUG:boto:Data: 2023-09-29T22:34:26.673 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.674 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.674 DEBUG:boto:Params: {} 2023-09-29T22:34:26.674 DEBUG:boto:Token: None 2023-09-29T22:34:26.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SccJitZNNoFeoP5R1CQiq3pAOwE= 2023-09-29T22:34:26.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SccJitZNNoFeoP5R1CQiq3pAOwE=', 'Content-Length': '0'} 2023-09-29T22:34:26.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f194c870f79e7a1-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.722 DEBUG:boto:path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.723 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.723 DEBUG:boto:Method: GET 2023-09-29T22:34:26.723 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.723 DEBUG:boto:Data: 2023-09-29T22:34:26.723 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.724 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.724 DEBUG:boto:Params: {} 2023-09-29T22:34:26.724 DEBUG:boto:Token: None 2023-09-29T22:34:26.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000006 2023-09-29T22:34:26.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SccJitZNNoFeoP5R1CQiq3pAOwE= 2023-09-29T22:34:26.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SccJitZNNoFeoP5R1CQiq3pAOwE=', 'Content-Length': '0'} 2023-09-29T22:34:26.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa73b5b11b702cd8-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.728 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.728 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.728 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.728 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.728 DEBUG:boto:Method: GET 2023-09-29T22:34:26.728 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.728 DEBUG:boto:Data: 2023-09-29T22:34:26.729 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.729 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.729 DEBUG:boto:Params: {} 2023-09-29T22:34:26.729 DEBUG:boto:Token: None 2023-09-29T22:34:26.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c12524add6a85e6-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.734 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.735 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.735 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.735 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.735 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.736 DEBUG:boto:Method: GET 2023-09-29T22:34:26.736 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000006 2023-09-29T22:34:26.736 DEBUG:boto:Data: 2023-09-29T22:34:26.736 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.736 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.736 DEBUG:boto:Params: {} 2023-09-29T22:34:26.737 DEBUG:boto:Token: None 2023-09-29T22:34:26.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.737 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f56dadd811d0b82-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.745 DEBUG:boto:b'hcnphz-58obj000000000000000000061000falseobj000000000000000000062023-09-29T22:34:14.431Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-09-29T22:34:26.745 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.745 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.746 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.746 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.746 DEBUG:boto:Method: GET 2023-09-29T22:34:26.746 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.746 DEBUG:boto:Data: 2023-09-29T22:34:26.746 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.747 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.747 DEBUG:boto:Params: {} 2023-09-29T22:34:26.747 DEBUG:boto:Token: None 2023-09-29T22:34:26.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.747 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI= 2023-09-29T22:34:26.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI=', 'Content-Length': '0'} 2023-09-29T22:34:26.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e78ad44278c3225c-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.750 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.750 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.750 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.750 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.750 DEBUG:boto:Method: GET 2023-09-29T22:34:26.751 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.751 DEBUG:boto:Data: 2023-09-29T22:34:26.751 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.751 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.751 DEBUG:boto:Params: {} 2023-09-29T22:34:26.751 DEBUG:boto:Token: None 2023-09-29T22:34:26.752 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.752 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI= 2023-09-29T22:34:26.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI=', 'Content-Length': '0'} 2023-09-29T22:34:26.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee379864cf1182ae-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.798 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.799 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.799 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.799 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.799 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.800 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.800 DEBUG:boto:Data: 2023-09-29T22:34:26.800 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.800 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.800 DEBUG:boto:Params: {} 2023-09-29T22:34:26.801 DEBUG:boto:Token: None 2023-09-29T22:34:26.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iXblvkoYrp8fszyp69Q1KoMDEV0= 2023-09-29T22:34:26.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iXblvkoYrp8fszyp69Q1KoMDEV0=', 'Content-Length': '0'} 2023-09-29T22:34:26.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9e7ad446aa582e4-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.805 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.805 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.805 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.806 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.806 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.806 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.806 DEBUG:boto:Data: 2023-09-29T22:34:26.806 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.806 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.807 DEBUG:boto:Params: {} 2023-09-29T22:34:26.807 DEBUG:boto:Token: None 2023-09-29T22:34:26.807 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iXblvkoYrp8fszyp69Q1KoMDEV0= 2023-09-29T22:34:26.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iXblvkoYrp8fszyp69Q1KoMDEV0=', 'Content-Length': '0'} 2023-09-29T22:34:26.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fed47946e37e3b0c-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-09-29T22:34:26.811 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.811 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.811 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.811 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.811 DEBUG:boto:Method: GET 2023-09-29T22:34:26.811 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.811 DEBUG:boto:Data: 2023-09-29T22:34:26.812 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.812 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.812 DEBUG:boto:Params: {} 2023-09-29T22:34:26.812 DEBUG:boto:Token: None 2023-09-29T22:34:26.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI= 2023-09-29T22:34:26.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI=', 'Content-Length': '0'} 2023-09-29T22:34:26.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f95c7c256846e541-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.816 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.816 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.816 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.816 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.816 DEBUG:boto:Method: GET 2023-09-29T22:34:26.817 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.817 DEBUG:boto:Data: 2023-09-29T22:34:26.817 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.817 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.817 DEBUG:boto:Params: {} 2023-09-29T22:34:26.817 DEBUG:boto:Token: None 2023-09-29T22:34:26.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007?versionId=null 2023-09-29T22:34:26.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI= 2023-09-29T22:34:26.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wzJhlhlQvEFIikwnafSQ0sg+6RI=', 'Content-Length': '0'} 2023-09-29T22:34:26.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037de93580bbd5c0a-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.821 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.821 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.821 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.821 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.822 DEBUG:boto:Method: GET 2023-09-29T22:34:26.822 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.822 DEBUG:boto:Data: 2023-09-29T22:34:26.822 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.822 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.822 DEBUG:boto:Params: {} 2023-09-29T22:34:26.822 DEBUG:boto:Token: None 2023-09-29T22:34:26.822 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.827 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000348f0c0ce484dbf7-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.874 DEBUG:boto:b'hcnphz-58obj000000000000000000071000falseobj000000000000000000072023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.875 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.875 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.875 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.876 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.876 DEBUG:boto:Method: GET 2023-09-29T22:34:26.876 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.876 DEBUG:boto:Data: 2023-09-29T22:34:26.876 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.876 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.876 DEBUG:boto:Params: {} 2023-09-29T22:34:26.877 DEBUG:boto:Token: None 2023-09-29T22:34:26.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.886 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daf71129b936e53b-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.887 DEBUG:boto:b'hcnphz-58obj000000000000000000071000falseobj000000000000000000072023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.887 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.887 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.887 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.888 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.888 DEBUG:boto:Data: 2023-09-29T22:34:26.888 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.888 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.888 DEBUG:boto:Params: {} 2023-09-29T22:34:26.888 DEBUG:boto:Token: None 2023-09-29T22:34:26.889 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PGEh6LlgkmBafUu96nxJzHcvMqs= 2023-09-29T22:34:26.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PGEh6LlgkmBafUu96nxJzHcvMqs=', 'Content-Length': '0'} 2023-09-29T22:34:26.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec823a0701727c5-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.892 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.892 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.892 DEBUG:boto:Method: HEAD 2023-09-29T22:34:26.892 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.892 DEBUG:boto:Data: 2023-09-29T22:34:26.892 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.893 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.893 DEBUG:boto:Params: {} 2023-09-29T22:34:26.893 DEBUG:boto:Token: None 2023-09-29T22:34:26.893 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PGEh6LlgkmBafUu96nxJzHcvMqs= 2023-09-29T22:34:26.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PGEh6LlgkmBafUu96nxJzHcvMqs=', 'Content-Length': '0'} 2023-09-29T22:34:26.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e76a4f884b9476a-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000007 2023-09-29T22:34:26.895 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.896 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.896 DEBUG:boto:Method: GET 2023-09-29T22:34:26.896 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.896 DEBUG:boto:Data: 2023-09-29T22:34:26.896 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.896 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.896 DEBUG:boto:Params: {} 2023-09-29T22:34:26.896 DEBUG:boto:Token: None 2023-09-29T22:34:26.897 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.897 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLl4r3oEdjQTJzqcQ3Q+nuPKvCI= 2023-09-29T22:34:26.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLl4r3oEdjQTJzqcQ3Q+nuPKvCI=', 'Content-Length': '0'} 2023-09-29T22:34:26.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007ad9083091f1864-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.899 DEBUG:boto:path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.899 DEBUG:boto:Method: GET 2023-09-29T22:34:26.899 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.900 DEBUG:boto:Data: 2023-09-29T22:34:26.900 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.900 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.900 DEBUG:boto:Params: {} 2023-09-29T22:34:26.900 DEBUG:boto:Token: None 2023-09-29T22:34:26.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000007 2023-09-29T22:34:26.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLl4r3oEdjQTJzqcQ3Q+nuPKvCI= 2023-09-29T22:34:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLl4r3oEdjQTJzqcQ3Q+nuPKvCI=', 'Content-Length': '0'} 2023-09-29T22:34:26.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f2972f8d7d2613e-00651750f2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.950 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.951 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.951 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.951 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.951 DEBUG:boto:Method: GET 2023-09-29T22:34:26.951 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.951 DEBUG:boto:Data: 2023-09-29T22:34:26.952 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.952 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.952 DEBUG:boto:Params: {} 2023-09-29T22:34:26.952 DEBUG:boto:Token: None 2023-09-29T22:34:26.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d747b6b9ed45115-00651750f2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.957 DEBUG:boto:b'hcnphz-58obj000000000000000000071000falseobj000000000000000000072023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.957 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:26.958 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:26.958 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.958 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.958 DEBUG:boto:Method: GET 2023-09-29T22:34:26.958 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000007 2023-09-29T22:34:26.958 DEBUG:boto:Data: 2023-09-29T22:34:26.958 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:26.958 DEBUG:boto:Port: 8000 2023-09-29T22:34:26.959 DEBUG:boto:Params: {} 2023-09-29T22:34:26.959 DEBUG:boto:Token: None 2023-09-29T22:34:26.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/ 2023-09-29T22:34:26.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw= 2023-09-29T22:34:26.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AgIVOTqsyUt4+w/MFRGRO56VRCw=', 'Content-Length': '0'} 2023-09-29T22:34:26.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020e0b941ffc2f819-00651750f2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:26.967 DEBUG:boto:b'hcnphz-58obj000000000000000000071000falseobj000000000000000000072023-09-29T22:34:16.364Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:26.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-09-29T22:34:26.967 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:26.967 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:26.968 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:26.968 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:26.968 DEBUG:boto:Method: GET 2023-09-29T22:34:26.968 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:26.968 DEBUG:boto:Data: 2023-09-29T22:34:26.968 DEBUG:boto:Headers: {} 2023-09-29T22:34:26.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:26.968 DEBUG:boto:Port: 8001 2023-09-29T22:34:26.968 DEBUG:boto:Params: {} 2023-09-29T22:34:26.969 DEBUG:boto:Token: None 2023-09-29T22:34:26.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:26 GMT /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:26.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W+V5o6omUOZKnTqI97vGwCT3FNg= 2023-09-29T22:34:26.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W+V5o6omUOZKnTqI97vGwCT3FNg=', 'Content-Length': '0'} 2023-09-29T22:34:26.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e7bdb40d104712-00651750f2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.014 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.015 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.015 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.015 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.015 DEBUG:boto:Method: GET 2023-09-29T22:34:27.015 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.015 DEBUG:boto:Data: 2023-09-29T22:34:27.015 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.015 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.016 DEBUG:boto:Params: {} 2023-09-29T22:34:27.016 DEBUG:boto:Token: None 2023-09-29T22:34:27.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Qg9ngWLCeBM1SjiyCjh+HDhyJQ0= 2023-09-29T22:34:27.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Qg9ngWLCeBM1SjiyCjh+HDhyJQ0=', 'Content-Length': '0'} 2023-09-29T22:34:27.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8270003104a4389-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.019 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.020 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.020 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.020 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.020 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.020 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.020 DEBUG:boto:Data: 2023-09-29T22:34:27.020 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.021 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.021 DEBUG:boto:Params: {} 2023-09-29T22:34:27.021 DEBUG:boto:Token: None 2023-09-29T22:34:27.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NbxfaxbwvEvbXqq8yjpLzJBqW0g= 2023-09-29T22:34:27.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NbxfaxbwvEvbXqq8yjpLzJBqW0g=', 'Content-Length': '0'} 2023-09-29T22:34:27.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a20e566e323698-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.025 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.025 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.025 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.025 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.025 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.025 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.025 DEBUG:boto:Data: 2023-09-29T22:34:27.026 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.026 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.026 DEBUG:boto:Params: {} 2023-09-29T22:34:27.026 DEBUG:boto:Token: None 2023-09-29T22:34:27.026 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NbxfaxbwvEvbXqq8yjpLzJBqW0g= 2023-09-29T22:34:27.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NbxfaxbwvEvbXqq8yjpLzJBqW0g=', 'Content-Length': '0'} 2023-09-29T22:34:27.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d701f78a29eacc-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-09-29T22:34:27.030 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.030 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.030 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.030 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.030 DEBUG:boto:Method: GET 2023-09-29T22:34:27.031 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.031 DEBUG:boto:Data: 2023-09-29T22:34:27.031 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.031 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.031 DEBUG:boto:Params: {} 2023-09-29T22:34:27.031 DEBUG:boto:Token: None 2023-09-29T22:34:27.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Qg9ngWLCeBM1SjiyCjh+HDhyJQ0= 2023-09-29T22:34:27.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Qg9ngWLCeBM1SjiyCjh+HDhyJQ0=', 'Content-Length': '0'} 2023-09-29T22:34:27.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061224ee6a24c6e01-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.035 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.035 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.035 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.035 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.035 DEBUG:boto:Method: GET 2023-09-29T22:34:27.035 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.035 DEBUG:boto:Data: 2023-09-29T22:34:27.035 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.036 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.036 DEBUG:boto:Params: {} 2023-09-29T22:34:27.036 DEBUG:boto:Token: None 2023-09-29T22:34:27.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008?versionId=null 2023-09-29T22:34:27.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Qg9ngWLCeBM1SjiyCjh+HDhyJQ0= 2023-09-29T22:34:27.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Qg9ngWLCeBM1SjiyCjh+HDhyJQ0=', 'Content-Length': '0'} 2023-09-29T22:34:27.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000405144caadb2ca9a-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.039 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.039 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.039 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.040 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.040 DEBUG:boto:Method: GET 2023-09-29T22:34:27.040 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.040 DEBUG:boto:Data: 2023-09-29T22:34:27.040 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.040 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.040 DEBUG:boto:Params: {} 2023-09-29T22:34:27.040 DEBUG:boto:Token: None 2023-09-29T22:34:27.041 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d03ced3f5f1fd58-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.090 DEBUG:boto:b'hcnphz-58obj000000000000000000081000falseobj000000000000000000082023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.091 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.091 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.091 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.091 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.092 DEBUG:boto:Method: GET 2023-09-29T22:34:27.092 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.092 DEBUG:boto:Data: 2023-09-29T22:34:27.092 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.092 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.092 DEBUG:boto:Params: {} 2023-09-29T22:34:27.092 DEBUG:boto:Token: None 2023-09-29T22:34:27.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006565c766e476c760-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.102 DEBUG:boto:b'hcnphz-58obj000000000000000000081000falseobj000000000000000000082023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.103 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.103 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.104 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.104 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.104 DEBUG:boto:Data: 2023-09-29T22:34:27.104 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.104 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.104 DEBUG:boto:Params: {} 2023-09-29T22:34:27.105 DEBUG:boto:Token: None 2023-09-29T22:34:27.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eZyXZd97LDjA4f4ksvHQDCjeFF8= 2023-09-29T22:34:27.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eZyXZd97LDjA4f4ksvHQDCjeFF8=', 'Content-Length': '0'} 2023-09-29T22:34:27.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ede5aa494a8b1f3-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.107 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.107 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.108 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.108 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.108 DEBUG:boto:Data: 2023-09-29T22:34:27.108 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.108 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.108 DEBUG:boto:Params: {} 2023-09-29T22:34:27.108 DEBUG:boto:Token: None 2023-09-29T22:34:27.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eZyXZd97LDjA4f4ksvHQDCjeFF8= 2023-09-29T22:34:27.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eZyXZd97LDjA4f4ksvHQDCjeFF8=', 'Content-Length': '0'} 2023-09-29T22:34:27.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2d065f9b475440-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000008 2023-09-29T22:34:27.111 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.111 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.111 DEBUG:boto:Method: GET 2023-09-29T22:34:27.111 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.111 DEBUG:boto:Data: 2023-09-29T22:34:27.112 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.112 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.112 DEBUG:boto:Params: {} 2023-09-29T22:34:27.112 DEBUG:boto:Token: None 2023-09-29T22:34:27.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FHPHWBuyUqiV+w28bCJT4pmk0H4= 2023-09-29T22:34:27.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FHPHWBuyUqiV+w28bCJT4pmk0H4=', 'Content-Length': '0'} 2023-09-29T22:34:27.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee7d92eb327e7fb1-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.115 DEBUG:boto:path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.115 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.115 DEBUG:boto:Method: GET 2023-09-29T22:34:27.115 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.115 DEBUG:boto:Data: 2023-09-29T22:34:27.115 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.115 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.116 DEBUG:boto:Params: {} 2023-09-29T22:34:27.116 DEBUG:boto:Token: None 2023-09-29T22:34:27.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000008 2023-09-29T22:34:27.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FHPHWBuyUqiV+w28bCJT4pmk0H4= 2023-09-29T22:34:27.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FHPHWBuyUqiV+w28bCJT4pmk0H4=', 'Content-Length': '0'} 2023-09-29T22:34:27.118 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1c3b22306bd313d-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.162 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.163 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.163 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.163 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.163 DEBUG:boto:Method: GET 2023-09-29T22:34:27.163 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.163 DEBUG:boto:Data: 2023-09-29T22:34:27.163 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.163 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.164 DEBUG:boto:Params: {} 2023-09-29T22:34:27.164 DEBUG:boto:Token: None 2023-09-29T22:34:27.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f1ed70a06f6532c-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.169 DEBUG:boto:b'hcnphz-58obj000000000000000000081000falseobj000000000000000000082023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.169 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.170 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.170 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.170 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.170 DEBUG:boto:Method: GET 2023-09-29T22:34:27.170 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000008 2023-09-29T22:34:27.170 DEBUG:boto:Data: 2023-09-29T22:34:27.170 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.170 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.170 DEBUG:boto:Params: {} 2023-09-29T22:34:27.171 DEBUG:boto:Token: None 2023-09-29T22:34:27.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f14bb3d97cf3b74-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.179 DEBUG:boto:b'hcnphz-58obj000000000000000000081000falseobj000000000000000000082023-09-29T22:34:16.294Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-09-29T22:34:27.180 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.180 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.180 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.180 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.180 DEBUG:boto:Method: GET 2023-09-29T22:34:27.180 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.180 DEBUG:boto:Data: 2023-09-29T22:34:27.180 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.181 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.181 DEBUG:boto:Params: {} 2023-09-29T22:34:27.181 DEBUG:boto:Token: None 2023-09-29T22:34:27.181 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc= 2023-09-29T22:34:27.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc=', 'Content-Length': '0'} 2023-09-29T22:34:27.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dc2cd45295afb4d-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.226 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.227 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.227 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.227 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.227 DEBUG:boto:Method: GET 2023-09-29T22:34:27.227 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.227 DEBUG:boto:Data: 2023-09-29T22:34:27.227 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.228 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.228 DEBUG:boto:Params: {} 2023-09-29T22:34:27.228 DEBUG:boto:Token: None 2023-09-29T22:34:27.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.228 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc= 2023-09-29T22:34:27.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc=', 'Content-Length': '0'} 2023-09-29T22:34:27.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b702ec81e5e1ac7d-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.232 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.232 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.232 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.232 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.232 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.232 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.233 DEBUG:boto:Data: 2023-09-29T22:34:27.233 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.233 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.233 DEBUG:boto:Params: {} 2023-09-29T22:34:27.233 DEBUG:boto:Token: None 2023-09-29T22:34:27.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:93nU9H0VfZ3ixAaKf+U+iq7IkUo= 2023-09-29T22:34:27.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:93nU9H0VfZ3ixAaKf+U+iq7IkUo=', 'Content-Length': '0'} 2023-09-29T22:34:27.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054c99b54acc4817d-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.237 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.237 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.237 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.237 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.237 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.237 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.237 DEBUG:boto:Data: 2023-09-29T22:34:27.238 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.238 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.238 DEBUG:boto:Params: {} 2023-09-29T22:34:27.238 DEBUG:boto:Token: None 2023-09-29T22:34:27.238 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:93nU9H0VfZ3ixAaKf+U+iq7IkUo= 2023-09-29T22:34:27.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:93nU9H0VfZ3ixAaKf+U+iq7IkUo=', 'Content-Length': '0'} 2023-09-29T22:34:27.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ceffcebcbcc4ff6-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-09-29T22:34:27.242 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.242 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.242 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.242 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.242 DEBUG:boto:Method: GET 2023-09-29T22:34:27.242 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.242 DEBUG:boto:Data: 2023-09-29T22:34:27.243 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.243 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.243 DEBUG:boto:Params: {} 2023-09-29T22:34:27.243 DEBUG:boto:Token: None 2023-09-29T22:34:27.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc= 2023-09-29T22:34:27.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc=', 'Content-Length': '0'} 2023-09-29T22:34:27.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000165ced72845e932f-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.246 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.247 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.247 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.247 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.247 DEBUG:boto:Method: GET 2023-09-29T22:34:27.247 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.248 DEBUG:boto:Data: 2023-09-29T22:34:27.248 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.248 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.248 DEBUG:boto:Params: {} 2023-09-29T22:34:27.248 DEBUG:boto:Token: None 2023-09-29T22:34:27.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009?versionId=null 2023-09-29T22:34:27.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc= 2023-09-29T22:34:27.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6rd2b0jRq9Z/SnffhhWezHkmc=', 'Content-Length': '0'} 2023-09-29T22:34:27.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b667098373ba7df-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.252 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.252 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.252 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.252 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.252 DEBUG:boto:Method: GET 2023-09-29T22:34:27.252 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.252 DEBUG:boto:Data: 2023-09-29T22:34:27.253 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.253 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.253 DEBUG:boto:Params: {} 2023-09-29T22:34:27.253 DEBUG:boto:Token: None 2023-09-29T22:34:27.253 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.253 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009cf0da699d65cd2a-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.258 DEBUG:boto:b'hcnphz-58obj000000000000000000091000falseobj000000000000000000092023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.259 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.259 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.259 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.259 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.259 DEBUG:boto:Method: GET 2023-09-29T22:34:27.259 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.260 DEBUG:boto:Data: 2023-09-29T22:34:27.260 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.260 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.260 DEBUG:boto:Params: {} 2023-09-29T22:34:27.260 DEBUG:boto:Token: None 2023-09-29T22:34:27.261 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.269 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cefd976519c9d700-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.310 DEBUG:boto:b'hcnphz-58obj000000000000000000091000falseobj000000000000000000092023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.311 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.311 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.311 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.311 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.311 DEBUG:boto:Data: 2023-09-29T22:34:27.311 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.312 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.312 DEBUG:boto:Params: {} 2023-09-29T22:34:27.312 DEBUG:boto:Token: None 2023-09-29T22:34:27.312 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:obfxuVuX68fCHzXJM1j/MVF/imU= 2023-09-29T22:34:27.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:obfxuVuX68fCHzXJM1j/MVF/imU=', 'Content-Length': '0'} 2023-09-29T22:34:27.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff8402d0e99d9ba4-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.315 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.315 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.316 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.316 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.316 DEBUG:boto:Data: 2023-09-29T22:34:27.316 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.316 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.316 DEBUG:boto:Params: {} 2023-09-29T22:34:27.316 DEBUG:boto:Token: None 2023-09-29T22:34:27.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:obfxuVuX68fCHzXJM1j/MVF/imU= 2023-09-29T22:34:27.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:obfxuVuX68fCHzXJM1j/MVF/imU=', 'Content-Length': '0'} 2023-09-29T22:34:27.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016ce8a21c863acdf-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000009 2023-09-29T22:34:27.320 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.320 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.320 DEBUG:boto:Method: GET 2023-09-29T22:34:27.320 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.320 DEBUG:boto:Data: 2023-09-29T22:34:27.320 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.321 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.321 DEBUG:boto:Params: {} 2023-09-29T22:34:27.321 DEBUG:boto:Token: None 2023-09-29T22:34:27.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b82PTRSU64f+J5peRXt2XPbnJsk= 2023-09-29T22:34:27.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b82PTRSU64f+J5peRXt2XPbnJsk=', 'Content-Length': '0'} 2023-09-29T22:34:27.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000301e2b3558d5929e-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.370 DEBUG:boto:path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.371 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.371 DEBUG:boto:Method: GET 2023-09-29T22:34:27.371 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.371 DEBUG:boto:Data: 2023-09-29T22:34:27.371 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.371 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.372 DEBUG:boto:Params: {} 2023-09-29T22:34:27.372 DEBUG:boto:Token: None 2023-09-29T22:34:27.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000009 2023-09-29T22:34:27.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b82PTRSU64f+J5peRXt2XPbnJsk= 2023-09-29T22:34:27.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b82PTRSU64f+J5peRXt2XPbnJsk=', 'Content-Length': '0'} 2023-09-29T22:34:27.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089bbb093db9f1f9a-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.376 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.376 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.376 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.376 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.376 DEBUG:boto:Method: GET 2023-09-29T22:34:27.376 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.376 DEBUG:boto:Data: 2023-09-29T22:34:27.376 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.377 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.377 DEBUG:boto:Params: {} 2023-09-29T22:34:27.377 DEBUG:boto:Token: None 2023-09-29T22:34:27.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5b85c427e840e05-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.383 DEBUG:boto:b'hcnphz-58obj000000000000000000091000falseobj000000000000000000092023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.383 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.384 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.384 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.384 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.384 DEBUG:boto:Method: GET 2023-09-29T22:34:27.384 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000009 2023-09-29T22:34:27.384 DEBUG:boto:Data: 2023-09-29T22:34:27.384 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.384 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.384 DEBUG:boto:Params: {} 2023-09-29T22:34:27.385 DEBUG:boto:Token: None 2023-09-29T22:34:27.385 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088c4768b9175fea7-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.394 DEBUG:boto:b'hcnphz-58obj000000000000000000091000falseobj000000000000000000092023-09-29T22:34:15.425Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-29T22:34:27.394 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.395 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.395 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.395 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.395 DEBUG:boto:Method: GET 2023-09-29T22:34:27.395 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.395 DEBUG:boto:Data: 2023-09-29T22:34:27.395 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.396 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.396 DEBUG:boto:Params: {} 2023-09-29T22:34:27.396 DEBUG:boto:Token: None 2023-09-29T22:34:27.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA= 2023-09-29T22:34:27.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA=', 'Content-Length': '0'} 2023-09-29T22:34:27.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000747698f55a8f972b-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.399 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.399 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.399 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.399 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.400 DEBUG:boto:Method: GET 2023-09-29T22:34:27.400 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.400 DEBUG:boto:Data: 2023-09-29T22:34:27.400 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.400 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.400 DEBUG:boto:Params: {} 2023-09-29T22:34:27.400 DEBUG:boto:Token: None 2023-09-29T22:34:27.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA= 2023-09-29T22:34:27.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA=', 'Content-Length': '0'} 2023-09-29T22:34:27.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000639e851869ef6f5c-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.450 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.451 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.451 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.451 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.451 DEBUG:boto:Data: 2023-09-29T22:34:27.451 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.452 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.452 DEBUG:boto:Params: {} 2023-09-29T22:34:27.452 DEBUG:boto:Token: None 2023-09-29T22:34:27.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OToHykzSF16j9oC5kAT7K9y5noM= 2023-09-29T22:34:27.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OToHykzSF16j9oC5kAT7K9y5noM=', 'Content-Length': '0'} 2023-09-29T22:34:27.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c838d287742b8f88-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.455 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.456 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.456 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.456 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.456 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.456 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.456 DEBUG:boto:Data: 2023-09-29T22:34:27.456 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.456 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.457 DEBUG:boto:Params: {} 2023-09-29T22:34:27.457 DEBUG:boto:Token: None 2023-09-29T22:34:27.457 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OToHykzSF16j9oC5kAT7K9y5noM= 2023-09-29T22:34:27.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OToHykzSF16j9oC5kAT7K9y5noM=', 'Content-Length': '0'} 2023-09-29T22:34:27.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000239b63984ee239c1-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-29T22:34:27.460 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.460 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.460 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.460 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.461 DEBUG:boto:Method: GET 2023-09-29T22:34:27.461 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.461 DEBUG:boto:Data: 2023-09-29T22:34:27.461 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.461 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.461 DEBUG:boto:Params: {} 2023-09-29T22:34:27.461 DEBUG:boto:Token: None 2023-09-29T22:34:27.462 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.462 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA= 2023-09-29T22:34:27.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA=', 'Content-Length': '0'} 2023-09-29T22:34:27.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000274ec7d8ad0d8f2-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.506 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.507 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.507 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.507 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.507 DEBUG:boto:Method: GET 2023-09-29T22:34:27.507 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.507 DEBUG:boto:Data: 2023-09-29T22:34:27.507 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.508 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.508 DEBUG:boto:Params: {} 2023-09-29T22:34:27.508 DEBUG:boto:Token: None 2023-09-29T22:34:27.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010?versionId=null 2023-09-29T22:34:27.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA= 2023-09-29T22:34:27.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K2M9FTe/EWEQnDszFle76fe7ATA=', 'Content-Length': '0'} 2023-09-29T22:34:27.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abe0a1802195ec7d-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.512 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.512 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.512 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.512 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.512 DEBUG:boto:Method: GET 2023-09-29T22:34:27.512 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.513 DEBUG:boto:Data: 2023-09-29T22:34:27.513 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.513 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.513 DEBUG:boto:Params: {} 2023-09-29T22:34:27.513 DEBUG:boto:Token: None 2023-09-29T22:34:27.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.518 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000067e31f3940a2e447-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.518 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.519 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.519 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.519 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.519 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.520 DEBUG:boto:Method: GET 2023-09-29T22:34:27.520 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.520 DEBUG:boto:Data: 2023-09-29T22:34:27.520 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.520 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.520 DEBUG:boto:Params: {} 2023-09-29T22:34:27.521 DEBUG:boto:Token: None 2023-09-29T22:34:27.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.530 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f39ec4aeab2ae2-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.530 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.531 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.531 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.531 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.531 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.531 DEBUG:boto:Data: 2023-09-29T22:34:27.531 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.532 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.532 DEBUG:boto:Params: {} 2023-09-29T22:34:27.532 DEBUG:boto:Token: None 2023-09-29T22:34:27.532 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uWzGJLlfOeYGaG0CzITfmz9dvcY= 2023-09-29T22:34:27.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uWzGJLlfOeYGaG0CzITfmz9dvcY=', 'Content-Length': '0'} 2023-09-29T22:34:27.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc5a8ba407a4bbeb-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.535 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.535 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.536 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.536 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.536 DEBUG:boto:Data: 2023-09-29T22:34:27.536 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.537 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.537 DEBUG:boto:Params: {} 2023-09-29T22:34:27.537 DEBUG:boto:Token: None 2023-09-29T22:34:27.537 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uWzGJLlfOeYGaG0CzITfmz9dvcY= 2023-09-29T22:34:27.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uWzGJLlfOeYGaG0CzITfmz9dvcY=', 'Content-Length': '0'} 2023-09-29T22:34:27.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000310f6ed4942d7c8b-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000010 2023-09-29T22:34:27.540 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.540 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.540 DEBUG:boto:Method: GET 2023-09-29T22:34:27.540 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.540 DEBUG:boto:Data: 2023-09-29T22:34:27.541 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.541 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.541 DEBUG:boto:Params: {} 2023-09-29T22:34:27.541 DEBUG:boto:Token: None 2023-09-29T22:34:27.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z23jpv217RgjCI97q1zGahpnVmc= 2023-09-29T22:34:27.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z23jpv217RgjCI97q1zGahpnVmc=', 'Content-Length': '0'} 2023-09-29T22:34:27.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003279450b25634d3f-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.590 DEBUG:boto:path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.591 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.591 DEBUG:boto:Method: GET 2023-09-29T22:34:27.591 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.591 DEBUG:boto:Data: 2023-09-29T22:34:27.591 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.591 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.591 DEBUG:boto:Params: {} 2023-09-29T22:34:27.591 DEBUG:boto:Token: None 2023-09-29T22:34:27.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000010 2023-09-29T22:34:27.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z23jpv217RgjCI97q1zGahpnVmc= 2023-09-29T22:34:27.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z23jpv217RgjCI97q1zGahpnVmc=', 'Content-Length': '0'} 2023-09-29T22:34:27.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c619cebe78449a41-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.595 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.595 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.595 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.595 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.596 DEBUG:boto:Method: GET 2023-09-29T22:34:27.596 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.596 DEBUG:boto:Data: 2023-09-29T22:34:27.596 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.596 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.596 DEBUG:boto:Params: {} 2023-09-29T22:34:27.596 DEBUG:boto:Token: None 2023-09-29T22:34:27.597 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d05a6e048077dd0-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.602 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.602 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.602 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.603 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.603 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.603 DEBUG:boto:Method: GET 2023-09-29T22:34:27.603 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000010 2023-09-29T22:34:27.603 DEBUG:boto:Data: 2023-09-29T22:34:27.603 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.603 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.603 DEBUG:boto:Params: {} 2023-09-29T22:34:27.604 DEBUG:boto:Token: None 2023-09-29T22:34:27.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a242094a31da4103-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.613 DEBUG:boto:b'hcnphz-58obj000000000000000000101000falseobj000000000000000000102023-09-29T22:34:15.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-29T22:34:27.613 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.613 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.613 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.614 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.614 DEBUG:boto:Method: GET 2023-09-29T22:34:27.614 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.614 DEBUG:boto:Data: 2023-09-29T22:34:27.614 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.614 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.614 DEBUG:boto:Params: {} 2023-09-29T22:34:27.614 DEBUG:boto:Token: None 2023-09-29T22:34:27.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno= 2023-09-29T22:34:27.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno=', 'Content-Length': '0'} 2023-09-29T22:34:27.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f1e1ee58886190f-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.617 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.618 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.618 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.618 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.618 DEBUG:boto:Method: GET 2023-09-29T22:34:27.618 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.618 DEBUG:boto:Data: 2023-09-29T22:34:27.618 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.619 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.619 DEBUG:boto:Params: {} 2023-09-29T22:34:27.619 DEBUG:boto:Token: None 2023-09-29T22:34:27.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno= 2023-09-29T22:34:27.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno=', 'Content-Length': '0'} 2023-09-29T22:34:27.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a998453971732a45-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.622 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.622 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.622 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.622 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.622 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.622 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.622 DEBUG:boto:Data: 2023-09-29T22:34:27.623 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.623 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.623 DEBUG:boto:Params: {} 2023-09-29T22:34:27.623 DEBUG:boto:Token: None 2023-09-29T22:34:27.623 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zlyBRI35Ae+/lcufvobiEst/ePU= 2023-09-29T22:34:27.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zlyBRI35Ae+/lcufvobiEst/ePU=', 'Content-Length': '0'} 2023-09-29T22:34:27.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f60fb95b2e3542-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.626 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.626 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.626 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.626 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.626 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.626 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.626 DEBUG:boto:Data: 2023-09-29T22:34:27.626 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.626 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.627 DEBUG:boto:Params: {} 2023-09-29T22:34:27.627 DEBUG:boto:Token: None 2023-09-29T22:34:27.627 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zlyBRI35Ae+/lcufvobiEst/ePU= 2023-09-29T22:34:27.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zlyBRI35Ae+/lcufvobiEst/ePU=', 'Content-Length': '0'} 2023-09-29T22:34:27.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cafec85a0e2f58b9-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-29T22:34:27.629 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.630 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.630 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.630 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.630 DEBUG:boto:Method: GET 2023-09-29T22:34:27.630 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.630 DEBUG:boto:Data: 2023-09-29T22:34:27.630 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.630 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.630 DEBUG:boto:Params: {} 2023-09-29T22:34:27.631 DEBUG:boto:Token: None 2023-09-29T22:34:27.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno= 2023-09-29T22:34:27.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno=', 'Content-Length': '0'} 2023-09-29T22:34:27.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003631c73e23590cce-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.633 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.634 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.634 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.634 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.634 DEBUG:boto:Method: GET 2023-09-29T22:34:27.634 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.634 DEBUG:boto:Data: 2023-09-29T22:34:27.634 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.634 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.635 DEBUG:boto:Params: {} 2023-09-29T22:34:27.635 DEBUG:boto:Token: None 2023-09-29T22:34:27.635 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011?versionId=null 2023-09-29T22:34:27.635 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno= 2023-09-29T22:34:27.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLY9E6sKXkM9spjnuRSj5VWHBno=', 'Content-Length': '0'} 2023-09-29T22:34:27.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db7b875727fa576a-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.638 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.638 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.638 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.638 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.638 DEBUG:boto:Method: GET 2023-09-29T22:34:27.638 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.639 DEBUG:boto:Data: 2023-09-29T22:34:27.639 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.639 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.639 DEBUG:boto:Params: {} 2023-09-29T22:34:27.639 DEBUG:boto:Token: None 2023-09-29T22:34:27.639 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126c78e4d53875be-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.643 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.644 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.644 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.644 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.644 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.644 DEBUG:boto:Method: GET 2023-09-29T22:34:27.644 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.645 DEBUG:boto:Data: 2023-09-29T22:34:27.645 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.645 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.645 DEBUG:boto:Params: {} 2023-09-29T22:34:27.645 DEBUG:boto:Token: None 2023-09-29T22:34:27.645 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fb06c58d0733ba8-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.653 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.654 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.654 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.654 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.654 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.654 DEBUG:boto:Data: 2023-09-29T22:34:27.654 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.655 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.655 DEBUG:boto:Params: {} 2023-09-29T22:34:27.655 DEBUG:boto:Token: None 2023-09-29T22:34:27.655 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KNDF6LL2EwNCq9k+vMOhEyFXI+8= 2023-09-29T22:34:27.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNDF6LL2EwNCq9k+vMOhEyFXI+8=', 'Content-Length': '0'} 2023-09-29T22:34:27.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007ff25015fb07f23-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.657 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.658 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.658 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.658 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.658 DEBUG:boto:Data: 2023-09-29T22:34:27.658 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.658 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.659 DEBUG:boto:Params: {} 2023-09-29T22:34:27.659 DEBUG:boto:Token: None 2023-09-29T22:34:27.659 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KNDF6LL2EwNCq9k+vMOhEyFXI+8= 2023-09-29T22:34:27.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNDF6LL2EwNCq9k+vMOhEyFXI+8=', 'Content-Length': '0'} 2023-09-29T22:34:27.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b61bddf4e6a4cc8-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000011 2023-09-29T22:34:27.661 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.662 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.662 DEBUG:boto:Method: GET 2023-09-29T22:34:27.662 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.662 DEBUG:boto:Data: 2023-09-29T22:34:27.662 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.662 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.662 DEBUG:boto:Params: {} 2023-09-29T22:34:27.662 DEBUG:boto:Token: None 2023-09-29T22:34:27.663 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WDodDIlRiiHxpgKWlRzH8Mrclug= 2023-09-29T22:34:27.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WDodDIlRiiHxpgKWlRzH8Mrclug=', 'Content-Length': '0'} 2023-09-29T22:34:27.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baeb517072b4e8f0-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.706 DEBUG:boto:path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.707 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.707 DEBUG:boto:Method: GET 2023-09-29T22:34:27.707 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.707 DEBUG:boto:Data: 2023-09-29T22:34:27.707 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.707 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.707 DEBUG:boto:Params: {} 2023-09-29T22:34:27.708 DEBUG:boto:Token: None 2023-09-29T22:34:27.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000011 2023-09-29T22:34:27.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WDodDIlRiiHxpgKWlRzH8Mrclug= 2023-09-29T22:34:27.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WDodDIlRiiHxpgKWlRzH8Mrclug=', 'Content-Length': '0'} 2023-09-29T22:34:27.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f845cca57be9a923-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.711 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.712 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.712 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.712 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.712 DEBUG:boto:Method: GET 2023-09-29T22:34:27.712 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.712 DEBUG:boto:Data: 2023-09-29T22:34:27.712 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.712 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.712 DEBUG:boto:Params: {} 2023-09-29T22:34:27.713 DEBUG:boto:Token: None 2023-09-29T22:34:27.713 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000feb57f29008c9440-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.718 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.718 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.718 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.719 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.719 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.719 DEBUG:boto:Method: GET 2023-09-29T22:34:27.719 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000011 2023-09-29T22:34:27.719 DEBUG:boto:Data: 2023-09-29T22:34:27.719 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.719 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.719 DEBUG:boto:Params: {} 2023-09-29T22:34:27.720 DEBUG:boto:Token: None 2023-09-29T22:34:27.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6813c9e6364704d-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.728 DEBUG:boto:b'hcnphz-58obj000000000000000000111000falseobj000000000000000000112023-09-29T22:34:15.976Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-29T22:34:27.729 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.729 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.729 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.729 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.730 DEBUG:boto:Method: GET 2023-09-29T22:34:27.730 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.730 DEBUG:boto:Data: 2023-09-29T22:34:27.730 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.730 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.730 DEBUG:boto:Params: {} 2023-09-29T22:34:27.730 DEBUG:boto:Token: None 2023-09-29T22:34:27.730 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.731 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI= 2023-09-29T22:34:27.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI=', 'Content-Length': '0'} 2023-09-29T22:34:27.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c14c553fefccdcc-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.733 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.733 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.733 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.733 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.734 DEBUG:boto:Method: GET 2023-09-29T22:34:27.734 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.734 DEBUG:boto:Data: 2023-09-29T22:34:27.734 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.734 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.734 DEBUG:boto:Params: {} 2023-09-29T22:34:27.734 DEBUG:boto:Token: None 2023-09-29T22:34:27.735 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI= 2023-09-29T22:34:27.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI=', 'Content-Length': '0'} 2023-09-29T22:34:27.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adbc4dd194437274-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.737 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.737 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.737 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.737 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.737 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.738 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.738 DEBUG:boto:Data: 2023-09-29T22:34:27.738 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.738 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.738 DEBUG:boto:Params: {} 2023-09-29T22:34:27.738 DEBUG:boto:Token: None 2023-09-29T22:34:27.738 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EP6TJYR0isKsQEL9cmw97eG+Lcg= 2023-09-29T22:34:27.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EP6TJYR0isKsQEL9cmw97eG+Lcg=', 'Content-Length': '0'} 2023-09-29T22:34:27.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f23a95c7ffb14e4-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.741 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.741 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.741 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.741 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.741 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.741 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.741 DEBUG:boto:Data: 2023-09-29T22:34:27.741 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.742 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.742 DEBUG:boto:Params: {} 2023-09-29T22:34:27.742 DEBUG:boto:Token: None 2023-09-29T22:34:27.742 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.742 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EP6TJYR0isKsQEL9cmw97eG+Lcg= 2023-09-29T22:34:27.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EP6TJYR0isKsQEL9cmw97eG+Lcg=', 'Content-Length': '0'} 2023-09-29T22:34:27.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018fd0bcfb6313568-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-29T22:34:27.745 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.745 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.745 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.745 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.745 DEBUG:boto:Method: GET 2023-09-29T22:34:27.745 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.745 DEBUG:boto:Data: 2023-09-29T22:34:27.745 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.746 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.746 DEBUG:boto:Params: {} 2023-09-29T22:34:27.746 DEBUG:boto:Token: None 2023-09-29T22:34:27.746 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.746 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI= 2023-09-29T22:34:27.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI=', 'Content-Length': '0'} 2023-09-29T22:34:27.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aa1f3b7d2896138-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.749 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.749 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.749 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.749 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.749 DEBUG:boto:Method: GET 2023-09-29T22:34:27.749 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.749 DEBUG:boto:Data: 2023-09-29T22:34:27.750 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.750 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.750 DEBUG:boto:Params: {} 2023-09-29T22:34:27.750 DEBUG:boto:Token: None 2023-09-29T22:34:27.750 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012?versionId=null 2023-09-29T22:34:27.750 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI= 2023-09-29T22:34:27.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KKoSQyPLV9Olq6EwdrYeg5TJxNI=', 'Content-Length': '0'} 2023-09-29T22:34:27.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143e2174a5149f96-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.753 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.753 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.753 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.753 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.753 DEBUG:boto:Method: GET 2023-09-29T22:34:27.754 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.754 DEBUG:boto:Data: 2023-09-29T22:34:27.754 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.754 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.754 DEBUG:boto:Params: {} 2023-09-29T22:34:27.754 DEBUG:boto:Token: None 2023-09-29T22:34:27.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.755 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1173c8cb4e2133-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.759 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.759 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.759 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.759 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.760 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.760 DEBUG:boto:Method: GET 2023-09-29T22:34:27.760 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.760 DEBUG:boto:Data: 2023-09-29T22:34:27.760 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.760 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.760 DEBUG:boto:Params: {} 2023-09-29T22:34:27.760 DEBUG:boto:Token: None 2023-09-29T22:34:27.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f291032e52a1f89-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.769 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.769 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.769 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.769 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.769 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.770 DEBUG:boto:Data: 2023-09-29T22:34:27.770 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.770 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.770 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.770 DEBUG:boto:Params: {} 2023-09-29T22:34:27.770 DEBUG:boto:Token: None 2023-09-29T22:34:27.770 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.770 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:crRQj4EdmxJUSqsXDRy9E4Vh8Xc= 2023-09-29T22:34:27.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:crRQj4EdmxJUSqsXDRy9E4Vh8Xc=', 'Content-Length': '0'} 2023-09-29T22:34:27.772 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003461a6cb1c94d0b4-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.773 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.773 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.773 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.773 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.773 DEBUG:boto:Data: 2023-09-29T22:34:27.773 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.773 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.773 DEBUG:boto:Params: {} 2023-09-29T22:34:27.774 DEBUG:boto:Token: None 2023-09-29T22:34:27.774 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:crRQj4EdmxJUSqsXDRy9E4Vh8Xc= 2023-09-29T22:34:27.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:crRQj4EdmxJUSqsXDRy9E4Vh8Xc=', 'Content-Length': '0'} 2023-09-29T22:34:27.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cd000d7a490e0c5-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000012 2023-09-29T22:34:27.776 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.776 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.776 DEBUG:boto:Method: GET 2023-09-29T22:34:27.776 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.776 DEBUG:boto:Data: 2023-09-29T22:34:27.777 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.777 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.777 DEBUG:boto:Params: {} 2023-09-29T22:34:27.777 DEBUG:boto:Token: None 2023-09-29T22:34:27.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/6+8uhAvlo2uLroPRvnhli+Zr+g= 2023-09-29T22:34:27.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/6+8uhAvlo2uLroPRvnhli+Zr+g=', 'Content-Length': '0'} 2023-09-29T22:34:27.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d54d9ab3de9c3f50-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.822 DEBUG:boto:path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.823 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.823 DEBUG:boto:Method: GET 2023-09-29T22:34:27.823 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.823 DEBUG:boto:Data: 2023-09-29T22:34:27.823 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.823 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.824 DEBUG:boto:Params: {} 2023-09-29T22:34:27.824 DEBUG:boto:Token: None 2023-09-29T22:34:27.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000012 2023-09-29T22:34:27.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/6+8uhAvlo2uLroPRvnhli+Zr+g= 2023-09-29T22:34:27.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/6+8uhAvlo2uLroPRvnhli+Zr+g=', 'Content-Length': '0'} 2023-09-29T22:34:27.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf71d08d34a5ef6-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.828 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.828 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.828 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.828 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.828 DEBUG:boto:Method: GET 2023-09-29T22:34:27.828 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.828 DEBUG:boto:Data: 2023-09-29T22:34:27.829 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.829 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.829 DEBUG:boto:Params: {} 2023-09-29T22:34:27.829 DEBUG:boto:Token: None 2023-09-29T22:34:27.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000591712650f16622c-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.836 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.836 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.836 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.837 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.837 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.837 DEBUG:boto:Method: GET 2023-09-29T22:34:27.837 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000012 2023-09-29T22:34:27.837 DEBUG:boto:Data: 2023-09-29T22:34:27.837 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.837 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.838 DEBUG:boto:Params: {} 2023-09-29T22:34:27.838 DEBUG:boto:Token: None 2023-09-29T22:34:27.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.844 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056501b44f11d9013-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.844 DEBUG:boto:b'hcnphz-58obj000000000000000000121000falseobj000000000000000000122023-09-29T22:34:15.446Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-29T22:34:27.845 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.845 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.845 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.845 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.845 DEBUG:boto:Method: GET 2023-09-29T22:34:27.845 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.845 DEBUG:boto:Data: 2023-09-29T22:34:27.846 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.846 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.846 DEBUG:boto:Params: {} 2023-09-29T22:34:27.846 DEBUG:boto:Token: None 2023-09-29T22:34:27.846 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.846 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA= 2023-09-29T22:34:27.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA=', 'Content-Length': '0'} 2023-09-29T22:34:27.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004745f407e619f204-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.849 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.849 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.849 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.849 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.849 DEBUG:boto:Method: GET 2023-09-29T22:34:27.849 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.850 DEBUG:boto:Data: 2023-09-29T22:34:27.850 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.850 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.850 DEBUG:boto:Params: {} 2023-09-29T22:34:27.850 DEBUG:boto:Token: None 2023-09-29T22:34:27.850 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA= 2023-09-29T22:34:27.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA=', 'Content-Length': '0'} 2023-09-29T22:34:27.853 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054135c6435e17f63-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.853 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.853 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.853 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.853 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.853 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.854 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.854 DEBUG:boto:Data: 2023-09-29T22:34:27.854 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.854 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.854 DEBUG:boto:Params: {} 2023-09-29T22:34:27.854 DEBUG:boto:Token: None 2023-09-29T22:34:27.855 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4rwVQ+Ns4C3GPSIX4NbdiP/5X4Q= 2023-09-29T22:34:27.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4rwVQ+Ns4C3GPSIX4NbdiP/5X4Q=', 'Content-Length': '0'} 2023-09-29T22:34:27.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ac5f2d8687e4f60-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.857 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.857 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.857 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.857 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.857 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.858 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.858 DEBUG:boto:Data: 2023-09-29T22:34:27.858 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.858 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.858 DEBUG:boto:Params: {} 2023-09-29T22:34:27.858 DEBUG:boto:Token: None 2023-09-29T22:34:27.858 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.859 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4rwVQ+Ns4C3GPSIX4NbdiP/5X4Q= 2023-09-29T22:34:27.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4rwVQ+Ns4C3GPSIX4NbdiP/5X4Q=', 'Content-Length': '0'} 2023-09-29T22:34:27.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024505fdc2049f223-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-29T22:34:27.861 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.861 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.862 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.862 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.862 DEBUG:boto:Method: GET 2023-09-29T22:34:27.862 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.862 DEBUG:boto:Data: 2023-09-29T22:34:27.862 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.862 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.862 DEBUG:boto:Params: {} 2023-09-29T22:34:27.863 DEBUG:boto:Token: None 2023-09-29T22:34:27.863 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.863 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA= 2023-09-29T22:34:27.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA=', 'Content-Length': '0'} 2023-09-29T22:34:27.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ce641beae352419-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.865 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.865 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.865 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.866 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.866 DEBUG:boto:Method: GET 2023-09-29T22:34:27.866 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.866 DEBUG:boto:Data: 2023-09-29T22:34:27.866 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.866 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.866 DEBUG:boto:Params: {} 2023-09-29T22:34:27.867 DEBUG:boto:Token: None 2023-09-29T22:34:27.867 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013?versionId=null 2023-09-29T22:34:27.867 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA= 2023-09-29T22:34:27.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pMC1Kt0DwrV8y+OnnweXhGYY1zA=', 'Content-Length': '0'} 2023-09-29T22:34:27.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846607c77b7ba984-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.870 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.870 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.870 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.870 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.870 DEBUG:boto:Method: GET 2023-09-29T22:34:27.870 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.870 DEBUG:boto:Data: 2023-09-29T22:34:27.870 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.871 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.871 DEBUG:boto:Params: {} 2023-09-29T22:34:27.871 DEBUG:boto:Token: None 2023-09-29T22:34:27.871 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.871 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebb6a97f1b8cbfa0-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.876 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.876 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.876 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.876 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.876 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.877 DEBUG:boto:Method: GET 2023-09-29T22:34:27.877 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.877 DEBUG:boto:Data: 2023-09-29T22:34:27.877 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.877 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.877 DEBUG:boto:Params: {} 2023-09-29T22:34:27.877 DEBUG:boto:Token: None 2023-09-29T22:34:27.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.885 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f072fa47d752f996-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.885 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.886 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.886 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.886 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.886 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.886 DEBUG:boto:Data: 2023-09-29T22:34:27.886 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.886 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.887 DEBUG:boto:Params: {} 2023-09-29T22:34:27.889 DEBUG:boto:Token: None 2023-09-29T22:34:27.889 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bJ6m9Uzasf0A6OVCLibSJgp3r8o= 2023-09-29T22:34:27.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bJ6m9Uzasf0A6OVCLibSJgp3r8o=', 'Content-Length': '0'} 2023-09-29T22:34:27.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d5266dca4763934-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.891 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.891 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.892 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.892 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.892 DEBUG:boto:Data: 2023-09-29T22:34:27.892 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.892 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.892 DEBUG:boto:Params: {} 2023-09-29T22:34:27.892 DEBUG:boto:Token: None 2023-09-29T22:34:27.892 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bJ6m9Uzasf0A6OVCLibSJgp3r8o= 2023-09-29T22:34:27.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bJ6m9Uzasf0A6OVCLibSJgp3r8o=', 'Content-Length': '0'} 2023-09-29T22:34:27.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5b49ac6516ac664-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000013 2023-09-29T22:34:27.895 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.895 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.895 DEBUG:boto:Method: GET 2023-09-29T22:34:27.895 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.895 DEBUG:boto:Data: 2023-09-29T22:34:27.895 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.896 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.896 DEBUG:boto:Params: {} 2023-09-29T22:34:27.896 DEBUG:boto:Token: None 2023-09-29T22:34:27.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnTnHm47C6bi/ijFOoce8hojcjQ= 2023-09-29T22:34:27.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnTnHm47C6bi/ijFOoce8hojcjQ=', 'Content-Length': '0'} 2023-09-29T22:34:27.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d14d736f228b08f-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.942 DEBUG:boto:path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.943 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.943 DEBUG:boto:Method: GET 2023-09-29T22:34:27.943 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.943 DEBUG:boto:Data: 2023-09-29T22:34:27.943 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.943 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.943 DEBUG:boto:Params: {} 2023-09-29T22:34:27.943 DEBUG:boto:Token: None 2023-09-29T22:34:27.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000013 2023-09-29T22:34:27.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnTnHm47C6bi/ijFOoce8hojcjQ= 2023-09-29T22:34:27.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnTnHm47C6bi/ijFOoce8hojcjQ=', 'Content-Length': '0'} 2023-09-29T22:34:27.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000803558835bb8edea-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.947 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.948 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.948 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.948 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.948 DEBUG:boto:Method: GET 2023-09-29T22:34:27.948 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.948 DEBUG:boto:Data: 2023-09-29T22:34:27.948 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.948 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.948 DEBUG:boto:Params: {} 2023-09-29T22:34:27.949 DEBUG:boto:Token: None 2023-09-29T22:34:27.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.954 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a8a43796c8117ae-00651750f3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.955 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.955 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:27.955 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:27.955 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.956 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.956 DEBUG:boto:Method: GET 2023-09-29T22:34:27.956 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000013 2023-09-29T22:34:27.956 DEBUG:boto:Data: 2023-09-29T22:34:27.956 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.956 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.956 DEBUG:boto:Params: {} 2023-09-29T22:34:27.956 DEBUG:boto:Token: None 2023-09-29T22:34:27.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/ 2023-09-29T22:34:27.957 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc= 2023-09-29T22:34:27.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Mjcu5SmMuwj58JvGnxGPIzDKbc=', 'Content-Length': '0'} 2023-09-29T22:34:27.964 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd2f579269821717-00651750f3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.964 DEBUG:boto:b'hcnphz-58obj000000000000000000131000falseobj000000000000000000132023-09-29T22:34:15.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:27.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-09-29T22:34:27.965 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.965 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.965 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.965 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.966 DEBUG:boto:Method: GET 2023-09-29T22:34:27.966 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.966 DEBUG:boto:Data: 2023-09-29T22:34:27.966 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.966 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.966 DEBUG:boto:Params: {} 2023-09-29T22:34:27.966 DEBUG:boto:Token: None 2023-09-29T22:34:27.966 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.967 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KNboAOR+Ijvla8cBAh9y9XDxOCw= 2023-09-29T22:34:27.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNboAOR+Ijvla8cBAh9y9XDxOCw=', 'Content-Length': '0'} 2023-09-29T22:34:27.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024cc228198aae6f9-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.969 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.969 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.970 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.970 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.970 DEBUG:boto:Method: GET 2023-09-29T22:34:27.970 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.970 DEBUG:boto:Data: 2023-09-29T22:34:27.970 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.970 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.970 DEBUG:boto:Params: {} 2023-09-29T22:34:27.971 DEBUG:boto:Token: None 2023-09-29T22:34:27.971 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.971 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KNboAOR+Ijvla8cBAh9y9XDxOCw= 2023-09-29T22:34:27.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNboAOR+Ijvla8cBAh9y9XDxOCw=', 'Content-Length': '0'} 2023-09-29T22:34:27.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d6cea47dd796ad-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.973 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.974 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.974 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.974 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.974 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.974 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.974 DEBUG:boto:Data: 2023-09-29T22:34:27.975 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.975 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.975 DEBUG:boto:Params: {} 2023-09-29T22:34:27.975 DEBUG:boto:Token: None 2023-09-29T22:34:27.976 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0Tfk0Q9WE3rFwywq1C808SBVLkQ= 2023-09-29T22:34:27.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0Tfk0Q9WE3rFwywq1C808SBVLkQ=', 'Content-Length': '0'} 2023-09-29T22:34:27.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000601b3ce5b80ab5a9-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.979 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.979 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.980 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.980 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.980 DEBUG:boto:Method: HEAD 2023-09-29T22:34:27.980 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.980 DEBUG:boto:Data: 2023-09-29T22:34:27.981 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:27.981 DEBUG:boto:Port: 8000 2023-09-29T22:34:27.981 DEBUG:boto:Params: {} 2023-09-29T22:34:27.982 DEBUG:boto:Token: None 2023-09-29T22:34:27.982 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.982 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0Tfk0Q9WE3rFwywq1C808SBVLkQ= 2023-09-29T22:34:27.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0Tfk0Q9WE3rFwywq1C808SBVLkQ=', 'Content-Length': '0'} 2023-09-29T22:34:27.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606b089fa45bc7bf-00651750f3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:27.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-09-29T22:34:27.986 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.986 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:27.986 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.986 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.986 DEBUG:boto:Method: GET 2023-09-29T22:34:27.987 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.987 DEBUG:boto:Data: 2023-09-29T22:34:27.987 DEBUG:boto:Headers: {} 2023-09-29T22:34:27.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:27.987 DEBUG:boto:Port: 8001 2023-09-29T22:34:27.988 DEBUG:boto:Params: {} 2023-09-29T22:34:27.988 DEBUG:boto:Token: None 2023-09-29T22:34:27.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:27 GMT /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:27.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KNboAOR+Ijvla8cBAh9y9XDxOCw= 2023-09-29T22:34:27.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNboAOR+Ijvla8cBAh9y9XDxOCw=', 'Content-Length': '0'} 2023-09-29T22:34:27.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccd1cb9b0fd88e62-00651750f3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.034 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.196 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.196 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:28.196 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:28.196 DEBUG:boto:Method: GET 2023-09-29T22:34:28.196 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:28.196 DEBUG:boto:Data: 2023-09-29T22:34:28.197 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.197 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.197 DEBUG:boto:Params: {} 2023-09-29T22:34:28.197 DEBUG:boto:Token: None 2023-09-29T22:34:28.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000014?versionId=null 2023-09-29T22:34:28.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3gbYlPBP/9lbsmdOK1kLE1Gvivk= 2023-09-29T22:34:28.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3gbYlPBP/9lbsmdOK1kLE1Gvivk=', 'Content-Length': '0'} 2023-09-29T22:34:28.201 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e6c97933cf6671f-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.202 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.202 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.202 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.203 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.203 DEBUG:boto:Method: GET 2023-09-29T22:34:28.203 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.204 DEBUG:boto:Data: 2023-09-29T22:34:28.204 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.204 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.205 DEBUG:boto:Params: {} 2023-09-29T22:34:28.205 DEBUG:boto:Token: None 2023-09-29T22:34:28.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.206 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7dff301c0190e60-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.212 DEBUG:boto:b'hcnphz-58obj000000000000000000141000falseobj000000000000000000142023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.213 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.213 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.213 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.213 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.213 DEBUG:boto:Method: GET 2023-09-29T22:34:28.214 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.214 DEBUG:boto:Data: 2023-09-29T22:34:28.214 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.215 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.215 DEBUG:boto:Params: {} 2023-09-29T22:34:28.215 DEBUG:boto:Token: None 2023-09-29T22:34:28.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.224 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afa4ee672bae6020-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.224 DEBUG:boto:b'hcnphz-58obj000000000000000000141000falseobj000000000000000000142023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.224 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.224 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.225 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.225 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.225 DEBUG:boto:Data: 2023-09-29T22:34:28.225 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.225 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.225 DEBUG:boto:Params: {} 2023-09-29T22:34:28.225 DEBUG:boto:Token: None 2023-09-29T22:34:28.226 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.226 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:htbq4y7P1RDyaZNk5rTWb3LtxMY= 2023-09-29T22:34:28.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:htbq4y7P1RDyaZNk5rTWb3LtxMY=', 'Content-Length': '0'} 2023-09-29T22:34:28.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fb6162037f51703-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.229 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.229 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.229 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.229 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.229 DEBUG:boto:Data: 2023-09-29T22:34:28.229 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.230 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.230 DEBUG:boto:Params: {} 2023-09-29T22:34:28.230 DEBUG:boto:Token: None 2023-09-29T22:34:28.230 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.230 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:htbq4y7P1RDyaZNk5rTWb3LtxMY= 2023-09-29T22:34:28.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:htbq4y7P1RDyaZNk5rTWb3LtxMY=', 'Content-Length': '0'} 2023-09-29T22:34:28.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db27bcb6af14b6f-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000014 2023-09-29T22:34:28.233 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.233 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.233 DEBUG:boto:Method: GET 2023-09-29T22:34:28.233 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.233 DEBUG:boto:Data: 2023-09-29T22:34:28.233 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.234 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.234 DEBUG:boto:Params: {} 2023-09-29T22:34:28.234 DEBUG:boto:Token: None 2023-09-29T22:34:28.234 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.235 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BdHQ6+COGCsMNyiawIV6XxPWmAg= 2023-09-29T22:34:28.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BdHQ6+COGCsMNyiawIV6XxPWmAg=', 'Content-Length': '0'} 2023-09-29T22:34:28.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452cc17ee5f4f79d-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.237 DEBUG:boto:path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.237 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.238 DEBUG:boto:Method: GET 2023-09-29T22:34:28.238 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.238 DEBUG:boto:Data: 2023-09-29T22:34:28.238 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.239 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.239 DEBUG:boto:Params: {} 2023-09-29T22:34:28.239 DEBUG:boto:Token: None 2023-09-29T22:34:28.239 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000014 2023-09-29T22:34:28.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BdHQ6+COGCsMNyiawIV6XxPWmAg= 2023-09-29T22:34:28.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BdHQ6+COGCsMNyiawIV6XxPWmAg=', 'Content-Length': '0'} 2023-09-29T22:34:28.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054e54f62e1e9f94b-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.286 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.287 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.287 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.287 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.287 DEBUG:boto:Method: GET 2023-09-29T22:34:28.287 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.287 DEBUG:boto:Data: 2023-09-29T22:34:28.287 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.287 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.288 DEBUG:boto:Params: {} 2023-09-29T22:34:28.288 DEBUG:boto:Token: None 2023-09-29T22:34:28.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5bca2de8fe2be03-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.293 DEBUG:boto:b'hcnphz-58obj000000000000000000141000falseobj000000000000000000142023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.431 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.432 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.432 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.432 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.432 DEBUG:boto:Method: GET 2023-09-29T22:34:28.432 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000014 2023-09-29T22:34:28.432 DEBUG:boto:Data: 2023-09-29T22:34:28.432 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.433 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.433 DEBUG:boto:Params: {} 2023-09-29T22:34:28.433 DEBUG:boto:Token: None 2023-09-29T22:34:28.433 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000849fa54507ba6886-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.442 DEBUG:boto:b'hcnphz-58obj000000000000000000141000falseobj000000000000000000142023-09-29T22:34:15.522Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-09-29T22:34:28.443 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.443 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.443 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.443 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.443 DEBUG:boto:Method: GET 2023-09-29T22:34:28.443 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.443 DEBUG:boto:Data: 2023-09-29T22:34:28.444 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.444 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.444 DEBUG:boto:Params: {} 2023-09-29T22:34:28.444 DEBUG:boto:Token: None 2023-09-29T22:34:28.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk= 2023-09-29T22:34:28.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk=', 'Content-Length': '0'} 2023-09-29T22:34:28.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b2498cf72c6a78f-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.447 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.447 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.447 DEBUG:boto:Method: GET 2023-09-29T22:34:28.447 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.447 DEBUG:boto:Data: 2023-09-29T22:34:28.448 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.448 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.448 DEBUG:boto:Params: {} 2023-09-29T22:34:28.448 DEBUG:boto:Token: None 2023-09-29T22:34:28.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk= 2023-09-29T22:34:28.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk=', 'Content-Length': '0'} 2023-09-29T22:34:28.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0b9cd47a9c3afa1-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.450 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.451 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.451 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.451 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.451 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.451 DEBUG:boto:Data: 2023-09-29T22:34:28.451 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.451 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.451 DEBUG:boto:Params: {} 2023-09-29T22:34:28.452 DEBUG:boto:Token: None 2023-09-29T22:34:28.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:32evbk7E4pWJFoPUQH/KY/lWZTg= 2023-09-29T22:34:28.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:32evbk7E4pWJFoPUQH/KY/lWZTg=', 'Content-Length': '0'} 2023-09-29T22:34:28.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8da183969732782-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.454 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.455 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.455 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.455 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.455 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.455 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.455 DEBUG:boto:Data: 2023-09-29T22:34:28.455 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.455 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.455 DEBUG:boto:Params: {} 2023-09-29T22:34:28.456 DEBUG:boto:Token: None 2023-09-29T22:34:28.456 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:32evbk7E4pWJFoPUQH/KY/lWZTg= 2023-09-29T22:34:28.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:32evbk7E4pWJFoPUQH/KY/lWZTg=', 'Content-Length': '0'} 2023-09-29T22:34:28.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042318922fb7fd79c-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-09-29T22:34:28.459 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.459 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.459 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.459 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.459 DEBUG:boto:Method: GET 2023-09-29T22:34:28.459 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.459 DEBUG:boto:Data: 2023-09-29T22:34:28.459 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.460 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.460 DEBUG:boto:Params: {} 2023-09-29T22:34:28.460 DEBUG:boto:Token: None 2023-09-29T22:34:28.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk= 2023-09-29T22:34:28.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk=', 'Content-Length': '0'} 2023-09-29T22:34:28.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3dff61419c1ef9-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.463 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.463 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.463 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.463 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.463 DEBUG:boto:Method: GET 2023-09-29T22:34:28.463 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.463 DEBUG:boto:Data: 2023-09-29T22:34:28.464 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.464 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.464 DEBUG:boto:Params: {} 2023-09-29T22:34:28.464 DEBUG:boto:Token: None 2023-09-29T22:34:28.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015?versionId=null 2023-09-29T22:34:28.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk= 2023-09-29T22:34:28.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UMwL3lWfiYcL2KsXU8ZW3uk/vnk=', 'Content-Length': '0'} 2023-09-29T22:34:28.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000423e52120a0439bf-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.467 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.467 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.467 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.467 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.468 DEBUG:boto:Method: GET 2023-09-29T22:34:28.468 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.468 DEBUG:boto:Data: 2023-09-29T22:34:28.468 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.468 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.468 DEBUG:boto:Params: {} 2023-09-29T22:34:28.468 DEBUG:boto:Token: None 2023-09-29T22:34:28.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.469 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea735b8f4d3666f4-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.514 DEBUG:boto:b'hcnphz-58obj000000000000000000151000falseobj000000000000000000152023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.515 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.515 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.515 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.515 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.515 DEBUG:boto:Method: GET 2023-09-29T22:34:28.515 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.516 DEBUG:boto:Data: 2023-09-29T22:34:28.516 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.516 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.516 DEBUG:boto:Params: {} 2023-09-29T22:34:28.516 DEBUG:boto:Token: None 2023-09-29T22:34:28.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.516 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b021a56621c23c-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.525 DEBUG:boto:b'hcnphz-58obj000000000000000000151000falseobj000000000000000000152023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.525 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.525 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.525 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.525 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.525 DEBUG:boto:Data: 2023-09-29T22:34:28.526 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.526 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.526 DEBUG:boto:Params: {} 2023-09-29T22:34:28.526 DEBUG:boto:Token: None 2023-09-29T22:34:28.526 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.526 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5E82gxkSiQXQ4Ie/Xgj/XjoAMOk= 2023-09-29T22:34:28.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5E82gxkSiQXQ4Ie/Xgj/XjoAMOk=', 'Content-Length': '0'} 2023-09-29T22:34:28.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e48e64a06eaaf478-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.529 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.529 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.529 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.529 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.529 DEBUG:boto:Data: 2023-09-29T22:34:28.530 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.530 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.530 DEBUG:boto:Params: {} 2023-09-29T22:34:28.530 DEBUG:boto:Token: None 2023-09-29T22:34:28.530 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5E82gxkSiQXQ4Ie/Xgj/XjoAMOk= 2023-09-29T22:34:28.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5E82gxkSiQXQ4Ie/Xgj/XjoAMOk=', 'Content-Length': '0'} 2023-09-29T22:34:28.533 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcb3e60fdabe237f-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000015 2023-09-29T22:34:28.533 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.533 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.533 DEBUG:boto:Method: GET 2023-09-29T22:34:28.534 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.534 DEBUG:boto:Data: 2023-09-29T22:34:28.534 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.534 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.534 DEBUG:boto:Params: {} 2023-09-29T22:34:28.534 DEBUG:boto:Token: None 2023-09-29T22:34:28.534 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.534 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hh0OgGm41LafLb8gkjlKjYS+Vpo= 2023-09-29T22:34:28.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hh0OgGm41LafLb8gkjlKjYS+Vpo=', 'Content-Length': '0'} 2023-09-29T22:34:28.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c7713244938abe7-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.537 DEBUG:boto:path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.537 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.537 DEBUG:boto:Method: GET 2023-09-29T22:34:28.538 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.538 DEBUG:boto:Data: 2023-09-29T22:34:28.538 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.538 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.538 DEBUG:boto:Params: {} 2023-09-29T22:34:28.538 DEBUG:boto:Token: None 2023-09-29T22:34:28.538 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000015 2023-09-29T22:34:28.539 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hh0OgGm41LafLb8gkjlKjYS+Vpo= 2023-09-29T22:34:28.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hh0OgGm41LafLb8gkjlKjYS+Vpo=', 'Content-Length': '0'} 2023-09-29T22:34:28.541 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e069e4670e823ea-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.582 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.583 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.583 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.583 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.583 DEBUG:boto:Method: GET 2023-09-29T22:34:28.583 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.583 DEBUG:boto:Data: 2023-09-29T22:34:28.583 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.584 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.584 DEBUG:boto:Params: {} 2023-09-29T22:34:28.584 DEBUG:boto:Token: None 2023-09-29T22:34:28.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049ac3fd87077903f-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.590 DEBUG:boto:b'hcnphz-58obj000000000000000000151000falseobj000000000000000000152023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.591 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.591 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.591 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.591 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.591 DEBUG:boto:Method: GET 2023-09-29T22:34:28.591 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000015 2023-09-29T22:34:28.591 DEBUG:boto:Data: 2023-09-29T22:34:28.591 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.592 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.592 DEBUG:boto:Params: {} 2023-09-29T22:34:28.592 DEBUG:boto:Token: None 2023-09-29T22:34:28.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.601 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0a2d4088194fa0a-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.601 DEBUG:boto:b'hcnphz-58obj000000000000000000151000falseobj000000000000000000152023-09-29T22:34:15.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-09-29T22:34:28.602 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.602 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.602 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.602 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.602 DEBUG:boto:Method: GET 2023-09-29T22:34:28.602 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.602 DEBUG:boto:Data: 2023-09-29T22:34:28.602 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.603 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.603 DEBUG:boto:Params: {} 2023-09-29T22:34:28.603 DEBUG:boto:Token: None 2023-09-29T22:34:28.603 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0= 2023-09-29T22:34:28.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0=', 'Content-Length': '0'} 2023-09-29T22:34:28.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df617673f1cd0f26-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.606 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.606 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.607 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.607 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.607 DEBUG:boto:Method: GET 2023-09-29T22:34:28.607 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.607 DEBUG:boto:Data: 2023-09-29T22:34:28.607 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.607 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.607 DEBUG:boto:Params: {} 2023-09-29T22:34:28.607 DEBUG:boto:Token: None 2023-09-29T22:34:28.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0= 2023-09-29T22:34:28.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0=', 'Content-Length': '0'} 2023-09-29T22:34:28.610 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d83cc7f69be85f0b-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.611 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.611 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.611 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.611 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.611 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.611 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.611 DEBUG:boto:Data: 2023-09-29T22:34:28.611 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.612 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.612 DEBUG:boto:Params: {} 2023-09-29T22:34:28.612 DEBUG:boto:Token: None 2023-09-29T22:34:28.612 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bw9S51MBuX6IubCZTOZ2CLl9vMw= 2023-09-29T22:34:28.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bw9S51MBuX6IubCZTOZ2CLl9vMw=', 'Content-Length': '0'} 2023-09-29T22:34:28.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007ad02eac08120ce-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.615 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.615 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.615 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.615 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.615 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.615 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.615 DEBUG:boto:Data: 2023-09-29T22:34:28.615 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.616 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.616 DEBUG:boto:Params: {} 2023-09-29T22:34:28.616 DEBUG:boto:Token: None 2023-09-29T22:34:28.616 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bw9S51MBuX6IubCZTOZ2CLl9vMw= 2023-09-29T22:34:28.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bw9S51MBuX6IubCZTOZ2CLl9vMw=', 'Content-Length': '0'} 2023-09-29T22:34:28.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6b55e437fc28ff-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-09-29T22:34:28.619 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.619 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.619 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.619 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.619 DEBUG:boto:Method: GET 2023-09-29T22:34:28.620 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.620 DEBUG:boto:Data: 2023-09-29T22:34:28.620 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.620 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.620 DEBUG:boto:Params: {} 2023-09-29T22:34:28.620 DEBUG:boto:Token: None 2023-09-29T22:34:28.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.621 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0= 2023-09-29T22:34:28.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0=', 'Content-Length': '0'} 2023-09-29T22:34:28.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009496f2e531f00acc-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.670 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.671 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.671 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.671 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.671 DEBUG:boto:Method: GET 2023-09-29T22:34:28.671 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.671 DEBUG:boto:Data: 2023-09-29T22:34:28.671 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.671 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.672 DEBUG:boto:Params: {} 2023-09-29T22:34:28.672 DEBUG:boto:Token: None 2023-09-29T22:34:28.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016?versionId=null 2023-09-29T22:34:28.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0= 2023-09-29T22:34:28.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o2Bu9ySYxhUuaQhtHf3fs8b9iN0=', 'Content-Length': '0'} 2023-09-29T22:34:28.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a47fefb104a59216-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.675 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.675 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.676 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.676 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.676 DEBUG:boto:Method: GET 2023-09-29T22:34:28.676 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.676 DEBUG:boto:Data: 2023-09-29T22:34:28.676 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.676 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.676 DEBUG:boto:Params: {} 2023-09-29T22:34:28.677 DEBUG:boto:Token: None 2023-09-29T22:34:28.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094af691bbd801a26-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.681 DEBUG:boto:b'hcnphz-58obj000000000000000000161000falseobj000000000000000000162023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.681 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.682 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.682 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.682 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.682 DEBUG:boto:Method: GET 2023-09-29T22:34:28.682 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.682 DEBUG:boto:Data: 2023-09-29T22:34:28.682 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.682 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.683 DEBUG:boto:Params: {} 2023-09-29T22:34:28.683 DEBUG:boto:Token: None 2023-09-29T22:34:28.683 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.683 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.689 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d541efee62fcf38-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.690 DEBUG:boto:b'hcnphz-58obj000000000000000000161000falseobj000000000000000000162023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.690 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.690 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.690 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.690 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.691 DEBUG:boto:Data: 2023-09-29T22:34:28.691 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.691 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.691 DEBUG:boto:Params: {} 2023-09-29T22:34:28.691 DEBUG:boto:Token: None 2023-09-29T22:34:28.691 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4pIBf+FZZ+Vgc1IHWB4bFl8ZJnA= 2023-09-29T22:34:28.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4pIBf+FZZ+Vgc1IHWB4bFl8ZJnA=', 'Content-Length': '0'} 2023-09-29T22:34:28.694 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001679069e6d5f5e04-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.694 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.694 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.694 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.694 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.694 DEBUG:boto:Data: 2023-09-29T22:34:28.694 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.695 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.695 DEBUG:boto:Params: {} 2023-09-29T22:34:28.695 DEBUG:boto:Token: None 2023-09-29T22:34:28.695 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.695 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4pIBf+FZZ+Vgc1IHWB4bFl8ZJnA= 2023-09-29T22:34:28.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4pIBf+FZZ+Vgc1IHWB4bFl8ZJnA=', 'Content-Length': '0'} 2023-09-29T22:34:28.698 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048c96e211d9609cb-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000016 2023-09-29T22:34:28.698 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.698 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.698 DEBUG:boto:Method: GET 2023-09-29T22:34:28.699 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.699 DEBUG:boto:Data: 2023-09-29T22:34:28.699 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.699 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.699 DEBUG:boto:Params: {} 2023-09-29T22:34:28.699 DEBUG:boto:Token: None 2023-09-29T22:34:28.699 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HQbRaVC6v0DEeFHjsE7rc+0Z7i4= 2023-09-29T22:34:28.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HQbRaVC6v0DEeFHjsE7rc+0Z7i4=', 'Content-Length': '0'} 2023-09-29T22:34:28.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500b927b0f4ea141-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.702 DEBUG:boto:path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.702 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.702 DEBUG:boto:Method: GET 2023-09-29T22:34:28.702 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.702 DEBUG:boto:Data: 2023-09-29T22:34:28.703 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.703 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.703 DEBUG:boto:Params: {} 2023-09-29T22:34:28.703 DEBUG:boto:Token: None 2023-09-29T22:34:28.703 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000016 2023-09-29T22:34:28.703 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HQbRaVC6v0DEeFHjsE7rc+0Z7i4= 2023-09-29T22:34:28.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HQbRaVC6v0DEeFHjsE7rc+0Z7i4=', 'Content-Length': '0'} 2023-09-29T22:34:28.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0072aa74ce970ca-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.706 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.706 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.706 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.707 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.707 DEBUG:boto:Method: GET 2023-09-29T22:34:28.707 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.707 DEBUG:boto:Data: 2023-09-29T22:34:28.707 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.707 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.707 DEBUG:boto:Params: {} 2023-09-29T22:34:28.707 DEBUG:boto:Token: None 2023-09-29T22:34:28.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002dea1cd4926dc551-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.712 DEBUG:boto:b'hcnphz-58obj000000000000000000161000falseobj000000000000000000162023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.712 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.712 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.712 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.713 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.713 DEBUG:boto:Method: GET 2023-09-29T22:34:28.713 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000016 2023-09-29T22:34:28.713 DEBUG:boto:Data: 2023-09-29T22:34:28.714 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.714 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.714 DEBUG:boto:Params: {} 2023-09-29T22:34:28.714 DEBUG:boto:Token: None 2023-09-29T22:34:28.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0b93acfbea6e71f-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.723 DEBUG:boto:b'hcnphz-58obj000000000000000000161000falseobj000000000000000000162023-09-29T22:34:15.903Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-09-29T22:34:28.724 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.724 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.724 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.724 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.724 DEBUG:boto:Method: GET 2023-09-29T22:34:28.725 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.725 DEBUG:boto:Data: 2023-09-29T22:34:28.725 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.725 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.725 DEBUG:boto:Params: {} 2023-09-29T22:34:28.725 DEBUG:boto:Token: None 2023-09-29T22:34:28.725 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E= 2023-09-29T22:34:28.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E=', 'Content-Length': '0'} 2023-09-29T22:34:28.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eba080fd3e84edb-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.774 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.775 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.775 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.775 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.775 DEBUG:boto:Method: GET 2023-09-29T22:34:28.775 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.775 DEBUG:boto:Data: 2023-09-29T22:34:28.775 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.776 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.776 DEBUG:boto:Params: {} 2023-09-29T22:34:28.776 DEBUG:boto:Token: None 2023-09-29T22:34:28.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E= 2023-09-29T22:34:28.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E=', 'Content-Length': '0'} 2023-09-29T22:34:28.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dfcd093e9d15d5f-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.780 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.780 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.780 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.781 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.781 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.781 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.781 DEBUG:boto:Data: 2023-09-29T22:34:28.781 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.781 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.781 DEBUG:boto:Params: {} 2023-09-29T22:34:28.782 DEBUG:boto:Token: None 2023-09-29T22:34:28.782 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.782 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1QDEdAXHr/so7JTUyy1M7zYfSec= 2023-09-29T22:34:28.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1QDEdAXHr/so7JTUyy1M7zYfSec=', 'Content-Length': '0'} 2023-09-29T22:34:28.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067f16d12169b7010-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.785 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.785 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.785 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.785 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.786 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.786 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.786 DEBUG:boto:Data: 2023-09-29T22:34:28.786 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.786 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.786 DEBUG:boto:Params: {} 2023-09-29T22:34:28.786 DEBUG:boto:Token: None 2023-09-29T22:34:28.787 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.787 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1QDEdAXHr/so7JTUyy1M7zYfSec= 2023-09-29T22:34:28.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1QDEdAXHr/so7JTUyy1M7zYfSec=', 'Content-Length': '0'} 2023-09-29T22:34:28.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b07b66e22d29cb-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-09-29T22:34:28.790 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.790 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.790 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.791 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.791 DEBUG:boto:Method: GET 2023-09-29T22:34:28.791 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.791 DEBUG:boto:Data: 2023-09-29T22:34:28.791 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.792 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.792 DEBUG:boto:Params: {} 2023-09-29T22:34:28.792 DEBUG:boto:Token: None 2023-09-29T22:34:28.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E= 2023-09-29T22:34:28.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E=', 'Content-Length': '0'} 2023-09-29T22:34:28.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b4272d621fabc2e-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.796 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.796 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.796 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.796 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.796 DEBUG:boto:Method: GET 2023-09-29T22:34:28.796 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.796 DEBUG:boto:Data: 2023-09-29T22:34:28.796 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.797 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.797 DEBUG:boto:Params: {} 2023-09-29T22:34:28.797 DEBUG:boto:Token: None 2023-09-29T22:34:28.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017?versionId=null 2023-09-29T22:34:28.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E= 2023-09-29T22:34:28.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fJQ9JPYXKkUE8z8lGFOJODLRy9E=', 'Content-Length': '0'} 2023-09-29T22:34:28.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065ea2f4dcbe805ae-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.842 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.843 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.843 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.843 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.843 DEBUG:boto:Method: GET 2023-09-29T22:34:28.843 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.843 DEBUG:boto:Data: 2023-09-29T22:34:28.844 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.844 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.844 DEBUG:boto:Params: {} 2023-09-29T22:34:28.844 DEBUG:boto:Token: None 2023-09-29T22:34:28.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f02ecee261697d79-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.849 DEBUG:boto:b'hcnphz-58obj000000000000000000171000falseobj000000000000000000172023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.850 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.850 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.850 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.850 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.850 DEBUG:boto:Method: GET 2023-09-29T22:34:28.851 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.851 DEBUG:boto:Data: 2023-09-29T22:34:28.851 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.851 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.851 DEBUG:boto:Params: {} 2023-09-29T22:34:28.851 DEBUG:boto:Token: None 2023-09-29T22:34:28.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e434c15cf6363819-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.860 DEBUG:boto:b'hcnphz-58obj000000000000000000171000falseobj000000000000000000172023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.860 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.860 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.861 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.861 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.861 DEBUG:boto:Data: 2023-09-29T22:34:28.861 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.861 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.861 DEBUG:boto:Params: {} 2023-09-29T22:34:28.861 DEBUG:boto:Token: None 2023-09-29T22:34:28.862 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.862 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+ezR/+MxiVF0XB+JogWUYpNarG4= 2023-09-29T22:34:28.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+ezR/+MxiVF0XB+JogWUYpNarG4=', 'Content-Length': '0'} 2023-09-29T22:34:28.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba050e435bc57602-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.864 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.864 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.864 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.865 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.865 DEBUG:boto:Data: 2023-09-29T22:34:28.865 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.865 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.865 DEBUG:boto:Params: {} 2023-09-29T22:34:28.865 DEBUG:boto:Token: None 2023-09-29T22:34:28.866 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.866 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+ezR/+MxiVF0XB+JogWUYpNarG4= 2023-09-29T22:34:28.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+ezR/+MxiVF0XB+JogWUYpNarG4=', 'Content-Length': '0'} 2023-09-29T22:34:28.868 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5a51f2920b05812-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000017 2023-09-29T22:34:28.868 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.869 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.869 DEBUG:boto:Method: GET 2023-09-29T22:34:28.869 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.869 DEBUG:boto:Data: 2023-09-29T22:34:28.869 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.869 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.869 DEBUG:boto:Params: {} 2023-09-29T22:34:28.870 DEBUG:boto:Token: None 2023-09-29T22:34:28.870 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.870 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BMphb2jCroD0FeS2FB+trltwK9k= 2023-09-29T22:34:28.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BMphb2jCroD0FeS2FB+trltwK9k=', 'Content-Length': '0'} 2023-09-29T22:34:28.872 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db9d275ae699840-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.872 DEBUG:boto:path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.872 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.873 DEBUG:boto:Method: GET 2023-09-29T22:34:28.873 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.873 DEBUG:boto:Data: 2023-09-29T22:34:28.873 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.873 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.873 DEBUG:boto:Params: {} 2023-09-29T22:34:28.874 DEBUG:boto:Token: None 2023-09-29T22:34:28.874 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000017 2023-09-29T22:34:28.874 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BMphb2jCroD0FeS2FB+trltwK9k= 2023-09-29T22:34:28.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BMphb2jCroD0FeS2FB+trltwK9k=', 'Content-Length': '0'} 2023-09-29T22:34:28.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075d70660986cfa82-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.877 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.877 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.877 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.877 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.877 DEBUG:boto:Method: GET 2023-09-29T22:34:28.877 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.878 DEBUG:boto:Data: 2023-09-29T22:34:28.878 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.878 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.878 DEBUG:boto:Params: {} 2023-09-29T22:34:28.878 DEBUG:boto:Token: None 2023-09-29T22:34:28.878 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.879 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055e94c3c37745db5-00651750f4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.883 DEBUG:boto:b'hcnphz-58obj000000000000000000171000falseobj000000000000000000172023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.883 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:28.883 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:28.883 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.884 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.884 DEBUG:boto:Method: GET 2023-09-29T22:34:28.884 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000017 2023-09-29T22:34:28.884 DEBUG:boto:Data: 2023-09-29T22:34:28.884 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.884 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.884 DEBUG:boto:Params: {} 2023-09-29T22:34:28.884 DEBUG:boto:Token: None 2023-09-29T22:34:28.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/ 2023-09-29T22:34:28.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE= 2023-09-29T22:34:28.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LGeDIRCEIT5js+YI+cj7k+RuIRE=', 'Content-Length': '0'} 2023-09-29T22:34:28.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001de0819405349113-00651750f4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.893 DEBUG:boto:b'hcnphz-58obj000000000000000000171000falseobj000000000000000000172023-09-29T22:34:15.750Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:28.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-09-29T22:34:28.893 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.894 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.894 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.894 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.894 DEBUG:boto:Method: GET 2023-09-29T22:34:28.894 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.894 DEBUG:boto:Data: 2023-09-29T22:34:28.894 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.895 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.895 DEBUG:boto:Params: {} 2023-09-29T22:34:28.895 DEBUG:boto:Token: None 2023-09-29T22:34:28.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco= 2023-09-29T22:34:28.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco=', 'Content-Length': '0'} 2023-09-29T22:34:28.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d25356fb71a8464-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.938 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.939 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.939 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.939 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.939 DEBUG:boto:Method: GET 2023-09-29T22:34:28.939 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.939 DEBUG:boto:Data: 2023-09-29T22:34:28.939 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.940 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.940 DEBUG:boto:Params: {} 2023-09-29T22:34:28.940 DEBUG:boto:Token: None 2023-09-29T22:34:28.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco= 2023-09-29T22:34:28.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco=', 'Content-Length': '0'} 2023-09-29T22:34:28.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050fc64bb108228b8-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.944 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.944 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.944 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.945 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.945 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.945 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.945 DEBUG:boto:Data: 2023-09-29T22:34:28.945 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.945 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.945 DEBUG:boto:Params: {} 2023-09-29T22:34:28.945 DEBUG:boto:Token: None 2023-09-29T22:34:28.946 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.946 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4aJ6ETYoUyl2tcghaRW1B1lvBBs= 2023-09-29T22:34:28.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4aJ6ETYoUyl2tcghaRW1B1lvBBs=', 'Content-Length': '0'} 2023-09-29T22:34:28.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004926f72915bebe7b-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.949 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.949 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.949 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.949 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.949 DEBUG:boto:Method: HEAD 2023-09-29T22:34:28.949 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.949 DEBUG:boto:Data: 2023-09-29T22:34:28.949 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.950 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.950 DEBUG:boto:Params: {} 2023-09-29T22:34:28.950 DEBUG:boto:Token: None 2023-09-29T22:34:28.950 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.950 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4aJ6ETYoUyl2tcghaRW1B1lvBBs= 2023-09-29T22:34:28.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4aJ6ETYoUyl2tcghaRW1B1lvBBs=', 'Content-Length': '0'} 2023-09-29T22:34:28.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b10b77940ee1b2-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.953 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-09-29T22:34:28.953 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.953 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.953 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.954 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.954 DEBUG:boto:Method: GET 2023-09-29T22:34:28.954 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.954 DEBUG:boto:Data: 2023-09-29T22:34:28.954 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:28.954 DEBUG:boto:Port: 8001 2023-09-29T22:34:28.954 DEBUG:boto:Params: {} 2023-09-29T22:34:28.954 DEBUG:boto:Token: None 2023-09-29T22:34:28.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco= 2023-09-29T22:34:28.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco=', 'Content-Length': '0'} 2023-09-29T22:34:28.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be40b5941ba24ac4-00651750f4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:28.958 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.958 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:28.958 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.958 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.958 DEBUG:boto:Method: GET 2023-09-29T22:34:28.958 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.958 DEBUG:boto:Data: 2023-09-29T22:34:28.959 DEBUG:boto:Headers: {} 2023-09-29T22:34:28.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:28.959 DEBUG:boto:Port: 8000 2023-09-29T22:34:28.959 DEBUG:boto:Params: {} 2023-09-29T22:34:28.959 DEBUG:boto:Token: None 2023-09-29T22:34:28.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:28 GMT /hcnphz-58/obj00000000000000000018?versionId=null 2023-09-29T22:34:28.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco= 2023-09-29T22:34:28.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4PV55psTuEdY2hSDs8kJqL3gYco=', 'Content-Length': '0'} 2023-09-29T22:34:28.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce871debd761ce8d-00651750f4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.010 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.011 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.011 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.011 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.011 DEBUG:boto:Method: GET 2023-09-29T22:34:29.011 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.011 DEBUG:boto:Data: 2023-09-29T22:34:29.011 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.011 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.011 DEBUG:boto:Params: {} 2023-09-29T22:34:29.012 DEBUG:boto:Token: None 2023-09-29T22:34:29.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099390cfc7c255f32-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.018 DEBUG:boto:b'hcnphz-58obj000000000000000000181000falseobj000000000000000000182023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.018 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.019 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.019 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.019 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.019 DEBUG:boto:Method: GET 2023-09-29T22:34:29.019 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.019 DEBUG:boto:Data: 2023-09-29T22:34:29.019 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.019 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.019 DEBUG:boto:Params: {} 2023-09-29T22:34:29.020 DEBUG:boto:Token: None 2023-09-29T22:34:29.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023be01470f945b7b-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.028 DEBUG:boto:b'hcnphz-58obj000000000000000000181000falseobj000000000000000000182023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.028 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.028 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.029 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.029 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.029 DEBUG:boto:Data: 2023-09-29T22:34:29.029 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.029 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.029 DEBUG:boto:Params: {} 2023-09-29T22:34:29.029 DEBUG:boto:Token: None 2023-09-29T22:34:29.030 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AdBlqypW8/x3ttvvx5B6iyeCo/I= 2023-09-29T22:34:29.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AdBlqypW8/x3ttvvx5B6iyeCo/I=', 'Content-Length': '0'} 2023-09-29T22:34:29.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008112b1e260706143-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.032 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.032 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.032 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.033 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.033 DEBUG:boto:Data: 2023-09-29T22:34:29.033 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.033 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.033 DEBUG:boto:Params: {} 2023-09-29T22:34:29.033 DEBUG:boto:Token: None 2023-09-29T22:34:29.033 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AdBlqypW8/x3ttvvx5B6iyeCo/I= 2023-09-29T22:34:29.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AdBlqypW8/x3ttvvx5B6iyeCo/I=', 'Content-Length': '0'} 2023-09-29T22:34:29.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a0c031e8d13db75-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000018 2023-09-29T22:34:29.036 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.036 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.036 DEBUG:boto:Method: GET 2023-09-29T22:34:29.036 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.036 DEBUG:boto:Data: 2023-09-29T22:34:29.037 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.037 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.037 DEBUG:boto:Params: {} 2023-09-29T22:34:29.037 DEBUG:boto:Token: None 2023-09-29T22:34:29.037 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.037 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xGlB5afozHeEdQ1uYoIMbQhPgIk= 2023-09-29T22:34:29.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xGlB5afozHeEdQ1uYoIMbQhPgIk=', 'Content-Length': '0'} 2023-09-29T22:34:29.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd71177721b5be8d-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.040 DEBUG:boto:path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.040 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.040 DEBUG:boto:Method: GET 2023-09-29T22:34:29.040 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.040 DEBUG:boto:Data: 2023-09-29T22:34:29.040 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.041 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.041 DEBUG:boto:Params: {} 2023-09-29T22:34:29.041 DEBUG:boto:Token: None 2023-09-29T22:34:29.041 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000018 2023-09-29T22:34:29.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xGlB5afozHeEdQ1uYoIMbQhPgIk= 2023-09-29T22:34:29.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xGlB5afozHeEdQ1uYoIMbQhPgIk=', 'Content-Length': '0'} 2023-09-29T22:34:29.044 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077b5a97090394f0d-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.044 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.044 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.044 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.044 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.044 DEBUG:boto:Method: GET 2023-09-29T22:34:29.044 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.045 DEBUG:boto:Data: 2023-09-29T22:34:29.045 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.045 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.045 DEBUG:boto:Params: {} 2023-09-29T22:34:29.045 DEBUG:boto:Token: None 2023-09-29T22:34:29.045 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.045 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a03d7025c24c755-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.090 DEBUG:boto:b'hcnphz-58obj000000000000000000181000falseobj000000000000000000182023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.091 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.091 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.091 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.091 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.091 DEBUG:boto:Method: GET 2023-09-29T22:34:29.092 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000018 2023-09-29T22:34:29.092 DEBUG:boto:Data: 2023-09-29T22:34:29.092 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.092 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.092 DEBUG:boto:Params: {} 2023-09-29T22:34:29.092 DEBUG:boto:Token: None 2023-09-29T22:34:29.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.099 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c9486b70bdecfef8-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.099 DEBUG:boto:b'hcnphz-58obj000000000000000000181000falseobj000000000000000000182023-09-29T22:34:16.646Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-09-29T22:34:29.099 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.100 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.100 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.100 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.100 DEBUG:boto:Method: GET 2023-09-29T22:34:29.100 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.100 DEBUG:boto:Data: 2023-09-29T22:34:29.100 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.100 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.100 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.101 DEBUG:boto:Params: {} 2023-09-29T22:34:29.101 DEBUG:boto:Token: None 2023-09-29T22:34:29.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.101 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI= 2023-09-29T22:34:29.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI=', 'Content-Length': '0'} 2023-09-29T22:34:29.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000590c6df119db7c99-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.104 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.104 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.104 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.104 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.104 DEBUG:boto:Method: GET 2023-09-29T22:34:29.104 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.104 DEBUG:boto:Data: 2023-09-29T22:34:29.104 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.105 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.105 DEBUG:boto:Params: {} 2023-09-29T22:34:29.105 DEBUG:boto:Token: None 2023-09-29T22:34:29.105 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI= 2023-09-29T22:34:29.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI=', 'Content-Length': '0'} 2023-09-29T22:34:29.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070dda2cc0a612efb-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.150 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.151 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.151 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.151 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.151 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.151 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.151 DEBUG:boto:Data: 2023-09-29T22:34:29.151 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.152 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.152 DEBUG:boto:Params: {} 2023-09-29T22:34:29.152 DEBUG:boto:Token: None 2023-09-29T22:34:29.152 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kN+F8ALAsVWPKeMhoz8fXHtkfak= 2023-09-29T22:34:29.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kN+F8ALAsVWPKeMhoz8fXHtkfak=', 'Content-Length': '0'} 2023-09-29T22:34:29.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025e2865dde3d59ae-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.155 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.155 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.156 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.156 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.156 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.156 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.156 DEBUG:boto:Data: 2023-09-29T22:34:29.156 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.156 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.156 DEBUG:boto:Params: {} 2023-09-29T22:34:29.157 DEBUG:boto:Token: None 2023-09-29T22:34:29.157 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kN+F8ALAsVWPKeMhoz8fXHtkfak= 2023-09-29T22:34:29.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kN+F8ALAsVWPKeMhoz8fXHtkfak=', 'Content-Length': '0'} 2023-09-29T22:34:29.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389046d52df47788-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-09-29T22:34:29.160 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.160 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.160 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.160 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.160 DEBUG:boto:Method: GET 2023-09-29T22:34:29.160 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.161 DEBUG:boto:Data: 2023-09-29T22:34:29.161 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.161 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.161 DEBUG:boto:Params: {} 2023-09-29T22:34:29.161 DEBUG:boto:Token: None 2023-09-29T22:34:29.161 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI= 2023-09-29T22:34:29.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI=', 'Content-Length': '0'} 2023-09-29T22:34:29.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0f311ca2860d8cb-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.165 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.165 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.165 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.165 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.165 DEBUG:boto:Method: GET 2023-09-29T22:34:29.165 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.165 DEBUG:boto:Data: 2023-09-29T22:34:29.165 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.166 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.166 DEBUG:boto:Params: {} 2023-09-29T22:34:29.166 DEBUG:boto:Token: None 2023-09-29T22:34:29.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019?versionId=null 2023-09-29T22:34:29.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI= 2023-09-29T22:34:29.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tYf+cfLj3msg1NY3qrVlBk61ZmI=', 'Content-Length': '0'} 2023-09-29T22:34:29.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d4c45d23fc8af1d-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.169 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.169 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.170 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.170 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.170 DEBUG:boto:Method: GET 2023-09-29T22:34:29.170 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.170 DEBUG:boto:Data: 2023-09-29T22:34:29.170 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.170 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.170 DEBUG:boto:Params: {} 2023-09-29T22:34:29.171 DEBUG:boto:Token: None 2023-09-29T22:34:29.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d396e2f45fc421b3-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.222 DEBUG:boto:b'hcnphz-58obj000000000000000000191000falseobj000000000000000000192023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.223 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.223 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.223 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.223 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.223 DEBUG:boto:Method: GET 2023-09-29T22:34:29.223 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.224 DEBUG:boto:Data: 2023-09-29T22:34:29.224 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.224 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.224 DEBUG:boto:Params: {} 2023-09-29T22:34:29.224 DEBUG:boto:Token: None 2023-09-29T22:34:29.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009de0a0ec1549f759-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.237 DEBUG:boto:b'hcnphz-58obj000000000000000000191000falseobj000000000000000000192023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.237 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.237 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.237 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.237 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.237 DEBUG:boto:Data: 2023-09-29T22:34:29.238 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.238 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.238 DEBUG:boto:Params: {} 2023-09-29T22:34:29.238 DEBUG:boto:Token: None 2023-09-29T22:34:29.238 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tVJa27FOPF24N4UPunjh7oPIENQ= 2023-09-29T22:34:29.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tVJa27FOPF24N4UPunjh7oPIENQ=', 'Content-Length': '0'} 2023-09-29T22:34:29.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050a325924b642d28-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.241 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.241 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.241 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.241 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.241 DEBUG:boto:Data: 2023-09-29T22:34:29.241 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.242 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.242 DEBUG:boto:Params: {} 2023-09-29T22:34:29.242 DEBUG:boto:Token: None 2023-09-29T22:34:29.242 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tVJa27FOPF24N4UPunjh7oPIENQ= 2023-09-29T22:34:29.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tVJa27FOPF24N4UPunjh7oPIENQ=', 'Content-Length': '0'} 2023-09-29T22:34:29.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b7484072f7af3e-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000019 2023-09-29T22:34:29.244 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.244 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.245 DEBUG:boto:Method: GET 2023-09-29T22:34:29.245 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.245 DEBUG:boto:Data: 2023-09-29T22:34:29.245 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.245 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.245 DEBUG:boto:Params: {} 2023-09-29T22:34:29.245 DEBUG:boto:Token: None 2023-09-29T22:34:29.245 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hiyhp6DBM58tUGeMQXkhKUDXJjs= 2023-09-29T22:34:29.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hiyhp6DBM58tUGeMQXkhKUDXJjs=', 'Content-Length': '0'} 2023-09-29T22:34:29.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd1283bac0c5e63f-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.248 DEBUG:boto:path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.248 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.248 DEBUG:boto:Method: GET 2023-09-29T22:34:29.248 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.248 DEBUG:boto:Data: 2023-09-29T22:34:29.248 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.248 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.248 DEBUG:boto:Params: {} 2023-09-29T22:34:29.249 DEBUG:boto:Token: None 2023-09-29T22:34:29.249 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000019 2023-09-29T22:34:29.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hiyhp6DBM58tUGeMQXkhKUDXJjs= 2023-09-29T22:34:29.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hiyhp6DBM58tUGeMQXkhKUDXJjs=', 'Content-Length': '0'} 2023-09-29T22:34:29.251 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f5e750e6755f9d-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.294 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.294 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.295 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.295 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.295 DEBUG:boto:Method: GET 2023-09-29T22:34:29.295 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.295 DEBUG:boto:Data: 2023-09-29T22:34:29.295 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.295 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.295 DEBUG:boto:Params: {} 2023-09-29T22:34:29.295 DEBUG:boto:Token: None 2023-09-29T22:34:29.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011d1a958f043d2a7-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.301 DEBUG:boto:b'hcnphz-58obj000000000000000000191000falseobj000000000000000000192023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.301 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.301 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.301 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.302 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.302 DEBUG:boto:Method: GET 2023-09-29T22:34:29.302 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000019 2023-09-29T22:34:29.302 DEBUG:boto:Data: 2023-09-29T22:34:29.302 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.302 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.302 DEBUG:boto:Params: {} 2023-09-29T22:34:29.302 DEBUG:boto:Token: None 2023-09-29T22:34:29.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008af522291b5934ed-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.311 DEBUG:boto:b'hcnphz-58obj000000000000000000191000falseobj000000000000000000192023-09-29T22:34:16.273Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-09-29T22:34:29.312 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.312 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.312 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.312 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.312 DEBUG:boto:Method: GET 2023-09-29T22:34:29.312 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.312 DEBUG:boto:Data: 2023-09-29T22:34:29.313 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.313 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.313 DEBUG:boto:Params: {} 2023-09-29T22:34:29.313 DEBUG:boto:Token: None 2023-09-29T22:34:29.313 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY= 2023-09-29T22:34:29.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY=', 'Content-Length': '0'} 2023-09-29T22:34:29.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039bab16438be7310-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.317 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.317 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.317 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.317 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.318 DEBUG:boto:Method: GET 2023-09-29T22:34:29.318 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.318 DEBUG:boto:Data: 2023-09-29T22:34:29.318 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.318 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.318 DEBUG:boto:Params: {} 2023-09-29T22:34:29.318 DEBUG:boto:Token: None 2023-09-29T22:34:29.318 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY= 2023-09-29T22:34:29.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY=', 'Content-Length': '0'} 2023-09-29T22:34:29.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000109f154e35e42a3-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.321 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.321 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.321 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.321 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.321 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.321 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.321 DEBUG:boto:Data: 2023-09-29T22:34:29.322 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.322 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.322 DEBUG:boto:Params: {} 2023-09-29T22:34:29.322 DEBUG:boto:Token: None 2023-09-29T22:34:29.322 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.322 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8OaIqFrjsCvap7WeGELIoH035q4= 2023-09-29T22:34:29.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8OaIqFrjsCvap7WeGELIoH035q4=', 'Content-Length': '0'} 2023-09-29T22:34:29.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059cd1ebc6c06bab2-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.324 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.325 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.325 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.325 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.325 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.325 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.325 DEBUG:boto:Data: 2023-09-29T22:34:29.325 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.325 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.326 DEBUG:boto:Params: {} 2023-09-29T22:34:29.326 DEBUG:boto:Token: None 2023-09-29T22:34:29.326 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.326 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8OaIqFrjsCvap7WeGELIoH035q4= 2023-09-29T22:34:29.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8OaIqFrjsCvap7WeGELIoH035q4=', 'Content-Length': '0'} 2023-09-29T22:34:29.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028d6866994ce565-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-09-29T22:34:29.329 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.329 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.329 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.329 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.329 DEBUG:boto:Method: GET 2023-09-29T22:34:29.329 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.329 DEBUG:boto:Data: 2023-09-29T22:34:29.329 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.329 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.330 DEBUG:boto:Params: {} 2023-09-29T22:34:29.330 DEBUG:boto:Token: None 2023-09-29T22:34:29.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY= 2023-09-29T22:34:29.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY=', 'Content-Length': '0'} 2023-09-29T22:34:29.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061fa54866b9ead79-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.374 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.374 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.375 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.375 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.375 DEBUG:boto:Method: GET 2023-09-29T22:34:29.375 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.375 DEBUG:boto:Data: 2023-09-29T22:34:29.375 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.375 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.375 DEBUG:boto:Params: {} 2023-09-29T22:34:29.376 DEBUG:boto:Token: None 2023-09-29T22:34:29.376 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020?versionId=null 2023-09-29T22:34:29.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY= 2023-09-29T22:34:29.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E0aiDPij9rEteR9LToyB7YRXufY=', 'Content-Length': '0'} 2023-09-29T22:34:29.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e612652ceb1b0423-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.379 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.379 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.379 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.379 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.380 DEBUG:boto:Method: GET 2023-09-29T22:34:29.380 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.380 DEBUG:boto:Data: 2023-09-29T22:34:29.380 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.380 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.380 DEBUG:boto:Params: {} 2023-09-29T22:34:29.380 DEBUG:boto:Token: None 2023-09-29T22:34:29.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095e9412031c2da35-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.385 DEBUG:boto:b'hcnphz-58obj000000000000000000201000falseobj000000000000000000202023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.386 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.386 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.386 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.386 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.386 DEBUG:boto:Method: GET 2023-09-29T22:34:29.386 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.386 DEBUG:boto:Data: 2023-09-29T22:34:29.386 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.387 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.387 DEBUG:boto:Params: {} 2023-09-29T22:34:29.387 DEBUG:boto:Token: None 2023-09-29T22:34:29.387 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.394 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000126a7e1e515264db-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.442 DEBUG:boto:b'hcnphz-58obj000000000000000000201000falseobj000000000000000000202023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.443 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.443 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.443 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.443 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.443 DEBUG:boto:Data: 2023-09-29T22:34:29.443 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.444 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.444 DEBUG:boto:Params: {} 2023-09-29T22:34:29.444 DEBUG:boto:Token: None 2023-09-29T22:34:29.444 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TM1r3rv5r7FqlC3PlM38pKPUq7s= 2023-09-29T22:34:29.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TM1r3rv5r7FqlC3PlM38pKPUq7s=', 'Content-Length': '0'} 2023-09-29T22:34:29.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a93b3cc23b5b6a7-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.447 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.448 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.448 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.448 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.448 DEBUG:boto:Data: 2023-09-29T22:34:29.448 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.448 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.448 DEBUG:boto:Params: {} 2023-09-29T22:34:29.448 DEBUG:boto:Token: None 2023-09-29T22:34:29.449 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.449 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TM1r3rv5r7FqlC3PlM38pKPUq7s= 2023-09-29T22:34:29.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TM1r3rv5r7FqlC3PlM38pKPUq7s=', 'Content-Length': '0'} 2023-09-29T22:34:29.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5743d078ed7d5b1-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000020 2023-09-29T22:34:29.452 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.452 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.452 DEBUG:boto:Method: GET 2023-09-29T22:34:29.452 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.453 DEBUG:boto:Data: 2023-09-29T22:34:29.453 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.453 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.453 DEBUG:boto:Params: {} 2023-09-29T22:34:29.453 DEBUG:boto:Token: None 2023-09-29T22:34:29.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PiiMoKJJ42K5C44TVTWwo6/kpQI= 2023-09-29T22:34:29.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PiiMoKJJ42K5C44TVTWwo6/kpQI=', 'Content-Length': '0'} 2023-09-29T22:34:29.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e75dccb1e0e5638-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.457 DEBUG:boto:path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.457 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.457 DEBUG:boto:Method: GET 2023-09-29T22:34:29.457 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.457 DEBUG:boto:Data: 2023-09-29T22:34:29.457 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.457 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.457 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.458 DEBUG:boto:Params: {} 2023-09-29T22:34:29.458 DEBUG:boto:Token: None 2023-09-29T22:34:29.458 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000020 2023-09-29T22:34:29.458 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PiiMoKJJ42K5C44TVTWwo6/kpQI= 2023-09-29T22:34:29.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PiiMoKJJ42K5C44TVTWwo6/kpQI=', 'Content-Length': '0'} 2023-09-29T22:34:29.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2406ff7dff20d1-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.461 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.461 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.461 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.462 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.462 DEBUG:boto:Method: GET 2023-09-29T22:34:29.462 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.462 DEBUG:boto:Data: 2023-09-29T22:34:29.462 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.462 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.462 DEBUG:boto:Params: {} 2023-09-29T22:34:29.462 DEBUG:boto:Token: None 2023-09-29T22:34:29.463 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.463 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.467 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000575a9ee71f8383f8-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.510 DEBUG:boto:b'hcnphz-58obj000000000000000000201000falseobj000000000000000000202023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.511 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.511 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.511 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.511 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.511 DEBUG:boto:Method: GET 2023-09-29T22:34:29.512 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000020 2023-09-29T22:34:29.512 DEBUG:boto:Data: 2023-09-29T22:34:29.512 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.512 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.512 DEBUG:boto:Params: {} 2023-09-29T22:34:29.513 DEBUG:boto:Token: None 2023-09-29T22:34:29.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afeab4dcf4682084-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.522 DEBUG:boto:b'hcnphz-58obj000000000000000000201000falseobj000000000000000000202023-09-29T22:34:16.591Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-09-29T22:34:29.522 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.523 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.523 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.523 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.523 DEBUG:boto:Method: GET 2023-09-29T22:34:29.523 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.523 DEBUG:boto:Data: 2023-09-29T22:34:29.523 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.524 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.524 DEBUG:boto:Params: {} 2023-09-29T22:34:29.524 DEBUG:boto:Token: None 2023-09-29T22:34:29.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k= 2023-09-29T22:34:29.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k=', 'Content-Length': '0'} 2023-09-29T22:34:29.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cac578f5aa219d1-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.527 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.527 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.528 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.528 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.528 DEBUG:boto:Method: GET 2023-09-29T22:34:29.528 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.528 DEBUG:boto:Data: 2023-09-29T22:34:29.528 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.528 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.528 DEBUG:boto:Params: {} 2023-09-29T22:34:29.529 DEBUG:boto:Token: None 2023-09-29T22:34:29.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k= 2023-09-29T22:34:29.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k=', 'Content-Length': '0'} 2023-09-29T22:34:29.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b558aab0cb1f0e4-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.574 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.575 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.575 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.575 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.575 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.575 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.575 DEBUG:boto:Data: 2023-09-29T22:34:29.575 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.576 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.576 DEBUG:boto:Params: {} 2023-09-29T22:34:29.576 DEBUG:boto:Token: None 2023-09-29T22:34:29.576 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OrYKddSvakPEw8UZ17hmu61VC0w= 2023-09-29T22:34:29.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OrYKddSvakPEw8UZ17hmu61VC0w=', 'Content-Length': '0'} 2023-09-29T22:34:29.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e51b2baba1371f36-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.580 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.580 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.580 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.580 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.580 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.581 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.581 DEBUG:boto:Data: 2023-09-29T22:34:29.581 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.581 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.581 DEBUG:boto:Params: {} 2023-09-29T22:34:29.581 DEBUG:boto:Token: None 2023-09-29T22:34:29.582 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OrYKddSvakPEw8UZ17hmu61VC0w= 2023-09-29T22:34:29.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OrYKddSvakPEw8UZ17hmu61VC0w=', 'Content-Length': '0'} 2023-09-29T22:34:29.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b85628f960046a2-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-09-29T22:34:29.585 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.585 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.585 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.586 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.586 DEBUG:boto:Method: GET 2023-09-29T22:34:29.586 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.586 DEBUG:boto:Data: 2023-09-29T22:34:29.586 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.586 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.586 DEBUG:boto:Params: {} 2023-09-29T22:34:29.587 DEBUG:boto:Token: None 2023-09-29T22:34:29.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k= 2023-09-29T22:34:29.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k=', 'Content-Length': '0'} 2023-09-29T22:34:29.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c99d60b7d012767-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.590 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.590 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.590 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.590 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.591 DEBUG:boto:Method: GET 2023-09-29T22:34:29.591 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.591 DEBUG:boto:Data: 2023-09-29T22:34:29.591 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.591 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.591 DEBUG:boto:Params: {} 2023-09-29T22:34:29.591 DEBUG:boto:Token: None 2023-09-29T22:34:29.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021?versionId=null 2023-09-29T22:34:29.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k= 2023-09-29T22:34:29.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xb6tYifvI7rxVc/TOywJJ2gG33k=', 'Content-Length': '0'} 2023-09-29T22:34:29.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056f5460925be7b0a-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.595 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.595 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.595 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.596 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.596 DEBUG:boto:Method: GET 2023-09-29T22:34:29.596 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.596 DEBUG:boto:Data: 2023-09-29T22:34:29.596 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.596 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.597 DEBUG:boto:Params: {} 2023-09-29T22:34:29.597 DEBUG:boto:Token: None 2023-09-29T22:34:29.597 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.602 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3c28e92c535d931-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.602 DEBUG:boto:b'hcnphz-58obj000000000000000000211000falseobj000000000000000000212023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.602 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.603 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.603 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.603 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.603 DEBUG:boto:Method: GET 2023-09-29T22:34:29.603 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.603 DEBUG:boto:Data: 2023-09-29T22:34:29.603 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.604 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.604 DEBUG:boto:Params: {} 2023-09-29T22:34:29.604 DEBUG:boto:Token: None 2023-09-29T22:34:29.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.612 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af0ff4ed44f515e9-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.612 DEBUG:boto:b'hcnphz-58obj000000000000000000211000falseobj000000000000000000212023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.612 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.612 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.613 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.613 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.613 DEBUG:boto:Data: 2023-09-29T22:34:29.613 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.613 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.613 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.613 DEBUG:boto:Params: {} 2023-09-29T22:34:29.614 DEBUG:boto:Token: None 2023-09-29T22:34:29.614 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.614 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9x1D0+Cu6bp/Ny8HG7Y5n7sIrL8= 2023-09-29T22:34:29.614 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9x1D0+Cu6bp/Ny8HG7Y5n7sIrL8=', 'Content-Length': '0'} 2023-09-29T22:34:29.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b7c131d6fa5fcde-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.616 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.617 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.617 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.617 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.617 DEBUG:boto:Data: 2023-09-29T22:34:29.617 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.617 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.617 DEBUG:boto:Params: {} 2023-09-29T22:34:29.618 DEBUG:boto:Token: None 2023-09-29T22:34:29.618 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.618 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9x1D0+Cu6bp/Ny8HG7Y5n7sIrL8= 2023-09-29T22:34:29.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9x1D0+Cu6bp/Ny8HG7Y5n7sIrL8=', 'Content-Length': '0'} 2023-09-29T22:34:29.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a452cf0bdfe57190-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000021 2023-09-29T22:34:29.621 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.621 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.621 DEBUG:boto:Method: GET 2023-09-29T22:34:29.621 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.621 DEBUG:boto:Data: 2023-09-29T22:34:29.621 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.622 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.622 DEBUG:boto:Params: {} 2023-09-29T22:34:29.622 DEBUG:boto:Token: None 2023-09-29T22:34:29.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9lfgPKShb/fwPSczbhn39Gv06tw= 2023-09-29T22:34:29.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9lfgPKShb/fwPSczbhn39Gv06tw=', 'Content-Length': '0'} 2023-09-29T22:34:29.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060e661aae694266b-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.666 DEBUG:boto:path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.667 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.667 DEBUG:boto:Method: GET 2023-09-29T22:34:29.667 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.667 DEBUG:boto:Data: 2023-09-29T22:34:29.667 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.667 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.667 DEBUG:boto:Params: {} 2023-09-29T22:34:29.667 DEBUG:boto:Token: None 2023-09-29T22:34:29.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000021 2023-09-29T22:34:29.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9lfgPKShb/fwPSczbhn39Gv06tw= 2023-09-29T22:34:29.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9lfgPKShb/fwPSczbhn39Gv06tw=', 'Content-Length': '0'} 2023-09-29T22:34:29.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee9e0aead9492fd3-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.671 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.671 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.671 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.671 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.672 DEBUG:boto:Method: GET 2023-09-29T22:34:29.672 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.672 DEBUG:boto:Data: 2023-09-29T22:34:29.672 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.672 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.672 DEBUG:boto:Params: {} 2023-09-29T22:34:29.672 DEBUG:boto:Token: None 2023-09-29T22:34:29.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b20b53fec3448887-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.678 DEBUG:boto:b'hcnphz-58obj000000000000000000211000falseobj000000000000000000212023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.678 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.678 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.679 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.679 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.679 DEBUG:boto:Method: GET 2023-09-29T22:34:29.679 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000021 2023-09-29T22:34:29.679 DEBUG:boto:Data: 2023-09-29T22:34:29.679 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.679 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.679 DEBUG:boto:Params: {} 2023-09-29T22:34:29.679 DEBUG:boto:Token: None 2023-09-29T22:34:29.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c99a62ee4757c2a-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.688 DEBUG:boto:b'hcnphz-58obj000000000000000000211000falseobj000000000000000000212023-09-29T22:34:14.064Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-09-29T22:34:29.688 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.688 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.688 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.689 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.689 DEBUG:boto:Method: GET 2023-09-29T22:34:29.689 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.689 DEBUG:boto:Data: 2023-09-29T22:34:29.689 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.689 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.689 DEBUG:boto:Params: {} 2023-09-29T22:34:29.689 DEBUG:boto:Token: None 2023-09-29T22:34:29.690 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.690 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI= 2023-09-29T22:34:29.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI=', 'Content-Length': '0'} 2023-09-29T22:34:29.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000272d34baf50d9a89-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.692 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.693 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.693 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.693 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.693 DEBUG:boto:Method: GET 2023-09-29T22:34:29.693 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.693 DEBUG:boto:Data: 2023-09-29T22:34:29.693 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.694 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.694 DEBUG:boto:Params: {} 2023-09-29T22:34:29.694 DEBUG:boto:Token: None 2023-09-29T22:34:29.694 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI= 2023-09-29T22:34:29.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI=', 'Content-Length': '0'} 2023-09-29T22:34:29.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7b3dd62044ae421-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.697 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.697 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.697 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.697 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.697 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.697 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.697 DEBUG:boto:Data: 2023-09-29T22:34:29.697 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.698 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.698 DEBUG:boto:Params: {} 2023-09-29T22:34:29.698 DEBUG:boto:Token: None 2023-09-29T22:34:29.698 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BD0qTED7nqS5DkHJAi1HRK1u6Wo= 2023-09-29T22:34:29.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BD0qTED7nqS5DkHJAi1HRK1u6Wo=', 'Content-Length': '0'} 2023-09-29T22:34:29.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df600274b3060a6f-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.700 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.700 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.701 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.701 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.701 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.701 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.701 DEBUG:boto:Data: 2023-09-29T22:34:29.701 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.701 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.701 DEBUG:boto:Params: {} 2023-09-29T22:34:29.702 DEBUG:boto:Token: None 2023-09-29T22:34:29.702 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BD0qTED7nqS5DkHJAi1HRK1u6Wo= 2023-09-29T22:34:29.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BD0qTED7nqS5DkHJAi1HRK1u6Wo=', 'Content-Length': '0'} 2023-09-29T22:34:29.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaed6357627840e7-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-09-29T22:34:29.704 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.704 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.705 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.705 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.705 DEBUG:boto:Method: GET 2023-09-29T22:34:29.705 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.705 DEBUG:boto:Data: 2023-09-29T22:34:29.705 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.705 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.705 DEBUG:boto:Params: {} 2023-09-29T22:34:29.705 DEBUG:boto:Token: None 2023-09-29T22:34:29.706 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.706 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI= 2023-09-29T22:34:29.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI=', 'Content-Length': '0'} 2023-09-29T22:34:29.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1fb0020e3b4aa4-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.709 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.709 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.709 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.709 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.709 DEBUG:boto:Method: GET 2023-09-29T22:34:29.709 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.709 DEBUG:boto:Data: 2023-09-29T22:34:29.709 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.710 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.710 DEBUG:boto:Params: {} 2023-09-29T22:34:29.710 DEBUG:boto:Token: None 2023-09-29T22:34:29.710 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022?versionId=null 2023-09-29T22:34:29.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI= 2023-09-29T22:34:29.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bqp1ZMYnsEJMw7weWbRDGkjqkoI=', 'Content-Length': '0'} 2023-09-29T22:34:29.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a5760ecb3dc5cb6-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.713 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.713 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.713 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.713 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.713 DEBUG:boto:Method: GET 2023-09-29T22:34:29.713 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.713 DEBUG:boto:Data: 2023-09-29T22:34:29.714 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.714 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.714 DEBUG:boto:Params: {} 2023-09-29T22:34:29.714 DEBUG:boto:Token: None 2023-09-29T22:34:29.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008680e55a56a1af85-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.718 DEBUG:boto:b'hcnphz-58obj000000000000000000221000falseobj000000000000000000222023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.718 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.718 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.719 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.719 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.719 DEBUG:boto:Method: GET 2023-09-29T22:34:29.719 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.719 DEBUG:boto:Data: 2023-09-29T22:34:29.719 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.719 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.719 DEBUG:boto:Params: {} 2023-09-29T22:34:29.720 DEBUG:boto:Token: None 2023-09-29T22:34:29.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.728 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008349da623e1b9889-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.728 DEBUG:boto:b'hcnphz-58obj000000000000000000221000falseobj000000000000000000222023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.728 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.728 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.728 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.729 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.729 DEBUG:boto:Data: 2023-09-29T22:34:29.729 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.729 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.729 DEBUG:boto:Params: {} 2023-09-29T22:34:29.729 DEBUG:boto:Token: None 2023-09-29T22:34:29.729 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Hky9/PqwFvNj4gMqrc7+GG1rXs= 2023-09-29T22:34:29.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Hky9/PqwFvNj4gMqrc7+GG1rXs=', 'Content-Length': '0'} 2023-09-29T22:34:29.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cdeda85623fe78a-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.732 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.732 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.732 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.732 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.732 DEBUG:boto:Data: 2023-09-29T22:34:29.733 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.733 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.733 DEBUG:boto:Params: {} 2023-09-29T22:34:29.733 DEBUG:boto:Token: None 2023-09-29T22:34:29.733 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Hky9/PqwFvNj4gMqrc7+GG1rXs= 2023-09-29T22:34:29.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Hky9/PqwFvNj4gMqrc7+GG1rXs=', 'Content-Length': '0'} 2023-09-29T22:34:29.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b8ca80470a4434d-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000022 2023-09-29T22:34:29.736 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.736 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.736 DEBUG:boto:Method: GET 2023-09-29T22:34:29.736 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.736 DEBUG:boto:Data: 2023-09-29T22:34:29.736 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.737 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.737 DEBUG:boto:Params: {} 2023-09-29T22:34:29.737 DEBUG:boto:Token: None 2023-09-29T22:34:29.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.737 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7XJJ34uVcn0A8zAShvu/ub48wOI= 2023-09-29T22:34:29.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7XJJ34uVcn0A8zAShvu/ub48wOI=', 'Content-Length': '0'} 2023-09-29T22:34:29.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0c64864706a552-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.786 DEBUG:boto:path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.787 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.787 DEBUG:boto:Method: GET 2023-09-29T22:34:29.787 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.787 DEBUG:boto:Data: 2023-09-29T22:34:29.787 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.787 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.787 DEBUG:boto:Params: {} 2023-09-29T22:34:29.788 DEBUG:boto:Token: None 2023-09-29T22:34:29.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000022 2023-09-29T22:34:29.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7XJJ34uVcn0A8zAShvu/ub48wOI= 2023-09-29T22:34:29.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7XJJ34uVcn0A8zAShvu/ub48wOI=', 'Content-Length': '0'} 2023-09-29T22:34:29.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7f3abb4dd6b39e-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.791 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.791 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.791 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.791 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.792 DEBUG:boto:Method: GET 2023-09-29T22:34:29.792 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.792 DEBUG:boto:Data: 2023-09-29T22:34:29.792 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.792 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.792 DEBUG:boto:Params: {} 2023-09-29T22:34:29.792 DEBUG:boto:Token: None 2023-09-29T22:34:29.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000097475f454063fe0-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.798 DEBUG:boto:b'hcnphz-58obj000000000000000000221000falseobj000000000000000000222023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.798 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.798 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.798 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.798 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.798 DEBUG:boto:Method: GET 2023-09-29T22:34:29.799 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000022 2023-09-29T22:34:29.799 DEBUG:boto:Data: 2023-09-29T22:34:29.799 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.799 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.799 DEBUG:boto:Params: {} 2023-09-29T22:34:29.799 DEBUG:boto:Token: None 2023-09-29T22:34:29.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.799 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e39efbe5a6defc74-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.807 DEBUG:boto:b'hcnphz-58obj000000000000000000221000falseobj000000000000000000222023-09-29T22:34:14.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-09-29T22:34:29.808 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.808 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.808 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.808 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.808 DEBUG:boto:Method: GET 2023-09-29T22:34:29.808 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.808 DEBUG:boto:Data: 2023-09-29T22:34:29.809 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.809 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.809 DEBUG:boto:Params: {} 2023-09-29T22:34:29.809 DEBUG:boto:Token: None 2023-09-29T22:34:29.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo= 2023-09-29T22:34:29.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo=', 'Content-Length': '0'} 2023-09-29T22:34:29.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b65c9cae1af51c8a-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.812 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.812 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.812 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.812 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.813 DEBUG:boto:Method: GET 2023-09-29T22:34:29.813 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.813 DEBUG:boto:Data: 2023-09-29T22:34:29.813 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.813 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.813 DEBUG:boto:Params: {} 2023-09-29T22:34:29.813 DEBUG:boto:Token: None 2023-09-29T22:34:29.813 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.814 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo= 2023-09-29T22:34:29.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo=', 'Content-Length': '0'} 2023-09-29T22:34:29.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003496c8dcdc5fe01f-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.816 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.817 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.817 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.817 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.817 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.817 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.817 DEBUG:boto:Data: 2023-09-29T22:34:29.817 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.817 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.817 DEBUG:boto:Params: {} 2023-09-29T22:34:29.818 DEBUG:boto:Token: None 2023-09-29T22:34:29.818 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zDuHMrDnMAlKC2M8sJUr5lQf1HQ= 2023-09-29T22:34:29.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zDuHMrDnMAlKC2M8sJUr5lQf1HQ=', 'Content-Length': '0'} 2023-09-29T22:34:29.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec3d33981b1b7664-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.820 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.821 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.821 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.821 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.821 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.821 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.821 DEBUG:boto:Data: 2023-09-29T22:34:29.821 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.821 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.821 DEBUG:boto:Params: {} 2023-09-29T22:34:29.822 DEBUG:boto:Token: None 2023-09-29T22:34:29.822 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.822 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zDuHMrDnMAlKC2M8sJUr5lQf1HQ= 2023-09-29T22:34:29.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zDuHMrDnMAlKC2M8sJUr5lQf1HQ=', 'Content-Length': '0'} 2023-09-29T22:34:29.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008dc9918721c1552-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-09-29T22:34:29.825 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.825 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.825 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.825 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.825 DEBUG:boto:Method: GET 2023-09-29T22:34:29.825 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.825 DEBUG:boto:Data: 2023-09-29T22:34:29.825 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.826 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.826 DEBUG:boto:Params: {} 2023-09-29T22:34:29.826 DEBUG:boto:Token: None 2023-09-29T22:34:29.826 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.826 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo= 2023-09-29T22:34:29.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo=', 'Content-Length': '0'} 2023-09-29T22:34:29.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012cbbb612fc721f7-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.870 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.871 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.871 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.871 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.871 DEBUG:boto:Method: GET 2023-09-29T22:34:29.871 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.871 DEBUG:boto:Data: 2023-09-29T22:34:29.871 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.872 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.872 DEBUG:boto:Params: {} 2023-09-29T22:34:29.872 DEBUG:boto:Token: None 2023-09-29T22:34:29.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023?versionId=null 2023-09-29T22:34:29.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo= 2023-09-29T22:34:29.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1OWlGyeUI8mNAJax4HtZjxT6jeo=', 'Content-Length': '0'} 2023-09-29T22:34:29.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df4ba28f400fe183-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.876 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.876 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.876 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.876 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.876 DEBUG:boto:Method: GET 2023-09-29T22:34:29.876 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.876 DEBUG:boto:Data: 2023-09-29T22:34:29.876 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.877 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.877 DEBUG:boto:Params: {} 2023-09-29T22:34:29.877 DEBUG:boto:Token: None 2023-09-29T22:34:29.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b69e4941c946e32-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.883 DEBUG:boto:b'hcnphz-58obj000000000000000000231000falseobj000000000000000000232023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.883 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.883 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.883 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.883 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.884 DEBUG:boto:Method: GET 2023-09-29T22:34:29.884 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.884 DEBUG:boto:Data: 2023-09-29T22:34:29.884 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.884 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.884 DEBUG:boto:Params: {} 2023-09-29T22:34:29.884 DEBUG:boto:Token: None 2023-09-29T22:34:29.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f8a8fb1659f6e0-00651750f5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.934 DEBUG:boto:b'hcnphz-58obj000000000000000000231000falseobj000000000000000000232023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:29.935 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.935 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.935 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.935 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.935 DEBUG:boto:Data: 2023-09-29T22:34:29.935 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.936 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.936 DEBUG:boto:Params: {} 2023-09-29T22:34:29.936 DEBUG:boto:Token: None 2023-09-29T22:34:29.936 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Um5wwnGSta2lj/Wb+Qzuvr1Waio= 2023-09-29T22:34:29.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Um5wwnGSta2lj/Wb+Qzuvr1Waio=', 'Content-Length': '0'} 2023-09-29T22:34:29.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bae509fe3a54ebf6-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.940 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.940 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.940 DEBUG:boto:Method: HEAD 2023-09-29T22:34:29.940 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.940 DEBUG:boto:Data: 2023-09-29T22:34:29.940 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.940 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.941 DEBUG:boto:Params: {} 2023-09-29T22:34:29.941 DEBUG:boto:Token: None 2023-09-29T22:34:29.941 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Um5wwnGSta2lj/Wb+Qzuvr1Waio= 2023-09-29T22:34:29.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Um5wwnGSta2lj/Wb+Qzuvr1Waio=', 'Content-Length': '0'} 2023-09-29T22:34:29.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ca2b93b0bd31b13-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000023 2023-09-29T22:34:29.944 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.944 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.944 DEBUG:boto:Method: GET 2023-09-29T22:34:29.945 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.945 DEBUG:boto:Data: 2023-09-29T22:34:29.945 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.945 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.945 DEBUG:boto:Params: {} 2023-09-29T22:34:29.945 DEBUG:boto:Token: None 2023-09-29T22:34:29.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7xtgnH710MOfBe+DreygQMMeLX0= 2023-09-29T22:34:29.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7xtgnH710MOfBe+DreygQMMeLX0=', 'Content-Length': '0'} 2023-09-29T22:34:29.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfe538f0c4689ab0-00651750f5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.948 DEBUG:boto:path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.948 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.949 DEBUG:boto:Method: GET 2023-09-29T22:34:29.949 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.949 DEBUG:boto:Data: 2023-09-29T22:34:29.949 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:29.949 DEBUG:boto:Port: 8000 2023-09-29T22:34:29.949 DEBUG:boto:Params: {} 2023-09-29T22:34:29.949 DEBUG:boto:Token: None 2023-09-29T22:34:29.950 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/obj00000000000000000023 2023-09-29T22:34:29.950 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7xtgnH710MOfBe+DreygQMMeLX0= 2023-09-29T22:34:29.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7xtgnH710MOfBe+DreygQMMeLX0=', 'Content-Length': '0'} 2023-09-29T22:34:29.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006470374ef4bf191a-00651750f5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:29.953 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:29.953 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:29.953 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.953 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.954 DEBUG:boto:Method: GET 2023-09-29T22:34:29.954 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:29.954 DEBUG:boto:Data: 2023-09-29T22:34:29.954 DEBUG:boto:Headers: {} 2023-09-29T22:34:29.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:29.954 DEBUG:boto:Port: 8001 2023-09-29T22:34:29.954 DEBUG:boto:Params: {} 2023-09-29T22:34:29.954 DEBUG:boto:Token: None 2023-09-29T22:34:29.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:29 GMT /hcnphz-58/ 2023-09-29T22:34:29.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4= 2023-09-29T22:34:29.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wZGQyovuNkQ2AnvcKlESKcW+De4=', 'Content-Length': '0'} 2023-09-29T22:34:29.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000479f65f5c5416986-00651750f5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.002 DEBUG:boto:b'hcnphz-58obj000000000000000000231000falseobj000000000000000000232023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.003 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.003 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.003 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:30.004 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:30.004 DEBUG:boto:Method: GET 2023-09-29T22:34:30.004 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000023 2023-09-29T22:34:30.004 DEBUG:boto:Data: 2023-09-29T22:34:30.004 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.004 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.004 DEBUG:boto:Params: {} 2023-09-29T22:34:30.004 DEBUG:boto:Token: None 2023-09-29T22:34:30.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001923ca5894e42284-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.014 DEBUG:boto:b'hcnphz-58obj000000000000000000231000falseobj000000000000000000232023-09-29T22:34:13.864Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-09-29T22:34:30.015 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.015 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.015 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.015 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.015 DEBUG:boto:Method: GET 2023-09-29T22:34:30.015 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.015 DEBUG:boto:Data: 2023-09-29T22:34:30.016 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.016 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.016 DEBUG:boto:Params: {} 2023-09-29T22:34:30.016 DEBUG:boto:Token: None 2023-09-29T22:34:30.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0= 2023-09-29T22:34:30.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0=', 'Content-Length': '0'} 2023-09-29T22:34:30.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b5ea54f58305a3-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.019 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.019 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.019 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.019 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.019 DEBUG:boto:Method: GET 2023-09-29T22:34:30.019 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.020 DEBUG:boto:Data: 2023-09-29T22:34:30.020 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.020 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.020 DEBUG:boto:Params: {} 2023-09-29T22:34:30.020 DEBUG:boto:Token: None 2023-09-29T22:34:30.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0= 2023-09-29T22:34:30.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0=', 'Content-Length': '0'} 2023-09-29T22:34:30.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092185d5d32d3f547-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.066 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.066 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.067 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.067 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.067 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.067 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.067 DEBUG:boto:Data: 2023-09-29T22:34:30.067 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.067 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.067 DEBUG:boto:Params: {} 2023-09-29T22:34:30.067 DEBUG:boto:Token: None 2023-09-29T22:34:30.068 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gQBRo3ZVGWGF7zbk/encCo6h+U4= 2023-09-29T22:34:30.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gQBRo3ZVGWGF7zbk/encCo6h+U4=', 'Content-Length': '0'} 2023-09-29T22:34:30.070 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9520ef8ef6479d8-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.071 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.071 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.071 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.071 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.071 DEBUG:boto:Data: 2023-09-29T22:34:30.072 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.072 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.072 DEBUG:boto:Params: {} 2023-09-29T22:34:30.072 DEBUG:boto:Token: None 2023-09-29T22:34:30.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gQBRo3ZVGWGF7zbk/encCo6h+U4= 2023-09-29T22:34:30.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gQBRo3ZVGWGF7zbk/encCo6h+U4=', 'Content-Length': '0'} 2023-09-29T22:34:30.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084b8027c8678c12a-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-09-29T22:34:30.075 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.075 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.075 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.076 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.076 DEBUG:boto:Method: GET 2023-09-29T22:34:30.076 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.076 DEBUG:boto:Data: 2023-09-29T22:34:30.076 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.076 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.076 DEBUG:boto:Params: {} 2023-09-29T22:34:30.076 DEBUG:boto:Token: None 2023-09-29T22:34:30.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.077 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0= 2023-09-29T22:34:30.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0=', 'Content-Length': '0'} 2023-09-29T22:34:30.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b65fc05fb733fab-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.080 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.080 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.080 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.080 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.080 DEBUG:boto:Method: GET 2023-09-29T22:34:30.080 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.081 DEBUG:boto:Data: 2023-09-29T22:34:30.081 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.081 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.081 DEBUG:boto:Params: {} 2023-09-29T22:34:30.081 DEBUG:boto:Token: None 2023-09-29T22:34:30.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024?versionId=null 2023-09-29T22:34:30.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0= 2023-09-29T22:34:30.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7C7dSSCp9ZaFK5nWzzlnadD7Zl0=', 'Content-Length': '0'} 2023-09-29T22:34:30.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f11cef9480fa07e1-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.084 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.085 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.085 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.085 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.085 DEBUG:boto:Method: GET 2023-09-29T22:34:30.085 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.085 DEBUG:boto:Data: 2023-09-29T22:34:30.085 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.085 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.085 DEBUG:boto:Params: {} 2023-09-29T22:34:30.086 DEBUG:boto:Token: None 2023-09-29T22:34:30.086 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.086 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aaedce75196bed9-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.134 DEBUG:boto:b'hcnphz-58obj000000000000000000241000falseobj000000000000000000242023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.135 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.135 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.135 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.135 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.135 DEBUG:boto:Method: GET 2023-09-29T22:34:30.136 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.136 DEBUG:boto:Data: 2023-09-29T22:34:30.136 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.136 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.136 DEBUG:boto:Params: {} 2023-09-29T22:34:30.136 DEBUG:boto:Token: None 2023-09-29T22:34:30.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.145 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000580cc9bffef4063-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.145 DEBUG:boto:b'hcnphz-58obj000000000000000000241000falseobj000000000000000000242023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.145 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.145 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.146 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.146 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.146 DEBUG:boto:Data: 2023-09-29T22:34:30.146 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.146 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.146 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.146 DEBUG:boto:Params: {} 2023-09-29T22:34:30.146 DEBUG:boto:Token: None 2023-09-29T22:34:30.146 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.147 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0OJAJENKwfxSZaX1jldS3h417K8= 2023-09-29T22:34:30.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0OJAJENKwfxSZaX1jldS3h417K8=', 'Content-Length': '0'} 2023-09-29T22:34:30.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365ae283382f7f90-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.149 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.149 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.149 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.150 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.150 DEBUG:boto:Data: 2023-09-29T22:34:30.150 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.150 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.150 DEBUG:boto:Params: {} 2023-09-29T22:34:30.150 DEBUG:boto:Token: None 2023-09-29T22:34:30.150 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.151 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0OJAJENKwfxSZaX1jldS3h417K8= 2023-09-29T22:34:30.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0OJAJENKwfxSZaX1jldS3h417K8=', 'Content-Length': '0'} 2023-09-29T22:34:30.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1043b0f5575ca56-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000024 2023-09-29T22:34:30.153 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.153 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.153 DEBUG:boto:Method: GET 2023-09-29T22:34:30.154 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.154 DEBUG:boto:Data: 2023-09-29T22:34:30.154 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.154 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.154 DEBUG:boto:Params: {} 2023-09-29T22:34:30.154 DEBUG:boto:Token: None 2023-09-29T22:34:30.154 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.154 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HwR5K4tojbiBhl6hafhRlUOqLzM= 2023-09-29T22:34:30.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HwR5K4tojbiBhl6hafhRlUOqLzM=', 'Content-Length': '0'} 2023-09-29T22:34:30.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de8e263992aa2f8a-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.157 DEBUG:boto:path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.157 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.157 DEBUG:boto:Method: GET 2023-09-29T22:34:30.157 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.157 DEBUG:boto:Data: 2023-09-29T22:34:30.157 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.158 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.158 DEBUG:boto:Params: {} 2023-09-29T22:34:30.158 DEBUG:boto:Token: None 2023-09-29T22:34:30.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000024 2023-09-29T22:34:30.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HwR5K4tojbiBhl6hafhRlUOqLzM= 2023-09-29T22:34:30.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HwR5K4tojbiBhl6hafhRlUOqLzM=', 'Content-Length': '0'} 2023-09-29T22:34:30.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a09219252d1be596-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.202 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.203 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.203 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.203 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.203 DEBUG:boto:Method: GET 2023-09-29T22:34:30.203 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.203 DEBUG:boto:Data: 2023-09-29T22:34:30.203 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.203 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.203 DEBUG:boto:Params: {} 2023-09-29T22:34:30.204 DEBUG:boto:Token: None 2023-09-29T22:34:30.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f9bc1feb74ba0b9-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.209 DEBUG:boto:b'hcnphz-58obj000000000000000000241000falseobj000000000000000000242023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.209 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.210 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.210 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.210 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.210 DEBUG:boto:Method: GET 2023-09-29T22:34:30.210 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000024 2023-09-29T22:34:30.210 DEBUG:boto:Data: 2023-09-29T22:34:30.210 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.210 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.211 DEBUG:boto:Params: {} 2023-09-29T22:34:30.211 DEBUG:boto:Token: None 2023-09-29T22:34:30.211 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.219 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ebc760d9ae350f5a-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.219 DEBUG:boto:b'hcnphz-58obj000000000000000000241000falseobj000000000000000000242023-09-29T22:34:15.957Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-09-29T22:34:30.220 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.220 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.220 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.220 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.220 DEBUG:boto:Method: GET 2023-09-29T22:34:30.220 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.221 DEBUG:boto:Data: 2023-09-29T22:34:30.221 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.221 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.221 DEBUG:boto:Params: {} 2023-09-29T22:34:30.221 DEBUG:boto:Token: None 2023-09-29T22:34:30.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM= 2023-09-29T22:34:30.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM=', 'Content-Length': '0'} 2023-09-29T22:34:30.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d9e949fb5e63ce0-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.224 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.224 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.224 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.225 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.225 DEBUG:boto:Method: GET 2023-09-29T22:34:30.225 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.225 DEBUG:boto:Data: 2023-09-29T22:34:30.225 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.225 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.225 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.225 DEBUG:boto:Params: {} 2023-09-29T22:34:30.225 DEBUG:boto:Token: None 2023-09-29T22:34:30.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.226 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM= 2023-09-29T22:34:30.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM=', 'Content-Length': '0'} 2023-09-29T22:34:30.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d2a5b349aeadbe3-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.228 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.228 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.229 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.229 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.229 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.229 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.229 DEBUG:boto:Data: 2023-09-29T22:34:30.229 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.229 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.229 DEBUG:boto:Params: {} 2023-09-29T22:34:30.229 DEBUG:boto:Token: None 2023-09-29T22:34:30.230 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.230 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q9eIXKYCQuuQAzt0ClOMg6AOSx4= 2023-09-29T22:34:30.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q9eIXKYCQuuQAzt0ClOMg6AOSx4=', 'Content-Length': '0'} 2023-09-29T22:34:30.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c58aec67b124a788-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.232 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.232 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.232 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.232 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.232 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.233 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.233 DEBUG:boto:Data: 2023-09-29T22:34:30.233 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.233 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.233 DEBUG:boto:Params: {} 2023-09-29T22:34:30.233 DEBUG:boto:Token: None 2023-09-29T22:34:30.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.234 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q9eIXKYCQuuQAzt0ClOMg6AOSx4= 2023-09-29T22:34:30.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q9eIXKYCQuuQAzt0ClOMg6AOSx4=', 'Content-Length': '0'} 2023-09-29T22:34:30.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a99f163ff061e9ad-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-09-29T22:34:30.236 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.236 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.236 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.236 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.236 DEBUG:boto:Method: GET 2023-09-29T22:34:30.236 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.237 DEBUG:boto:Data: 2023-09-29T22:34:30.237 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.237 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.237 DEBUG:boto:Params: {} 2023-09-29T22:34:30.237 DEBUG:boto:Token: None 2023-09-29T22:34:30.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM= 2023-09-29T22:34:30.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM=', 'Content-Length': '0'} 2023-09-29T22:34:30.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f504d4a8cc3bc150-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.282 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.283 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.283 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.283 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.283 DEBUG:boto:Method: GET 2023-09-29T22:34:30.283 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.283 DEBUG:boto:Data: 2023-09-29T22:34:30.283 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.283 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.284 DEBUG:boto:Params: {} 2023-09-29T22:34:30.284 DEBUG:boto:Token: None 2023-09-29T22:34:30.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025?versionId=null 2023-09-29T22:34:30.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM= 2023-09-29T22:34:30.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FH9GcJEz7mkYkuMyH/TYDh8q2KM=', 'Content-Length': '0'} 2023-09-29T22:34:30.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000802489e43d21d6e3-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.287 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.288 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.288 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.288 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.288 DEBUG:boto:Method: GET 2023-09-29T22:34:30.288 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.288 DEBUG:boto:Data: 2023-09-29T22:34:30.288 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.288 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.288 DEBUG:boto:Params: {} 2023-09-29T22:34:30.289 DEBUG:boto:Token: None 2023-09-29T22:34:30.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9ff229a5278e2ba-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.294 DEBUG:boto:b'hcnphz-58obj000000000000000000251000falseobj000000000000000000252023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.294 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.294 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.295 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.295 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.295 DEBUG:boto:Method: GET 2023-09-29T22:34:30.295 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.295 DEBUG:boto:Data: 2023-09-29T22:34:30.295 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.295 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.296 DEBUG:boto:Params: {} 2023-09-29T22:34:30.296 DEBUG:boto:Token: None 2023-09-29T22:34:30.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.305 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db8a5545bfa2e9ea-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.346 DEBUG:boto:b'hcnphz-58obj000000000000000000251000falseobj000000000000000000252023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.347 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.347 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.347 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.347 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.348 DEBUG:boto:Data: 2023-09-29T22:34:30.348 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.348 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.348 DEBUG:boto:Params: {} 2023-09-29T22:34:30.348 DEBUG:boto:Token: None 2023-09-29T22:34:30.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9X2qj0+izTPWq4x91YxpUE/faiQ= 2023-09-29T22:34:30.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9X2qj0+izTPWq4x91YxpUE/faiQ=', 'Content-Length': '0'} 2023-09-29T22:34:30.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ca4d44e804adf67-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.351 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.352 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.352 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.352 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.352 DEBUG:boto:Data: 2023-09-29T22:34:30.352 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.352 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.352 DEBUG:boto:Params: {} 2023-09-29T22:34:30.352 DEBUG:boto:Token: None 2023-09-29T22:34:30.353 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.353 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9X2qj0+izTPWq4x91YxpUE/faiQ= 2023-09-29T22:34:30.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9X2qj0+izTPWq4x91YxpUE/faiQ=', 'Content-Length': '0'} 2023-09-29T22:34:30.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002de575fbafb47f92-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000025 2023-09-29T22:34:30.356 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.356 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.356 DEBUG:boto:Method: GET 2023-09-29T22:34:30.356 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.356 DEBUG:boto:Data: 2023-09-29T22:34:30.357 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.357 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.357 DEBUG:boto:Params: {} 2023-09-29T22:34:30.357 DEBUG:boto:Token: None 2023-09-29T22:34:30.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.357 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:isYXuc+R3KqgNL+xCj6YqP2O7yI= 2023-09-29T22:34:30.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:isYXuc+R3KqgNL+xCj6YqP2O7yI=', 'Content-Length': '0'} 2023-09-29T22:34:30.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b467bf693cfa870-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.361 DEBUG:boto:path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.361 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.361 DEBUG:boto:Method: GET 2023-09-29T22:34:30.361 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.361 DEBUG:boto:Data: 2023-09-29T22:34:30.361 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.361 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.362 DEBUG:boto:Params: {} 2023-09-29T22:34:30.362 DEBUG:boto:Token: None 2023-09-29T22:34:30.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000025 2023-09-29T22:34:30.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:isYXuc+R3KqgNL+xCj6YqP2O7yI= 2023-09-29T22:34:30.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:isYXuc+R3KqgNL+xCj6YqP2O7yI=', 'Content-Length': '0'} 2023-09-29T22:34:30.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000782d76ba00e90ed-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.365 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.365 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.366 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.366 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.366 DEBUG:boto:Method: GET 2023-09-29T22:34:30.366 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.366 DEBUG:boto:Data: 2023-09-29T22:34:30.366 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.366 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.366 DEBUG:boto:Params: {} 2023-09-29T22:34:30.367 DEBUG:boto:Token: None 2023-09-29T22:34:30.367 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.367 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f88866caa40b68bf-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.414 DEBUG:boto:b'hcnphz-58obj000000000000000000251000falseobj000000000000000000252023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.415 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.415 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.415 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.415 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.415 DEBUG:boto:Method: GET 2023-09-29T22:34:30.416 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000025 2023-09-29T22:34:30.416 DEBUG:boto:Data: 2023-09-29T22:34:30.416 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.416 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.416 DEBUG:boto:Params: {} 2023-09-29T22:34:30.416 DEBUG:boto:Token: None 2023-09-29T22:34:30.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc91d6b52309bf94-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.425 DEBUG:boto:b'hcnphz-58obj000000000000000000251000falseobj000000000000000000252023-09-29T22:34:13.396Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-09-29T22:34:30.426 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.426 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.426 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.426 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.426 DEBUG:boto:Method: GET 2023-09-29T22:34:30.426 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.427 DEBUG:boto:Data: 2023-09-29T22:34:30.427 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.427 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.427 DEBUG:boto:Params: {} 2023-09-29T22:34:30.427 DEBUG:boto:Token: None 2023-09-29T22:34:30.427 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w= 2023-09-29T22:34:30.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w=', 'Content-Length': '0'} 2023-09-29T22:34:30.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092d4fa41eda27f3b-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.430 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.430 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.430 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.431 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.431 DEBUG:boto:Method: GET 2023-09-29T22:34:30.431 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.431 DEBUG:boto:Data: 2023-09-29T22:34:30.431 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.431 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.431 DEBUG:boto:Params: {} 2023-09-29T22:34:30.431 DEBUG:boto:Token: None 2023-09-29T22:34:30.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w= 2023-09-29T22:34:30.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w=', 'Content-Length': '0'} 2023-09-29T22:34:30.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3a6369adbff8d35-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.478 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.479 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.479 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.479 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.479 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.479 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.479 DEBUG:boto:Data: 2023-09-29T22:34:30.479 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.479 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.480 DEBUG:boto:Params: {} 2023-09-29T22:34:30.480 DEBUG:boto:Token: None 2023-09-29T22:34:30.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZXZcYq60a3c1mu5QvOlY2GVYM2o= 2023-09-29T22:34:30.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZXZcYq60a3c1mu5QvOlY2GVYM2o=', 'Content-Length': '0'} 2023-09-29T22:34:30.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddef37b590abb0bf-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.483 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.483 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.483 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.483 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.483 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.484 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.484 DEBUG:boto:Data: 2023-09-29T22:34:30.484 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.484 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.484 DEBUG:boto:Params: {} 2023-09-29T22:34:30.484 DEBUG:boto:Token: None 2023-09-29T22:34:30.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZXZcYq60a3c1mu5QvOlY2GVYM2o= 2023-09-29T22:34:30.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZXZcYq60a3c1mu5QvOlY2GVYM2o=', 'Content-Length': '0'} 2023-09-29T22:34:30.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031cfc5fb385f61f1-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-09-29T22:34:30.488 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.488 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.488 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.488 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.488 DEBUG:boto:Method: GET 2023-09-29T22:34:30.488 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.489 DEBUG:boto:Data: 2023-09-29T22:34:30.489 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.489 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.489 DEBUG:boto:Params: {} 2023-09-29T22:34:30.489 DEBUG:boto:Token: None 2023-09-29T22:34:30.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w= 2023-09-29T22:34:30.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w=', 'Content-Length': '0'} 2023-09-29T22:34:30.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d15dc33f00071474-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.492 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.492 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.493 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.493 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.493 DEBUG:boto:Method: GET 2023-09-29T22:34:30.493 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.493 DEBUG:boto:Data: 2023-09-29T22:34:30.493 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.493 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.493 DEBUG:boto:Params: {} 2023-09-29T22:34:30.494 DEBUG:boto:Token: None 2023-09-29T22:34:30.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026?versionId=null 2023-09-29T22:34:30.494 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w= 2023-09-29T22:34:30.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Flftd9KqfKeJsc5SieWv4XNV86w=', 'Content-Length': '0'} 2023-09-29T22:34:30.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009778d1dc1c52a481-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.497 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.497 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.497 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.497 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.497 DEBUG:boto:Method: GET 2023-09-29T22:34:30.497 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.498 DEBUG:boto:Data: 2023-09-29T22:34:30.498 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.498 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.498 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.498 DEBUG:boto:Params: {} 2023-09-29T22:34:30.498 DEBUG:boto:Token: None 2023-09-29T22:34:30.498 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.498 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.499 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5771105e223c50f-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.550 DEBUG:boto:b'hcnphz-58obj000000000000000000261000falseobj000000000000000000262023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.551 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.551 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.551 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.551 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.551 DEBUG:boto:Method: GET 2023-09-29T22:34:30.552 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.552 DEBUG:boto:Data: 2023-09-29T22:34:30.552 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.552 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.552 DEBUG:boto:Params: {} 2023-09-29T22:34:30.552 DEBUG:boto:Token: None 2023-09-29T22:34:30.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35f5c2605bac9c0-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.562 DEBUG:boto:b'hcnphz-58obj000000000000000000261000falseobj000000000000000000262023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.562 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.562 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.563 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.563 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.563 DEBUG:boto:Data: 2023-09-29T22:34:30.563 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.563 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.563 DEBUG:boto:Params: {} 2023-09-29T22:34:30.563 DEBUG:boto:Token: None 2023-09-29T22:34:30.564 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Gj5pO9G1w4WZmH5mqZ9N8dm4h4= 2023-09-29T22:34:30.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Gj5pO9G1w4WZmH5mqZ9N8dm4h4=', 'Content-Length': '0'} 2023-09-29T22:34:30.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a71445f62babd3ad-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.567 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.567 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.567 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.567 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.567 DEBUG:boto:Data: 2023-09-29T22:34:30.567 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.568 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.568 DEBUG:boto:Params: {} 2023-09-29T22:34:30.568 DEBUG:boto:Token: None 2023-09-29T22:34:30.568 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Gj5pO9G1w4WZmH5mqZ9N8dm4h4= 2023-09-29T22:34:30.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Gj5pO9G1w4WZmH5mqZ9N8dm4h4=', 'Content-Length': '0'} 2023-09-29T22:34:30.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004413acf4bf326899-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000026 2023-09-29T22:34:30.571 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.571 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.571 DEBUG:boto:Method: GET 2023-09-29T22:34:30.571 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.571 DEBUG:boto:Data: 2023-09-29T22:34:30.571 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.572 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.572 DEBUG:boto:Params: {} 2023-09-29T22:34:30.572 DEBUG:boto:Token: None 2023-09-29T22:34:30.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xsupLn8/GgS8fULueQP4jEyVKtY= 2023-09-29T22:34:30.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xsupLn8/GgS8fULueQP4jEyVKtY=', 'Content-Length': '0'} 2023-09-29T22:34:30.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ecf1a0b42327f9c-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.575 DEBUG:boto:path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.575 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.575 DEBUG:boto:Method: GET 2023-09-29T22:34:30.575 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.575 DEBUG:boto:Data: 2023-09-29T22:34:30.575 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.575 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.576 DEBUG:boto:Params: {} 2023-09-29T22:34:30.576 DEBUG:boto:Token: None 2023-09-29T22:34:30.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000026 2023-09-29T22:34:30.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xsupLn8/GgS8fULueQP4jEyVKtY= 2023-09-29T22:34:30.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xsupLn8/GgS8fULueQP4jEyVKtY=', 'Content-Length': '0'} 2023-09-29T22:34:30.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813df57a9088f505-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.626 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.627 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.627 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.627 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.627 DEBUG:boto:Method: GET 2023-09-29T22:34:30.627 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.627 DEBUG:boto:Data: 2023-09-29T22:34:30.627 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.628 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.628 DEBUG:boto:Params: {} 2023-09-29T22:34:30.628 DEBUG:boto:Token: None 2023-09-29T22:34:30.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd8f4108d9b3db5a-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.635 DEBUG:boto:b'hcnphz-58obj000000000000000000261000falseobj000000000000000000262023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.636 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.636 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.636 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.636 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.636 DEBUG:boto:Method: GET 2023-09-29T22:34:30.636 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000026 2023-09-29T22:34:30.637 DEBUG:boto:Data: 2023-09-29T22:34:30.637 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.637 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.637 DEBUG:boto:Params: {} 2023-09-29T22:34:30.637 DEBUG:boto:Token: None 2023-09-29T22:34:30.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9393b74b3fb8e89-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.643 DEBUG:boto:b'hcnphz-58obj000000000000000000261000falseobj000000000000000000262023-09-29T22:34:16.436Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-09-29T22:34:30.644 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.644 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.644 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.644 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.644 DEBUG:boto:Method: GET 2023-09-29T22:34:30.644 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.645 DEBUG:boto:Data: 2023-09-29T22:34:30.645 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.645 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.645 DEBUG:boto:Params: {} 2023-09-29T22:34:30.645 DEBUG:boto:Token: None 2023-09-29T22:34:30.645 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8= 2023-09-29T22:34:30.646 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8=', 'Content-Length': '0'} 2023-09-29T22:34:30.648 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdcdd9a033e1e05e-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.694 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.695 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.695 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.695 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.695 DEBUG:boto:Method: GET 2023-09-29T22:34:30.695 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.695 DEBUG:boto:Data: 2023-09-29T22:34:30.695 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.695 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.696 DEBUG:boto:Params: {} 2023-09-29T22:34:30.696 DEBUG:boto:Token: None 2023-09-29T22:34:30.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8= 2023-09-29T22:34:30.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8=', 'Content-Length': '0'} 2023-09-29T22:34:30.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000038e9d7601cd2b68-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.700 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.700 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.700 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.700 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.700 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.700 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.700 DEBUG:boto:Data: 2023-09-29T22:34:30.700 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.701 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.701 DEBUG:boto:Params: {} 2023-09-29T22:34:30.701 DEBUG:boto:Token: None 2023-09-29T22:34:30.701 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ghepCOCcpD38Zc7n2jQRa/cVqMw= 2023-09-29T22:34:30.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ghepCOCcpD38Zc7n2jQRa/cVqMw=', 'Content-Length': '0'} 2023-09-29T22:34:30.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064ec8b73b7be19a6-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.704 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.705 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.705 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.705 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.705 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.705 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.705 DEBUG:boto:Data: 2023-09-29T22:34:30.705 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.706 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.706 DEBUG:boto:Params: {} 2023-09-29T22:34:30.706 DEBUG:boto:Token: None 2023-09-29T22:34:30.706 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.706 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ghepCOCcpD38Zc7n2jQRa/cVqMw= 2023-09-29T22:34:30.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ghepCOCcpD38Zc7n2jQRa/cVqMw=', 'Content-Length': '0'} 2023-09-29T22:34:30.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a42d7b163b248bf3-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-09-29T22:34:30.709 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.710 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.710 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.710 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.710 DEBUG:boto:Method: GET 2023-09-29T22:34:30.710 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.710 DEBUG:boto:Data: 2023-09-29T22:34:30.710 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.711 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.711 DEBUG:boto:Params: {} 2023-09-29T22:34:30.711 DEBUG:boto:Token: None 2023-09-29T22:34:30.711 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8= 2023-09-29T22:34:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8=', 'Content-Length': '0'} 2023-09-29T22:34:30.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fab68528deea4c27-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.714 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.715 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.715 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.715 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.715 DEBUG:boto:Method: GET 2023-09-29T22:34:30.715 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.715 DEBUG:boto:Data: 2023-09-29T22:34:30.715 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.716 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.716 DEBUG:boto:Params: {} 2023-09-29T22:34:30.716 DEBUG:boto:Token: None 2023-09-29T22:34:30.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027?versionId=null 2023-09-29T22:34:30.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8= 2023-09-29T22:34:30.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3zLpi8Pccz+bD07V/yY6Fs3OIz8=', 'Content-Length': '0'} 2023-09-29T22:34:30.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d15c5234c2ce8b7-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.719 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.719 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.720 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.720 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.720 DEBUG:boto:Method: GET 2023-09-29T22:34:30.720 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.720 DEBUG:boto:Data: 2023-09-29T22:34:30.720 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.720 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.720 DEBUG:boto:Params: {} 2023-09-29T22:34:30.721 DEBUG:boto:Token: None 2023-09-29T22:34:30.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000729d07a27c4e1a47-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.766 DEBUG:boto:b'hcnphz-58obj000000000000000000271000falseobj000000000000000000272023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.767 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.767 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.767 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.767 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.767 DEBUG:boto:Method: GET 2023-09-29T22:34:30.767 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.767 DEBUG:boto:Data: 2023-09-29T22:34:30.768 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.768 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.768 DEBUG:boto:Params: {} 2023-09-29T22:34:30.768 DEBUG:boto:Token: None 2023-09-29T22:34:30.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051c784a23c8cd8f4-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.778 DEBUG:boto:b'hcnphz-58obj000000000000000000271000falseobj000000000000000000272023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.778 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.778 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.779 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.779 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.779 DEBUG:boto:Data: 2023-09-29T22:34:30.779 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.779 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.779 DEBUG:boto:Params: {} 2023-09-29T22:34:30.779 DEBUG:boto:Token: None 2023-09-29T22:34:30.779 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qjRKYzGzAXmgb8DF0y5Rt31HSg= 2023-09-29T22:34:30.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qjRKYzGzAXmgb8DF0y5Rt31HSg=', 'Content-Length': '0'} 2023-09-29T22:34:30.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005957c7a73e8bcf28-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.783 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.783 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.783 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.783 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.783 DEBUG:boto:Data: 2023-09-29T22:34:30.783 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.783 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.784 DEBUG:boto:Params: {} 2023-09-29T22:34:30.784 DEBUG:boto:Token: None 2023-09-29T22:34:30.784 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qjRKYzGzAXmgb8DF0y5Rt31HSg= 2023-09-29T22:34:30.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qjRKYzGzAXmgb8DF0y5Rt31HSg=', 'Content-Length': '0'} 2023-09-29T22:34:30.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004345cbfda990fb60-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000027 2023-09-29T22:34:30.786 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.787 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.787 DEBUG:boto:Method: GET 2023-09-29T22:34:30.787 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.787 DEBUG:boto:Data: 2023-09-29T22:34:30.787 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.787 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.787 DEBUG:boto:Params: {} 2023-09-29T22:34:30.787 DEBUG:boto:Token: None 2023-09-29T22:34:30.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AhnT7Jl3B796EUQ9JzlOsDDNBzE= 2023-09-29T22:34:30.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AhnT7Jl3B796EUQ9JzlOsDDNBzE=', 'Content-Length': '0'} 2023-09-29T22:34:30.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e07a88319746e27-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.790 DEBUG:boto:path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.790 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.790 DEBUG:boto:Method: GET 2023-09-29T22:34:30.790 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.791 DEBUG:boto:Data: 2023-09-29T22:34:30.791 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.791 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.791 DEBUG:boto:Params: {} 2023-09-29T22:34:30.791 DEBUG:boto:Token: None 2023-09-29T22:34:30.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000027 2023-09-29T22:34:30.791 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AhnT7Jl3B796EUQ9JzlOsDDNBzE= 2023-09-29T22:34:30.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AhnT7Jl3B796EUQ9JzlOsDDNBzE=', 'Content-Length': '0'} 2023-09-29T22:34:30.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000363a70b71492f4d1-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.834 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.835 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.835 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.835 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.835 DEBUG:boto:Method: GET 2023-09-29T22:34:30.835 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.835 DEBUG:boto:Data: 2023-09-29T22:34:30.835 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.835 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.835 DEBUG:boto:Params: {} 2023-09-29T22:34:30.836 DEBUG:boto:Token: None 2023-09-29T22:34:30.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da50f42c885ccac9-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.841 DEBUG:boto:b'hcnphz-58obj000000000000000000271000falseobj000000000000000000272023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.841 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.842 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.842 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.842 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.842 DEBUG:boto:Method: GET 2023-09-29T22:34:30.842 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000027 2023-09-29T22:34:30.842 DEBUG:boto:Data: 2023-09-29T22:34:30.842 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.842 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.843 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.843 DEBUG:boto:Params: {} 2023-09-29T22:34:30.843 DEBUG:boto:Token: None 2023-09-29T22:34:30.843 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.851 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a458525ab3b4b95-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.851 DEBUG:boto:b'hcnphz-58obj000000000000000000271000falseobj000000000000000000272023-09-29T22:34:15.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-09-29T22:34:30.852 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.852 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.852 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.852 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.852 DEBUG:boto:Method: GET 2023-09-29T22:34:30.852 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.852 DEBUG:boto:Data: 2023-09-29T22:34:30.852 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.853 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.853 DEBUG:boto:Params: {} 2023-09-29T22:34:30.853 DEBUG:boto:Token: None 2023-09-29T22:34:30.853 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.853 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU= 2023-09-29T22:34:30.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU=', 'Content-Length': '0'} 2023-09-29T22:34:30.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5015cf204f77ace-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.898 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.899 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.899 DEBUG:boto:Method: GET 2023-09-29T22:34:30.899 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.900 DEBUG:boto:Data: 2023-09-29T22:34:30.900 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.900 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.900 DEBUG:boto:Params: {} 2023-09-29T22:34:30.901 DEBUG:boto:Token: None 2023-09-29T22:34:30.901 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU= 2023-09-29T22:34:30.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU=', 'Content-Length': '0'} 2023-09-29T22:34:30.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b0fbfcb61b3f753-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.904 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.904 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.905 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.905 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.905 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.905 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.905 DEBUG:boto:Data: 2023-09-29T22:34:30.906 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.906 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.906 DEBUG:boto:Params: {} 2023-09-29T22:34:30.906 DEBUG:boto:Token: None 2023-09-29T22:34:30.907 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.907 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RZxNZXHl5ugFYa7mT3w2fOAHuFQ= 2023-09-29T22:34:30.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RZxNZXHl5ugFYa7mT3w2fOAHuFQ=', 'Content-Length': '0'} 2023-09-29T22:34:30.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b5e48d708896da5-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.910 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.910 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.911 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.911 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.911 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.911 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.911 DEBUG:boto:Data: 2023-09-29T22:34:30.912 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.912 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.912 DEBUG:boto:Params: {} 2023-09-29T22:34:30.912 DEBUG:boto:Token: None 2023-09-29T22:34:30.913 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RZxNZXHl5ugFYa7mT3w2fOAHuFQ= 2023-09-29T22:34:30.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RZxNZXHl5ugFYa7mT3w2fOAHuFQ=', 'Content-Length': '0'} 2023-09-29T22:34:30.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006779faf954b3f83-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-09-29T22:34:30.916 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.916 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.917 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.917 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.917 DEBUG:boto:Method: GET 2023-09-29T22:34:30.917 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.917 DEBUG:boto:Data: 2023-09-29T22:34:30.918 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.918 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.918 DEBUG:boto:Params: {} 2023-09-29T22:34:30.918 DEBUG:boto:Token: None 2023-09-29T22:34:30.919 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.919 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU= 2023-09-29T22:34:30.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU=', 'Content-Length': '0'} 2023-09-29T22:34:30.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cebf9e49d17d11b-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.922 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.922 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.922 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.922 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.922 DEBUG:boto:Method: GET 2023-09-29T22:34:30.922 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.923 DEBUG:boto:Data: 2023-09-29T22:34:30.923 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.923 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.923 DEBUG:boto:Params: {} 2023-09-29T22:34:30.923 DEBUG:boto:Token: None 2023-09-29T22:34:30.923 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028?versionId=null 2023-09-29T22:34:30.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU= 2023-09-29T22:34:30.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZURDp6+M5Nvi2/E9pex7dbp2nkU=', 'Content-Length': '0'} 2023-09-29T22:34:30.926 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a028f9db43d7521-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.966 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.967 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.967 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:30.967 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:30.967 DEBUG:boto:Method: GET 2023-09-29T22:34:30.968 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:30.968 DEBUG:boto:Data: 2023-09-29T22:34:30.968 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.968 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.968 DEBUG:boto:Params: {} 2023-09-29T22:34:30.969 DEBUG:boto:Token: None 2023-09-29T22:34:30.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f1a8c30136e986-00651750f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.974 DEBUG:boto:b'hcnphz-58obj000000000000000000281000falseobj000000000000000000282023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.975 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:30.975 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:30.975 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:30.975 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:30.975 DEBUG:boto:Method: GET 2023-09-29T22:34:30.975 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:30.975 DEBUG:boto:Data: 2023-09-29T22:34:30.976 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.976 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.976 DEBUG:boto:Params: {} 2023-09-29T22:34:30.976 DEBUG:boto:Token: None 2023-09-29T22:34:30.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/ 2023-09-29T22:34:30.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes= 2023-09-29T22:34:30.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbRP5PV8QIri0o8WAwlU5IEfdes=', 'Content-Length': '0'} 2023-09-29T22:34:30.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d7ff6e4e282821-00651750f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.985 DEBUG:boto:b'hcnphz-58obj000000000000000000281000falseobj000000000000000000282023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:30.985 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.985 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.985 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.986 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.986 DEBUG:boto:Data: 2023-09-29T22:34:30.986 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.986 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.986 DEBUG:boto:Params: {} 2023-09-29T22:34:30.986 DEBUG:boto:Token: None 2023-09-29T22:34:30.986 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.987 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GJ121dxLVl1Dh4i9o9mCASLIQYY= 2023-09-29T22:34:30.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GJ121dxLVl1Dh4i9o9mCASLIQYY=', 'Content-Length': '0'} 2023-09-29T22:34:30.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcb93625d442ee59-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.989 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.989 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.989 DEBUG:boto:Method: HEAD 2023-09-29T22:34:30.989 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.990 DEBUG:boto:Data: 2023-09-29T22:34:30.990 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.990 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.990 DEBUG:boto:Params: {} 2023-09-29T22:34:30.990 DEBUG:boto:Token: None 2023-09-29T22:34:30.990 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.990 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GJ121dxLVl1Dh4i9o9mCASLIQYY= 2023-09-29T22:34:30.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GJ121dxLVl1Dh4i9o9mCASLIQYY=', 'Content-Length': '0'} 2023-09-29T22:34:30.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000394824a3799affd0-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000028 2023-09-29T22:34:30.993 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.993 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.993 DEBUG:boto:Method: GET 2023-09-29T22:34:30.993 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.994 DEBUG:boto:Data: 2023-09-29T22:34:30.994 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:30.994 DEBUG:boto:Port: 8001 2023-09-29T22:34:30.994 DEBUG:boto:Params: {} 2023-09-29T22:34:30.994 DEBUG:boto:Token: None 2023-09-29T22:34:30.994 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.994 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a4KxlHupjfI6vyhbcX7AMHYasfI= 2023-09-29T22:34:30.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a4KxlHupjfI6vyhbcX7AMHYasfI=', 'Content-Length': '0'} 2023-09-29T22:34:30.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c932a955154ed06f-00651750f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:30.997 DEBUG:boto:path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.997 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.997 DEBUG:boto:Method: GET 2023-09-29T22:34:30.997 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.997 DEBUG:boto:Data: 2023-09-29T22:34:30.998 DEBUG:boto:Headers: {} 2023-09-29T22:34:30.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:30.998 DEBUG:boto:Port: 8000 2023-09-29T22:34:30.998 DEBUG:boto:Params: {} 2023-09-29T22:34:30.998 DEBUG:boto:Token: None 2023-09-29T22:34:30.998 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:30 GMT /hcnphz-58/obj00000000000000000028 2023-09-29T22:34:30.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a4KxlHupjfI6vyhbcX7AMHYasfI= 2023-09-29T22:34:30.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a4KxlHupjfI6vyhbcX7AMHYasfI=', 'Content-Length': '0'} 2023-09-29T22:34:31.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cac5ccd1d5000b2f-00651750f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.001 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.001 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.001 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:31.001 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:31.002 DEBUG:boto:Method: GET 2023-09-29T22:34:31.002 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:31.002 DEBUG:boto:Data: 2023-09-29T22:34:31.002 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.002 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.002 DEBUG:boto:Params: {} 2023-09-29T22:34:31.002 DEBUG:boto:Token: None 2023-09-29T22:34:31.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064478cb219af708e-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.006 DEBUG:boto:b'hcnphz-58obj000000000000000000281000falseobj000000000000000000282023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.006 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.007 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.007 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:31.007 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:31.007 DEBUG:boto:Method: GET 2023-09-29T22:34:31.007 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000028 2023-09-29T22:34:31.007 DEBUG:boto:Data: 2023-09-29T22:34:31.007 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.007 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.008 DEBUG:boto:Params: {} 2023-09-29T22:34:31.008 DEBUG:boto:Token: None 2023-09-29T22:34:31.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007911a80788182d18-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.015 DEBUG:boto:b'hcnphz-58obj000000000000000000281000falseobj000000000000000000282023-09-29T22:34:09.550Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-09-29T22:34:31.016 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.016 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.016 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.016 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.016 DEBUG:boto:Method: GET 2023-09-29T22:34:31.017 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.017 DEBUG:boto:Data: 2023-09-29T22:34:31.017 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.017 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.017 DEBUG:boto:Params: {} 2023-09-29T22:34:31.017 DEBUG:boto:Token: None 2023-09-29T22:34:31.017 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0= 2023-09-29T22:34:31.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0=', 'Content-Length': '0'} 2023-09-29T22:34:31.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5303de98fa42fa1-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.020 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.020 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.021 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.021 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.021 DEBUG:boto:Method: GET 2023-09-29T22:34:31.021 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.021 DEBUG:boto:Data: 2023-09-29T22:34:31.021 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.021 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.021 DEBUG:boto:Params: {} 2023-09-29T22:34:31.021 DEBUG:boto:Token: None 2023-09-29T22:34:31.022 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.022 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0= 2023-09-29T22:34:31.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0=', 'Content-Length': '0'} 2023-09-29T22:34:31.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000364cf6bc948282fb-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.066 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.067 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.067 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.067 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.067 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.067 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.067 DEBUG:boto:Data: 2023-09-29T22:34:31.067 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.068 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.068 DEBUG:boto:Params: {} 2023-09-29T22:34:31.068 DEBUG:boto:Token: None 2023-09-29T22:34:31.068 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o+SagwLnJMStbHFDoi3zJMvcGGk= 2023-09-29T22:34:31.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o+SagwLnJMStbHFDoi3zJMvcGGk=', 'Content-Length': '0'} 2023-09-29T22:34:31.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7af90a7efc8922a-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.071 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.071 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.071 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.072 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.072 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.072 DEBUG:boto:Data: 2023-09-29T22:34:31.072 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.072 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.072 DEBUG:boto:Params: {} 2023-09-29T22:34:31.072 DEBUG:boto:Token: None 2023-09-29T22:34:31.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o+SagwLnJMStbHFDoi3zJMvcGGk= 2023-09-29T22:34:31.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o+SagwLnJMStbHFDoi3zJMvcGGk=', 'Content-Length': '0'} 2023-09-29T22:34:31.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce3fc3a4bf8f1f18-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-09-29T22:34:31.076 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.076 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.076 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.076 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.077 DEBUG:boto:Method: GET 2023-09-29T22:34:31.077 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.077 DEBUG:boto:Data: 2023-09-29T22:34:31.077 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.077 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.077 DEBUG:boto:Params: {} 2023-09-29T22:34:31.077 DEBUG:boto:Token: None 2023-09-29T22:34:31.077 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.078 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0= 2023-09-29T22:34:31.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0=', 'Content-Length': '0'} 2023-09-29T22:34:31.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b67231772c6f21d-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.122 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.123 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.123 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.123 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.123 DEBUG:boto:Method: GET 2023-09-29T22:34:31.123 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.123 DEBUG:boto:Data: 2023-09-29T22:34:31.123 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.123 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.123 DEBUG:boto:Params: {} 2023-09-29T22:34:31.124 DEBUG:boto:Token: None 2023-09-29T22:34:31.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029?versionId=null 2023-09-29T22:34:31.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0= 2023-09-29T22:34:31.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fUxa4CZhk1dRgGM6+uD+SEMTLK0=', 'Content-Length': '0'} 2023-09-29T22:34:31.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f5a9b256dce591-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.127 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.128 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.128 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.128 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.128 DEBUG:boto:Method: GET 2023-09-29T22:34:31.128 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.128 DEBUG:boto:Data: 2023-09-29T22:34:31.128 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.128 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.129 DEBUG:boto:Params: {} 2023-09-29T22:34:31.129 DEBUG:boto:Token: None 2023-09-29T22:34:31.129 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.134 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000026349092712d6c55-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.134 DEBUG:boto:b'hcnphz-58obj000000000000000000291000falseobj000000000000000000292023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.135 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.135 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.135 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.135 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.135 DEBUG:boto:Method: GET 2023-09-29T22:34:31.135 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.135 DEBUG:boto:Data: 2023-09-29T22:34:31.135 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.135 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.136 DEBUG:boto:Params: {} 2023-09-29T22:34:31.136 DEBUG:boto:Token: None 2023-09-29T22:34:31.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.144 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008264b078376abbfa-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.144 DEBUG:boto:b'hcnphz-58obj000000000000000000291000falseobj000000000000000000292023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.145 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.145 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.145 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.145 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.145 DEBUG:boto:Data: 2023-09-29T22:34:31.145 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.145 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.145 DEBUG:boto:Params: {} 2023-09-29T22:34:31.146 DEBUG:boto:Token: None 2023-09-29T22:34:31.146 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MW1EQ3KLp/efCxgDeuaWwHqBmV0= 2023-09-29T22:34:31.146 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MW1EQ3KLp/efCxgDeuaWwHqBmV0=', 'Content-Length': '0'} 2023-09-29T22:34:31.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbe4abf6741c3a62-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.149 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.149 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.149 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.149 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.149 DEBUG:boto:Data: 2023-09-29T22:34:31.149 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.149 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.150 DEBUG:boto:Params: {} 2023-09-29T22:34:31.150 DEBUG:boto:Token: None 2023-09-29T22:34:31.150 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.150 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MW1EQ3KLp/efCxgDeuaWwHqBmV0= 2023-09-29T22:34:31.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MW1EQ3KLp/efCxgDeuaWwHqBmV0=', 'Content-Length': '0'} 2023-09-29T22:34:31.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f769d726adb4f631-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000029 2023-09-29T22:34:31.152 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.152 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.152 DEBUG:boto:Method: GET 2023-09-29T22:34:31.153 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.153 DEBUG:boto:Data: 2023-09-29T22:34:31.153 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.153 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.153 DEBUG:boto:Params: {} 2023-09-29T22:34:31.153 DEBUG:boto:Token: None 2023-09-29T22:34:31.153 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FfJZVIPxR0AXbjVABIEZFHuX6qg= 2023-09-29T22:34:31.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FfJZVIPxR0AXbjVABIEZFHuX6qg=', 'Content-Length': '0'} 2023-09-29T22:34:31.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065073b22675b2db2-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.156 DEBUG:boto:path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.156 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.156 DEBUG:boto:Method: GET 2023-09-29T22:34:31.156 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.156 DEBUG:boto:Data: 2023-09-29T22:34:31.156 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.156 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.157 DEBUG:boto:Params: {} 2023-09-29T22:34:31.157 DEBUG:boto:Token: None 2023-09-29T22:34:31.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000029 2023-09-29T22:34:31.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FfJZVIPxR0AXbjVABIEZFHuX6qg= 2023-09-29T22:34:31.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FfJZVIPxR0AXbjVABIEZFHuX6qg=', 'Content-Length': '0'} 2023-09-29T22:34:31.159 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016c64a4e0d57bd8e-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.206 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.206 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.207 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.207 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.207 DEBUG:boto:Method: GET 2023-09-29T22:34:31.207 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.207 DEBUG:boto:Data: 2023-09-29T22:34:31.207 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.207 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.207 DEBUG:boto:Params: {} 2023-09-29T22:34:31.208 DEBUG:boto:Token: None 2023-09-29T22:34:31.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c434dcd6c44cf33f-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.213 DEBUG:boto:b'hcnphz-58obj000000000000000000291000falseobj000000000000000000292023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.213 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.213 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.214 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.214 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.214 DEBUG:boto:Method: GET 2023-09-29T22:34:31.214 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000029 2023-09-29T22:34:31.214 DEBUG:boto:Data: 2023-09-29T22:34:31.214 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.214 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.214 DEBUG:boto:Params: {} 2023-09-29T22:34:31.215 DEBUG:boto:Token: None 2023-09-29T22:34:31.215 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.215 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e8a89ad1881b0c2-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.223 DEBUG:boto:b'hcnphz-58obj000000000000000000291000falseobj000000000000000000292023-09-29T22:34:14.353Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-09-29T22:34:31.223 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.224 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.224 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.224 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.224 DEBUG:boto:Method: GET 2023-09-29T22:34:31.224 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.224 DEBUG:boto:Data: 2023-09-29T22:34:31.224 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.224 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.224 DEBUG:boto:Params: {} 2023-09-29T22:34:31.225 DEBUG:boto:Token: None 2023-09-29T22:34:31.225 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798= 2023-09-29T22:34:31.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798=', 'Content-Length': '0'} 2023-09-29T22:34:31.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b1fb916b2e2b388-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.274 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.274 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.275 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.275 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.275 DEBUG:boto:Method: GET 2023-09-29T22:34:31.275 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.275 DEBUG:boto:Data: 2023-09-29T22:34:31.275 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.275 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.275 DEBUG:boto:Params: {} 2023-09-29T22:34:31.276 DEBUG:boto:Token: None 2023-09-29T22:34:31.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798= 2023-09-29T22:34:31.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798=', 'Content-Length': '0'} 2023-09-29T22:34:31.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b67c6cde6d1bff-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.279 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.279 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.280 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.280 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.280 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.280 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.280 DEBUG:boto:Data: 2023-09-29T22:34:31.280 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.280 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.280 DEBUG:boto:Params: {} 2023-09-29T22:34:31.281 DEBUG:boto:Token: None 2023-09-29T22:34:31.281 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c43T+X4kDwCQInzFTZ1pNaszu5c= 2023-09-29T22:34:31.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c43T+X4kDwCQInzFTZ1pNaszu5c=', 'Content-Length': '0'} 2023-09-29T22:34:31.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ff4336292eb4c0-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.284 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.284 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.284 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.284 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.284 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.284 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.285 DEBUG:boto:Data: 2023-09-29T22:34:31.285 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.285 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.285 DEBUG:boto:Params: {} 2023-09-29T22:34:31.285 DEBUG:boto:Token: None 2023-09-29T22:34:31.285 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.286 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c43T+X4kDwCQInzFTZ1pNaszu5c= 2023-09-29T22:34:31.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c43T+X4kDwCQInzFTZ1pNaszu5c=', 'Content-Length': '0'} 2023-09-29T22:34:31.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9157ddfffa2cdc9-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-09-29T22:34:31.289 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.289 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.289 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.289 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.289 DEBUG:boto:Method: GET 2023-09-29T22:34:31.290 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.290 DEBUG:boto:Data: 2023-09-29T22:34:31.290 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.290 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.290 DEBUG:boto:Params: {} 2023-09-29T22:34:31.290 DEBUG:boto:Token: None 2023-09-29T22:34:31.290 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798= 2023-09-29T22:34:31.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798=', 'Content-Length': '0'} 2023-09-29T22:34:31.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d0b9ec7d7d1f42-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.293 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.294 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.294 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.294 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.294 DEBUG:boto:Method: GET 2023-09-29T22:34:31.294 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.294 DEBUG:boto:Data: 2023-09-29T22:34:31.294 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.294 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.295 DEBUG:boto:Params: {} 2023-09-29T22:34:31.295 DEBUG:boto:Token: None 2023-09-29T22:34:31.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030?versionId=null 2023-09-29T22:34:31.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798= 2023-09-29T22:34:31.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f5PucvpvaYR8GMx7ALAsbqlV798=', 'Content-Length': '0'} 2023-09-29T22:34:31.298 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018b13b4a54929e03-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.299 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.299 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.299 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.299 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.299 DEBUG:boto:Method: GET 2023-09-29T22:34:31.300 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.300 DEBUG:boto:Data: 2023-09-29T22:34:31.300 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.300 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.300 DEBUG:boto:Params: {} 2023-09-29T22:34:31.300 DEBUG:boto:Token: None 2023-09-29T22:34:31.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.301 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e102c83e774954db-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.306 DEBUG:boto:b'hcnphz-58obj000000000000000000301000falseobj000000000000000000302023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.306 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.306 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.307 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.307 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.307 DEBUG:boto:Method: GET 2023-09-29T22:34:31.307 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.307 DEBUG:boto:Data: 2023-09-29T22:34:31.307 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.307 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.307 DEBUG:boto:Params: {} 2023-09-29T22:34:31.308 DEBUG:boto:Token: None 2023-09-29T22:34:31.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.308 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5ee5ac40f5c570d-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.358 DEBUG:boto:b'hcnphz-58obj000000000000000000301000falseobj000000000000000000302023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.362 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.362 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.362 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.362 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.362 DEBUG:boto:Data: 2023-09-29T22:34:31.363 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.363 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.363 DEBUG:boto:Params: {} 2023-09-29T22:34:31.363 DEBUG:boto:Token: None 2023-09-29T22:34:31.363 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.363 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YzH6biDX7gq9aKpPPOlqi5uZOos= 2023-09-29T22:34:31.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YzH6biDX7gq9aKpPPOlqi5uZOos=', 'Content-Length': '0'} 2023-09-29T22:34:31.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0cb7fdf5d4569b4-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.367 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.368 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.368 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.368 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.368 DEBUG:boto:Data: 2023-09-29T22:34:31.368 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.368 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.368 DEBUG:boto:Params: {} 2023-09-29T22:34:31.368 DEBUG:boto:Token: None 2023-09-29T22:34:31.369 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.369 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YzH6biDX7gq9aKpPPOlqi5uZOos= 2023-09-29T22:34:31.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YzH6biDX7gq9aKpPPOlqi5uZOos=', 'Content-Length': '0'} 2023-09-29T22:34:31.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001588c7d937ce691c-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000030 2023-09-29T22:34:31.371 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.372 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.372 DEBUG:boto:Method: GET 2023-09-29T22:34:31.372 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.372 DEBUG:boto:Data: 2023-09-29T22:34:31.372 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.372 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.372 DEBUG:boto:Params: {} 2023-09-29T22:34:31.372 DEBUG:boto:Token: None 2023-09-29T22:34:31.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GUSgUk+ryV1Anw8fhfzaEeB1248= 2023-09-29T22:34:31.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GUSgUk+ryV1Anw8fhfzaEeB1248=', 'Content-Length': '0'} 2023-09-29T22:34:31.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258c5dbdae8bbb16-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.418 DEBUG:boto:path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.419 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.419 DEBUG:boto:Method: GET 2023-09-29T22:34:31.419 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.419 DEBUG:boto:Data: 2023-09-29T22:34:31.419 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.419 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.419 DEBUG:boto:Params: {} 2023-09-29T22:34:31.419 DEBUG:boto:Token: None 2023-09-29T22:34:31.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000030 2023-09-29T22:34:31.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GUSgUk+ryV1Anw8fhfzaEeB1248= 2023-09-29T22:34:31.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GUSgUk+ryV1Anw8fhfzaEeB1248=', 'Content-Length': '0'} 2023-09-29T22:34:31.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f1637057a946baf-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.423 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.423 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.424 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.424 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.424 DEBUG:boto:Method: GET 2023-09-29T22:34:31.424 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.424 DEBUG:boto:Data: 2023-09-29T22:34:31.424 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.424 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.424 DEBUG:boto:Params: {} 2023-09-29T22:34:31.424 DEBUG:boto:Token: None 2023-09-29T22:34:31.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.430 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000195c970a360e6433-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.430 DEBUG:boto:b'hcnphz-58obj000000000000000000301000falseobj000000000000000000302023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.430 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.430 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.431 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.431 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.431 DEBUG:boto:Method: GET 2023-09-29T22:34:31.431 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000030 2023-09-29T22:34:31.431 DEBUG:boto:Data: 2023-09-29T22:34:31.431 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.431 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.431 DEBUG:boto:Params: {} 2023-09-29T22:34:31.432 DEBUG:boto:Token: None 2023-09-29T22:34:31.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.439 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069d6beb2fbaa6d8c-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.440 DEBUG:boto:b'hcnphz-58obj000000000000000000301000falseobj000000000000000000302023-09-29T22:34:16.037Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-09-29T22:34:31.440 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.440 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.440 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.440 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.441 DEBUG:boto:Method: GET 2023-09-29T22:34:31.441 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.441 DEBUG:boto:Data: 2023-09-29T22:34:31.441 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.441 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.441 DEBUG:boto:Params: {} 2023-09-29T22:34:31.441 DEBUG:boto:Token: None 2023-09-29T22:34:31.441 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE= 2023-09-29T22:34:31.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE=', 'Content-Length': '0'} 2023-09-29T22:34:31.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005178b16acfae6418-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.444 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.444 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.444 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.444 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.444 DEBUG:boto:Method: GET 2023-09-29T22:34:31.444 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.445 DEBUG:boto:Data: 2023-09-29T22:34:31.445 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.445 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.445 DEBUG:boto:Params: {} 2023-09-29T22:34:31.445 DEBUG:boto:Token: None 2023-09-29T22:34:31.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE= 2023-09-29T22:34:31.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE=', 'Content-Length': '0'} 2023-09-29T22:34:31.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebae95a97710c361-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.490 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.490 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.491 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.491 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.491 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.491 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.491 DEBUG:boto:Data: 2023-09-29T22:34:31.491 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.491 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.491 DEBUG:boto:Params: {} 2023-09-29T22:34:31.492 DEBUG:boto:Token: None 2023-09-29T22:34:31.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nrdvCKYUzqACl4Emyo5FT5b6iGo= 2023-09-29T22:34:31.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nrdvCKYUzqACl4Emyo5FT5b6iGo=', 'Content-Length': '0'} 2023-09-29T22:34:31.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bdb7f4bff976040-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.495 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.495 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.495 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.495 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.495 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.495 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.496 DEBUG:boto:Data: 2023-09-29T22:34:31.496 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.496 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.496 DEBUG:boto:Params: {} 2023-09-29T22:34:31.496 DEBUG:boto:Token: None 2023-09-29T22:34:31.496 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nrdvCKYUzqACl4Emyo5FT5b6iGo= 2023-09-29T22:34:31.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nrdvCKYUzqACl4Emyo5FT5b6iGo=', 'Content-Length': '0'} 2023-09-29T22:34:31.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c7b17fab8dccb9b-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-09-29T22:34:31.499 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.500 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.500 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.500 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.500 DEBUG:boto:Method: GET 2023-09-29T22:34:31.500 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.500 DEBUG:boto:Data: 2023-09-29T22:34:31.500 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.500 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.501 DEBUG:boto:Params: {} 2023-09-29T22:34:31.501 DEBUG:boto:Token: None 2023-09-29T22:34:31.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE= 2023-09-29T22:34:31.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE=', 'Content-Length': '0'} 2023-09-29T22:34:31.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e904b18342351a04-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.550 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.550 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.551 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.551 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.551 DEBUG:boto:Method: GET 2023-09-29T22:34:31.551 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.551 DEBUG:boto:Data: 2023-09-29T22:34:31.551 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.551 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.551 DEBUG:boto:Params: {} 2023-09-29T22:34:31.552 DEBUG:boto:Token: None 2023-09-29T22:34:31.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031?versionId=null 2023-09-29T22:34:31.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE= 2023-09-29T22:34:31.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o71oXBMuwNHRRotu66soeo3+AJE=', 'Content-Length': '0'} 2023-09-29T22:34:31.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca683d51d9860fcf-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.555 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.555 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.555 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.555 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.555 DEBUG:boto:Method: GET 2023-09-29T22:34:31.555 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.556 DEBUG:boto:Data: 2023-09-29T22:34:31.556 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.556 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.556 DEBUG:boto:Params: {} 2023-09-29T22:34:31.556 DEBUG:boto:Token: None 2023-09-29T22:34:31.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.561 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6276ac7330b6a24-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.561 DEBUG:boto:b'hcnphz-58obj000000000000000000311000falseobj000000000000000000312023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.562 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.562 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.562 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.562 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.562 DEBUG:boto:Method: GET 2023-09-29T22:34:31.562 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.562 DEBUG:boto:Data: 2023-09-29T22:34:31.563 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.563 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.563 DEBUG:boto:Params: {} 2023-09-29T22:34:31.563 DEBUG:boto:Token: None 2023-09-29T22:34:31.563 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.563 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001896473c01e4a636-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.571 DEBUG:boto:b'hcnphz-58obj000000000000000000311000falseobj000000000000000000312023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.571 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.571 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.572 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.572 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.572 DEBUG:boto:Data: 2023-09-29T22:34:31.572 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.572 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.572 DEBUG:boto:Params: {} 2023-09-29T22:34:31.572 DEBUG:boto:Token: None 2023-09-29T22:34:31.572 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K2NC7gX50IPHLvsjy+byu/0v1JI= 2023-09-29T22:34:31.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K2NC7gX50IPHLvsjy+byu/0v1JI=', 'Content-Length': '0'} 2023-09-29T22:34:31.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da0cae4884dcea9b-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.575 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.575 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.575 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.575 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.575 DEBUG:boto:Data: 2023-09-29T22:34:31.576 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.576 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.576 DEBUG:boto:Params: {} 2023-09-29T22:34:31.576 DEBUG:boto:Token: None 2023-09-29T22:34:31.576 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K2NC7gX50IPHLvsjy+byu/0v1JI= 2023-09-29T22:34:31.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K2NC7gX50IPHLvsjy+byu/0v1JI=', 'Content-Length': '0'} 2023-09-29T22:34:31.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005778f160ac7bc95c-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000031 2023-09-29T22:34:31.579 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.579 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.579 DEBUG:boto:Method: GET 2023-09-29T22:34:31.579 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.579 DEBUG:boto:Data: 2023-09-29T22:34:31.579 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.579 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.579 DEBUG:boto:Params: {} 2023-09-29T22:34:31.580 DEBUG:boto:Token: None 2023-09-29T22:34:31.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l4gArWSKVfOuO+tVIibLQIEX8/0= 2023-09-29T22:34:31.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l4gArWSKVfOuO+tVIibLQIEX8/0=', 'Content-Length': '0'} 2023-09-29T22:34:31.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a425024e62f6c76-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.626 DEBUG:boto:path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.626 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.627 DEBUG:boto:Method: GET 2023-09-29T22:34:31.627 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.627 DEBUG:boto:Data: 2023-09-29T22:34:31.627 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.627 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.627 DEBUG:boto:Params: {} 2023-09-29T22:34:31.627 DEBUG:boto:Token: None 2023-09-29T22:34:31.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000031 2023-09-29T22:34:31.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l4gArWSKVfOuO+tVIibLQIEX8/0= 2023-09-29T22:34:31.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l4gArWSKVfOuO+tVIibLQIEX8/0=', 'Content-Length': '0'} 2023-09-29T22:34:31.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db4ca5539cdb457-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.631 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.631 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.631 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.631 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.631 DEBUG:boto:Method: GET 2023-09-29T22:34:31.631 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.631 DEBUG:boto:Data: 2023-09-29T22:34:31.631 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.632 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.632 DEBUG:boto:Params: {} 2023-09-29T22:34:31.632 DEBUG:boto:Token: None 2023-09-29T22:34:31.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.636 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048c35dfa8191ab24-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.637 DEBUG:boto:b'hcnphz-58obj000000000000000000311000falseobj000000000000000000312023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.637 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.637 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.637 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.637 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.637 DEBUG:boto:Method: GET 2023-09-29T22:34:31.638 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000031 2023-09-29T22:34:31.638 DEBUG:boto:Data: 2023-09-29T22:34:31.638 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.638 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.638 DEBUG:boto:Params: {} 2023-09-29T22:34:31.638 DEBUG:boto:Token: None 2023-09-29T22:34:31.638 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.647 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088ea10ca5a42905e-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.647 DEBUG:boto:b'hcnphz-58obj000000000000000000311000falseobj000000000000000000312023-09-29T22:34:12.919Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-09-29T22:34:31.647 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.648 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.648 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.648 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.648 DEBUG:boto:Method: GET 2023-09-29T22:34:31.648 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.648 DEBUG:boto:Data: 2023-09-29T22:34:31.648 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.648 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.648 DEBUG:boto:Params: {} 2023-09-29T22:34:31.649 DEBUG:boto:Token: None 2023-09-29T22:34:31.649 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8= 2023-09-29T22:34:31.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8=', 'Content-Length': '0'} 2023-09-29T22:34:31.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000080e282f12a7f47f-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.652 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.652 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.652 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.652 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.652 DEBUG:boto:Method: GET 2023-09-29T22:34:31.652 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.652 DEBUG:boto:Data: 2023-09-29T22:34:31.652 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.653 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.653 DEBUG:boto:Params: {} 2023-09-29T22:34:31.653 DEBUG:boto:Token: None 2023-09-29T22:34:31.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8= 2023-09-29T22:34:31.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8=', 'Content-Length': '0'} 2023-09-29T22:34:31.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091309e03d8ec19a0-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.655 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.655 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.655 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.656 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.656 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.656 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.656 DEBUG:boto:Data: 2023-09-29T22:34:31.656 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.656 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.656 DEBUG:boto:Params: {} 2023-09-29T22:34:31.656 DEBUG:boto:Token: None 2023-09-29T22:34:31.657 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2cS88DgPmeW0X+dMlbwpk6Zyuw= 2023-09-29T22:34:31.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2cS88DgPmeW0X+dMlbwpk6Zyuw=', 'Content-Length': '0'} 2023-09-29T22:34:31.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000515b57e44262af99-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.659 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.659 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.659 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.659 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.659 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.659 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.660 DEBUG:boto:Data: 2023-09-29T22:34:31.660 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.660 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.660 DEBUG:boto:Params: {} 2023-09-29T22:34:31.660 DEBUG:boto:Token: None 2023-09-29T22:34:31.660 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2cS88DgPmeW0X+dMlbwpk6Zyuw= 2023-09-29T22:34:31.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2cS88DgPmeW0X+dMlbwpk6Zyuw=', 'Content-Length': '0'} 2023-09-29T22:34:31.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085d2991ef33d74c5-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-09-29T22:34:31.663 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.663 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.663 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.663 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.663 DEBUG:boto:Method: GET 2023-09-29T22:34:31.664 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.664 DEBUG:boto:Data: 2023-09-29T22:34:31.664 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.664 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.664 DEBUG:boto:Params: {} 2023-09-29T22:34:31.664 DEBUG:boto:Token: None 2023-09-29T22:34:31.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8= 2023-09-29T22:34:31.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8=', 'Content-Length': '0'} 2023-09-29T22:34:31.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c110e44b73b7b53-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.714 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.714 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.715 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.715 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.715 DEBUG:boto:Method: GET 2023-09-29T22:34:31.715 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.715 DEBUG:boto:Data: 2023-09-29T22:34:31.715 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.715 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.715 DEBUG:boto:Params: {} 2023-09-29T22:34:31.715 DEBUG:boto:Token: None 2023-09-29T22:34:31.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032?versionId=null 2023-09-29T22:34:31.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8= 2023-09-29T22:34:31.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qn99EneTQT5rO8E78FpAfw6GDP8=', 'Content-Length': '0'} 2023-09-29T22:34:31.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0bd754de9ce498f-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.719 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.719 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.719 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.719 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.719 DEBUG:boto:Method: GET 2023-09-29T22:34:31.719 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.720 DEBUG:boto:Data: 2023-09-29T22:34:31.720 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.720 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.720 DEBUG:boto:Params: {} 2023-09-29T22:34:31.720 DEBUG:boto:Token: None 2023-09-29T22:34:31.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000637143a7b45df329-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.725 DEBUG:boto:b'hcnphz-58obj000000000000000000321000falseobj000000000000000000322023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.725 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.726 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.726 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.726 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.726 DEBUG:boto:Method: GET 2023-09-29T22:34:31.726 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.726 DEBUG:boto:Data: 2023-09-29T22:34:31.726 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.726 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.727 DEBUG:boto:Params: {} 2023-09-29T22:34:31.727 DEBUG:boto:Token: None 2023-09-29T22:34:31.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.735 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2f4b8c4354d5f2d-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.735 DEBUG:boto:b'hcnphz-58obj000000000000000000321000falseobj000000000000000000322023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.735 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.735 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.736 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.736 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.736 DEBUG:boto:Data: 2023-09-29T22:34:31.736 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.736 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.736 DEBUG:boto:Params: {} 2023-09-29T22:34:31.736 DEBUG:boto:Token: None 2023-09-29T22:34:31.736 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LWMDXH4dOCOMQdbNyrrsq/43pRc= 2023-09-29T22:34:31.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LWMDXH4dOCOMQdbNyrrsq/43pRc=', 'Content-Length': '0'} 2023-09-29T22:34:31.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1fc8d9b0f8adff-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.739 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.739 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.739 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.739 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.739 DEBUG:boto:Data: 2023-09-29T22:34:31.739 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.739 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.739 DEBUG:boto:Params: {} 2023-09-29T22:34:31.740 DEBUG:boto:Token: None 2023-09-29T22:34:31.740 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LWMDXH4dOCOMQdbNyrrsq/43pRc= 2023-09-29T22:34:31.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LWMDXH4dOCOMQdbNyrrsq/43pRc=', 'Content-Length': '0'} 2023-09-29T22:34:31.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000754da65b249467de-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000032 2023-09-29T22:34:31.742 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.742 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.743 DEBUG:boto:Method: GET 2023-09-29T22:34:31.743 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.743 DEBUG:boto:Data: 2023-09-29T22:34:31.743 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.743 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.743 DEBUG:boto:Params: {} 2023-09-29T22:34:31.743 DEBUG:boto:Token: None 2023-09-29T22:34:31.743 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.743 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FiyHOOlLz3+AIWYcGUl6rkzEQ8Q= 2023-09-29T22:34:31.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FiyHOOlLz3+AIWYcGUl6rkzEQ8Q=', 'Content-Length': '0'} 2023-09-29T22:34:31.745 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081a0332ff2826abb-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.786 DEBUG:boto:path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.787 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.787 DEBUG:boto:Method: GET 2023-09-29T22:34:31.787 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.787 DEBUG:boto:Data: 2023-09-29T22:34:31.787 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.787 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.787 DEBUG:boto:Params: {} 2023-09-29T22:34:31.788 DEBUG:boto:Token: None 2023-09-29T22:34:31.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000032 2023-09-29T22:34:31.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FiyHOOlLz3+AIWYcGUl6rkzEQ8Q= 2023-09-29T22:34:31.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FiyHOOlLz3+AIWYcGUl6rkzEQ8Q=', 'Content-Length': '0'} 2023-09-29T22:34:31.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bc940420410c46a-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.791 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.791 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.792 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.792 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.792 DEBUG:boto:Method: GET 2023-09-29T22:34:31.792 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.792 DEBUG:boto:Data: 2023-09-29T22:34:31.792 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.792 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.793 DEBUG:boto:Params: {} 2023-09-29T22:34:31.793 DEBUG:boto:Token: None 2023-09-29T22:34:31.793 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000001e09c16facd2739-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.798 DEBUG:boto:b'hcnphz-58obj000000000000000000321000falseobj000000000000000000322023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.798 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.799 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.799 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.799 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.799 DEBUG:boto:Method: GET 2023-09-29T22:34:31.799 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000032 2023-09-29T22:34:31.799 DEBUG:boto:Data: 2023-09-29T22:34:31.799 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.799 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.799 DEBUG:boto:Params: {} 2023-09-29T22:34:31.800 DEBUG:boto:Token: None 2023-09-29T22:34:31.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d762c94f4f954ce6-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.854 DEBUG:boto:b'hcnphz-58obj000000000000000000321000falseobj000000000000000000322023-09-29T22:34:11.833Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-09-29T22:34:31.855 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.855 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.855 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.855 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.855 DEBUG:boto:Method: GET 2023-09-29T22:34:31.856 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.856 DEBUG:boto:Data: 2023-09-29T22:34:31.856 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.856 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.856 DEBUG:boto:Params: {} 2023-09-29T22:34:31.856 DEBUG:boto:Token: None 2023-09-29T22:34:31.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw= 2023-09-29T22:34:31.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw=', 'Content-Length': '0'} 2023-09-29T22:34:31.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005bfadccbf582a28-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.860 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.860 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.860 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.860 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.860 DEBUG:boto:Method: GET 2023-09-29T22:34:31.861 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.861 DEBUG:boto:Data: 2023-09-29T22:34:31.861 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.861 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.861 DEBUG:boto:Params: {} 2023-09-29T22:34:31.861 DEBUG:boto:Token: None 2023-09-29T22:34:31.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.862 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw= 2023-09-29T22:34:31.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw=', 'Content-Length': '0'} 2023-09-29T22:34:31.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ea08b1781cb34e-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.865 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.865 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.865 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.865 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.865 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.866 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.866 DEBUG:boto:Data: 2023-09-29T22:34:31.866 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.866 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.866 DEBUG:boto:Params: {} 2023-09-29T22:34:31.866 DEBUG:boto:Token: None 2023-09-29T22:34:31.866 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.867 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lwupTi/mRuHY1r1hE6RBRfc4hw0= 2023-09-29T22:34:31.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lwupTi/mRuHY1r1hE6RBRfc4hw0=', 'Content-Length': '0'} 2023-09-29T22:34:31.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b41f89c034b04e2-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.869 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.869 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.870 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.870 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.870 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.870 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.870 DEBUG:boto:Data: 2023-09-29T22:34:31.871 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.871 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.871 DEBUG:boto:Params: {} 2023-09-29T22:34:31.871 DEBUG:boto:Token: None 2023-09-29T22:34:31.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lwupTi/mRuHY1r1hE6RBRfc4hw0= 2023-09-29T22:34:31.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lwupTi/mRuHY1r1hE6RBRfc4hw0=', 'Content-Length': '0'} 2023-09-29T22:34:31.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d56ce29e6998312-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-09-29T22:34:31.876 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.876 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.876 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.876 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.877 DEBUG:boto:Method: GET 2023-09-29T22:34:31.877 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.877 DEBUG:boto:Data: 2023-09-29T22:34:31.877 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.878 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.878 DEBUG:boto:Params: {} 2023-09-29T22:34:31.878 DEBUG:boto:Token: None 2023-09-29T22:34:31.878 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.879 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw= 2023-09-29T22:34:31.879 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw=', 'Content-Length': '0'} 2023-09-29T22:34:31.882 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc04b64d42a928f3-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.882 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.882 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.883 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.883 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.883 DEBUG:boto:Method: GET 2023-09-29T22:34:31.883 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.883 DEBUG:boto:Data: 2023-09-29T22:34:31.884 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.884 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.884 DEBUG:boto:Params: {} 2023-09-29T22:34:31.884 DEBUG:boto:Token: None 2023-09-29T22:34:31.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033?versionId=null 2023-09-29T22:34:31.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw= 2023-09-29T22:34:31.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mbSBKLqrDICXM7z1wjiKNCyduWw=', 'Content-Length': '0'} 2023-09-29T22:34:31.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1b2f72e923b961d-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.889 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.889 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.890 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:31.890 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:31.890 DEBUG:boto:Method: GET 2023-09-29T22:34:31.890 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:31.891 DEBUG:boto:Data: 2023-09-29T22:34:31.891 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.891 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.891 DEBUG:boto:Params: {} 2023-09-29T22:34:31.892 DEBUG:boto:Token: None 2023-09-29T22:34:31.892 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.892 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020473874467a51ef-00651750f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.898 DEBUG:boto:b'hcnphz-58obj000000000000000000331000falseobj000000000000000000332023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.898 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:31.899 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:31.899 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:31.899 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:31.899 DEBUG:boto:Method: GET 2023-09-29T22:34:31.899 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:31.899 DEBUG:boto:Data: 2023-09-29T22:34:31.899 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.899 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.899 DEBUG:boto:Params: {} 2023-09-29T22:34:31.900 DEBUG:boto:Token: None 2023-09-29T22:34:31.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/ 2023-09-29T22:34:31.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc= 2023-09-29T22:34:31.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J/3Zd4ofL1+x9j92eg1ceZj2rOc=', 'Content-Length': '0'} 2023-09-29T22:34:31.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fb65a034ac5f4253-00651750f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.950 DEBUG:boto:b'hcnphz-58obj000000000000000000331000falseobj000000000000000000332023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:31.951 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.951 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.951 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.951 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.951 DEBUG:boto:Data: 2023-09-29T22:34:31.951 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.952 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.952 DEBUG:boto:Params: {} 2023-09-29T22:34:31.952 DEBUG:boto:Token: None 2023-09-29T22:34:31.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5gCKBhp5uYiiJVCwYu+W/Buhgqc= 2023-09-29T22:34:31.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5gCKBhp5uYiiJVCwYu+W/Buhgqc=', 'Content-Length': '0'} 2023-09-29T22:34:31.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000546e4e645a0173c0-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.956 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.956 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.956 DEBUG:boto:Method: HEAD 2023-09-29T22:34:31.956 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.956 DEBUG:boto:Data: 2023-09-29T22:34:31.956 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:31.957 DEBUG:boto:Port: 8000 2023-09-29T22:34:31.957 DEBUG:boto:Params: {} 2023-09-29T22:34:31.957 DEBUG:boto:Token: None 2023-09-29T22:34:31.957 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.957 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5gCKBhp5uYiiJVCwYu+W/Buhgqc= 2023-09-29T22:34:31.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5gCKBhp5uYiiJVCwYu+W/Buhgqc=', 'Content-Length': '0'} 2023-09-29T22:34:31.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042e8db9effb32430-00651750f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:31.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000033 2023-09-29T22:34:31.960 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.960 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.960 DEBUG:boto:Method: GET 2023-09-29T22:34:31.961 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.961 DEBUG:boto:Data: 2023-09-29T22:34:31.961 DEBUG:boto:Headers: {} 2023-09-29T22:34:31.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:31.961 DEBUG:boto:Port: 8001 2023-09-29T22:34:31.961 DEBUG:boto:Params: {} 2023-09-29T22:34:31.961 DEBUG:boto:Token: None 2023-09-29T22:34:31.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:31 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:31.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6KSfpPdMRqA08bR2kshTbt+X7Lk= 2023-09-29T22:34:31.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6KSfpPdMRqA08bR2kshTbt+X7Lk=', 'Content-Length': '0'} 2023-09-29T22:34:31.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009678d70a54faee6c-00651750f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.006 DEBUG:boto:path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:32.006 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000033 2023-09-29T22:34:32.007 DEBUG:boto:Method: GET 2023-09-29T22:34:32.007 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:32.007 DEBUG:boto:Data: 2023-09-29T22:34:32.007 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.007 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.007 DEBUG:boto:Params: {} 2023-09-29T22:34:32.007 DEBUG:boto:Token: None 2023-09-29T22:34:32.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000033 2023-09-29T22:34:32.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8YDSpA5TEh0HuRxfeTJZ87A7sSw= 2023-09-29T22:34:32.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8YDSpA5TEh0HuRxfeTJZ87A7sSw=', 'Content-Length': '0'} 2023-09-29T22:34:32.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000850fed159cee597-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.011 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.011 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.011 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:32.011 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:32.011 DEBUG:boto:Method: GET 2023-09-29T22:34:32.011 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:32.011 DEBUG:boto:Data: 2023-09-29T22:34:32.011 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.011 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.012 DEBUG:boto:Params: {} 2023-09-29T22:34:32.012 DEBUG:boto:Token: None 2023-09-29T22:34:32.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002071c36d97e42aba-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.017 DEBUG:boto:b'hcnphz-58obj000000000000000000331000falseobj000000000000000000332023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.018 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.018 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.018 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:32.018 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:32.018 DEBUG:boto:Method: GET 2023-09-29T22:34:32.018 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000033 2023-09-29T22:34:32.018 DEBUG:boto:Data: 2023-09-29T22:34:32.018 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.019 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.019 DEBUG:boto:Params: {} 2023-09-29T22:34:32.019 DEBUG:boto:Token: None 2023-09-29T22:34:32.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db35d7feeb1b5e0d-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.028 DEBUG:boto:b'hcnphz-58obj000000000000000000331000falseobj000000000000000000332023-09-29T22:34:15.581Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-09-29T22:34:32.028 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.029 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.029 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.029 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.029 DEBUG:boto:Method: GET 2023-09-29T22:34:32.029 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.029 DEBUG:boto:Data: 2023-09-29T22:34:32.029 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.029 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.029 DEBUG:boto:Params: {} 2023-09-29T22:34:32.030 DEBUG:boto:Token: None 2023-09-29T22:34:32.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54= 2023-09-29T22:34:32.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54=', 'Content-Length': '0'} 2023-09-29T22:34:32.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b960c97c1e712b-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.032 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.032 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.033 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.033 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.033 DEBUG:boto:Method: GET 2023-09-29T22:34:32.033 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.033 DEBUG:boto:Data: 2023-09-29T22:34:32.033 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.033 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.033 DEBUG:boto:Params: {} 2023-09-29T22:34:32.033 DEBUG:boto:Token: None 2023-09-29T22:34:32.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54= 2023-09-29T22:34:32.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54=', 'Content-Length': '0'} 2023-09-29T22:34:32.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015dccb5bcde334ab-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.078 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.078 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.078 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.079 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.079 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.079 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.079 DEBUG:boto:Data: 2023-09-29T22:34:32.079 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.079 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.079 DEBUG:boto:Params: {} 2023-09-29T22:34:32.079 DEBUG:boto:Token: None 2023-09-29T22:34:32.080 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fsuaOyYkK6IivImFobuafvi1E50= 2023-09-29T22:34:32.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fsuaOyYkK6IivImFobuafvi1E50=', 'Content-Length': '0'} 2023-09-29T22:34:32.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f77894603cd8647-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.083 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.083 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.083 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.083 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.084 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.084 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.084 DEBUG:boto:Data: 2023-09-29T22:34:32.084 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.084 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.084 DEBUG:boto:Params: {} 2023-09-29T22:34:32.084 DEBUG:boto:Token: None 2023-09-29T22:34:32.084 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fsuaOyYkK6IivImFobuafvi1E50= 2023-09-29T22:34:32.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fsuaOyYkK6IivImFobuafvi1E50=', 'Content-Length': '0'} 2023-09-29T22:34:32.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000318a0761f3adfe70-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-09-29T22:34:32.087 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.088 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.088 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.088 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.088 DEBUG:boto:Method: GET 2023-09-29T22:34:32.088 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.088 DEBUG:boto:Data: 2023-09-29T22:34:32.088 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.088 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.088 DEBUG:boto:Params: {} 2023-09-29T22:34:32.088 DEBUG:boto:Token: None 2023-09-29T22:34:32.089 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54= 2023-09-29T22:34:32.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54=', 'Content-Length': '0'} 2023-09-29T22:34:32.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000991e7a505b68f55f-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.138 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.138 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.138 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.139 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.139 DEBUG:boto:Method: GET 2023-09-29T22:34:32.139 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.139 DEBUG:boto:Data: 2023-09-29T22:34:32.139 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.139 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.139 DEBUG:boto:Params: {} 2023-09-29T22:34:32.139 DEBUG:boto:Token: None 2023-09-29T22:34:32.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036?versionId=null 2023-09-29T22:34:32.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54= 2023-09-29T22:34:32.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XsubLT2OiaV/eHZ8U+IFMq6lS54=', 'Content-Length': '0'} 2023-09-29T22:34:32.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e17dd6d5935313a5-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.143 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.143 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.143 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.143 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.143 DEBUG:boto:Method: GET 2023-09-29T22:34:32.143 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.144 DEBUG:boto:Data: 2023-09-29T22:34:32.144 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.144 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.144 DEBUG:boto:Params: {} 2023-09-29T22:34:32.144 DEBUG:boto:Token: None 2023-09-29T22:34:32.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.149 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052bec12b07fe32e6-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.149 DEBUG:boto:b'hcnphz-58obj000000000000000000361000falseobj000000000000000000362023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.150 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.150 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.150 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.150 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.150 DEBUG:boto:Method: GET 2023-09-29T22:34:32.151 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.151 DEBUG:boto:Data: 2023-09-29T22:34:32.151 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.151 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.151 DEBUG:boto:Params: {} 2023-09-29T22:34:32.152 DEBUG:boto:Token: None 2023-09-29T22:34:32.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.160 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa012b88bda02a86-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.160 DEBUG:boto:b'hcnphz-58obj000000000000000000361000falseobj000000000000000000362023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.160 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.161 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.161 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.161 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.161 DEBUG:boto:Data: 2023-09-29T22:34:32.161 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.162 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.162 DEBUG:boto:Params: {} 2023-09-29T22:34:32.162 DEBUG:boto:Token: None 2023-09-29T22:34:32.162 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.162 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y/Wffo2kRsGG7mYdV5WvgUk+ppQ= 2023-09-29T22:34:32.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y/Wffo2kRsGG7mYdV5WvgUk+ppQ=', 'Content-Length': '0'} 2023-09-29T22:34:32.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075c90e746e858c3c-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.165 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.165 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.165 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.165 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.165 DEBUG:boto:Data: 2023-09-29T22:34:32.165 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.166 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.166 DEBUG:boto:Params: {} 2023-09-29T22:34:32.166 DEBUG:boto:Token: None 2023-09-29T22:34:32.166 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y/Wffo2kRsGG7mYdV5WvgUk+ppQ= 2023-09-29T22:34:32.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y/Wffo2kRsGG7mYdV5WvgUk+ppQ=', 'Content-Length': '0'} 2023-09-29T22:34:32.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008286f7604b9bf32d-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000036 2023-09-29T22:34:32.169 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.169 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.169 DEBUG:boto:Method: GET 2023-09-29T22:34:32.169 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.169 DEBUG:boto:Data: 2023-09-29T22:34:32.170 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.170 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.170 DEBUG:boto:Params: {} 2023-09-29T22:34:32.170 DEBUG:boto:Token: None 2023-09-29T22:34:32.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U27QWWDKLkRm+Bb9AF7lhdeLYHw= 2023-09-29T22:34:32.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U27QWWDKLkRm+Bb9AF7lhdeLYHw=', 'Content-Length': '0'} 2023-09-29T22:34:32.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000953bd69a421f7cc7-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.173 DEBUG:boto:path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.173 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.173 DEBUG:boto:Method: GET 2023-09-29T22:34:32.174 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.174 DEBUG:boto:Data: 2023-09-29T22:34:32.174 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.174 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.174 DEBUG:boto:Params: {} 2023-09-29T22:34:32.174 DEBUG:boto:Token: None 2023-09-29T22:34:32.174 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000036 2023-09-29T22:34:32.175 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U27QWWDKLkRm+Bb9AF7lhdeLYHw= 2023-09-29T22:34:32.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U27QWWDKLkRm+Bb9AF7lhdeLYHw=', 'Content-Length': '0'} 2023-09-29T22:34:32.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052e1d44584e84833-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.222 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.223 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.223 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.223 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.223 DEBUG:boto:Method: GET 2023-09-29T22:34:32.223 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.223 DEBUG:boto:Data: 2023-09-29T22:34:32.223 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.224 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.224 DEBUG:boto:Params: {} 2023-09-29T22:34:32.224 DEBUG:boto:Token: None 2023-09-29T22:34:32.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.230 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dccb84a3f7b7331-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.230 DEBUG:boto:b'hcnphz-58obj000000000000000000361000falseobj000000000000000000362023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.231 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.231 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.231 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.231 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.231 DEBUG:boto:Method: GET 2023-09-29T22:34:32.232 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000036 2023-09-29T22:34:32.232 DEBUG:boto:Data: 2023-09-29T22:34:32.232 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.232 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.232 DEBUG:boto:Params: {} 2023-09-29T22:34:32.232 DEBUG:boto:Token: None 2023-09-29T22:34:32.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000071847d579c74d24-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.239 DEBUG:boto:b'hcnphz-58obj000000000000000000361000falseobj000000000000000000362023-09-29T22:34:16.348Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-09-29T22:34:32.240 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.240 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.240 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.240 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.240 DEBUG:boto:Method: GET 2023-09-29T22:34:32.240 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.240 DEBUG:boto:Data: 2023-09-29T22:34:32.240 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.240 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.241 DEBUG:boto:Params: {} 2023-09-29T22:34:32.241 DEBUG:boto:Token: None 2023-09-29T22:34:32.241 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U= 2023-09-29T22:34:32.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U=', 'Content-Length': '0'} 2023-09-29T22:34:32.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a455b1415a8f2f8-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.286 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.287 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.287 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.287 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.287 DEBUG:boto:Method: GET 2023-09-29T22:34:32.287 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.287 DEBUG:boto:Data: 2023-09-29T22:34:32.287 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.288 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.288 DEBUG:boto:Params: {} 2023-09-29T22:34:32.288 DEBUG:boto:Token: None 2023-09-29T22:34:32.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U= 2023-09-29T22:34:32.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U=', 'Content-Length': '0'} 2023-09-29T22:34:32.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a612f2bdb531b1-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.291 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.292 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.292 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.292 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.292 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.292 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.292 DEBUG:boto:Data: 2023-09-29T22:34:32.292 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.292 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.292 DEBUG:boto:Params: {} 2023-09-29T22:34:32.293 DEBUG:boto:Token: None 2023-09-29T22:34:32.293 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OxWOcuRB8MWufSAwWB7cBhUxBSI= 2023-09-29T22:34:32.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OxWOcuRB8MWufSAwWB7cBhUxBSI=', 'Content-Length': '0'} 2023-09-29T22:34:32.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9ccc1d95c6510bb-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.296 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.296 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.296 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.296 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.296 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.297 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.297 DEBUG:boto:Data: 2023-09-29T22:34:32.297 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.297 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.297 DEBUG:boto:Params: {} 2023-09-29T22:34:32.297 DEBUG:boto:Token: None 2023-09-29T22:34:32.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OxWOcuRB8MWufSAwWB7cBhUxBSI= 2023-09-29T22:34:32.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OxWOcuRB8MWufSAwWB7cBhUxBSI=', 'Content-Length': '0'} 2023-09-29T22:34:32.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f001ac7e3bc41ebd-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-09-29T22:34:32.300 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.301 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.301 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.301 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.301 DEBUG:boto:Method: GET 2023-09-29T22:34:32.301 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.301 DEBUG:boto:Data: 2023-09-29T22:34:32.301 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.302 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.302 DEBUG:boto:Params: {} 2023-09-29T22:34:32.302 DEBUG:boto:Token: None 2023-09-29T22:34:32.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U= 2023-09-29T22:34:32.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U=', 'Content-Length': '0'} 2023-09-29T22:34:32.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007433d8429ea84a28-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.305 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.305 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.306 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.306 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.306 DEBUG:boto:Method: GET 2023-09-29T22:34:32.306 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.306 DEBUG:boto:Data: 2023-09-29T22:34:32.306 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.306 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.306 DEBUG:boto:Params: {} 2023-09-29T22:34:32.307 DEBUG:boto:Token: None 2023-09-29T22:34:32.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037?versionId=null 2023-09-29T22:34:32.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U= 2023-09-29T22:34:32.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fevTAQ/AciOC9GdVChfVYOhUO4U=', 'Content-Length': '0'} 2023-09-29T22:34:32.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f2f67d045d51c00-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.350 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.351 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.351 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.351 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.351 DEBUG:boto:Method: GET 2023-09-29T22:34:32.351 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.351 DEBUG:boto:Data: 2023-09-29T22:34:32.351 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.351 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.352 DEBUG:boto:Params: {} 2023-09-29T22:34:32.352 DEBUG:boto:Token: None 2023-09-29T22:34:32.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.357 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000902822ffb5853bc3-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.357 DEBUG:boto:b'hcnphz-58obj000000000000000000371000falseobj000000000000000000372023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.358 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.358 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.358 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.358 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.358 DEBUG:boto:Method: GET 2023-09-29T22:34:32.358 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.358 DEBUG:boto:Data: 2023-09-29T22:34:32.358 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.359 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.359 DEBUG:boto:Params: {} 2023-09-29T22:34:32.359 DEBUG:boto:Token: None 2023-09-29T22:34:32.359 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.359 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.359 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.367 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e030a142b0975cd1-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.367 DEBUG:boto:b'hcnphz-58obj000000000000000000371000falseobj000000000000000000372023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.367 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.368 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.368 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.368 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.368 DEBUG:boto:Data: 2023-09-29T22:34:32.368 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.368 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.368 DEBUG:boto:Params: {} 2023-09-29T22:34:32.369 DEBUG:boto:Token: None 2023-09-29T22:34:32.369 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.369 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ct9+s9AMn2CDTBWBroM87HahwHY= 2023-09-29T22:34:32.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ct9+s9AMn2CDTBWBroM87HahwHY=', 'Content-Length': '0'} 2023-09-29T22:34:32.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063fc9a3aa9f23336-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.371 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.371 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.371 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.372 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.372 DEBUG:boto:Data: 2023-09-29T22:34:32.372 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.372 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.372 DEBUG:boto:Params: {} 2023-09-29T22:34:32.372 DEBUG:boto:Token: None 2023-09-29T22:34:32.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ct9+s9AMn2CDTBWBroM87HahwHY= 2023-09-29T22:34:32.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ct9+s9AMn2CDTBWBroM87HahwHY=', 'Content-Length': '0'} 2023-09-29T22:34:32.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098335fa2c8cc75c7-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000037 2023-09-29T22:34:32.375 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.375 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.375 DEBUG:boto:Method: GET 2023-09-29T22:34:32.375 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.375 DEBUG:boto:Data: 2023-09-29T22:34:32.375 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.376 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.376 DEBUG:boto:Params: {} 2023-09-29T22:34:32.376 DEBUG:boto:Token: None 2023-09-29T22:34:32.376 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gjslsry/5Tzbtbt/xq4AeIeYMwk= 2023-09-29T22:34:32.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gjslsry/5Tzbtbt/xq4AeIeYMwk=', 'Content-Length': '0'} 2023-09-29T22:34:32.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045c710e80932e93b-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.426 DEBUG:boto:path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.427 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.427 DEBUG:boto:Method: GET 2023-09-29T22:34:32.427 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.427 DEBUG:boto:Data: 2023-09-29T22:34:32.427 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.427 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.427 DEBUG:boto:Params: {} 2023-09-29T22:34:32.428 DEBUG:boto:Token: None 2023-09-29T22:34:32.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000037 2023-09-29T22:34:32.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gjslsry/5Tzbtbt/xq4AeIeYMwk= 2023-09-29T22:34:32.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gjslsry/5Tzbtbt/xq4AeIeYMwk=', 'Content-Length': '0'} 2023-09-29T22:34:32.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cb329f87a44aa6b-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.431 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.431 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.431 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.431 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.432 DEBUG:boto:Method: GET 2023-09-29T22:34:32.432 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.432 DEBUG:boto:Data: 2023-09-29T22:34:32.432 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.432 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.432 DEBUG:boto:Params: {} 2023-09-29T22:34:32.432 DEBUG:boto:Token: None 2023-09-29T22:34:32.433 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008261adbc2c25bd9f-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.438 DEBUG:boto:b'hcnphz-58obj000000000000000000371000falseobj000000000000000000372023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.438 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.438 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.438 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.439 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.439 DEBUG:boto:Method: GET 2023-09-29T22:34:32.439 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000037 2023-09-29T22:34:32.439 DEBUG:boto:Data: 2023-09-29T22:34:32.439 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.439 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.439 DEBUG:boto:Params: {} 2023-09-29T22:34:32.439 DEBUG:boto:Token: None 2023-09-29T22:34:32.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4dd1438c9437820-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.490 DEBUG:boto:b'hcnphz-58obj000000000000000000371000falseobj000000000000000000372023-09-29T22:34:13.095Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-09-29T22:34:32.491 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.491 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.492 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.492 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.492 DEBUG:boto:Method: GET 2023-09-29T22:34:32.492 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.492 DEBUG:boto:Data: 2023-09-29T22:34:32.492 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.492 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.493 DEBUG:boto:Params: {} 2023-09-29T22:34:32.493 DEBUG:boto:Token: None 2023-09-29T22:34:32.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE= 2023-09-29T22:34:32.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE=', 'Content-Length': '0'} 2023-09-29T22:34:32.496 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009744826cc88de136-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.497 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.497 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.497 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.497 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.497 DEBUG:boto:Method: GET 2023-09-29T22:34:32.497 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.497 DEBUG:boto:Data: 2023-09-29T22:34:32.497 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.497 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.498 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.498 DEBUG:boto:Params: {} 2023-09-29T22:34:32.498 DEBUG:boto:Token: None 2023-09-29T22:34:32.498 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.498 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE= 2023-09-29T22:34:32.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE=', 'Content-Length': '0'} 2023-09-29T22:34:32.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012d557ae1d89b743-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.502 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.502 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.502 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.502 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.502 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.502 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.503 DEBUG:boto:Data: 2023-09-29T22:34:32.503 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.503 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.503 DEBUG:boto:Params: {} 2023-09-29T22:34:32.503 DEBUG:boto:Token: None 2023-09-29T22:34:32.503 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2bB64+orDYBOCDt6RXy17GxPnR8= 2023-09-29T22:34:32.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2bB64+orDYBOCDt6RXy17GxPnR8=', 'Content-Length': '0'} 2023-09-29T22:34:32.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad18f0d7cc9acbf-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.506 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.506 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.506 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.507 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.507 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.507 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.507 DEBUG:boto:Data: 2023-09-29T22:34:32.507 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.507 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.507 DEBUG:boto:Params: {} 2023-09-29T22:34:32.507 DEBUG:boto:Token: None 2023-09-29T22:34:32.507 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2bB64+orDYBOCDt6RXy17GxPnR8= 2023-09-29T22:34:32.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2bB64+orDYBOCDt6RXy17GxPnR8=', 'Content-Length': '0'} 2023-09-29T22:34:32.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000143025a97930f5da-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-09-29T22:34:32.511 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.511 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.511 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.511 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.511 DEBUG:boto:Method: GET 2023-09-29T22:34:32.512 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.512 DEBUG:boto:Data: 2023-09-29T22:34:32.512 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.512 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.512 DEBUG:boto:Params: {} 2023-09-29T22:34:32.512 DEBUG:boto:Token: None 2023-09-29T22:34:32.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE= 2023-09-29T22:34:32.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE=', 'Content-Length': '0'} 2023-09-29T22:34:32.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040fb141543dbd651-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.516 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.516 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.516 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.517 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.517 DEBUG:boto:Method: GET 2023-09-29T22:34:32.517 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.517 DEBUG:boto:Data: 2023-09-29T22:34:32.517 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.517 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.517 DEBUG:boto:Params: {} 2023-09-29T22:34:32.518 DEBUG:boto:Token: None 2023-09-29T22:34:32.518 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039?versionId=null 2023-09-29T22:34:32.518 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE= 2023-09-29T22:34:32.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZYzmIQ37kAXhfXXhZ1dJMsMybuE=', 'Content-Length': '0'} 2023-09-29T22:34:32.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000552ba9f67699a2b6-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.521 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.521 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.521 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.521 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.522 DEBUG:boto:Method: GET 2023-09-29T22:34:32.522 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.522 DEBUG:boto:Data: 2023-09-29T22:34:32.522 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.522 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.522 DEBUG:boto:Params: {} 2023-09-29T22:34:32.523 DEBUG:boto:Token: None 2023-09-29T22:34:32.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.527 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000333c75c2bfbd8b38-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.570 DEBUG:boto:b'hcnphz-58obj000000000000000000391000falseobj000000000000000000392023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.571 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.571 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.571 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.571 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.571 DEBUG:boto:Method: GET 2023-09-29T22:34:32.572 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.572 DEBUG:boto:Data: 2023-09-29T22:34:32.572 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.572 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.572 DEBUG:boto:Params: {} 2023-09-29T22:34:32.572 DEBUG:boto:Token: None 2023-09-29T22:34:32.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.583 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f003bfb0cc30c47e-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.583 DEBUG:boto:b'hcnphz-58obj000000000000000000391000falseobj000000000000000000392023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.583 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.583 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.584 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.584 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.584 DEBUG:boto:Data: 2023-09-29T22:34:32.584 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.584 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.584 DEBUG:boto:Params: {} 2023-09-29T22:34:32.584 DEBUG:boto:Token: None 2023-09-29T22:34:32.585 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.585 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QbCCAq8cED0LV87mzn15VafUTcc= 2023-09-29T22:34:32.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QbCCAq8cED0LV87mzn15VafUTcc=', 'Content-Length': '0'} 2023-09-29T22:34:32.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004920f5b48143b548-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.589 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.590 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.590 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.590 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.590 DEBUG:boto:Data: 2023-09-29T22:34:32.590 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.590 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.590 DEBUG:boto:Params: {} 2023-09-29T22:34:32.591 DEBUG:boto:Token: None 2023-09-29T22:34:32.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QbCCAq8cED0LV87mzn15VafUTcc= 2023-09-29T22:34:32.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QbCCAq8cED0LV87mzn15VafUTcc=', 'Content-Length': '0'} 2023-09-29T22:34:32.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5bc97726a0615ca-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000039 2023-09-29T22:34:32.593 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.594 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.594 DEBUG:boto:Method: GET 2023-09-29T22:34:32.594 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.594 DEBUG:boto:Data: 2023-09-29T22:34:32.594 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.594 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.594 DEBUG:boto:Params: {} 2023-09-29T22:34:32.594 DEBUG:boto:Token: None 2023-09-29T22:34:32.595 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.595 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rdsef0qDE4CUBaHtfDNEGWw5PO8= 2023-09-29T22:34:32.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rdsef0qDE4CUBaHtfDNEGWw5PO8=', 'Content-Length': '0'} 2023-09-29T22:34:32.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093b42b39e4358fd0-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.597 DEBUG:boto:path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.597 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.597 DEBUG:boto:Method: GET 2023-09-29T22:34:32.598 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.598 DEBUG:boto:Data: 2023-09-29T22:34:32.598 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.598 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.598 DEBUG:boto:Params: {} 2023-09-29T22:34:32.598 DEBUG:boto:Token: None 2023-09-29T22:34:32.599 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000039 2023-09-29T22:34:32.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rdsef0qDE4CUBaHtfDNEGWw5PO8= 2023-09-29T22:34:32.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rdsef0qDE4CUBaHtfDNEGWw5PO8=', 'Content-Length': '0'} 2023-09-29T22:34:32.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c343b52c175b21b-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.642 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.643 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.643 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.643 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.643 DEBUG:boto:Method: GET 2023-09-29T22:34:32.643 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.643 DEBUG:boto:Data: 2023-09-29T22:34:32.643 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.643 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.644 DEBUG:boto:Params: {} 2023-09-29T22:34:32.644 DEBUG:boto:Token: None 2023-09-29T22:34:32.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005420ab8d093c1aa5-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.649 DEBUG:boto:b'hcnphz-58obj000000000000000000391000falseobj000000000000000000392023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.650 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.650 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.650 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.650 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.650 DEBUG:boto:Method: GET 2023-09-29T22:34:32.650 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000039 2023-09-29T22:34:32.650 DEBUG:boto:Data: 2023-09-29T22:34:32.650 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.651 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.651 DEBUG:boto:Params: {} 2023-09-29T22:34:32.651 DEBUG:boto:Token: None 2023-09-29T22:34:32.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.659 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015f18783f02a1050-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.659 DEBUG:boto:b'hcnphz-58obj000000000000000000391000falseobj000000000000000000392023-09-29T22:34:16.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-09-29T22:34:32.660 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.660 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.660 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.660 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.660 DEBUG:boto:Method: GET 2023-09-29T22:34:32.660 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.660 DEBUG:boto:Data: 2023-09-29T22:34:32.661 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.661 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.661 DEBUG:boto:Params: {} 2023-09-29T22:34:32.661 DEBUG:boto:Token: None 2023-09-29T22:34:32.661 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q= 2023-09-29T22:34:32.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q=', 'Content-Length': '0'} 2023-09-29T22:34:32.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002012a942d20298ea-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.706 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.706 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.707 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.707 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.707 DEBUG:boto:Method: GET 2023-09-29T22:34:32.707 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.707 DEBUG:boto:Data: 2023-09-29T22:34:32.707 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.707 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.707 DEBUG:boto:Params: {} 2023-09-29T22:34:32.707 DEBUG:boto:Token: None 2023-09-29T22:34:32.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q= 2023-09-29T22:34:32.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q=', 'Content-Length': '0'} 2023-09-29T22:34:32.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdee183b3d3c43b5-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.711 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.711 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.711 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.711 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.711 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.711 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.711 DEBUG:boto:Data: 2023-09-29T22:34:32.711 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.712 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.712 DEBUG:boto:Params: {} 2023-09-29T22:34:32.712 DEBUG:boto:Token: None 2023-09-29T22:34:32.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2188sQKkTCzB36A9YD6scwLz+6U= 2023-09-29T22:34:32.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2188sQKkTCzB36A9YD6scwLz+6U=', 'Content-Length': '0'} 2023-09-29T22:34:32.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9072df168994b69-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.716 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.716 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.716 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.716 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.716 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.717 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.717 DEBUG:boto:Data: 2023-09-29T22:34:32.717 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.717 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.717 DEBUG:boto:Params: {} 2023-09-29T22:34:32.718 DEBUG:boto:Token: None 2023-09-29T22:34:32.718 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2188sQKkTCzB36A9YD6scwLz+6U= 2023-09-29T22:34:32.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2188sQKkTCzB36A9YD6scwLz+6U=', 'Content-Length': '0'} 2023-09-29T22:34:32.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009073a43fee7c9a36-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-09-29T22:34:32.721 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.721 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.721 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.721 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.722 DEBUG:boto:Method: GET 2023-09-29T22:34:32.722 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.722 DEBUG:boto:Data: 2023-09-29T22:34:32.722 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.722 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.722 DEBUG:boto:Params: {} 2023-09-29T22:34:32.722 DEBUG:boto:Token: None 2023-09-29T22:34:32.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q= 2023-09-29T22:34:32.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q=', 'Content-Length': '0'} 2023-09-29T22:34:32.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5d7ee6c6f6f282a-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.726 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.726 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.726 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.726 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.726 DEBUG:boto:Method: GET 2023-09-29T22:34:32.727 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.727 DEBUG:boto:Data: 2023-09-29T22:34:32.727 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.727 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.727 DEBUG:boto:Params: {} 2023-09-29T22:34:32.727 DEBUG:boto:Token: None 2023-09-29T22:34:32.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040?versionId=null 2023-09-29T22:34:32.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q= 2023-09-29T22:34:32.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PKYAtBL2VESmsB/1EKcnjyzXs3Q=', 'Content-Length': '0'} 2023-09-29T22:34:32.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eaaa60088467decc-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.730 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.731 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.731 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.731 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.731 DEBUG:boto:Method: GET 2023-09-29T22:34:32.731 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.731 DEBUG:boto:Data: 2023-09-29T22:34:32.731 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.731 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.731 DEBUG:boto:Params: {} 2023-09-29T22:34:32.732 DEBUG:boto:Token: None 2023-09-29T22:34:32.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000010c1214fabdfa25-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.737 DEBUG:boto:b'hcnphz-58obj000000000000000000401000falseobj000000000000000000402023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.737 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.737 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.737 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.738 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.738 DEBUG:boto:Method: GET 2023-09-29T22:34:32.738 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.738 DEBUG:boto:Data: 2023-09-29T22:34:32.738 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.738 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.738 DEBUG:boto:Params: {} 2023-09-29T22:34:32.739 DEBUG:boto:Token: None 2023-09-29T22:34:32.739 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000289ea3346b5fb7a3-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.790 DEBUG:boto:b'hcnphz-58obj000000000000000000401000falseobj000000000000000000402023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.791 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.791 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.791 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.791 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.791 DEBUG:boto:Data: 2023-09-29T22:34:32.791 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.791 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.792 DEBUG:boto:Params: {} 2023-09-29T22:34:32.792 DEBUG:boto:Token: None 2023-09-29T22:34:32.792 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aafOshmKwmVbr98Br82Aw6ZDkA0= 2023-09-29T22:34:32.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aafOshmKwmVbr98Br82Aw6ZDkA0=', 'Content-Length': '0'} 2023-09-29T22:34:32.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3219beafa841e74-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.795 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.795 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.795 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.795 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.795 DEBUG:boto:Data: 2023-09-29T22:34:32.796 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.796 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.796 DEBUG:boto:Params: {} 2023-09-29T22:34:32.796 DEBUG:boto:Token: None 2023-09-29T22:34:32.796 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aafOshmKwmVbr98Br82Aw6ZDkA0= 2023-09-29T22:34:32.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aafOshmKwmVbr98Br82Aw6ZDkA0=', 'Content-Length': '0'} 2023-09-29T22:34:32.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f70485c3b7c5c62b-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000040 2023-09-29T22:34:32.799 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.800 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.800 DEBUG:boto:Method: GET 2023-09-29T22:34:32.800 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.800 DEBUG:boto:Data: 2023-09-29T22:34:32.800 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.800 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.800 DEBUG:boto:Params: {} 2023-09-29T22:34:32.800 DEBUG:boto:Token: None 2023-09-29T22:34:32.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zcgcnR/K3xoKyTIvB1Gv7YNUS5k= 2023-09-29T22:34:32.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zcgcnR/K3xoKyTIvB1Gv7YNUS5k=', 'Content-Length': '0'} 2023-09-29T22:34:32.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a22fc19dd1e44393-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.846 DEBUG:boto:path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.846 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.847 DEBUG:boto:Method: GET 2023-09-29T22:34:32.847 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.847 DEBUG:boto:Data: 2023-09-29T22:34:32.847 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.847 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.847 DEBUG:boto:Params: {} 2023-09-29T22:34:32.847 DEBUG:boto:Token: None 2023-09-29T22:34:32.847 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000040 2023-09-29T22:34:32.847 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zcgcnR/K3xoKyTIvB1Gv7YNUS5k= 2023-09-29T22:34:32.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zcgcnR/K3xoKyTIvB1Gv7YNUS5k=', 'Content-Length': '0'} 2023-09-29T22:34:32.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10cfeded4308c42-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.851 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.851 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.851 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.851 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.851 DEBUG:boto:Method: GET 2023-09-29T22:34:32.852 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.852 DEBUG:boto:Data: 2023-09-29T22:34:32.852 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.852 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.852 DEBUG:boto:Params: {} 2023-09-29T22:34:32.852 DEBUG:boto:Token: None 2023-09-29T22:34:32.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054fd5481337a0fb5-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.857 DEBUG:boto:b'hcnphz-58obj000000000000000000401000falseobj000000000000000000402023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.857 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.858 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.858 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.858 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.858 DEBUG:boto:Method: GET 2023-09-29T22:34:32.858 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000040 2023-09-29T22:34:32.858 DEBUG:boto:Data: 2023-09-29T22:34:32.858 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.858 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.858 DEBUG:boto:Params: {} 2023-09-29T22:34:32.859 DEBUG:boto:Token: None 2023-09-29T22:34:32.859 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.859 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.867 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f13e4c709d247a3-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.867 DEBUG:boto:b'hcnphz-58obj000000000000000000401000falseobj000000000000000000402023-09-29T22:34:11.404Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-09-29T22:34:32.867 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.867 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.867 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.868 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.868 DEBUG:boto:Method: GET 2023-09-29T22:34:32.868 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.868 DEBUG:boto:Data: 2023-09-29T22:34:32.868 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.868 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.868 DEBUG:boto:Params: {} 2023-09-29T22:34:32.868 DEBUG:boto:Token: None 2023-09-29T22:34:32.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo= 2023-09-29T22:34:32.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo=', 'Content-Length': '0'} 2023-09-29T22:34:32.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a9e4269edbfc72-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.871 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.872 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.872 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.872 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.872 DEBUG:boto:Method: GET 2023-09-29T22:34:32.872 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.872 DEBUG:boto:Data: 2023-09-29T22:34:32.872 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.872 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.872 DEBUG:boto:Params: {} 2023-09-29T22:34:32.873 DEBUG:boto:Token: None 2023-09-29T22:34:32.873 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.873 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo= 2023-09-29T22:34:32.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo=', 'Content-Length': '0'} 2023-09-29T22:34:32.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd6dd37eb4fc9354-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.922 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.923 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.923 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.923 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.923 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.923 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.923 DEBUG:boto:Data: 2023-09-29T22:34:32.923 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.923 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.924 DEBUG:boto:Params: {} 2023-09-29T22:34:32.924 DEBUG:boto:Token: None 2023-09-29T22:34:32.924 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/fqDYAiGmhf2RdIE+FvLZ02e120= 2023-09-29T22:34:32.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/fqDYAiGmhf2RdIE+FvLZ02e120=', 'Content-Length': '0'} 2023-09-29T22:34:32.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005499824b820da9b2-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.927 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.927 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.927 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.927 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.927 DEBUG:boto:Method: HEAD 2023-09-29T22:34:32.927 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.927 DEBUG:boto:Data: 2023-09-29T22:34:32.928 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.928 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.928 DEBUG:boto:Params: {} 2023-09-29T22:34:32.928 DEBUG:boto:Token: None 2023-09-29T22:34:32.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/fqDYAiGmhf2RdIE+FvLZ02e120= 2023-09-29T22:34:32.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/fqDYAiGmhf2RdIE+FvLZ02e120=', 'Content-Length': '0'} 2023-09-29T22:34:32.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acbc7a06e0ef9965-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-09-29T22:34:32.931 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.931 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.932 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.932 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.932 DEBUG:boto:Method: GET 2023-09-29T22:34:32.932 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.932 DEBUG:boto:Data: 2023-09-29T22:34:32.932 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.932 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.932 DEBUG:boto:Params: {} 2023-09-29T22:34:32.932 DEBUG:boto:Token: None 2023-09-29T22:34:32.933 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.933 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo= 2023-09-29T22:34:32.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo=', 'Content-Length': '0'} 2023-09-29T22:34:32.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b187bdc8475d38bf-00651750f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.978 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.978 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:32.978 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.979 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.979 DEBUG:boto:Method: GET 2023-09-29T22:34:32.979 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.979 DEBUG:boto:Data: 2023-09-29T22:34:32.979 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.979 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.979 DEBUG:boto:Params: {} 2023-09-29T22:34:32.979 DEBUG:boto:Token: None 2023-09-29T22:34:32.980 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/obj00000000000000000042?versionId=null 2023-09-29T22:34:32.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo= 2023-09-29T22:34:32.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FN1pU818tlvDeYSO00U7Iz0Gjzo=', 'Content-Length': '0'} 2023-09-29T22:34:32.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1ae2a305bf174e7-00651750f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.983 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.983 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.983 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:32.983 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:32.983 DEBUG:boto:Method: GET 2023-09-29T22:34:32.983 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:32.983 DEBUG:boto:Data: 2023-09-29T22:34:32.983 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:32.984 DEBUG:boto:Port: 8001 2023-09-29T22:34:32.984 DEBUG:boto:Params: {} 2023-09-29T22:34:32.984 DEBUG:boto:Token: None 2023-09-29T22:34:32.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:32.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c758fc1c28be0dc-00651750f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:32.990 DEBUG:boto:b'hcnphz-58obj000000000000000000421000falseobj000000000000000000422023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:32.991 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:32.991 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:32.991 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:32.991 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:32.991 DEBUG:boto:Method: GET 2023-09-29T22:34:32.991 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:32.991 DEBUG:boto:Data: 2023-09-29T22:34:32.991 DEBUG:boto:Headers: {} 2023-09-29T22:34:32.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:32.991 DEBUG:boto:Port: 8000 2023-09-29T22:34:32.992 DEBUG:boto:Params: {} 2023-09-29T22:34:32.992 DEBUG:boto:Token: None 2023-09-29T22:34:32.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:32 GMT /hcnphz-58/ 2023-09-29T22:34:32.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU= 2023-09-29T22:34:32.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S9dTONilLowoHR1c52Ejszr8ULU=', 'Content-Length': '0'} 2023-09-29T22:34:33.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000139763bb21cbdacc-00651750f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.002 DEBUG:boto:b'hcnphz-58obj000000000000000000421000falseobj000000000000000000422023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.002 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.002 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.002 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.003 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.003 DEBUG:boto:Data: 2023-09-29T22:34:33.003 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.003 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.003 DEBUG:boto:Params: {} 2023-09-29T22:34:33.003 DEBUG:boto:Token: None 2023-09-29T22:34:33.003 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WM4RVJzr3sTzQpy6g1WrjS1hU/8= 2023-09-29T22:34:33.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WM4RVJzr3sTzQpy6g1WrjS1hU/8=', 'Content-Length': '0'} 2023-09-29T22:34:33.006 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff6451ba28d5e783-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.006 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.006 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.006 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.006 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.006 DEBUG:boto:Data: 2023-09-29T22:34:33.006 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.007 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.007 DEBUG:boto:Params: {} 2023-09-29T22:34:33.007 DEBUG:boto:Token: None 2023-09-29T22:34:33.007 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WM4RVJzr3sTzQpy6g1WrjS1hU/8= 2023-09-29T22:34:33.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WM4RVJzr3sTzQpy6g1WrjS1hU/8=', 'Content-Length': '0'} 2023-09-29T22:34:33.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5ef7f63056d3303-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000042 2023-09-29T22:34:33.010 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.010 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.010 DEBUG:boto:Method: GET 2023-09-29T22:34:33.010 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.010 DEBUG:boto:Data: 2023-09-29T22:34:33.010 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.011 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.011 DEBUG:boto:Params: {} 2023-09-29T22:34:33.011 DEBUG:boto:Token: None 2023-09-29T22:34:33.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DMh9UtWWpNypPjUyhEgi545GcHo= 2023-09-29T22:34:33.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DMh9UtWWpNypPjUyhEgi545GcHo=', 'Content-Length': '0'} 2023-09-29T22:34:33.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058a8d4f805c9d939-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.014 DEBUG:boto:path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.014 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.014 DEBUG:boto:Method: GET 2023-09-29T22:34:33.014 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.014 DEBUG:boto:Data: 2023-09-29T22:34:33.014 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.014 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.014 DEBUG:boto:Params: {} 2023-09-29T22:34:33.015 DEBUG:boto:Token: None 2023-09-29T22:34:33.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000042 2023-09-29T22:34:33.015 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DMh9UtWWpNypPjUyhEgi545GcHo= 2023-09-29T22:34:33.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DMh9UtWWpNypPjUyhEgi545GcHo=', 'Content-Length': '0'} 2023-09-29T22:34:33.017 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb394ba650499365-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.062 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.062 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.063 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:33.063 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:33.063 DEBUG:boto:Method: GET 2023-09-29T22:34:33.063 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:33.063 DEBUG:boto:Data: 2023-09-29T22:34:33.063 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.063 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.063 DEBUG:boto:Params: {} 2023-09-29T22:34:33.063 DEBUG:boto:Token: None 2023-09-29T22:34:33.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.068 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d13aaea38272fd3-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.069 DEBUG:boto:b'hcnphz-58obj000000000000000000421000falseobj000000000000000000422023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.069 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.069 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.069 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:33.069 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:33.070 DEBUG:boto:Method: GET 2023-09-29T22:34:33.070 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000042 2023-09-29T22:34:33.070 DEBUG:boto:Data: 2023-09-29T22:34:33.070 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.070 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.070 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.070 DEBUG:boto:Params: {} 2023-09-29T22:34:33.070 DEBUG:boto:Token: None 2023-09-29T22:34:33.071 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.071 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.071 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074892cb7618a9352-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.078 DEBUG:boto:b'hcnphz-58obj000000000000000000421000falseobj000000000000000000422023-09-29T22:34:13.167Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-09-29T22:34:33.079 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.079 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.079 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.079 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.079 DEBUG:boto:Method: GET 2023-09-29T22:34:33.080 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.080 DEBUG:boto:Data: 2023-09-29T22:34:33.080 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.080 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.080 DEBUG:boto:Params: {} 2023-09-29T22:34:33.080 DEBUG:boto:Token: None 2023-09-29T22:34:33.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8= 2023-09-29T22:34:33.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8=', 'Content-Length': '0'} 2023-09-29T22:34:33.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f080a35758eb238-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.130 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.130 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.130 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.131 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.131 DEBUG:boto:Method: GET 2023-09-29T22:34:33.131 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.131 DEBUG:boto:Data: 2023-09-29T22:34:33.131 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.131 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.131 DEBUG:boto:Params: {} 2023-09-29T22:34:33.131 DEBUG:boto:Token: None 2023-09-29T22:34:33.132 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8= 2023-09-29T22:34:33.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8=', 'Content-Length': '0'} 2023-09-29T22:34:33.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003adfbb7381990992-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.135 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.135 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.135 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.135 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.135 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.135 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.135 DEBUG:boto:Data: 2023-09-29T22:34:33.136 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.136 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.136 DEBUG:boto:Params: {} 2023-09-29T22:34:33.136 DEBUG:boto:Token: None 2023-09-29T22:34:33.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mXbicV8lchJ/lDamaoambDWByCo= 2023-09-29T22:34:33.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mXbicV8lchJ/lDamaoambDWByCo=', 'Content-Length': '0'} 2023-09-29T22:34:33.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cec98b50e8a7b4a-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.139 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.139 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.139 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.139 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.140 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.140 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.140 DEBUG:boto:Data: 2023-09-29T22:34:33.140 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.140 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.140 DEBUG:boto:Params: {} 2023-09-29T22:34:33.140 DEBUG:boto:Token: None 2023-09-29T22:34:33.140 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mXbicV8lchJ/lDamaoambDWByCo= 2023-09-29T22:34:33.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mXbicV8lchJ/lDamaoambDWByCo=', 'Content-Length': '0'} 2023-09-29T22:34:33.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000095b208455fb3d66-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-09-29T22:34:33.144 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.144 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.144 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.144 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.144 DEBUG:boto:Method: GET 2023-09-29T22:34:33.144 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.144 DEBUG:boto:Data: 2023-09-29T22:34:33.144 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.145 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.145 DEBUG:boto:Params: {} 2023-09-29T22:34:33.145 DEBUG:boto:Token: None 2023-09-29T22:34:33.145 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8= 2023-09-29T22:34:33.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8=', 'Content-Length': '0'} 2023-09-29T22:34:33.148 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541bacf7f984448e-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.148 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.148 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.149 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.149 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.149 DEBUG:boto:Method: GET 2023-09-29T22:34:33.149 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.149 DEBUG:boto:Data: 2023-09-29T22:34:33.149 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.149 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.149 DEBUG:boto:Params: {} 2023-09-29T22:34:33.149 DEBUG:boto:Token: None 2023-09-29T22:34:33.150 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045?versionId=null 2023-09-29T22:34:33.150 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8= 2023-09-29T22:34:33.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WMCdwht2fiQKGz/SMa9+ZaSNOI8=', 'Content-Length': '0'} 2023-09-29T22:34:33.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8fd13b59a5533d-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.194 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.194 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.195 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.195 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.195 DEBUG:boto:Method: GET 2023-09-29T22:34:33.195 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.195 DEBUG:boto:Data: 2023-09-29T22:34:33.195 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.195 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.195 DEBUG:boto:Params: {} 2023-09-29T22:34:33.195 DEBUG:boto:Token: None 2023-09-29T22:34:33.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa7b7d4d78d39cf6-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.201 DEBUG:boto:b'hcnphz-58obj000000000000000000451000falseobj000000000000000000452023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.201 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.201 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.201 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.201 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.202 DEBUG:boto:Method: GET 2023-09-29T22:34:33.202 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.202 DEBUG:boto:Data: 2023-09-29T22:34:33.202 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.202 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.202 DEBUG:boto:Params: {} 2023-09-29T22:34:33.202 DEBUG:boto:Token: None 2023-09-29T22:34:33.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.203 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e715b9feea80f2f-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.211 DEBUG:boto:b'hcnphz-58obj000000000000000000451000falseobj000000000000000000452023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.212 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.212 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.212 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.213 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.213 DEBUG:boto:Data: 2023-09-29T22:34:33.213 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.213 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.213 DEBUG:boto:Params: {} 2023-09-29T22:34:33.213 DEBUG:boto:Token: None 2023-09-29T22:34:33.213 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BzOwBURk4DyHYFTiKS77cxvKZog= 2023-09-29T22:34:33.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BzOwBURk4DyHYFTiKS77cxvKZog=', 'Content-Length': '0'} 2023-09-29T22:34:33.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fa8e5987450bfe3-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.216 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.216 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.216 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.216 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.216 DEBUG:boto:Data: 2023-09-29T22:34:33.216 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.216 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.217 DEBUG:boto:Params: {} 2023-09-29T22:34:33.217 DEBUG:boto:Token: None 2023-09-29T22:34:33.217 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BzOwBURk4DyHYFTiKS77cxvKZog= 2023-09-29T22:34:33.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BzOwBURk4DyHYFTiKS77cxvKZog=', 'Content-Length': '0'} 2023-09-29T22:34:33.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8e55b62102f349-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000045 2023-09-29T22:34:33.219 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.219 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.219 DEBUG:boto:Method: GET 2023-09-29T22:34:33.220 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.220 DEBUG:boto:Data: 2023-09-29T22:34:33.220 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.220 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.220 DEBUG:boto:Params: {} 2023-09-29T22:34:33.220 DEBUG:boto:Token: None 2023-09-29T22:34:33.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r16JXBpV+bHMP9PCIuxArixB5p4= 2023-09-29T22:34:33.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r16JXBpV+bHMP9PCIuxArixB5p4=', 'Content-Length': '0'} 2023-09-29T22:34:33.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a7e491187e9ab9-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.266 DEBUG:boto:path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.267 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.267 DEBUG:boto:Method: GET 2023-09-29T22:34:33.267 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.267 DEBUG:boto:Data: 2023-09-29T22:34:33.267 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.267 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.267 DEBUG:boto:Params: {} 2023-09-29T22:34:33.267 DEBUG:boto:Token: None 2023-09-29T22:34:33.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000045 2023-09-29T22:34:33.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r16JXBpV+bHMP9PCIuxArixB5p4= 2023-09-29T22:34:33.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r16JXBpV+bHMP9PCIuxArixB5p4=', 'Content-Length': '0'} 2023-09-29T22:34:33.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022719dfb98b003d3-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.271 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.271 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.271 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.271 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.271 DEBUG:boto:Method: GET 2023-09-29T22:34:33.271 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.272 DEBUG:boto:Data: 2023-09-29T22:34:33.272 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.272 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.272 DEBUG:boto:Params: {} 2023-09-29T22:34:33.272 DEBUG:boto:Token: None 2023-09-29T22:34:33.272 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b08260ecbd4bcc1a-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.277 DEBUG:boto:b'hcnphz-58obj000000000000000000451000falseobj000000000000000000452023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.277 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.277 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.277 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.278 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.278 DEBUG:boto:Method: GET 2023-09-29T22:34:33.278 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000045 2023-09-29T22:34:33.278 DEBUG:boto:Data: 2023-09-29T22:34:33.278 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.278 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.278 DEBUG:boto:Params: {} 2023-09-29T22:34:33.278 DEBUG:boto:Token: None 2023-09-29T22:34:33.278 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2b0ef0f917dc3a4-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.326 DEBUG:boto:b'hcnphz-58obj000000000000000000451000falseobj000000000000000000452023-09-29T22:34:15.728Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-09-29T22:34:33.327 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.327 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.327 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.327 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.327 DEBUG:boto:Method: GET 2023-09-29T22:34:33.327 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.327 DEBUG:boto:Data: 2023-09-29T22:34:33.327 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.328 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.328 DEBUG:boto:Params: {} 2023-09-29T22:34:33.328 DEBUG:boto:Token: None 2023-09-29T22:34:33.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg= 2023-09-29T22:34:33.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg=', 'Content-Length': '0'} 2023-09-29T22:34:33.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006940a702022800c8-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.332 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.332 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.332 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.332 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.332 DEBUG:boto:Method: GET 2023-09-29T22:34:33.332 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.332 DEBUG:boto:Data: 2023-09-29T22:34:33.332 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.333 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.333 DEBUG:boto:Params: {} 2023-09-29T22:34:33.333 DEBUG:boto:Token: None 2023-09-29T22:34:33.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg= 2023-09-29T22:34:33.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg=', 'Content-Length': '0'} 2023-09-29T22:34:33.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24c44fc715c9f19-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.337 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.337 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.337 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.337 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.337 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.337 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.337 DEBUG:boto:Data: 2023-09-29T22:34:33.337 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.338 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.338 DEBUG:boto:Params: {} 2023-09-29T22:34:33.338 DEBUG:boto:Token: None 2023-09-29T22:34:33.338 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:361bYzXUk0GkQRfjyVG1J40YpR4= 2023-09-29T22:34:33.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:361bYzXUk0GkQRfjyVG1J40YpR4=', 'Content-Length': '0'} 2023-09-29T22:34:33.341 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360cf60ac1a77aeb-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.341 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.341 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.341 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.341 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.341 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.342 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.342 DEBUG:boto:Data: 2023-09-29T22:34:33.342 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.342 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.342 DEBUG:boto:Params: {} 2023-09-29T22:34:33.342 DEBUG:boto:Token: None 2023-09-29T22:34:33.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:361bYzXUk0GkQRfjyVG1J40YpR4= 2023-09-29T22:34:33.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:361bYzXUk0GkQRfjyVG1J40YpR4=', 'Content-Length': '0'} 2023-09-29T22:34:33.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db0e8d674494479a-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-09-29T22:34:33.346 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.346 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.346 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.346 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.346 DEBUG:boto:Method: GET 2023-09-29T22:34:33.346 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.347 DEBUG:boto:Data: 2023-09-29T22:34:33.347 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.347 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.347 DEBUG:boto:Params: {} 2023-09-29T22:34:33.347 DEBUG:boto:Token: None 2023-09-29T22:34:33.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg= 2023-09-29T22:34:33.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg=', 'Content-Length': '0'} 2023-09-29T22:34:33.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000480c8b4ebeeac0b0-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.350 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.350 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.351 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.351 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.351 DEBUG:boto:Method: GET 2023-09-29T22:34:33.351 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.351 DEBUG:boto:Data: 2023-09-29T22:34:33.351 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.351 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.351 DEBUG:boto:Params: {} 2023-09-29T22:34:33.352 DEBUG:boto:Token: None 2023-09-29T22:34:33.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046?versionId=null 2023-09-29T22:34:33.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg= 2023-09-29T22:34:33.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uFHACMg5WaPFGObeNoV0lqLAZFg=', 'Content-Length': '0'} 2023-09-29T22:34:33.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5e30d2ac3619730-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.355 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.355 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.355 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.355 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.356 DEBUG:boto:Method: GET 2023-09-29T22:34:33.356 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.356 DEBUG:boto:Data: 2023-09-29T22:34:33.356 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.356 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.356 DEBUG:boto:Params: {} 2023-09-29T22:34:33.356 DEBUG:boto:Token: None 2023-09-29T22:34:33.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f12f0b624a707b80-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.361 DEBUG:boto:b'hcnphz-58obj000000000000000000461000falseobj000000000000000000462023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.362 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.362 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.362 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.362 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.362 DEBUG:boto:Method: GET 2023-09-29T22:34:33.362 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.362 DEBUG:boto:Data: 2023-09-29T22:34:33.363 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.363 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.363 DEBUG:boto:Params: {} 2023-09-29T22:34:33.363 DEBUG:boto:Token: None 2023-09-29T22:34:33.363 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.363 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c996a797dd529934-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.418 DEBUG:boto:b'hcnphz-58obj000000000000000000461000falseobj000000000000000000462023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.419 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.419 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.419 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.419 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.419 DEBUG:boto:Data: 2023-09-29T22:34:33.419 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.420 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.420 DEBUG:boto:Params: {} 2023-09-29T22:34:33.420 DEBUG:boto:Token: None 2023-09-29T22:34:33.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JLOBf1iLP/l+irb+146Uw6ca/QM= 2023-09-29T22:34:33.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLOBf1iLP/l+irb+146Uw6ca/QM=', 'Content-Length': '0'} 2023-09-29T22:34:33.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac3e47ada8b13e84-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.423 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.423 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.423 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.423 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.424 DEBUG:boto:Data: 2023-09-29T22:34:33.424 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.424 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.424 DEBUG:boto:Params: {} 2023-09-29T22:34:33.424 DEBUG:boto:Token: None 2023-09-29T22:34:33.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JLOBf1iLP/l+irb+146Uw6ca/QM= 2023-09-29T22:34:33.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLOBf1iLP/l+irb+146Uw6ca/QM=', 'Content-Length': '0'} 2023-09-29T22:34:33.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000353e7562da205187-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000046 2023-09-29T22:34:33.427 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.428 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.428 DEBUG:boto:Method: GET 2023-09-29T22:34:33.428 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.428 DEBUG:boto:Data: 2023-09-29T22:34:33.428 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.428 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.428 DEBUG:boto:Params: {} 2023-09-29T22:34:33.428 DEBUG:boto:Token: None 2023-09-29T22:34:33.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFGr2V8k0U7gNzLa9tHyePT+H1s= 2023-09-29T22:34:33.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFGr2V8k0U7gNzLa9tHyePT+H1s=', 'Content-Length': '0'} 2023-09-29T22:34:33.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005562002254670fa6-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.474 DEBUG:boto:path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.475 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.475 DEBUG:boto:Method: GET 2023-09-29T22:34:33.475 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.475 DEBUG:boto:Data: 2023-09-29T22:34:33.475 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.475 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.475 DEBUG:boto:Params: {} 2023-09-29T22:34:33.475 DEBUG:boto:Token: None 2023-09-29T22:34:33.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000046 2023-09-29T22:34:33.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gFGr2V8k0U7gNzLa9tHyePT+H1s= 2023-09-29T22:34:33.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gFGr2V8k0U7gNzLa9tHyePT+H1s=', 'Content-Length': '0'} 2023-09-29T22:34:33.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a93deb119b3f331-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.479 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.479 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.480 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.480 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.480 DEBUG:boto:Method: GET 2023-09-29T22:34:33.480 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.480 DEBUG:boto:Data: 2023-09-29T22:34:33.480 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.480 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.480 DEBUG:boto:Params: {} 2023-09-29T22:34:33.481 DEBUG:boto:Token: None 2023-09-29T22:34:33.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000953d43537d2b9542-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.486 DEBUG:boto:b'hcnphz-58obj000000000000000000461000falseobj000000000000000000462023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.486 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.487 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.487 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.487 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.487 DEBUG:boto:Method: GET 2023-09-29T22:34:33.487 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000046 2023-09-29T22:34:33.487 DEBUG:boto:Data: 2023-09-29T22:34:33.487 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.487 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.487 DEBUG:boto:Params: {} 2023-09-29T22:34:33.488 DEBUG:boto:Token: None 2023-09-29T22:34:33.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c04d4e1b256a4b9-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.494 DEBUG:boto:b'hcnphz-58obj000000000000000000461000falseobj000000000000000000462023-09-29T22:34:11.534Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-09-29T22:34:33.495 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.495 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.495 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.495 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.495 DEBUG:boto:Method: GET 2023-09-29T22:34:33.495 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.495 DEBUG:boto:Data: 2023-09-29T22:34:33.495 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.496 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.496 DEBUG:boto:Params: {} 2023-09-29T22:34:33.496 DEBUG:boto:Token: None 2023-09-29T22:34:33.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0= 2023-09-29T22:34:33.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0=', 'Content-Length': '0'} 2023-09-29T22:34:33.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006762673c3d473da1-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.499 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.499 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.499 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.500 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.500 DEBUG:boto:Method: GET 2023-09-29T22:34:33.500 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.500 DEBUG:boto:Data: 2023-09-29T22:34:33.500 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.500 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.500 DEBUG:boto:Params: {} 2023-09-29T22:34:33.500 DEBUG:boto:Token: None 2023-09-29T22:34:33.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0= 2023-09-29T22:34:33.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0=', 'Content-Length': '0'} 2023-09-29T22:34:33.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc7ddf35a59e9a44-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.546 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.546 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.547 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.547 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.547 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.547 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.547 DEBUG:boto:Data: 2023-09-29T22:34:33.547 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.547 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.547 DEBUG:boto:Params: {} 2023-09-29T22:34:33.548 DEBUG:boto:Token: None 2023-09-29T22:34:33.548 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U+sh0VQYBec574sO/NJKXIzWxI8= 2023-09-29T22:34:33.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U+sh0VQYBec574sO/NJKXIzWxI8=', 'Content-Length': '0'} 2023-09-29T22:34:33.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a83adb403f37da3f-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.551 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.551 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.552 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.552 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.552 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.552 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.552 DEBUG:boto:Data: 2023-09-29T22:34:33.552 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.552 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.552 DEBUG:boto:Params: {} 2023-09-29T22:34:33.553 DEBUG:boto:Token: None 2023-09-29T22:34:33.553 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U+sh0VQYBec574sO/NJKXIzWxI8= 2023-09-29T22:34:33.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U+sh0VQYBec574sO/NJKXIzWxI8=', 'Content-Length': '0'} 2023-09-29T22:34:33.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9f8f8d6095380d-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-09-29T22:34:33.556 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.556 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.556 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.557 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.557 DEBUG:boto:Method: GET 2023-09-29T22:34:33.557 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.557 DEBUG:boto:Data: 2023-09-29T22:34:33.557 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.557 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.557 DEBUG:boto:Params: {} 2023-09-29T22:34:33.557 DEBUG:boto:Token: None 2023-09-29T22:34:33.558 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.558 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0= 2023-09-29T22:34:33.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0=', 'Content-Length': '0'} 2023-09-29T22:34:33.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073f9910c3b5f1251-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.606 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.606 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.607 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.607 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.607 DEBUG:boto:Method: GET 2023-09-29T22:34:33.607 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.607 DEBUG:boto:Data: 2023-09-29T22:34:33.607 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.607 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.607 DEBUG:boto:Params: {} 2023-09-29T22:34:33.608 DEBUG:boto:Token: None 2023-09-29T22:34:33.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048?versionId=null 2023-09-29T22:34:33.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0= 2023-09-29T22:34:33.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iuUWJwGld6iQlePx5CerfhhXV/0=', 'Content-Length': '0'} 2023-09-29T22:34:33.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0f48312a5c30d22-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.611 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.611 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.613 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.613 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.613 DEBUG:boto:Method: GET 2023-09-29T22:34:33.614 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.615 DEBUG:boto:Data: 2023-09-29T22:34:33.615 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.615 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.615 DEBUG:boto:Params: {} 2023-09-29T22:34:33.615 DEBUG:boto:Token: None 2023-09-29T22:34:33.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.621 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abeab3064b768f43-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.621 DEBUG:boto:b'hcnphz-58obj000000000000000000481000falseobj000000000000000000482023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.622 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.622 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.622 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.622 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.622 DEBUG:boto:Method: GET 2023-09-29T22:34:33.622 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.622 DEBUG:boto:Data: 2023-09-29T22:34:33.622 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.623 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.623 DEBUG:boto:Params: {} 2023-09-29T22:34:33.623 DEBUG:boto:Token: None 2023-09-29T22:34:33.623 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.631 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000882a5a3d086da3d8-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.631 DEBUG:boto:b'hcnphz-58obj000000000000000000481000falseobj000000000000000000482023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.631 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.631 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.631 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.631 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.632 DEBUG:boto:Data: 2023-09-29T22:34:33.632 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.632 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.632 DEBUG:boto:Params: {} 2023-09-29T22:34:33.632 DEBUG:boto:Token: None 2023-09-29T22:34:33.632 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3hHWzxCHZI7VgIOq2yhd2ZzvSb8= 2023-09-29T22:34:33.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3hHWzxCHZI7VgIOq2yhd2ZzvSb8=', 'Content-Length': '0'} 2023-09-29T22:34:33.634 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d98e2f22253bb71e-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.635 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.635 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.635 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.635 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.635 DEBUG:boto:Data: 2023-09-29T22:34:33.635 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.636 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.636 DEBUG:boto:Params: {} 2023-09-29T22:34:33.636 DEBUG:boto:Token: None 2023-09-29T22:34:33.636 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3hHWzxCHZI7VgIOq2yhd2ZzvSb8= 2023-09-29T22:34:33.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3hHWzxCHZI7VgIOq2yhd2ZzvSb8=', 'Content-Length': '0'} 2023-09-29T22:34:33.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007da57ab1ffa05693-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000048 2023-09-29T22:34:33.639 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.639 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.639 DEBUG:boto:Method: GET 2023-09-29T22:34:33.639 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.639 DEBUG:boto:Data: 2023-09-29T22:34:33.639 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.639 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.639 DEBUG:boto:Params: {} 2023-09-29T22:34:33.640 DEBUG:boto:Token: None 2023-09-29T22:34:33.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:natEgHAh1ZIIJQmb2GeIH120mmc= 2023-09-29T22:34:33.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:natEgHAh1ZIIJQmb2GeIH120mmc=', 'Content-Length': '0'} 2023-09-29T22:34:33.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081526dd5900a5068-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.686 DEBUG:boto:path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.686 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.687 DEBUG:boto:Method: GET 2023-09-29T22:34:33.687 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.687 DEBUG:boto:Data: 2023-09-29T22:34:33.687 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.687 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.687 DEBUG:boto:Params: {} 2023-09-29T22:34:33.687 DEBUG:boto:Token: None 2023-09-29T22:34:33.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000048 2023-09-29T22:34:33.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:natEgHAh1ZIIJQmb2GeIH120mmc= 2023-09-29T22:34:33.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:natEgHAh1ZIIJQmb2GeIH120mmc=', 'Content-Length': '0'} 2023-09-29T22:34:33.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1139d68593b0e98-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.691 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.691 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.692 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.692 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.692 DEBUG:boto:Method: GET 2023-09-29T22:34:33.692 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.692 DEBUG:boto:Data: 2023-09-29T22:34:33.692 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.692 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.692 DEBUG:boto:Params: {} 2023-09-29T22:34:33.692 DEBUG:boto:Token: None 2023-09-29T22:34:33.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.693 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee163911b1a439e8-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.697 DEBUG:boto:b'hcnphz-58obj000000000000000000481000falseobj000000000000000000482023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.698 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.698 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.698 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.698 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.698 DEBUG:boto:Method: GET 2023-09-29T22:34:33.698 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000048 2023-09-29T22:34:33.698 DEBUG:boto:Data: 2023-09-29T22:34:33.698 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.699 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.699 DEBUG:boto:Params: {} 2023-09-29T22:34:33.699 DEBUG:boto:Token: None 2023-09-29T22:34:33.699 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b0b7213ccb0aa2d-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.708 DEBUG:boto:b'hcnphz-58obj000000000000000000481000falseobj000000000000000000482023-09-29T22:34:16.493Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-09-29T22:34:33.708 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.708 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.708 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.709 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.709 DEBUG:boto:Method: GET 2023-09-29T22:34:33.709 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.709 DEBUG:boto:Data: 2023-09-29T22:34:33.709 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.709 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.709 DEBUG:boto:Params: {} 2023-09-29T22:34:33.709 DEBUG:boto:Token: None 2023-09-29T22:34:33.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8= 2023-09-29T22:34:33.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8=', 'Content-Length': '0'} 2023-09-29T22:34:33.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e22c5c421422fde-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.712 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.712 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.713 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.713 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.713 DEBUG:boto:Method: GET 2023-09-29T22:34:33.713 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.713 DEBUG:boto:Data: 2023-09-29T22:34:33.713 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.713 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.713 DEBUG:boto:Params: {} 2023-09-29T22:34:33.713 DEBUG:boto:Token: None 2023-09-29T22:34:33.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8= 2023-09-29T22:34:33.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8=', 'Content-Length': '0'} 2023-09-29T22:34:33.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000541e4810cfa2e4cb-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.716 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.717 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.717 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.717 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.717 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.717 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.717 DEBUG:boto:Data: 2023-09-29T22:34:33.717 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.717 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.718 DEBUG:boto:Params: {} 2023-09-29T22:34:33.718 DEBUG:boto:Token: None 2023-09-29T22:34:33.718 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IcImX3tNwZV2BKXuLOfo8I/TjRQ= 2023-09-29T22:34:33.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IcImX3tNwZV2BKXuLOfo8I/TjRQ=', 'Content-Length': '0'} 2023-09-29T22:34:33.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2b48152c65e439c-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.720 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.721 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.721 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.721 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.721 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.721 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.721 DEBUG:boto:Data: 2023-09-29T22:34:33.721 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.721 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.721 DEBUG:boto:Params: {} 2023-09-29T22:34:33.722 DEBUG:boto:Token: None 2023-09-29T22:34:33.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IcImX3tNwZV2BKXuLOfo8I/TjRQ= 2023-09-29T22:34:33.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IcImX3tNwZV2BKXuLOfo8I/TjRQ=', 'Content-Length': '0'} 2023-09-29T22:34:33.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000082ff031e5c54e80-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-09-29T22:34:33.725 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.725 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.725 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.725 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.725 DEBUG:boto:Method: GET 2023-09-29T22:34:33.725 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.725 DEBUG:boto:Data: 2023-09-29T22:34:33.725 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.725 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.726 DEBUG:boto:Params: {} 2023-09-29T22:34:33.726 DEBUG:boto:Token: None 2023-09-29T22:34:33.726 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8= 2023-09-29T22:34:33.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8=', 'Content-Length': '0'} 2023-09-29T22:34:33.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ec6bb0fb7610b2-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.728 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.729 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.729 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.729 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.729 DEBUG:boto:Method: GET 2023-09-29T22:34:33.729 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.729 DEBUG:boto:Data: 2023-09-29T22:34:33.729 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.729 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.729 DEBUG:boto:Params: {} 2023-09-29T22:34:33.730 DEBUG:boto:Token: None 2023-09-29T22:34:33.730 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050?versionId=null 2023-09-29T22:34:33.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8= 2023-09-29T22:34:33.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+60KcoaN/l78x9EFoWK+g5grpB8=', 'Content-Length': '0'} 2023-09-29T22:34:33.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e42ff6c10125d2fc-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.733 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.733 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.733 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.733 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.733 DEBUG:boto:Method: GET 2023-09-29T22:34:33.733 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.733 DEBUG:boto:Data: 2023-09-29T22:34:33.733 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.734 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.734 DEBUG:boto:Params: {} 2023-09-29T22:34:33.734 DEBUG:boto:Token: None 2023-09-29T22:34:33.734 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.734 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.738 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044557683fc5b1ad8-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.738 DEBUG:boto:b'hcnphz-58obj000000000000000000501000falseobj000000000000000000502023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.738 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.739 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.739 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.739 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.739 DEBUG:boto:Method: GET 2023-09-29T22:34:33.739 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.739 DEBUG:boto:Data: 2023-09-29T22:34:33.739 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.739 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.739 DEBUG:boto:Params: {} 2023-09-29T22:34:33.740 DEBUG:boto:Token: None 2023-09-29T22:34:33.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.748 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a8f1e4a49deeff88-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.748 DEBUG:boto:b'hcnphz-58obj000000000000000000501000falseobj000000000000000000502023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.749 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.749 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.749 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.749 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.749 DEBUG:boto:Data: 2023-09-29T22:34:33.749 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.750 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.750 DEBUG:boto:Params: {} 2023-09-29T22:34:33.750 DEBUG:boto:Token: None 2023-09-29T22:34:33.750 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.750 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mMo0wUG83KKbOAxFmI4YEcdElDM= 2023-09-29T22:34:33.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mMo0wUG83KKbOAxFmI4YEcdElDM=', 'Content-Length': '0'} 2023-09-29T22:34:33.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8fcd740bb3ffb43-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.753 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.753 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.753 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.753 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.753 DEBUG:boto:Data: 2023-09-29T22:34:33.753 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.753 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.754 DEBUG:boto:Params: {} 2023-09-29T22:34:33.754 DEBUG:boto:Token: None 2023-09-29T22:34:33.754 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.754 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mMo0wUG83KKbOAxFmI4YEcdElDM= 2023-09-29T22:34:33.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mMo0wUG83KKbOAxFmI4YEcdElDM=', 'Content-Length': '0'} 2023-09-29T22:34:33.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b28641a44e73f23-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000050 2023-09-29T22:34:33.756 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.756 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.757 DEBUG:boto:Method: GET 2023-09-29T22:34:33.757 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.757 DEBUG:boto:Data: 2023-09-29T22:34:33.757 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.757 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.757 DEBUG:boto:Params: {} 2023-09-29T22:34:33.757 DEBUG:boto:Token: None 2023-09-29T22:34:33.757 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.758 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1suMSc3bWWtFpvcbKSTIMkLiDM4= 2023-09-29T22:34:33.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1suMSc3bWWtFpvcbKSTIMkLiDM4=', 'Content-Length': '0'} 2023-09-29T22:34:33.760 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a5c6237e4f97610-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.760 DEBUG:boto:path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.760 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.760 DEBUG:boto:Method: GET 2023-09-29T22:34:33.760 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.760 DEBUG:boto:Data: 2023-09-29T22:34:33.761 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.761 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.761 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.761 DEBUG:boto:Params: {} 2023-09-29T22:34:33.761 DEBUG:boto:Token: None 2023-09-29T22:34:33.761 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000050 2023-09-29T22:34:33.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1suMSc3bWWtFpvcbKSTIMkLiDM4= 2023-09-29T22:34:33.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1suMSc3bWWtFpvcbKSTIMkLiDM4=', 'Content-Length': '0'} 2023-09-29T22:34:33.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2e108cd5c907893-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.806 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.807 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.807 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.807 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.807 DEBUG:boto:Method: GET 2023-09-29T22:34:33.807 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.807 DEBUG:boto:Data: 2023-09-29T22:34:33.807 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.807 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.807 DEBUG:boto:Params: {} 2023-09-29T22:34:33.808 DEBUG:boto:Token: None 2023-09-29T22:34:33.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.811 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070ae9e7e6d9647e4-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.811 DEBUG:boto:b'hcnphz-58obj000000000000000000501000falseobj000000000000000000502023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.812 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.812 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.812 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.812 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.812 DEBUG:boto:Method: GET 2023-09-29T22:34:33.812 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000050 2023-09-29T22:34:33.812 DEBUG:boto:Data: 2023-09-29T22:34:33.813 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.813 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.813 DEBUG:boto:Params: {} 2023-09-29T22:34:33.813 DEBUG:boto:Token: None 2023-09-29T22:34:33.813 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000875ea636906c45a6-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.821 DEBUG:boto:b'hcnphz-58obj000000000000000000501000falseobj000000000000000000502023-09-29T22:34:14.753Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-09-29T22:34:33.821 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.821 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.821 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.822 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.822 DEBUG:boto:Method: GET 2023-09-29T22:34:33.822 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.822 DEBUG:boto:Data: 2023-09-29T22:34:33.822 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.822 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.823 DEBUG:boto:Params: {} 2023-09-29T22:34:33.823 DEBUG:boto:Token: None 2023-09-29T22:34:33.823 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI= 2023-09-29T22:34:33.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI=', 'Content-Length': '0'} 2023-09-29T22:34:33.825 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9578c93c041de9e-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.870 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.871 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.871 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.871 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.871 DEBUG:boto:Method: GET 2023-09-29T22:34:33.871 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.871 DEBUG:boto:Data: 2023-09-29T22:34:33.871 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.872 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.872 DEBUG:boto:Params: {} 2023-09-29T22:34:33.872 DEBUG:boto:Token: None 2023-09-29T22:34:33.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI= 2023-09-29T22:34:33.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI=', 'Content-Length': '0'} 2023-09-29T22:34:33.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a590c675f16a7720-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.876 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.876 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.876 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.876 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.876 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.876 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.876 DEBUG:boto:Data: 2023-09-29T22:34:33.876 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.877 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.877 DEBUG:boto:Params: {} 2023-09-29T22:34:33.877 DEBUG:boto:Token: None 2023-09-29T22:34:33.877 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MxJh0acX9wipnb8MeMJ4BgMEVno= 2023-09-29T22:34:33.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MxJh0acX9wipnb8MeMJ4BgMEVno=', 'Content-Length': '0'} 2023-09-29T22:34:33.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f90b9e546f64d384-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.880 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.881 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.881 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.881 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.881 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.881 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.881 DEBUG:boto:Data: 2023-09-29T22:34:33.881 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.881 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.881 DEBUG:boto:Params: {} 2023-09-29T22:34:33.882 DEBUG:boto:Token: None 2023-09-29T22:34:33.882 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.882 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MxJh0acX9wipnb8MeMJ4BgMEVno= 2023-09-29T22:34:33.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MxJh0acX9wipnb8MeMJ4BgMEVno=', 'Content-Length': '0'} 2023-09-29T22:34:33.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b716bb5b8e74c3f0-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-09-29T22:34:33.885 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.885 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.886 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.886 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.886 DEBUG:boto:Method: GET 2023-09-29T22:34:33.886 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.886 DEBUG:boto:Data: 2023-09-29T22:34:33.887 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.887 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.887 DEBUG:boto:Params: {} 2023-09-29T22:34:33.887 DEBUG:boto:Token: None 2023-09-29T22:34:33.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI= 2023-09-29T22:34:33.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI=', 'Content-Length': '0'} 2023-09-29T22:34:33.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6ae831ffb8f0601-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.891 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.891 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.892 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.892 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.892 DEBUG:boto:Method: GET 2023-09-29T22:34:33.892 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.892 DEBUG:boto:Data: 2023-09-29T22:34:33.893 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.893 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.893 DEBUG:boto:Params: {} 2023-09-29T22:34:33.893 DEBUG:boto:Token: None 2023-09-29T22:34:33.894 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053?versionId=null 2023-09-29T22:34:33.894 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI= 2023-09-29T22:34:33.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jV/EAdhikA+IWpDEQTM/iX7TanI=', 'Content-Length': '0'} 2023-09-29T22:34:33.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094702b94cebc3bdf-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.897 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.897 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.898 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:33.898 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:33.898 DEBUG:boto:Method: GET 2023-09-29T22:34:33.898 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:33.899 DEBUG:boto:Data: 2023-09-29T22:34:33.899 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.899 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.899 DEBUG:boto:Params: {} 2023-09-29T22:34:33.899 DEBUG:boto:Token: None 2023-09-29T22:34:33.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.905 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096e3f61044d1f896-00651750f9-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.950 DEBUG:boto:b'hcnphz-58obj000000000000000000531000falseobj000000000000000000532023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.951 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:33.951 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:33.951 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:33.951 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:33.952 DEBUG:boto:Method: GET 2023-09-29T22:34:33.952 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:33.952 DEBUG:boto:Data: 2023-09-29T22:34:33.952 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.952 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.952 DEBUG:boto:Params: {} 2023-09-29T22:34:33.952 DEBUG:boto:Token: None 2023-09-29T22:34:33.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/ 2023-09-29T22:34:33.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU= 2023-09-29T22:34:33.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIgwaldbEox/5OePbYgVu0iUydU=', 'Content-Length': '0'} 2023-09-29T22:34:33.962 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea193c71153d0dec-00651750f9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.962 DEBUG:boto:b'hcnphz-58obj000000000000000000531000falseobj000000000000000000532023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:33.962 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.962 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.963 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.963 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.963 DEBUG:boto:Data: 2023-09-29T22:34:33.963 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.963 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.963 DEBUG:boto:Params: {} 2023-09-29T22:34:33.963 DEBUG:boto:Token: None 2023-09-29T22:34:33.964 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kAPrkfmpX32bMMHKoRbTklvAZgs= 2023-09-29T22:34:33.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kAPrkfmpX32bMMHKoRbTklvAZgs=', 'Content-Length': '0'} 2023-09-29T22:34:33.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591a1cf09e5d12a8-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.966 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.967 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.967 DEBUG:boto:Method: HEAD 2023-09-29T22:34:33.967 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.967 DEBUG:boto:Data: 2023-09-29T22:34:33.967 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.967 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.967 DEBUG:boto:Params: {} 2023-09-29T22:34:33.967 DEBUG:boto:Token: None 2023-09-29T22:34:33.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kAPrkfmpX32bMMHKoRbTklvAZgs= 2023-09-29T22:34:33.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kAPrkfmpX32bMMHKoRbTklvAZgs=', 'Content-Length': '0'} 2023-09-29T22:34:33.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000291cd146a16c3f58-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000053 2023-09-29T22:34:33.971 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.971 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.971 DEBUG:boto:Method: GET 2023-09-29T22:34:33.971 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.971 DEBUG:boto:Data: 2023-09-29T22:34:33.972 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:33.972 DEBUG:boto:Port: 8001 2023-09-29T22:34:33.972 DEBUG:boto:Params: {} 2023-09-29T22:34:33.972 DEBUG:boto:Token: None 2023-09-29T22:34:33.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/pp8NAWiDfFUeYEbxjT9SOErJqo= 2023-09-29T22:34:33.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/pp8NAWiDfFUeYEbxjT9SOErJqo=', 'Content-Length': '0'} 2023-09-29T22:34:33.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035863d2baa1beceb-00651750f9-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:33.974 DEBUG:boto:path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.975 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.975 DEBUG:boto:Method: GET 2023-09-29T22:34:33.975 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.975 DEBUG:boto:Data: 2023-09-29T22:34:33.975 DEBUG:boto:Headers: {} 2023-09-29T22:34:33.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:33.975 DEBUG:boto:Port: 8000 2023-09-29T22:34:33.975 DEBUG:boto:Params: {} 2023-09-29T22:34:33.975 DEBUG:boto:Token: None 2023-09-29T22:34:33.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:33 GMT /hcnphz-58/obj00000000000000000053 2023-09-29T22:34:33.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/pp8NAWiDfFUeYEbxjT9SOErJqo= 2023-09-29T22:34:33.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/pp8NAWiDfFUeYEbxjT9SOErJqo=', 'Content-Length': '0'} 2023-09-29T22:34:33.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff70997a752f6f8-00651750f9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.018 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.019 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.019 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:34.019 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:34.019 DEBUG:boto:Method: GET 2023-09-29T22:34:34.021 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:34.021 DEBUG:boto:Data: 2023-09-29T22:34:34.021 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.021 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.022 DEBUG:boto:Params: {} 2023-09-29T22:34:34.022 DEBUG:boto:Token: None 2023-09-29T22:34:34.022 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2601b215de4767-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.027 DEBUG:boto:b'hcnphz-58obj000000000000000000531000falseobj000000000000000000532023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.028 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.028 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.028 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:34.028 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:34.028 DEBUG:boto:Method: GET 2023-09-29T22:34:34.028 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000053 2023-09-29T22:34:34.028 DEBUG:boto:Data: 2023-09-29T22:34:34.029 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.029 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.029 DEBUG:boto:Params: {} 2023-09-29T22:34:34.029 DEBUG:boto:Token: None 2023-09-29T22:34:34.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b63a5942888d9cd4-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.038 DEBUG:boto:b'hcnphz-58obj000000000000000000531000falseobj000000000000000000532023-09-29T22:34:16.418Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-09-29T22:34:34.038 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.038 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.038 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.039 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.039 DEBUG:boto:Method: GET 2023-09-29T22:34:34.039 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.039 DEBUG:boto:Data: 2023-09-29T22:34:34.039 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.039 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.039 DEBUG:boto:Params: {} 2023-09-29T22:34:34.039 DEBUG:boto:Token: None 2023-09-29T22:34:34.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw= 2023-09-29T22:34:34.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw=', 'Content-Length': '0'} 2023-09-29T22:34:34.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8763f8354b27781-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.086 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.087 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.087 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.087 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.087 DEBUG:boto:Method: GET 2023-09-29T22:34:34.087 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.087 DEBUG:boto:Data: 2023-09-29T22:34:34.087 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.087 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.088 DEBUG:boto:Params: {} 2023-09-29T22:34:34.088 DEBUG:boto:Token: None 2023-09-29T22:34:34.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw= 2023-09-29T22:34:34.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw=', 'Content-Length': '0'} 2023-09-29T22:34:34.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ffd536400e80205-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.092 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.092 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.092 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.092 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.092 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.092 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.092 DEBUG:boto:Data: 2023-09-29T22:34:34.093 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.093 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.093 DEBUG:boto:Params: {} 2023-09-29T22:34:34.093 DEBUG:boto:Token: None 2023-09-29T22:34:34.093 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NpcLe99KlNiLCPOQeA0K2WQTZAQ= 2023-09-29T22:34:34.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NpcLe99KlNiLCPOQeA0K2WQTZAQ=', 'Content-Length': '0'} 2023-09-29T22:34:34.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000592b0692cad00b7-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.096 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.096 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.096 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.096 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.097 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.097 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.097 DEBUG:boto:Data: 2023-09-29T22:34:34.097 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.097 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.097 DEBUG:boto:Params: {} 2023-09-29T22:34:34.097 DEBUG:boto:Token: None 2023-09-29T22:34:34.097 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NpcLe99KlNiLCPOQeA0K2WQTZAQ= 2023-09-29T22:34:34.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NpcLe99KlNiLCPOQeA0K2WQTZAQ=', 'Content-Length': '0'} 2023-09-29T22:34:34.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb94057817995dfe-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-09-29T22:34:34.102 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.102 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.102 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.102 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.103 DEBUG:boto:Method: GET 2023-09-29T22:34:34.103 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.103 DEBUG:boto:Data: 2023-09-29T22:34:34.103 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.103 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.103 DEBUG:boto:Params: {} 2023-09-29T22:34:34.103 DEBUG:boto:Token: None 2023-09-29T22:34:34.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw= 2023-09-29T22:34:34.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw=', 'Content-Length': '0'} 2023-09-29T22:34:34.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035df32cfc985daae-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.107 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.107 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.107 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.107 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.107 DEBUG:boto:Method: GET 2023-09-29T22:34:34.107 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.108 DEBUG:boto:Data: 2023-09-29T22:34:34.108 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.108 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.108 DEBUG:boto:Params: {} 2023-09-29T22:34:34.108 DEBUG:boto:Token: None 2023-09-29T22:34:34.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055?versionId=null 2023-09-29T22:34:34.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw= 2023-09-29T22:34:34.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b2n7FMy9/XUHk86RYfPH4wpA3bw=', 'Content-Length': '0'} 2023-09-29T22:34:34.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea56c794a53a70ee-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.111 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.111 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.111 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.111 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.111 DEBUG:boto:Method: GET 2023-09-29T22:34:34.111 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.112 DEBUG:boto:Data: 2023-09-29T22:34:34.112 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.112 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.112 DEBUG:boto:Params: {} 2023-09-29T22:34:34.112 DEBUG:boto:Token: None 2023-09-29T22:34:34.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.115 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006804b8fd7c08e4b7-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.162 DEBUG:boto:b'hcnphz-58obj000000000000000000551000falseobj000000000000000000552023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.163 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.163 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.163 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.163 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.163 DEBUG:boto:Method: GET 2023-09-29T22:34:34.163 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.164 DEBUG:boto:Data: 2023-09-29T22:34:34.164 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.164 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.164 DEBUG:boto:Params: {} 2023-09-29T22:34:34.164 DEBUG:boto:Token: None 2023-09-29T22:34:34.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041f1291177badc25-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.173 DEBUG:boto:b'hcnphz-58obj000000000000000000551000falseobj000000000000000000552023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.174 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.174 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.174 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.174 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.174 DEBUG:boto:Data: 2023-09-29T22:34:34.174 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.175 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.175 DEBUG:boto:Params: {} 2023-09-29T22:34:34.175 DEBUG:boto:Token: None 2023-09-29T22:34:34.175 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.175 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iZbxvmMHd65fnY5UiPdu4+mv3P4= 2023-09-29T22:34:34.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iZbxvmMHd65fnY5UiPdu4+mv3P4=', 'Content-Length': '0'} 2023-09-29T22:34:34.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b80d449afdf3efc8-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.178 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.178 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.179 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.179 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.179 DEBUG:boto:Data: 2023-09-29T22:34:34.179 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.179 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.179 DEBUG:boto:Params: {} 2023-09-29T22:34:34.179 DEBUG:boto:Token: None 2023-09-29T22:34:34.179 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iZbxvmMHd65fnY5UiPdu4+mv3P4= 2023-09-29T22:34:34.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iZbxvmMHd65fnY5UiPdu4+mv3P4=', 'Content-Length': '0'} 2023-09-29T22:34:34.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000237f5875ed145a2b-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000055 2023-09-29T22:34:34.182 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.182 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.182 DEBUG:boto:Method: GET 2023-09-29T22:34:34.182 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.182 DEBUG:boto:Data: 2023-09-29T22:34:34.182 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.183 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.183 DEBUG:boto:Params: {} 2023-09-29T22:34:34.183 DEBUG:boto:Token: None 2023-09-29T22:34:34.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y7CABHTAVrLj+LXjNzO14rxRIzE= 2023-09-29T22:34:34.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y7CABHTAVrLj+LXjNzO14rxRIzE=', 'Content-Length': '0'} 2023-09-29T22:34:34.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9bfeb924740f42-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.185 DEBUG:boto:path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.185 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.186 DEBUG:boto:Method: GET 2023-09-29T22:34:34.186 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.186 DEBUG:boto:Data: 2023-09-29T22:34:34.186 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.186 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.186 DEBUG:boto:Params: {} 2023-09-29T22:34:34.186 DEBUG:boto:Token: None 2023-09-29T22:34:34.186 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000055 2023-09-29T22:34:34.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y7CABHTAVrLj+LXjNzO14rxRIzE= 2023-09-29T22:34:34.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y7CABHTAVrLj+LXjNzO14rxRIzE=', 'Content-Length': '0'} 2023-09-29T22:34:34.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000702817b36fb96fac-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.230 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.231 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.231 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.231 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.231 DEBUG:boto:Method: GET 2023-09-29T22:34:34.231 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.231 DEBUG:boto:Data: 2023-09-29T22:34:34.231 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.232 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.232 DEBUG:boto:Params: {} 2023-09-29T22:34:34.232 DEBUG:boto:Token: None 2023-09-29T22:34:34.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c67b6bd887e05229-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.238 DEBUG:boto:b'hcnphz-58obj000000000000000000551000falseobj000000000000000000552023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.238 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.238 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.238 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.238 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.239 DEBUG:boto:Method: GET 2023-09-29T22:34:34.239 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000055 2023-09-29T22:34:34.239 DEBUG:boto:Data: 2023-09-29T22:34:34.239 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.239 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.239 DEBUG:boto:Params: {} 2023-09-29T22:34:34.239 DEBUG:boto:Token: None 2023-09-29T22:34:34.239 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.248 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d92fcafebcb499fa-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.248 DEBUG:boto:b'hcnphz-58obj000000000000000000551000falseobj000000000000000000552023-09-29T22:34:11.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-09-29T22:34:34.248 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.248 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.249 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.249 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.249 DEBUG:boto:Method: GET 2023-09-29T22:34:34.249 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.249 DEBUG:boto:Data: 2023-09-29T22:34:34.249 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.249 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.249 DEBUG:boto:Params: {} 2023-09-29T22:34:34.250 DEBUG:boto:Token: None 2023-09-29T22:34:34.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960= 2023-09-29T22:34:34.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960=', 'Content-Length': '0'} 2023-09-29T22:34:34.252 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000116691760415c19b-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.252 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.252 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.253 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.253 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.253 DEBUG:boto:Method: GET 2023-09-29T22:34:34.253 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.253 DEBUG:boto:Data: 2023-09-29T22:34:34.253 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.253 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.253 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.253 DEBUG:boto:Params: {} 2023-09-29T22:34:34.254 DEBUG:boto:Token: None 2023-09-29T22:34:34.254 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.254 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960= 2023-09-29T22:34:34.254 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960=', 'Content-Length': '0'} 2023-09-29T22:34:34.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bbd60adad37bcdf-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.256 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.256 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.256 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.256 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.256 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.257 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.257 DEBUG:boto:Data: 2023-09-29T22:34:34.257 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.257 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.257 DEBUG:boto:Params: {} 2023-09-29T22:34:34.257 DEBUG:boto:Token: None 2023-09-29T22:34:34.257 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U9wPPYOfSha/vuRq3aMGKcHzqkA= 2023-09-29T22:34:34.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U9wPPYOfSha/vuRq3aMGKcHzqkA=', 'Content-Length': '0'} 2023-09-29T22:34:34.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000806ddd3ad7f931af-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.259 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.259 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.260 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.260 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.260 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.260 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.260 DEBUG:boto:Data: 2023-09-29T22:34:34.260 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.260 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.260 DEBUG:boto:Params: {} 2023-09-29T22:34:34.261 DEBUG:boto:Token: None 2023-09-29T22:34:34.261 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U9wPPYOfSha/vuRq3aMGKcHzqkA= 2023-09-29T22:34:34.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U9wPPYOfSha/vuRq3aMGKcHzqkA=', 'Content-Length': '0'} 2023-09-29T22:34:34.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9ef313c2558fce6-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-09-29T22:34:34.263 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.263 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.264 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.264 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.264 DEBUG:boto:Method: GET 2023-09-29T22:34:34.264 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.264 DEBUG:boto:Data: 2023-09-29T22:34:34.264 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.264 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.264 DEBUG:boto:Params: {} 2023-09-29T22:34:34.264 DEBUG:boto:Token: None 2023-09-29T22:34:34.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960= 2023-09-29T22:34:34.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960=', 'Content-Length': '0'} 2023-09-29T22:34:34.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb36704cd9616d5-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.310 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.311 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.311 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.311 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.311 DEBUG:boto:Method: GET 2023-09-29T22:34:34.311 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.312 DEBUG:boto:Data: 2023-09-29T22:34:34.312 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.312 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.312 DEBUG:boto:Params: {} 2023-09-29T22:34:34.312 DEBUG:boto:Token: None 2023-09-29T22:34:34.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056?versionId=null 2023-09-29T22:34:34.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960= 2023-09-29T22:34:34.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BnOgS4a4NNSfWG8Q7H4/YjCS960=', 'Content-Length': '0'} 2023-09-29T22:34:34.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b114130ff5917573-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.316 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.316 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.316 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.316 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.316 DEBUG:boto:Method: GET 2023-09-29T22:34:34.316 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.317 DEBUG:boto:Data: 2023-09-29T22:34:34.317 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.317 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.317 DEBUG:boto:Params: {} 2023-09-29T22:34:34.317 DEBUG:boto:Token: None 2023-09-29T22:34:34.317 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000934351bf044cc0ce-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.322 DEBUG:boto:b'hcnphz-58obj000000000000000000561000falseobj000000000000000000562023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.323 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.323 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.323 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.323 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.323 DEBUG:boto:Method: GET 2023-09-29T22:34:34.323 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.323 DEBUG:boto:Data: 2023-09-29T22:34:34.323 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.323 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.324 DEBUG:boto:Params: {} 2023-09-29T22:34:34.324 DEBUG:boto:Token: None 2023-09-29T22:34:34.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.332 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008fca4f0b96f01adb-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.374 DEBUG:boto:b'hcnphz-58obj000000000000000000561000falseobj000000000000000000562023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.375 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.375 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.375 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.375 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.375 DEBUG:boto:Data: 2023-09-29T22:34:34.375 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.376 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.376 DEBUG:boto:Params: {} 2023-09-29T22:34:34.376 DEBUG:boto:Token: None 2023-09-29T22:34:34.376 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CQ/v9Bc9uxRlW9S/NXKctrJpRXk= 2023-09-29T22:34:34.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CQ/v9Bc9uxRlW9S/NXKctrJpRXk=', 'Content-Length': '0'} 2023-09-29T22:34:34.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef5debff40ed7c5f-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.379 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.380 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.380 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.380 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.380 DEBUG:boto:Data: 2023-09-29T22:34:34.380 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.380 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.380 DEBUG:boto:Params: {} 2023-09-29T22:34:34.380 DEBUG:boto:Token: None 2023-09-29T22:34:34.381 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CQ/v9Bc9uxRlW9S/NXKctrJpRXk= 2023-09-29T22:34:34.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CQ/v9Bc9uxRlW9S/NXKctrJpRXk=', 'Content-Length': '0'} 2023-09-29T22:34:34.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1d46fd09ca2c75d-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000056 2023-09-29T22:34:34.384 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.384 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.384 DEBUG:boto:Method: GET 2023-09-29T22:34:34.384 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.384 DEBUG:boto:Data: 2023-09-29T22:34:34.384 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.384 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.385 DEBUG:boto:Params: {} 2023-09-29T22:34:34.385 DEBUG:boto:Token: None 2023-09-29T22:34:34.385 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UjVqmvTtCoeBq1sfC/T96tiVKjM= 2023-09-29T22:34:34.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UjVqmvTtCoeBq1sfC/T96tiVKjM=', 'Content-Length': '0'} 2023-09-29T22:34:34.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e24a27fa5be83683-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.388 DEBUG:boto:path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.388 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.388 DEBUG:boto:Method: GET 2023-09-29T22:34:34.388 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.388 DEBUG:boto:Data: 2023-09-29T22:34:34.388 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.389 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.389 DEBUG:boto:Params: {} 2023-09-29T22:34:34.389 DEBUG:boto:Token: None 2023-09-29T22:34:34.389 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000056 2023-09-29T22:34:34.389 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UjVqmvTtCoeBq1sfC/T96tiVKjM= 2023-09-29T22:34:34.389 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UjVqmvTtCoeBq1sfC/T96tiVKjM=', 'Content-Length': '0'} 2023-09-29T22:34:34.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000899117c3831c41a9-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.392 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.392 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.393 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.393 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.393 DEBUG:boto:Method: GET 2023-09-29T22:34:34.393 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.393 DEBUG:boto:Data: 2023-09-29T22:34:34.393 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.393 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.393 DEBUG:boto:Params: {} 2023-09-29T22:34:34.393 DEBUG:boto:Token: None 2023-09-29T22:34:34.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.399 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000487d6ef12408b59d-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.442 DEBUG:boto:b'hcnphz-58obj000000000000000000561000falseobj000000000000000000562023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.443 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.443 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.443 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.443 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.443 DEBUG:boto:Method: GET 2023-09-29T22:34:34.443 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000056 2023-09-29T22:34:34.444 DEBUG:boto:Data: 2023-09-29T22:34:34.444 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.444 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.444 DEBUG:boto:Params: {} 2023-09-29T22:34:34.444 DEBUG:boto:Token: None 2023-09-29T22:34:34.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006723cb8f2184b65f-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.454 DEBUG:boto:b'hcnphz-58obj000000000000000000561000falseobj000000000000000000562023-09-29T22:34:14.147Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-09-29T22:34:34.455 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.455 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.455 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.455 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.455 DEBUG:boto:Method: GET 2023-09-29T22:34:34.455 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.455 DEBUG:boto:Data: 2023-09-29T22:34:34.455 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.455 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.456 DEBUG:boto:Params: {} 2023-09-29T22:34:34.456 DEBUG:boto:Token: None 2023-09-29T22:34:34.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc= 2023-09-29T22:34:34.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc=', 'Content-Length': '0'} 2023-09-29T22:34:34.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca2310e5dfcbf450-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.459 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.459 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.459 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.460 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.460 DEBUG:boto:Method: GET 2023-09-29T22:34:34.460 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.460 DEBUG:boto:Data: 2023-09-29T22:34:34.460 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.460 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.460 DEBUG:boto:Params: {} 2023-09-29T22:34:34.460 DEBUG:boto:Token: None 2023-09-29T22:34:34.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc= 2023-09-29T22:34:34.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc=', 'Content-Length': '0'} 2023-09-29T22:34:34.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a55969fc8d3820a-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.506 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.506 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.507 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.507 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.507 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.507 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.507 DEBUG:boto:Data: 2023-09-29T22:34:34.507 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.507 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.507 DEBUG:boto:Params: {} 2023-09-29T22:34:34.508 DEBUG:boto:Token: None 2023-09-29T22:34:34.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sGIMsYL/A9jW8hb/4OLIfFuSZQs= 2023-09-29T22:34:34.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sGIMsYL/A9jW8hb/4OLIfFuSZQs=', 'Content-Length': '0'} 2023-09-29T22:34:34.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5993547c490bacf-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.511 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.511 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.511 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.512 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.512 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.512 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.512 DEBUG:boto:Data: 2023-09-29T22:34:34.512 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.512 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.512 DEBUG:boto:Params: {} 2023-09-29T22:34:34.512 DEBUG:boto:Token: None 2023-09-29T22:34:34.513 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sGIMsYL/A9jW8hb/4OLIfFuSZQs= 2023-09-29T22:34:34.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sGIMsYL/A9jW8hb/4OLIfFuSZQs=', 'Content-Length': '0'} 2023-09-29T22:34:34.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009705aeb6900e4ab4-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-09-29T22:34:34.516 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.516 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.516 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.516 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.517 DEBUG:boto:Method: GET 2023-09-29T22:34:34.517 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.517 DEBUG:boto:Data: 2023-09-29T22:34:34.517 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.517 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.517 DEBUG:boto:Params: {} 2023-09-29T22:34:34.517 DEBUG:boto:Token: None 2023-09-29T22:34:34.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.518 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc= 2023-09-29T22:34:34.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc=', 'Content-Length': '0'} 2023-09-29T22:34:34.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2fcf4f30e7ae235-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.520 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.521 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.521 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.521 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.521 DEBUG:boto:Method: GET 2023-09-29T22:34:34.521 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.521 DEBUG:boto:Data: 2023-09-29T22:34:34.521 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.521 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.522 DEBUG:boto:Params: {} 2023-09-29T22:34:34.522 DEBUG:boto:Token: None 2023-09-29T22:34:34.522 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057?versionId=null 2023-09-29T22:34:34.522 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc= 2023-09-29T22:34:34.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/NNZ+fHaaWrdGl+hRI67wwEXuPc=', 'Content-Length': '0'} 2023-09-29T22:34:34.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bdb1e2ccdc82ea4-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.525 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.525 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.525 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.525 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.526 DEBUG:boto:Method: GET 2023-09-29T22:34:34.526 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.526 DEBUG:boto:Data: 2023-09-29T22:34:34.526 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.526 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.526 DEBUG:boto:Params: {} 2023-09-29T22:34:34.526 DEBUG:boto:Token: None 2023-09-29T22:34:34.526 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f76a851606e88a47-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.578 DEBUG:boto:b'hcnphz-58obj000000000000000000571000falseobj000000000000000000572023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.579 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.579 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.579 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.579 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.579 DEBUG:boto:Method: GET 2023-09-29T22:34:34.580 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.580 DEBUG:boto:Data: 2023-09-29T22:34:34.580 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.580 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.580 DEBUG:boto:Params: {} 2023-09-29T22:34:34.580 DEBUG:boto:Token: None 2023-09-29T22:34:34.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d8ec128181bef2d-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.590 DEBUG:boto:b'hcnphz-58obj000000000000000000571000falseobj000000000000000000572023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.591 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.591 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.591 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.591 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.591 DEBUG:boto:Data: 2023-09-29T22:34:34.591 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.592 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.592 DEBUG:boto:Params: {} 2023-09-29T22:34:34.592 DEBUG:boto:Token: None 2023-09-29T22:34:34.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WV4mpk6/YirpblTOta+r56kLSH4= 2023-09-29T22:34:34.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WV4mpk6/YirpblTOta+r56kLSH4=', 'Content-Length': '0'} 2023-09-29T22:34:34.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4729b3c4e94a1dd-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.595 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.595 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.595 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.595 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.595 DEBUG:boto:Data: 2023-09-29T22:34:34.596 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.596 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.596 DEBUG:boto:Params: {} 2023-09-29T22:34:34.596 DEBUG:boto:Token: None 2023-09-29T22:34:34.596 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WV4mpk6/YirpblTOta+r56kLSH4= 2023-09-29T22:34:34.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WV4mpk6/YirpblTOta+r56kLSH4=', 'Content-Length': '0'} 2023-09-29T22:34:34.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab8cf542b842a9e0-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000057 2023-09-29T22:34:34.599 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.599 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.599 DEBUG:boto:Method: GET 2023-09-29T22:34:34.599 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.599 DEBUG:boto:Data: 2023-09-29T22:34:34.599 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.600 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.600 DEBUG:boto:Params: {} 2023-09-29T22:34:34.600 DEBUG:boto:Token: None 2023-09-29T22:34:34.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kh7iuv8RGfg/CmQGRI/eX6bpYKo= 2023-09-29T22:34:34.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kh7iuv8RGfg/CmQGRI/eX6bpYKo=', 'Content-Length': '0'} 2023-09-29T22:34:34.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000755133ac76ebf5f1-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.602 DEBUG:boto:path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.602 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.603 DEBUG:boto:Method: GET 2023-09-29T22:34:34.603 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.603 DEBUG:boto:Data: 2023-09-29T22:34:34.603 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.603 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.603 DEBUG:boto:Params: {} 2023-09-29T22:34:34.603 DEBUG:boto:Token: None 2023-09-29T22:34:34.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000057 2023-09-29T22:34:34.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kh7iuv8RGfg/CmQGRI/eX6bpYKo= 2023-09-29T22:34:34.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kh7iuv8RGfg/CmQGRI/eX6bpYKo=', 'Content-Length': '0'} 2023-09-29T22:34:34.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004563301dd04c907-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.646 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.647 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.647 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.647 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.647 DEBUG:boto:Method: GET 2023-09-29T22:34:34.647 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.647 DEBUG:boto:Data: 2023-09-29T22:34:34.647 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.648 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.648 DEBUG:boto:Params: {} 2023-09-29T22:34:34.648 DEBUG:boto:Token: None 2023-09-29T22:34:34.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0c053a66268d20f-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.654 DEBUG:boto:b'hcnphz-58obj000000000000000000571000falseobj000000000000000000572023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.654 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.654 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.655 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.655 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.655 DEBUG:boto:Method: GET 2023-09-29T22:34:34.655 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000057 2023-09-29T22:34:34.655 DEBUG:boto:Data: 2023-09-29T22:34:34.655 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.655 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.655 DEBUG:boto:Params: {} 2023-09-29T22:34:34.656 DEBUG:boto:Token: None 2023-09-29T22:34:34.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eb87f0fa259f59fe-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.664 DEBUG:boto:b'hcnphz-58obj000000000000000000571000falseobj000000000000000000572023-09-29T22:34:15.505Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-09-29T22:34:34.665 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.665 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.665 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.665 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.665 DEBUG:boto:Method: GET 2023-09-29T22:34:34.665 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.666 DEBUG:boto:Data: 2023-09-29T22:34:34.666 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.666 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.666 DEBUG:boto:Params: {} 2023-09-29T22:34:34.666 DEBUG:boto:Token: None 2023-09-29T22:34:34.666 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.667 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw= 2023-09-29T22:34:34.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw=', 'Content-Length': '0'} 2023-09-29T22:34:34.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa1feaa4824ebcbe-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.714 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.715 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.715 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.715 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.715 DEBUG:boto:Method: GET 2023-09-29T22:34:34.715 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.715 DEBUG:boto:Data: 2023-09-29T22:34:34.715 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.716 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.716 DEBUG:boto:Params: {} 2023-09-29T22:34:34.716 DEBUG:boto:Token: None 2023-09-29T22:34:34.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw= 2023-09-29T22:34:34.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw=', 'Content-Length': '0'} 2023-09-29T22:34:34.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d561325e6d5868-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.720 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.720 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.720 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.720 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.720 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.720 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.720 DEBUG:boto:Data: 2023-09-29T22:34:34.721 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.721 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.721 DEBUG:boto:Params: {} 2023-09-29T22:34:34.721 DEBUG:boto:Token: None 2023-09-29T22:34:34.721 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S0mhwflFqa/Z6vDyP6PlYvKYFHs= 2023-09-29T22:34:34.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S0mhwflFqa/Z6vDyP6PlYvKYFHs=', 'Content-Length': '0'} 2023-09-29T22:34:34.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fd395bb78caaa2e-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.724 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.725 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.725 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.725 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.725 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.725 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.725 DEBUG:boto:Data: 2023-09-29T22:34:34.725 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.726 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.726 DEBUG:boto:Params: {} 2023-09-29T22:34:34.726 DEBUG:boto:Token: None 2023-09-29T22:34:34.726 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S0mhwflFqa/Z6vDyP6PlYvKYFHs= 2023-09-29T22:34:34.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S0mhwflFqa/Z6vDyP6PlYvKYFHs=', 'Content-Length': '0'} 2023-09-29T22:34:34.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f848678b51ee21a2-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-09-29T22:34:34.729 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.729 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.729 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.729 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.730 DEBUG:boto:Method: GET 2023-09-29T22:34:34.730 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.730 DEBUG:boto:Data: 2023-09-29T22:34:34.730 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.730 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.730 DEBUG:boto:Params: {} 2023-09-29T22:34:34.730 DEBUG:boto:Token: None 2023-09-29T22:34:34.731 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.731 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw= 2023-09-29T22:34:34.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw=', 'Content-Length': '0'} 2023-09-29T22:34:34.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094e73b96de57e2ed-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.734 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.734 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.734 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.734 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.735 DEBUG:boto:Method: GET 2023-09-29T22:34:34.735 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.735 DEBUG:boto:Data: 2023-09-29T22:34:34.735 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.735 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.735 DEBUG:boto:Params: {} 2023-09-29T22:34:34.735 DEBUG:boto:Token: None 2023-09-29T22:34:34.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058?versionId=null 2023-09-29T22:34:34.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw= 2023-09-29T22:34:34.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7oerCMqpzH8+u6o4ilR6unPSRbw=', 'Content-Length': '0'} 2023-09-29T22:34:34.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020894e81aadd9a9a-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.739 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.739 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.739 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.739 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.739 DEBUG:boto:Method: GET 2023-09-29T22:34:34.739 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.740 DEBUG:boto:Data: 2023-09-29T22:34:34.740 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.740 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.740 DEBUG:boto:Params: {} 2023-09-29T22:34:34.740 DEBUG:boto:Token: None 2023-09-29T22:34:34.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.741 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003862816434b091af-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.746 DEBUG:boto:b'hcnphz-58obj000000000000000000581000falseobj000000000000000000582023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.746 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.746 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.747 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.747 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.747 DEBUG:boto:Method: GET 2023-09-29T22:34:34.747 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.747 DEBUG:boto:Data: 2023-09-29T22:34:34.747 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.747 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.747 DEBUG:boto:Params: {} 2023-09-29T22:34:34.748 DEBUG:boto:Token: None 2023-09-29T22:34:34.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.755 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005196ed0bd0ef03c1-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.802 DEBUG:boto:b'hcnphz-58obj000000000000000000581000falseobj000000000000000000582023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.803 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.803 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.803 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.803 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.803 DEBUG:boto:Data: 2023-09-29T22:34:34.804 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.804 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.804 DEBUG:boto:Params: {} 2023-09-29T22:34:34.804 DEBUG:boto:Token: None 2023-09-29T22:34:34.804 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CaZy0qoXDz7czGSrEHcPVwGD3Cg= 2023-09-29T22:34:34.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CaZy0qoXDz7czGSrEHcPVwGD3Cg=', 'Content-Length': '0'} 2023-09-29T22:34:34.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab6e6a698e617c07-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.808 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.808 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.808 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.808 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.808 DEBUG:boto:Data: 2023-09-29T22:34:34.809 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.809 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.809 DEBUG:boto:Params: {} 2023-09-29T22:34:34.809 DEBUG:boto:Token: None 2023-09-29T22:34:34.809 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CaZy0qoXDz7czGSrEHcPVwGD3Cg= 2023-09-29T22:34:34.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CaZy0qoXDz7czGSrEHcPVwGD3Cg=', 'Content-Length': '0'} 2023-09-29T22:34:34.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004964b91bc1d92e9-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000058 2023-09-29T22:34:34.813 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.813 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.813 DEBUG:boto:Method: GET 2023-09-29T22:34:34.813 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.813 DEBUG:boto:Data: 2023-09-29T22:34:34.813 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.813 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.813 DEBUG:boto:Params: {} 2023-09-29T22:34:34.814 DEBUG:boto:Token: None 2023-09-29T22:34:34.814 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.814 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9SbTrtfEQo87C59z5if6P7AH2A8= 2023-09-29T22:34:34.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9SbTrtfEQo87C59z5if6P7AH2A8=', 'Content-Length': '0'} 2023-09-29T22:34:34.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab2276121a6413ee-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.859 DEBUG:boto:path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.860 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.860 DEBUG:boto:Method: GET 2023-09-29T22:34:34.860 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.860 DEBUG:boto:Data: 2023-09-29T22:34:34.860 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.861 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.861 DEBUG:boto:Params: {} 2023-09-29T22:34:34.861 DEBUG:boto:Token: None 2023-09-29T22:34:34.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000058 2023-09-29T22:34:34.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9SbTrtfEQo87C59z5if6P7AH2A8= 2023-09-29T22:34:34.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9SbTrtfEQo87C59z5if6P7AH2A8=', 'Content-Length': '0'} 2023-09-29T22:34:34.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:11 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf65f86b019b6171-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.864 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.864 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.865 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.865 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.865 DEBUG:boto:Method: GET 2023-09-29T22:34:34.865 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.865 DEBUG:boto:Data: 2023-09-29T22:34:34.865 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.865 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.865 DEBUG:boto:Params: {} 2023-09-29T22:34:34.866 DEBUG:boto:Token: None 2023-09-29T22:34:34.866 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.866 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055bb392a166062ac-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.871 DEBUG:boto:b'hcnphz-58obj000000000000000000581000falseobj000000000000000000582023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.871 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.871 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.871 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.872 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.872 DEBUG:boto:Method: GET 2023-09-29T22:34:34.872 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000058 2023-09-29T22:34:34.872 DEBUG:boto:Data: 2023-09-29T22:34:34.872 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.872 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.872 DEBUG:boto:Params: {} 2023-09-29T22:34:34.872 DEBUG:boto:Token: None 2023-09-29T22:34:34.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.873 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:34.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae9dbaca30301cb0-00651750fa-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.882 DEBUG:boto:b'hcnphz-58obj000000000000000000581000falseobj000000000000000000582023-09-29T22:34:11.934Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:34.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-09-29T22:34:34.883 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.883 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.883 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.883 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.883 DEBUG:boto:Method: GET 2023-09-29T22:34:34.883 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.884 DEBUG:boto:Data: 2023-09-29T22:34:34.884 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.884 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.884 DEBUG:boto:Params: {} 2023-09-29T22:34:34.884 DEBUG:boto:Token: None 2023-09-29T22:34:34.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek= 2023-09-29T22:34:34.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek=', 'Content-Length': '0'} 2023-09-29T22:34:34.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbfd2f5fe671a65c-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.887 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.887 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.887 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.887 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.887 DEBUG:boto:Method: GET 2023-09-29T22:34:34.888 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.888 DEBUG:boto:Data: 2023-09-29T22:34:34.888 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.888 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.888 DEBUG:boto:Params: {} 2023-09-29T22:34:34.888 DEBUG:boto:Token: None 2023-09-29T22:34:34.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek= 2023-09-29T22:34:34.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek=', 'Content-Length': '0'} 2023-09-29T22:34:34.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac7f0020d9cc3b2-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.934 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.935 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.935 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.935 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.935 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.935 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.935 DEBUG:boto:Data: 2023-09-29T22:34:34.935 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.935 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.936 DEBUG:boto:Params: {} 2023-09-29T22:34:34.936 DEBUG:boto:Token: None 2023-09-29T22:34:34.936 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3wwmZu/5bbitb6P8b/1hr2eZ1aw= 2023-09-29T22:34:34.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wwmZu/5bbitb6P8b/1hr2eZ1aw=', 'Content-Length': '0'} 2023-09-29T22:34:34.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e63232043a1e3f69-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.940 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.940 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.940 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.940 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.940 DEBUG:boto:Method: HEAD 2023-09-29T22:34:34.940 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.940 DEBUG:boto:Data: 2023-09-29T22:34:34.940 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.941 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.941 DEBUG:boto:Params: {} 2023-09-29T22:34:34.941 DEBUG:boto:Token: None 2023-09-29T22:34:34.941 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3wwmZu/5bbitb6P8b/1hr2eZ1aw= 2023-09-29T22:34:34.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wwmZu/5bbitb6P8b/1hr2eZ1aw=', 'Content-Length': '0'} 2023-09-29T22:34:34.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa166155be7440e-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-09-29T22:34:34.944 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.945 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.945 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.945 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.945 DEBUG:boto:Method: GET 2023-09-29T22:34:34.945 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.945 DEBUG:boto:Data: 2023-09-29T22:34:34.945 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.945 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.945 DEBUG:boto:Params: {} 2023-09-29T22:34:34.946 DEBUG:boto:Token: None 2023-09-29T22:34:34.946 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.946 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek= 2023-09-29T22:34:34.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek=', 'Content-Length': '0'} 2023-09-29T22:34:34.949 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001269539eb83cdd7f-00651750fa-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.990 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.990 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:34.991 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.991 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.991 DEBUG:boto:Method: GET 2023-09-29T22:34:34.991 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.992 DEBUG:boto:Data: 2023-09-29T22:34:34.992 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:34.992 DEBUG:boto:Port: 8000 2023-09-29T22:34:34.992 DEBUG:boto:Params: {} 2023-09-29T22:34:34.992 DEBUG:boto:Token: None 2023-09-29T22:34:34.993 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/obj00000000000000000059?versionId=null 2023-09-29T22:34:34.993 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek= 2023-09-29T22:34:34.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yQZVtU+10TMkuYXTUIF85CeC6Ek=', 'Content-Length': '0'} 2023-09-29T22:34:34.996 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c922a04edc0e22a3-00651750fa-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:34.996 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:34.997 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:34.997 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:34.997 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:34.997 DEBUG:boto:Method: GET 2023-09-29T22:34:34.997 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:34.997 DEBUG:boto:Data: 2023-09-29T22:34:34.998 DEBUG:boto:Headers: {} 2023-09-29T22:34:34.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:34.998 DEBUG:boto:Port: 8001 2023-09-29T22:34:34.998 DEBUG:boto:Params: {} 2023-09-29T22:34:34.999 DEBUG:boto:Token: None 2023-09-29T22:34:34.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:34 GMT /hcnphz-58/ 2023-09-29T22:34:34.999 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0= 2023-09-29T22:34:34.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k9EEJKHOoyqvBKZsoIazVUxc0q0=', 'Content-Length': '0'} 2023-09-29T22:34:35.004 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009257b6d62fecbd82-00651750fa-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.005 DEBUG:boto:b'hcnphz-58obj000000000000000000591000falseobj000000000000000000592023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.005 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.005 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.006 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.006 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.006 DEBUG:boto:Method: GET 2023-09-29T22:34:35.006 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.006 DEBUG:boto:Data: 2023-09-29T22:34:35.006 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.006 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.006 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.007 DEBUG:boto:Params: {} 2023-09-29T22:34:35.007 DEBUG:boto:Token: None 2023-09-29T22:34:35.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e8386ba0ff663fdf-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.015 DEBUG:boto:b'hcnphz-58obj000000000000000000591000falseobj000000000000000000592023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.015 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.016 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.016 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.016 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.016 DEBUG:boto:Data: 2023-09-29T22:34:35.016 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.016 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.016 DEBUG:boto:Params: {} 2023-09-29T22:34:35.016 DEBUG:boto:Token: None 2023-09-29T22:34:35.017 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nEAmIS5Re82OmVKyM5RIj3PK2C8= 2023-09-29T22:34:35.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nEAmIS5Re82OmVKyM5RIj3PK2C8=', 'Content-Length': '0'} 2023-09-29T22:34:35.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ad6a77319bde5a9-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.019 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.019 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.019 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.019 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.020 DEBUG:boto:Data: 2023-09-29T22:34:35.020 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.020 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.020 DEBUG:boto:Params: {} 2023-09-29T22:34:35.020 DEBUG:boto:Token: None 2023-09-29T22:34:35.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nEAmIS5Re82OmVKyM5RIj3PK2C8= 2023-09-29T22:34:35.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nEAmIS5Re82OmVKyM5RIj3PK2C8=', 'Content-Length': '0'} 2023-09-29T22:34:35.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de7cf1bdbfd58fa0-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000059 2023-09-29T22:34:35.023 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.023 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.023 DEBUG:boto:Method: GET 2023-09-29T22:34:35.023 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.023 DEBUG:boto:Data: 2023-09-29T22:34:35.023 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.024 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.024 DEBUG:boto:Params: {} 2023-09-29T22:34:35.024 DEBUG:boto:Token: None 2023-09-29T22:34:35.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jvHJZe2jVTHtdMHXCB9ZZBzag90= 2023-09-29T22:34:35.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jvHJZe2jVTHtdMHXCB9ZZBzag90=', 'Content-Length': '0'} 2023-09-29T22:34:35.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f59769177896e6c-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.074 DEBUG:boto:path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.075 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.075 DEBUG:boto:Method: GET 2023-09-29T22:34:35.075 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.075 DEBUG:boto:Data: 2023-09-29T22:34:35.075 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.075 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.076 DEBUG:boto:Params: {} 2023-09-29T22:34:35.076 DEBUG:boto:Token: None 2023-09-29T22:34:35.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000059 2023-09-29T22:34:35.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jvHJZe2jVTHtdMHXCB9ZZBzag90= 2023-09-29T22:34:35.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jvHJZe2jVTHtdMHXCB9ZZBzag90=', 'Content-Length': '0'} 2023-09-29T22:34:35.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a91bf3da304345-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.079 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.080 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.080 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.080 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.080 DEBUG:boto:Method: GET 2023-09-29T22:34:35.080 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.080 DEBUG:boto:Data: 2023-09-29T22:34:35.080 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.081 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.081 DEBUG:boto:Params: {} 2023-09-29T22:34:35.081 DEBUG:boto:Token: None 2023-09-29T22:34:35.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000086a5f7b759be958d-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.086 DEBUG:boto:b'hcnphz-58obj000000000000000000591000falseobj000000000000000000592023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.086 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.086 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.086 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.087 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.087 DEBUG:boto:Method: GET 2023-09-29T22:34:35.087 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000059 2023-09-29T22:34:35.087 DEBUG:boto:Data: 2023-09-29T22:34:35.087 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.087 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.087 DEBUG:boto:Params: {} 2023-09-29T22:34:35.088 DEBUG:boto:Token: None 2023-09-29T22:34:35.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004193675d6281c1bd-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.095 DEBUG:boto:b'hcnphz-58obj000000000000000000591000falseobj000000000000000000592023-09-29T22:34:13.245Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-09-29T22:34:35.095 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.096 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.096 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.096 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.096 DEBUG:boto:Method: GET 2023-09-29T22:34:35.096 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.096 DEBUG:boto:Data: 2023-09-29T22:34:35.096 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.096 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.096 DEBUG:boto:Params: {} 2023-09-29T22:34:35.097 DEBUG:boto:Token: None 2023-09-29T22:34:35.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk= 2023-09-29T22:34:35.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk=', 'Content-Length': '0'} 2023-09-29T22:34:35.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096ce07a20578810e-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.100 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.100 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.100 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.100 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.100 DEBUG:boto:Method: GET 2023-09-29T22:34:35.100 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.100 DEBUG:boto:Data: 2023-09-29T22:34:35.100 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.101 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.101 DEBUG:boto:Params: {} 2023-09-29T22:34:35.101 DEBUG:boto:Token: None 2023-09-29T22:34:35.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.101 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk= 2023-09-29T22:34:35.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk=', 'Content-Length': '0'} 2023-09-29T22:34:35.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011af20d53a6d1201-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.104 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.104 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.104 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.104 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.104 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.104 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.104 DEBUG:boto:Data: 2023-09-29T22:34:35.104 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.105 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.105 DEBUG:boto:Params: {} 2023-09-29T22:34:35.105 DEBUG:boto:Token: None 2023-09-29T22:34:35.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kIqTrCNVifzg7xakYEr7i84OPek= 2023-09-29T22:34:35.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kIqTrCNVifzg7xakYEr7i84OPek=', 'Content-Length': '0'} 2023-09-29T22:34:35.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b72fffb483d1599-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.107 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.107 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.108 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.108 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.108 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.108 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.108 DEBUG:boto:Data: 2023-09-29T22:34:35.108 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.108 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.108 DEBUG:boto:Params: {} 2023-09-29T22:34:35.109 DEBUG:boto:Token: None 2023-09-29T22:34:35.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kIqTrCNVifzg7xakYEr7i84OPek= 2023-09-29T22:34:35.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kIqTrCNVifzg7xakYEr7i84OPek=', 'Content-Length': '0'} 2023-09-29T22:34:35.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c77760ac3efd52ce-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-09-29T22:34:35.111 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.112 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.112 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.112 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.112 DEBUG:boto:Method: GET 2023-09-29T22:34:35.112 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.112 DEBUG:boto:Data: 2023-09-29T22:34:35.112 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.112 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.112 DEBUG:boto:Params: {} 2023-09-29T22:34:35.113 DEBUG:boto:Token: None 2023-09-29T22:34:35.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk= 2023-09-29T22:34:35.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk=', 'Content-Length': '0'} 2023-09-29T22:34:35.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085715bec94d0ec48-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.162 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.163 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.163 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.163 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.163 DEBUG:boto:Method: GET 2023-09-29T22:34:35.163 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.163 DEBUG:boto:Data: 2023-09-29T22:34:35.163 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.163 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.164 DEBUG:boto:Params: {} 2023-09-29T22:34:35.164 DEBUG:boto:Token: None 2023-09-29T22:34:35.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060?versionId=null 2023-09-29T22:34:35.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk= 2023-09-29T22:34:35.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9m7/n9Fr1GdhmQfzcDR8Nqj2/mk=', 'Content-Length': '0'} 2023-09-29T22:34:35.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d80d2c133160263d-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.167 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.168 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.168 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.168 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.168 DEBUG:boto:Method: GET 2023-09-29T22:34:35.168 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.168 DEBUG:boto:Data: 2023-09-29T22:34:35.168 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.168 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.168 DEBUG:boto:Params: {} 2023-09-29T22:34:35.169 DEBUG:boto:Token: None 2023-09-29T22:34:35.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.173 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe1824cdcdd3e451-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.174 DEBUG:boto:b'hcnphz-58obj000000000000000000601000falseobj000000000000000000602023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.174 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.174 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.174 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.174 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.174 DEBUG:boto:Method: GET 2023-09-29T22:34:35.175 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.175 DEBUG:boto:Data: 2023-09-29T22:34:35.175 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.175 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.175 DEBUG:boto:Params: {} 2023-09-29T22:34:35.175 DEBUG:boto:Token: None 2023-09-29T22:34:35.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.175 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000714b88f266eb6a0d-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.183 DEBUG:boto:b'hcnphz-58obj000000000000000000601000falseobj000000000000000000602023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.183 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.183 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.183 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.183 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.184 DEBUG:boto:Data: 2023-09-29T22:34:35.184 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.184 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.184 DEBUG:boto:Params: {} 2023-09-29T22:34:35.184 DEBUG:boto:Token: None 2023-09-29T22:34:35.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQJYPdxIQSGSwG/zKU6wHL35CMU= 2023-09-29T22:34:35.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQJYPdxIQSGSwG/zKU6wHL35CMU=', 'Content-Length': '0'} 2023-09-29T22:34:35.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d26b5fd9451969b2-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.187 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.187 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.187 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.187 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.187 DEBUG:boto:Data: 2023-09-29T22:34:35.187 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.188 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.188 DEBUG:boto:Params: {} 2023-09-29T22:34:35.188 DEBUG:boto:Token: None 2023-09-29T22:34:35.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQJYPdxIQSGSwG/zKU6wHL35CMU= 2023-09-29T22:34:35.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQJYPdxIQSGSwG/zKU6wHL35CMU=', 'Content-Length': '0'} 2023-09-29T22:34:35.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ce484bfaa18e20-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000060 2023-09-29T22:34:35.190 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.191 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.191 DEBUG:boto:Method: GET 2023-09-29T22:34:35.191 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.191 DEBUG:boto:Data: 2023-09-29T22:34:35.191 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.191 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.191 DEBUG:boto:Params: {} 2023-09-29T22:34:35.191 DEBUG:boto:Token: None 2023-09-29T22:34:35.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QcHTBkvA+n2RKSYwwowGMm2iAH4= 2023-09-29T22:34:35.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QcHTBkvA+n2RKSYwwowGMm2iAH4=', 'Content-Length': '0'} 2023-09-29T22:34:35.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc2e93373acfc95a-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.234 DEBUG:boto:path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.235 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.235 DEBUG:boto:Method: GET 2023-09-29T22:34:35.235 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.235 DEBUG:boto:Data: 2023-09-29T22:34:35.235 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.235 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.236 DEBUG:boto:Params: {} 2023-09-29T22:34:35.236 DEBUG:boto:Token: None 2023-09-29T22:34:35.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000060 2023-09-29T22:34:35.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QcHTBkvA+n2RKSYwwowGMm2iAH4= 2023-09-29T22:34:35.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QcHTBkvA+n2RKSYwwowGMm2iAH4=', 'Content-Length': '0'} 2023-09-29T22:34:35.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f03422c51ce0a477-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.239 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.239 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.239 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.239 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.240 DEBUG:boto:Method: GET 2023-09-29T22:34:35.240 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.240 DEBUG:boto:Data: 2023-09-29T22:34:35.240 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.240 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.240 DEBUG:boto:Params: {} 2023-09-29T22:34:35.240 DEBUG:boto:Token: None 2023-09-29T22:34:35.240 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000826c427b19533ed-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.246 DEBUG:boto:b'hcnphz-58obj000000000000000000601000falseobj000000000000000000602023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.247 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.247 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.247 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.247 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.247 DEBUG:boto:Method: GET 2023-09-29T22:34:35.247 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000060 2023-09-29T22:34:35.247 DEBUG:boto:Data: 2023-09-29T22:34:35.248 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.248 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.248 DEBUG:boto:Params: {} 2023-09-29T22:34:35.248 DEBUG:boto:Token: None 2023-09-29T22:34:35.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ec5bef2f5dfdc0b-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.298 DEBUG:boto:b'hcnphz-58obj000000000000000000601000falseobj000000000000000000602023-09-29T22:34:15.370Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-09-29T22:34:35.299 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.299 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.299 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.299 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.299 DEBUG:boto:Method: GET 2023-09-29T22:34:35.299 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.300 DEBUG:boto:Data: 2023-09-29T22:34:35.300 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.300 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.300 DEBUG:boto:Params: {} 2023-09-29T22:34:35.300 DEBUG:boto:Token: None 2023-09-29T22:34:35.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig= 2023-09-29T22:34:35.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig=', 'Content-Length': '0'} 2023-09-29T22:34:35.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040a8fd572a7e0911-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.304 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.304 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.304 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.305 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.305 DEBUG:boto:Method: GET 2023-09-29T22:34:35.305 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.305 DEBUG:boto:Data: 2023-09-29T22:34:35.305 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.305 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.305 DEBUG:boto:Params: {} 2023-09-29T22:34:35.305 DEBUG:boto:Token: None 2023-09-29T22:34:35.305 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig= 2023-09-29T22:34:35.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig=', 'Content-Length': '0'} 2023-09-29T22:34:35.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef4f22e82bc096aa-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.309 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.309 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.309 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.309 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.310 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.310 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.310 DEBUG:boto:Data: 2023-09-29T22:34:35.310 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.310 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.310 DEBUG:boto:Params: {} 2023-09-29T22:34:35.310 DEBUG:boto:Token: None 2023-09-29T22:34:35.310 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6LXFugmudYK1O5G+FZmt3ZhJ66s= 2023-09-29T22:34:35.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6LXFugmudYK1O5G+FZmt3ZhJ66s=', 'Content-Length': '0'} 2023-09-29T22:34:35.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6e5494212f9303-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.314 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.314 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.314 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.314 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.314 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.314 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.314 DEBUG:boto:Data: 2023-09-29T22:34:35.315 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.315 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.315 DEBUG:boto:Params: {} 2023-09-29T22:34:35.315 DEBUG:boto:Token: None 2023-09-29T22:34:35.315 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.315 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6LXFugmudYK1O5G+FZmt3ZhJ66s= 2023-09-29T22:34:35.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6LXFugmudYK1O5G+FZmt3ZhJ66s=', 'Content-Length': '0'} 2023-09-29T22:34:35.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e9c4451503bbb9-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-09-29T22:34:35.318 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.319 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.319 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.319 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.319 DEBUG:boto:Method: GET 2023-09-29T22:34:35.319 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.319 DEBUG:boto:Data: 2023-09-29T22:34:35.319 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.319 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.319 DEBUG:boto:Params: {} 2023-09-29T22:34:35.320 DEBUG:boto:Token: None 2023-09-29T22:34:35.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig= 2023-09-29T22:34:35.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig=', 'Content-Length': '0'} 2023-09-29T22:34:35.322 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097f9a81aaf32d03c-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.323 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.323 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.323 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.323 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.323 DEBUG:boto:Method: GET 2023-09-29T22:34:35.323 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.323 DEBUG:boto:Data: 2023-09-29T22:34:35.324 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.324 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.324 DEBUG:boto:Params: {} 2023-09-29T22:34:35.324 DEBUG:boto:Token: None 2023-09-29T22:34:35.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061?versionId=null 2023-09-29T22:34:35.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig= 2023-09-29T22:34:35.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/gFPesXHrvB4JXHlOa9nVE8Iig=', 'Content-Length': '0'} 2023-09-29T22:34:35.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d909cfc6422c11fe-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.327 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.328 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.328 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.328 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.328 DEBUG:boto:Method: GET 2023-09-29T22:34:35.328 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.328 DEBUG:boto:Data: 2023-09-29T22:34:35.328 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.328 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.329 DEBUG:boto:Params: {} 2023-09-29T22:34:35.329 DEBUG:boto:Token: None 2023-09-29T22:34:35.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004055afec71cbe6cd-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.378 DEBUG:boto:b'hcnphz-58obj000000000000000000611000falseobj000000000000000000612023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.379 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.379 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.379 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.379 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.379 DEBUG:boto:Method: GET 2023-09-29T22:34:35.380 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.380 DEBUG:boto:Data: 2023-09-29T22:34:35.380 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.380 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.380 DEBUG:boto:Params: {} 2023-09-29T22:34:35.380 DEBUG:boto:Token: None 2023-09-29T22:34:35.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000070a0025b6f773cb7-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.389 DEBUG:boto:b'hcnphz-58obj000000000000000000611000falseobj000000000000000000612023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.389 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.390 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.390 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.390 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.390 DEBUG:boto:Data: 2023-09-29T22:34:35.390 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.390 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.390 DEBUG:boto:Params: {} 2023-09-29T22:34:35.390 DEBUG:boto:Token: None 2023-09-29T22:34:35.391 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4NpMMwrXFv1mvp9z5v6P0Myd4hI= 2023-09-29T22:34:35.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4NpMMwrXFv1mvp9z5v6P0Myd4hI=', 'Content-Length': '0'} 2023-09-29T22:34:35.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc72331a692c23a5-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.393 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.394 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.394 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.394 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.394 DEBUG:boto:Data: 2023-09-29T22:34:35.394 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.394 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.394 DEBUG:boto:Params: {} 2023-09-29T22:34:35.394 DEBUG:boto:Token: None 2023-09-29T22:34:35.395 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4NpMMwrXFv1mvp9z5v6P0Myd4hI= 2023-09-29T22:34:35.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4NpMMwrXFv1mvp9z5v6P0Myd4hI=', 'Content-Length': '0'} 2023-09-29T22:34:35.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c9c363cece90e4a2-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000061 2023-09-29T22:34:35.397 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.397 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.397 DEBUG:boto:Method: GET 2023-09-29T22:34:35.397 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.397 DEBUG:boto:Data: 2023-09-29T22:34:35.397 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.398 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.398 DEBUG:boto:Params: {} 2023-09-29T22:34:35.398 DEBUG:boto:Token: None 2023-09-29T22:34:35.398 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.398 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XztFnSGCqOyuX7IlyMoQS2CH/WI= 2023-09-29T22:34:35.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XztFnSGCqOyuX7IlyMoQS2CH/WI=', 'Content-Length': '0'} 2023-09-29T22:34:35.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ff2b3d1ae36f43-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.401 DEBUG:boto:path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.401 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.401 DEBUG:boto:Method: GET 2023-09-29T22:34:35.401 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.401 DEBUG:boto:Data: 2023-09-29T22:34:35.401 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.401 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.401 DEBUG:boto:Params: {} 2023-09-29T22:34:35.402 DEBUG:boto:Token: None 2023-09-29T22:34:35.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000061 2023-09-29T22:34:35.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XztFnSGCqOyuX7IlyMoQS2CH/WI= 2023-09-29T22:34:35.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XztFnSGCqOyuX7IlyMoQS2CH/WI=', 'Content-Length': '0'} 2023-09-29T22:34:35.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c12daee43a8935-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.404 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.404 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.405 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.405 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.405 DEBUG:boto:Method: GET 2023-09-29T22:34:35.405 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.405 DEBUG:boto:Data: 2023-09-29T22:34:35.405 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.405 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.405 DEBUG:boto:Params: {} 2023-09-29T22:34:35.405 DEBUG:boto:Token: None 2023-09-29T22:34:35.406 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0c8858e56519b47-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.410 DEBUG:boto:b'hcnphz-58obj000000000000000000611000falseobj000000000000000000612023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.410 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.410 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.411 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.411 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.411 DEBUG:boto:Method: GET 2023-09-29T22:34:35.411 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000061 2023-09-29T22:34:35.411 DEBUG:boto:Data: 2023-09-29T22:34:35.411 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.411 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.411 DEBUG:boto:Params: {} 2023-09-29T22:34:35.411 DEBUG:boto:Token: None 2023-09-29T22:34:35.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008357fcad7dbe24eb-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.420 DEBUG:boto:b'hcnphz-58obj000000000000000000611000falseobj000000000000000000612023-09-29T22:34:14.656Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-09-29T22:34:35.420 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.421 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.421 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.421 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.421 DEBUG:boto:Method: GET 2023-09-29T22:34:35.421 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.421 DEBUG:boto:Data: 2023-09-29T22:34:35.421 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.421 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.422 DEBUG:boto:Params: {} 2023-09-29T22:34:35.422 DEBUG:boto:Token: None 2023-09-29T22:34:35.422 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k= 2023-09-29T22:34:35.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k=', 'Content-Length': '0'} 2023-09-29T22:34:35.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078ee57326b85d062-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.425 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.425 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.425 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.425 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.425 DEBUG:boto:Method: GET 2023-09-29T22:34:35.425 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.425 DEBUG:boto:Data: 2023-09-29T22:34:35.425 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.425 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.426 DEBUG:boto:Params: {} 2023-09-29T22:34:35.426 DEBUG:boto:Token: None 2023-09-29T22:34:35.426 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k= 2023-09-29T22:34:35.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k=', 'Content-Length': '0'} 2023-09-29T22:34:35.428 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000128ae19b6d981e1e-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.470 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.470 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.471 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.471 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.471 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.471 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.471 DEBUG:boto:Data: 2023-09-29T22:34:35.471 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.471 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.471 DEBUG:boto:Params: {} 2023-09-29T22:34:35.472 DEBUG:boto:Token: None 2023-09-29T22:34:35.472 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V/l9o58jF8gP7DUfxTOljp+7Br4= 2023-09-29T22:34:35.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V/l9o58jF8gP7DUfxTOljp+7Br4=', 'Content-Length': '0'} 2023-09-29T22:34:35.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002284d191f1a2af42-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.475 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.475 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.475 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.475 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.475 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.475 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.475 DEBUG:boto:Data: 2023-09-29T22:34:35.476 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.476 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.476 DEBUG:boto:Params: {} 2023-09-29T22:34:35.476 DEBUG:boto:Token: None 2023-09-29T22:34:35.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V/l9o58jF8gP7DUfxTOljp+7Br4= 2023-09-29T22:34:35.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V/l9o58jF8gP7DUfxTOljp+7Br4=', 'Content-Length': '0'} 2023-09-29T22:34:35.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e26eb81a1c3a3fe9-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-09-29T22:34:35.479 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.480 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.480 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.480 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.480 DEBUG:boto:Method: GET 2023-09-29T22:34:35.480 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.480 DEBUG:boto:Data: 2023-09-29T22:34:35.480 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.480 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.480 DEBUG:boto:Params: {} 2023-09-29T22:34:35.481 DEBUG:boto:Token: None 2023-09-29T22:34:35.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k= 2023-09-29T22:34:35.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k=', 'Content-Length': '0'} 2023-09-29T22:34:35.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8dbf557fe23be3e-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.484 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.484 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.484 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.484 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.484 DEBUG:boto:Method: GET 2023-09-29T22:34:35.485 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.485 DEBUG:boto:Data: 2023-09-29T22:34:35.485 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.485 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.485 DEBUG:boto:Params: {} 2023-09-29T22:34:35.485 DEBUG:boto:Token: None 2023-09-29T22:34:35.485 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062?versionId=null 2023-09-29T22:34:35.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k= 2023-09-29T22:34:35.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BWplubd0QfKGHUrkeZxVKYYOQ0k=', 'Content-Length': '0'} 2023-09-29T22:34:35.488 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081038d881c75d130-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.489 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.489 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.489 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.489 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.489 DEBUG:boto:Method: GET 2023-09-29T22:34:35.489 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.489 DEBUG:boto:Data: 2023-09-29T22:34:35.489 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.490 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.490 DEBUG:boto:Params: {} 2023-09-29T22:34:35.490 DEBUG:boto:Token: None 2023-09-29T22:34:35.490 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.490 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.495 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d82d440fa70044d-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.542 DEBUG:boto:b'hcnphz-58obj000000000000000000621000falseobj000000000000000000622023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.543 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.543 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.543 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.543 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.543 DEBUG:boto:Method: GET 2023-09-29T22:34:35.544 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.544 DEBUG:boto:Data: 2023-09-29T22:34:35.544 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.544 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.544 DEBUG:boto:Params: {} 2023-09-29T22:34:35.544 DEBUG:boto:Token: None 2023-09-29T22:34:35.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6639bafd1792b36-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.553 DEBUG:boto:b'hcnphz-58obj000000000000000000621000falseobj000000000000000000622023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.554 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.554 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.554 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.554 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.554 DEBUG:boto:Data: 2023-09-29T22:34:35.555 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.555 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.555 DEBUG:boto:Params: {} 2023-09-29T22:34:35.555 DEBUG:boto:Token: None 2023-09-29T22:34:35.555 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lZF0M5BlXZ7X5H26EgM8rS92Y0o= 2023-09-29T22:34:35.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lZF0M5BlXZ7X5H26EgM8rS92Y0o=', 'Content-Length': '0'} 2023-09-29T22:34:35.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041e5402f5c1de4c5-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.558 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.559 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.559 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.559 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.559 DEBUG:boto:Data: 2023-09-29T22:34:35.559 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.559 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.559 DEBUG:boto:Params: {} 2023-09-29T22:34:35.560 DEBUG:boto:Token: None 2023-09-29T22:34:35.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lZF0M5BlXZ7X5H26EgM8rS92Y0o= 2023-09-29T22:34:35.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lZF0M5BlXZ7X5H26EgM8rS92Y0o=', 'Content-Length': '0'} 2023-09-29T22:34:35.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069e7db3b423644ed-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000062 2023-09-29T22:34:35.562 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.562 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.562 DEBUG:boto:Method: GET 2023-09-29T22:34:35.563 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.563 DEBUG:boto:Data: 2023-09-29T22:34:35.563 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.563 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.563 DEBUG:boto:Params: {} 2023-09-29T22:34:35.563 DEBUG:boto:Token: None 2023-09-29T22:34:35.563 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.563 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/MY4PUjfYriiIFI0jBvejh38lME= 2023-09-29T22:34:35.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/MY4PUjfYriiIFI0jBvejh38lME=', 'Content-Length': '0'} 2023-09-29T22:34:35.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a75175f70d0f9ab-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.566 DEBUG:boto:path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.566 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.566 DEBUG:boto:Method: GET 2023-09-29T22:34:35.566 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.566 DEBUG:boto:Data: 2023-09-29T22:34:35.566 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.567 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.567 DEBUG:boto:Params: {} 2023-09-29T22:34:35.567 DEBUG:boto:Token: None 2023-09-29T22:34:35.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000062 2023-09-29T22:34:35.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/MY4PUjfYriiIFI0jBvejh38lME= 2023-09-29T22:34:35.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/MY4PUjfYriiIFI0jBvejh38lME=', 'Content-Length': '0'} 2023-09-29T22:34:35.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:16 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8e0f4792e5c5f2-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.610 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.611 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.611 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.611 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.611 DEBUG:boto:Method: GET 2023-09-29T22:34:35.611 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.611 DEBUG:boto:Data: 2023-09-29T22:34:35.611 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.612 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.612 DEBUG:boto:Params: {} 2023-09-29T22:34:35.612 DEBUG:boto:Token: None 2023-09-29T22:34:35.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9f58af4a718f9e1-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.617 DEBUG:boto:b'hcnphz-58obj000000000000000000621000falseobj000000000000000000622023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.618 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.618 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.618 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.618 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.618 DEBUG:boto:Method: GET 2023-09-29T22:34:35.618 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000062 2023-09-29T22:34:35.618 DEBUG:boto:Data: 2023-09-29T22:34:35.618 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.619 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.619 DEBUG:boto:Params: {} 2023-09-29T22:34:35.619 DEBUG:boto:Token: None 2023-09-29T22:34:35.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.627 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1a25d582eb66af4-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.627 DEBUG:boto:b'hcnphz-58obj000000000000000000621000falseobj000000000000000000622023-09-29T22:34:16.059Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-09-29T22:34:35.628 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.628 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.628 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.628 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.628 DEBUG:boto:Method: GET 2023-09-29T22:34:35.628 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.628 DEBUG:boto:Data: 2023-09-29T22:34:35.628 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.629 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.629 DEBUG:boto:Params: {} 2023-09-29T22:34:35.629 DEBUG:boto:Token: None 2023-09-29T22:34:35.629 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk= 2023-09-29T22:34:35.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk=', 'Content-Length': '0'} 2023-09-29T22:34:35.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27dc8cdd620ee5b-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.632 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.632 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.632 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.633 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.633 DEBUG:boto:Method: GET 2023-09-29T22:34:35.633 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.633 DEBUG:boto:Data: 2023-09-29T22:34:35.633 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.633 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.633 DEBUG:boto:Params: {} 2023-09-29T22:34:35.633 DEBUG:boto:Token: None 2023-09-29T22:34:35.633 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.634 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk= 2023-09-29T22:34:35.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk=', 'Content-Length': '0'} 2023-09-29T22:34:35.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000154f4b2abc6312f0-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.636 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.636 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.636 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.636 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.637 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.637 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.637 DEBUG:boto:Data: 2023-09-29T22:34:35.637 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.637 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.637 DEBUG:boto:Params: {} 2023-09-29T22:34:35.637 DEBUG:boto:Token: None 2023-09-29T22:34:35.638 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kz5RlWdhyApX3mA/BXrUOrb4IAE= 2023-09-29T22:34:35.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kz5RlWdhyApX3mA/BXrUOrb4IAE=', 'Content-Length': '0'} 2023-09-29T22:34:35.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf3ea03b541b950e-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.640 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.640 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.640 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.640 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.640 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.640 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.640 DEBUG:boto:Data: 2023-09-29T22:34:35.641 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.641 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.641 DEBUG:boto:Params: {} 2023-09-29T22:34:35.641 DEBUG:boto:Token: None 2023-09-29T22:34:35.641 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kz5RlWdhyApX3mA/BXrUOrb4IAE= 2023-09-29T22:34:35.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kz5RlWdhyApX3mA/BXrUOrb4IAE=', 'Content-Length': '0'} 2023-09-29T22:34:35.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba678928dd206d3e-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-09-29T22:34:35.644 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.644 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.644 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.644 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.644 DEBUG:boto:Method: GET 2023-09-29T22:34:35.644 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.644 DEBUG:boto:Data: 2023-09-29T22:34:35.644 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.645 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.645 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.645 DEBUG:boto:Params: {} 2023-09-29T22:34:35.645 DEBUG:boto:Token: None 2023-09-29T22:34:35.645 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk= 2023-09-29T22:34:35.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk=', 'Content-Length': '0'} 2023-09-29T22:34:35.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000699decf157e7853d-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.694 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.695 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.695 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.695 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.695 DEBUG:boto:Method: GET 2023-09-29T22:34:35.695 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.695 DEBUG:boto:Data: 2023-09-29T22:34:35.695 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.695 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.696 DEBUG:boto:Params: {} 2023-09-29T22:34:35.696 DEBUG:boto:Token: None 2023-09-29T22:34:35.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065?versionId=null 2023-09-29T22:34:35.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk= 2023-09-29T22:34:35.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x1YAFNjlE724/UJQFB8HIvnQ1Zk=', 'Content-Length': '0'} 2023-09-29T22:34:35.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004342bee498fb6723-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.699 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.699 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.699 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.700 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.700 DEBUG:boto:Method: GET 2023-09-29T22:34:35.700 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.700 DEBUG:boto:Data: 2023-09-29T22:34:35.700 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.701 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.701 DEBUG:boto:Params: {} 2023-09-29T22:34:35.701 DEBUG:boto:Token: None 2023-09-29T22:34:35.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.706 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea9b6d7e59f924d9-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.706 DEBUG:boto:b'hcnphz-58obj000000000000000000651000falseobj000000000000000000652023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.706 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.707 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.707 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.707 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.707 DEBUG:boto:Method: GET 2023-09-29T22:34:35.707 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.707 DEBUG:boto:Data: 2023-09-29T22:34:35.707 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.707 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.708 DEBUG:boto:Params: {} 2023-09-29T22:34:35.708 DEBUG:boto:Token: None 2023-09-29T22:34:35.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000792c0fd712d95dae-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.762 DEBUG:boto:b'hcnphz-58obj000000000000000000651000falseobj000000000000000000652023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.763 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.763 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.763 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.763 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.763 DEBUG:boto:Data: 2023-09-29T22:34:35.764 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.764 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.764 DEBUG:boto:Params: {} 2023-09-29T22:34:35.764 DEBUG:boto:Token: None 2023-09-29T22:34:35.764 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wuMLZfDpA4APG/sGXEtPm4dY7Ac= 2023-09-29T22:34:35.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wuMLZfDpA4APG/sGXEtPm4dY7Ac=', 'Content-Length': '0'} 2023-09-29T22:34:35.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ec41f2a80b8446-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.767 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.768 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.768 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.768 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.768 DEBUG:boto:Data: 2023-09-29T22:34:35.768 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.768 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.768 DEBUG:boto:Params: {} 2023-09-29T22:34:35.768 DEBUG:boto:Token: None 2023-09-29T22:34:35.769 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.769 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wuMLZfDpA4APG/sGXEtPm4dY7Ac= 2023-09-29T22:34:35.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wuMLZfDpA4APG/sGXEtPm4dY7Ac=', 'Content-Length': '0'} 2023-09-29T22:34:35.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000857315c2f76967cb-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000065 2023-09-29T22:34:35.772 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.772 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.772 DEBUG:boto:Method: GET 2023-09-29T22:34:35.772 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.772 DEBUG:boto:Data: 2023-09-29T22:34:35.772 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.773 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.773 DEBUG:boto:Params: {} 2023-09-29T22:34:35.773 DEBUG:boto:Token: None 2023-09-29T22:34:35.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U/6Ckc7TDMoF2SvvE3Yr+TW7vws= 2023-09-29T22:34:35.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U/6Ckc7TDMoF2SvvE3Yr+TW7vws=', 'Content-Length': '0'} 2023-09-29T22:34:35.776 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a752c7963a122c2d-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.776 DEBUG:boto:path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.776 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.776 DEBUG:boto:Method: GET 2023-09-29T22:34:35.776 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.777 DEBUG:boto:Data: 2023-09-29T22:34:35.777 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.777 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.777 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.777 DEBUG:boto:Params: {} 2023-09-29T22:34:35.777 DEBUG:boto:Token: None 2023-09-29T22:34:35.777 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000065 2023-09-29T22:34:35.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U/6Ckc7TDMoF2SvvE3Yr+TW7vws= 2023-09-29T22:34:35.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U/6Ckc7TDMoF2SvvE3Yr+TW7vws=', 'Content-Length': '0'} 2023-09-29T22:34:35.780 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5ac52da5238dc43-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.780 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.780 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.780 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.780 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.781 DEBUG:boto:Method: GET 2023-09-29T22:34:35.781 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.781 DEBUG:boto:Data: 2023-09-29T22:34:35.781 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.781 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.781 DEBUG:boto:Params: {} 2023-09-29T22:34:35.781 DEBUG:boto:Token: None 2023-09-29T22:34:35.781 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.782 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.786 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062a8fa75fd04758d-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.830 DEBUG:boto:b'hcnphz-58obj000000000000000000651000falseobj000000000000000000652023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.831 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.831 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.831 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.831 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.832 DEBUG:boto:Method: GET 2023-09-29T22:34:35.832 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000065 2023-09-29T22:34:35.832 DEBUG:boto:Data: 2023-09-29T22:34:35.832 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.832 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.832 DEBUG:boto:Params: {} 2023-09-29T22:34:35.832 DEBUG:boto:Token: None 2023-09-29T22:34:35.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.843 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c20d6b1675b08de4-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.843 DEBUG:boto:b'hcnphz-58obj000000000000000000651000falseobj000000000000000000652023-09-29T22:34:15.810Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-09-29T22:34:35.844 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.844 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.844 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.844 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.844 DEBUG:boto:Method: GET 2023-09-29T22:34:35.844 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.844 DEBUG:boto:Data: 2023-09-29T22:34:35.845 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.845 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.845 DEBUG:boto:Params: {} 2023-09-29T22:34:35.845 DEBUG:boto:Token: None 2023-09-29T22:34:35.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU= 2023-09-29T22:34:35.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU=', 'Content-Length': '0'} 2023-09-29T22:34:35.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc54ba82cbf2c7d4-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.848 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.848 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.848 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.849 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.849 DEBUG:boto:Method: GET 2023-09-29T22:34:35.849 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.849 DEBUG:boto:Data: 2023-09-29T22:34:35.849 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.849 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.849 DEBUG:boto:Params: {} 2023-09-29T22:34:35.849 DEBUG:boto:Token: None 2023-09-29T22:34:35.850 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU= 2023-09-29T22:34:35.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU=', 'Content-Length': '0'} 2023-09-29T22:34:35.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5bcb6d03a1ca370-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.898 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.899 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.899 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.899 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.899 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.899 DEBUG:boto:Data: 2023-09-29T22:34:35.899 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.900 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.900 DEBUG:boto:Params: {} 2023-09-29T22:34:35.900 DEBUG:boto:Token: None 2023-09-29T22:34:35.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uYf0vJOndoBOSlLOTYHiYVE1TtU= 2023-09-29T22:34:35.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uYf0vJOndoBOSlLOTYHiYVE1TtU=', 'Content-Length': '0'} 2023-09-29T22:34:35.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edb0209d559dcbf8-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.904 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.904 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.904 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.904 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.904 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.904 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.904 DEBUG:boto:Data: 2023-09-29T22:34:35.904 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.905 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.905 DEBUG:boto:Params: {} 2023-09-29T22:34:35.905 DEBUG:boto:Token: None 2023-09-29T22:34:35.905 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uYf0vJOndoBOSlLOTYHiYVE1TtU= 2023-09-29T22:34:35.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uYf0vJOndoBOSlLOTYHiYVE1TtU=', 'Content-Length': '0'} 2023-09-29T22:34:35.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341e9fa93c8e9c41-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-09-29T22:34:35.908 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.908 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.908 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.908 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.909 DEBUG:boto:Method: GET 2023-09-29T22:34:35.909 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.909 DEBUG:boto:Data: 2023-09-29T22:34:35.909 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.909 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.909 DEBUG:boto:Params: {} 2023-09-29T22:34:35.909 DEBUG:boto:Token: None 2023-09-29T22:34:35.909 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU= 2023-09-29T22:34:35.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU=', 'Content-Length': '0'} 2023-09-29T22:34:35.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4efb2ff35e5e743-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.913 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.913 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.913 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.913 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.913 DEBUG:boto:Method: GET 2023-09-29T22:34:35.913 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.913 DEBUG:boto:Data: 2023-09-29T22:34:35.913 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.914 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.914 DEBUG:boto:Params: {} 2023-09-29T22:34:35.914 DEBUG:boto:Token: None 2023-09-29T22:34:35.914 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076?versionId=null 2023-09-29T22:34:35.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU= 2023-09-29T22:34:35.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kf1RiOpGrF4umm8zeKetyY96ljU=', 'Content-Length': '0'} 2023-09-29T22:34:35.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cf13bfdaf80107-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.917 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.918 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.918 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.918 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.918 DEBUG:boto:Method: GET 2023-09-29T22:34:35.918 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.918 DEBUG:boto:Data: 2023-09-29T22:34:35.918 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.918 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.919 DEBUG:boto:Params: {} 2023-09-29T22:34:35.919 DEBUG:boto:Token: None 2023-09-29T22:34:35.919 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.919 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d644f2a13acb40-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.925 DEBUG:boto:b'hcnphz-58obj000000000000000000761000falseobj000000000000000000762023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.925 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.925 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.925 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.926 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.926 DEBUG:boto:Method: GET 2023-09-29T22:34:35.926 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.926 DEBUG:boto:Data: 2023-09-29T22:34:35.926 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.926 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.926 DEBUG:boto:Params: {} 2023-09-29T22:34:35.926 DEBUG:boto:Token: None 2023-09-29T22:34:35.927 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:35.934 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ebeb583fa046bef-00651750fb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.934 DEBUG:boto:b'hcnphz-58obj000000000000000000761000falseobj000000000000000000762023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:35.935 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.935 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.935 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.935 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.935 DEBUG:boto:Data: 2023-09-29T22:34:35.935 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.936 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.936 DEBUG:boto:Params: {} 2023-09-29T22:34:35.936 DEBUG:boto:Token: None 2023-09-29T22:34:35.936 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AxfECeYF5eoNTVWZlfQC7BORO5g= 2023-09-29T22:34:35.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AxfECeYF5eoNTVWZlfQC7BORO5g=', 'Content-Length': '0'} 2023-09-29T22:34:35.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de21a257ebcf69c6-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.939 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.939 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.939 DEBUG:boto:Method: HEAD 2023-09-29T22:34:35.939 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.939 DEBUG:boto:Data: 2023-09-29T22:34:35.940 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.940 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.940 DEBUG:boto:Params: {} 2023-09-29T22:34:35.940 DEBUG:boto:Token: None 2023-09-29T22:34:35.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AxfECeYF5eoNTVWZlfQC7BORO5g= 2023-09-29T22:34:35.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AxfECeYF5eoNTVWZlfQC7BORO5g=', 'Content-Length': '0'} 2023-09-29T22:34:35.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfd4d82f1e2a4250-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000076 2023-09-29T22:34:35.943 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.943 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.943 DEBUG:boto:Method: GET 2023-09-29T22:34:35.943 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.943 DEBUG:boto:Data: 2023-09-29T22:34:35.943 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.944 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.944 DEBUG:boto:Params: {} 2023-09-29T22:34:35.944 DEBUG:boto:Token: None 2023-09-29T22:34:35.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ePrVJ+UbI6lh3esGLKABarGpco= 2023-09-29T22:34:35.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ePrVJ+UbI6lh3esGLKABarGpco=', 'Content-Length': '0'} 2023-09-29T22:34:35.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042f5b6a2977d3dc0-00651750fb-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.990 DEBUG:boto:path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.991 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.991 DEBUG:boto:Method: GET 2023-09-29T22:34:35.991 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.991 DEBUG:boto:Data: 2023-09-29T22:34:35.991 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:35.991 DEBUG:boto:Port: 8000 2023-09-29T22:34:35.991 DEBUG:boto:Params: {} 2023-09-29T22:34:35.992 DEBUG:boto:Token: None 2023-09-29T22:34:35.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/obj00000000000000000076 2023-09-29T22:34:35.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ePrVJ+UbI6lh3esGLKABarGpco= 2023-09-29T22:34:35.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ePrVJ+UbI6lh3esGLKABarGpco=', 'Content-Length': '0'} 2023-09-29T22:34:35.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164f1bb9ee0e8a4c-00651750fb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:35.995 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:35.995 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:35.995 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.996 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.996 DEBUG:boto:Method: GET 2023-09-29T22:34:35.996 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:35.996 DEBUG:boto:Data: 2023-09-29T22:34:35.996 DEBUG:boto:Headers: {} 2023-09-29T22:34:35.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:35.996 DEBUG:boto:Port: 8001 2023-09-29T22:34:35.996 DEBUG:boto:Params: {} 2023-09-29T22:34:35.996 DEBUG:boto:Token: None 2023-09-29T22:34:35.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:35 GMT /hcnphz-58/ 2023-09-29T22:34:35.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g= 2023-09-29T22:34:35.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgbXqIt0/toAUEDYolGA+loi/2g=', 'Content-Length': '0'} 2023-09-29T22:34:36.001 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000446f4485383b2a3c-00651750fb-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.001 DEBUG:boto:b'hcnphz-58obj000000000000000000761000falseobj000000000000000000762023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.002 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.002 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.002 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:36.002 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:36.002 DEBUG:boto:Method: GET 2023-09-29T22:34:36.003 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000076 2023-09-29T22:34:36.003 DEBUG:boto:Data: 2023-09-29T22:34:36.003 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.003 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.003 DEBUG:boto:Params: {} 2023-09-29T22:34:36.003 DEBUG:boto:Token: None 2023-09-29T22:34:36.003 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000504bf9d796d0d7c0-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.011 DEBUG:boto:b'hcnphz-58obj000000000000000000761000falseobj000000000000000000762023-09-29T22:34:09.827Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-09-29T22:34:36.012 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.012 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.012 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.012 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.012 DEBUG:boto:Method: GET 2023-09-29T22:34:36.012 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.012 DEBUG:boto:Data: 2023-09-29T22:34:36.012 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.012 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.013 DEBUG:boto:Params: {} 2023-09-29T22:34:36.013 DEBUG:boto:Token: None 2023-09-29T22:34:36.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ= 2023-09-29T22:34:36.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0b1fadc26df228-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.016 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.016 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.016 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.016 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.016 DEBUG:boto:Method: GET 2023-09-29T22:34:36.016 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.016 DEBUG:boto:Data: 2023-09-29T22:34:36.016 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.017 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.017 DEBUG:boto:Params: {} 2023-09-29T22:34:36.017 DEBUG:boto:Token: None 2023-09-29T22:34:36.017 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ= 2023-09-29T22:34:36.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022eed3544f2db968-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.020 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.020 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.020 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.020 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.020 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.020 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.021 DEBUG:boto:Data: 2023-09-29T22:34:36.021 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.021 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.021 DEBUG:boto:Params: {} 2023-09-29T22:34:36.021 DEBUG:boto:Token: None 2023-09-29T22:34:36.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lv1tWVJvmyErNCvaJDcu3lxhUzc= 2023-09-29T22:34:36.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lv1tWVJvmyErNCvaJDcu3lxhUzc=', 'Content-Length': '0'} 2023-09-29T22:34:36.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b77db63e5eb4d62-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.024 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.024 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.024 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.025 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.025 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.025 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.025 DEBUG:boto:Data: 2023-09-29T22:34:36.025 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.025 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.025 DEBUG:boto:Params: {} 2023-09-29T22:34:36.025 DEBUG:boto:Token: None 2023-09-29T22:34:36.026 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lv1tWVJvmyErNCvaJDcu3lxhUzc= 2023-09-29T22:34:36.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lv1tWVJvmyErNCvaJDcu3lxhUzc=', 'Content-Length': '0'} 2023-09-29T22:34:36.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000881078c442918bc3-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-09-29T22:34:36.028 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.028 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.029 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.029 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.029 DEBUG:boto:Method: GET 2023-09-29T22:34:36.029 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.029 DEBUG:boto:Data: 2023-09-29T22:34:36.029 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.029 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.029 DEBUG:boto:Params: {} 2023-09-29T22:34:36.029 DEBUG:boto:Token: None 2023-09-29T22:34:36.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ= 2023-09-29T22:34:36.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ad3d4e8a72f2385-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.074 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.075 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.077 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.077 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.077 DEBUG:boto:Method: GET 2023-09-29T22:34:36.077 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.077 DEBUG:boto:Data: 2023-09-29T22:34:36.077 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.078 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.078 DEBUG:boto:Params: {} 2023-09-29T22:34:36.078 DEBUG:boto:Token: None 2023-09-29T22:34:36.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133?versionId=null 2023-09-29T22:34:36.079 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ= 2023-09-29T22:34:36.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/qlCkxuQL+gilHw57wlhc+5jgBQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007821546847d5a32c-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.082 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.083 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.083 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.083 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.083 DEBUG:boto:Method: GET 2023-09-29T22:34:36.083 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.083 DEBUG:boto:Data: 2023-09-29T22:34:36.083 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.083 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.083 DEBUG:boto:Params: {} 2023-09-29T22:34:36.084 DEBUG:boto:Token: None 2023-09-29T22:34:36.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.088 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000246c03c9cd29dd66-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.088 DEBUG:boto:b'hcnphz-58obj000000000000000001331000falseobj000000000000000001332023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.089 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.089 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.089 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.089 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.089 DEBUG:boto:Method: GET 2023-09-29T22:34:36.089 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.089 DEBUG:boto:Data: 2023-09-29T22:34:36.089 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.090 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.090 DEBUG:boto:Params: {} 2023-09-29T22:34:36.090 DEBUG:boto:Token: None 2023-09-29T22:34:36.090 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.090 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a13fb04db3a3517d-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.098 DEBUG:boto:b'hcnphz-58obj000000000000000001331000falseobj000000000000000001332023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.098 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.099 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.099 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.099 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.099 DEBUG:boto:Data: 2023-09-29T22:34:36.099 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.099 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.099 DEBUG:boto:Params: {} 2023-09-29T22:34:36.099 DEBUG:boto:Token: None 2023-09-29T22:34:36.100 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NVPUCc6/7ibcbxydYNyfnITHeck= 2023-09-29T22:34:36.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NVPUCc6/7ibcbxydYNyfnITHeck=', 'Content-Length': '0'} 2023-09-29T22:34:36.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c640f75e8d6038a9-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.102 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.102 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.102 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.102 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.103 DEBUG:boto:Data: 2023-09-29T22:34:36.103 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.103 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.103 DEBUG:boto:Params: {} 2023-09-29T22:34:36.103 DEBUG:boto:Token: None 2023-09-29T22:34:36.103 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.103 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NVPUCc6/7ibcbxydYNyfnITHeck= 2023-09-29T22:34:36.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NVPUCc6/7ibcbxydYNyfnITHeck=', 'Content-Length': '0'} 2023-09-29T22:34:36.106 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098b318a549643883-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000133 2023-09-29T22:34:36.106 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.106 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.106 DEBUG:boto:Method: GET 2023-09-29T22:34:36.106 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.107 DEBUG:boto:Data: 2023-09-29T22:34:36.107 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.107 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.107 DEBUG:boto:Params: {} 2023-09-29T22:34:36.107 DEBUG:boto:Token: None 2023-09-29T22:34:36.107 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.107 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NyInnWwZxlTlLChGCTKF8fm4JH0= 2023-09-29T22:34:36.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NyInnWwZxlTlLChGCTKF8fm4JH0=', 'Content-Length': '0'} 2023-09-29T22:34:36.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f84e55d1dcb94b2-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.110 DEBUG:boto:path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.110 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.110 DEBUG:boto:Method: GET 2023-09-29T22:34:36.110 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.110 DEBUG:boto:Data: 2023-09-29T22:34:36.110 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.111 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.111 DEBUG:boto:Params: {} 2023-09-29T22:34:36.111 DEBUG:boto:Token: None 2023-09-29T22:34:36.111 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000133 2023-09-29T22:34:36.111 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NyInnWwZxlTlLChGCTKF8fm4JH0= 2023-09-29T22:34:36.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NyInnWwZxlTlLChGCTKF8fm4JH0=', 'Content-Length': '0'} 2023-09-29T22:34:36.113 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958e3e68850fe8a7-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.113 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.114 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.114 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.114 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.114 DEBUG:boto:Method: GET 2023-09-29T22:34:36.114 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.114 DEBUG:boto:Data: 2023-09-29T22:34:36.114 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.114 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.114 DEBUG:boto:Params: {} 2023-09-29T22:34:36.115 DEBUG:boto:Token: None 2023-09-29T22:34:36.115 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.115 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.118 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013d15942db316c58-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.118 DEBUG:boto:b'hcnphz-58obj000000000000000001331000falseobj000000000000000001332023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.118 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.118 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.118 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.119 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.119 DEBUG:boto:Method: GET 2023-09-29T22:34:36.119 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000133 2023-09-29T22:34:36.119 DEBUG:boto:Data: 2023-09-29T22:34:36.119 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.119 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.119 DEBUG:boto:Params: {} 2023-09-29T22:34:36.119 DEBUG:boto:Token: None 2023-09-29T22:34:36.119 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a86135126a454db-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.125 DEBUG:boto:b'hcnphz-58obj000000000000000001331000falseobj000000000000000001332023-09-29T22:34:14.832Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-09-29T22:34:36.125 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.125 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.125 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.125 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.126 DEBUG:boto:Method: GET 2023-09-29T22:34:36.126 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.126 DEBUG:boto:Data: 2023-09-29T22:34:36.126 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.126 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.126 DEBUG:boto:Params: {} 2023-09-29T22:34:36.126 DEBUG:boto:Token: None 2023-09-29T22:34:36.126 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.127 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ= 2023-09-29T22:34:36.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.129 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c86cf459b0ad646-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.130 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.130 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.130 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.130 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.130 DEBUG:boto:Method: GET 2023-09-29T22:34:36.130 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.130 DEBUG:boto:Data: 2023-09-29T22:34:36.130 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.130 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.131 DEBUG:boto:Params: {} 2023-09-29T22:34:36.131 DEBUG:boto:Token: None 2023-09-29T22:34:36.131 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.131 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ= 2023-09-29T22:34:36.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079bdc8ef4eb068d9-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.134 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.134 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.134 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.135 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.135 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.135 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.135 DEBUG:boto:Data: 2023-09-29T22:34:36.135 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.135 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.135 DEBUG:boto:Params: {} 2023-09-29T22:34:36.135 DEBUG:boto:Token: None 2023-09-29T22:34:36.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uM35QBp7dsh8Wk1RiVfkJe64Ijc= 2023-09-29T22:34:36.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uM35QBp7dsh8Wk1RiVfkJe64Ijc=', 'Content-Length': '0'} 2023-09-29T22:34:36.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000305149ed10545a5d-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.138 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.138 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.139 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.139 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.139 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.139 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.139 DEBUG:boto:Data: 2023-09-29T22:34:36.139 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.139 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.139 DEBUG:boto:Params: {} 2023-09-29T22:34:36.140 DEBUG:boto:Token: None 2023-09-29T22:34:36.140 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uM35QBp7dsh8Wk1RiVfkJe64Ijc= 2023-09-29T22:34:36.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uM35QBp7dsh8Wk1RiVfkJe64Ijc=', 'Content-Length': '0'} 2023-09-29T22:34:36.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067d2d4a365c2d480-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-09-29T22:34:36.143 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.143 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.143 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.143 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.143 DEBUG:boto:Method: GET 2023-09-29T22:34:36.143 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.143 DEBUG:boto:Data: 2023-09-29T22:34:36.143 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.143 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.144 DEBUG:boto:Params: {} 2023-09-29T22:34:36.144 DEBUG:boto:Token: None 2023-09-29T22:34:36.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ= 2023-09-29T22:34:36.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246c43d06dffb2c7-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.147 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.147 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.147 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.147 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.147 DEBUG:boto:Method: GET 2023-09-29T22:34:36.147 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.147 DEBUG:boto:Data: 2023-09-29T22:34:36.147 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.148 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.148 DEBUG:boto:Params: {} 2023-09-29T22:34:36.148 DEBUG:boto:Token: None 2023-09-29T22:34:36.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162?versionId=null 2023-09-29T22:34:36.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ= 2023-09-29T22:34:36.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WHS5GrV23PRQ0Jfa+zDq3dRx8aQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d98c5f59b76290f-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.151 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.151 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.151 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.151 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.151 DEBUG:boto:Method: GET 2023-09-29T22:34:36.151 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.151 DEBUG:boto:Data: 2023-09-29T22:34:36.151 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.152 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.152 DEBUG:boto:Params: {} 2023-09-29T22:34:36.152 DEBUG:boto:Token: None 2023-09-29T22:34:36.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d669fbfdb6993dd8-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.198 DEBUG:boto:b'hcnphz-58obj000000000000000001621000falseobj000000000000000001622023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.199 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.199 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.199 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.200 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.200 DEBUG:boto:Method: GET 2023-09-29T22:34:36.200 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.200 DEBUG:boto:Data: 2023-09-29T22:34:36.200 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.200 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.200 DEBUG:boto:Params: {} 2023-09-29T22:34:36.201 DEBUG:boto:Token: None 2023-09-29T22:34:36.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.210 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb643b8e957683ce-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.211 DEBUG:boto:b'hcnphz-58obj000000000000000001621000falseobj000000000000000001622023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.211 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.211 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.211 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.211 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.212 DEBUG:boto:Data: 2023-09-29T22:34:36.212 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.212 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.212 DEBUG:boto:Params: {} 2023-09-29T22:34:36.212 DEBUG:boto:Token: None 2023-09-29T22:34:36.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VgEjmGSiOBvwihxM+fNOSkbr9EA= 2023-09-29T22:34:36.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VgEjmGSiOBvwihxM+fNOSkbr9EA=', 'Content-Length': '0'} 2023-09-29T22:34:36.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000235781ac7ea768e9-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.215 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.215 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.215 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.215 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.215 DEBUG:boto:Data: 2023-09-29T22:34:36.215 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.216 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.216 DEBUG:boto:Params: {} 2023-09-29T22:34:36.216 DEBUG:boto:Token: None 2023-09-29T22:34:36.216 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VgEjmGSiOBvwihxM+fNOSkbr9EA= 2023-09-29T22:34:36.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VgEjmGSiOBvwihxM+fNOSkbr9EA=', 'Content-Length': '0'} 2023-09-29T22:34:36.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c091343777b492e8-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000162 2023-09-29T22:34:36.219 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.219 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.219 DEBUG:boto:Method: GET 2023-09-29T22:34:36.219 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.219 DEBUG:boto:Data: 2023-09-29T22:34:36.219 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.220 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.220 DEBUG:boto:Params: {} 2023-09-29T22:34:36.220 DEBUG:boto:Token: None 2023-09-29T22:34:36.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JnBh8+DIaqiiTtdtMelZKTrIj08= 2023-09-29T22:34:36.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JnBh8+DIaqiiTtdtMelZKTrIj08=', 'Content-Length': '0'} 2023-09-29T22:34:36.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a83a3b186d38fd7-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.223 DEBUG:boto:path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.223 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.223 DEBUG:boto:Method: GET 2023-09-29T22:34:36.223 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.223 DEBUG:boto:Data: 2023-09-29T22:34:36.223 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.224 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.224 DEBUG:boto:Params: {} 2023-09-29T22:34:36.224 DEBUG:boto:Token: None 2023-09-29T22:34:36.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000162 2023-09-29T22:34:36.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JnBh8+DIaqiiTtdtMelZKTrIj08= 2023-09-29T22:34:36.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JnBh8+DIaqiiTtdtMelZKTrIj08=', 'Content-Length': '0'} 2023-09-29T22:34:36.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9c766e77d4fa54a-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.274 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.275 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.275 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.275 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.275 DEBUG:boto:Method: GET 2023-09-29T22:34:36.275 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.275 DEBUG:boto:Data: 2023-09-29T22:34:36.275 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.276 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.276 DEBUG:boto:Params: {} 2023-09-29T22:34:36.276 DEBUG:boto:Token: None 2023-09-29T22:34:36.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000958ba787311b289b-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.282 DEBUG:boto:b'hcnphz-58obj000000000000000001621000falseobj000000000000000001622023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.282 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.282 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.282 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.282 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.283 DEBUG:boto:Method: GET 2023-09-29T22:34:36.283 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000162 2023-09-29T22:34:36.283 DEBUG:boto:Data: 2023-09-29T22:34:36.283 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.283 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.283 DEBUG:boto:Params: {} 2023-09-29T22:34:36.283 DEBUG:boto:Token: None 2023-09-29T22:34:36.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d2fad0435536b84-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.292 DEBUG:boto:b'hcnphz-58obj000000000000000001621000falseobj000000000000000001622023-09-29T22:34:14.673Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-09-29T22:34:36.292 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.293 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.293 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.293 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.293 DEBUG:boto:Method: GET 2023-09-29T22:34:36.293 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.293 DEBUG:boto:Data: 2023-09-29T22:34:36.293 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.293 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.293 DEBUG:boto:Params: {} 2023-09-29T22:34:36.294 DEBUG:boto:Token: None 2023-09-29T22:34:36.294 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM= 2023-09-29T22:34:36.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM=', 'Content-Length': '0'} 2023-09-29T22:34:36.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a3a46f2073b072-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.342 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.343 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.343 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.343 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.343 DEBUG:boto:Method: GET 2023-09-29T22:34:36.343 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.343 DEBUG:boto:Data: 2023-09-29T22:34:36.343 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.344 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.344 DEBUG:boto:Params: {} 2023-09-29T22:34:36.344 DEBUG:boto:Token: None 2023-09-29T22:34:36.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM= 2023-09-29T22:34:36.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM=', 'Content-Length': '0'} 2023-09-29T22:34:36.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007624a3fd6683e134-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.347 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.348 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.348 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.348 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.348 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.348 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.348 DEBUG:boto:Data: 2023-09-29T22:34:36.348 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.348 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.349 DEBUG:boto:Params: {} 2023-09-29T22:34:36.349 DEBUG:boto:Token: None 2023-09-29T22:34:36.349 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4hDco7jVHv6s/p9l4x3jUzBeDpY= 2023-09-29T22:34:36.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4hDco7jVHv6s/p9l4x3jUzBeDpY=', 'Content-Length': '0'} 2023-09-29T22:34:36.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f6c595b270d1fd6-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.352 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.352 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.352 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.352 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.352 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.353 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.353 DEBUG:boto:Data: 2023-09-29T22:34:36.353 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.353 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.353 DEBUG:boto:Params: {} 2023-09-29T22:34:36.353 DEBUG:boto:Token: None 2023-09-29T22:34:36.353 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.353 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4hDco7jVHv6s/p9l4x3jUzBeDpY= 2023-09-29T22:34:36.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4hDco7jVHv6s/p9l4x3jUzBeDpY=', 'Content-Length': '0'} 2023-09-29T22:34:36.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2f0dcbe3acc3ec-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-09-29T22:34:36.356 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.356 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.357 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.357 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.357 DEBUG:boto:Method: GET 2023-09-29T22:34:36.357 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.357 DEBUG:boto:Data: 2023-09-29T22:34:36.357 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.357 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.357 DEBUG:boto:Params: {} 2023-09-29T22:34:36.358 DEBUG:boto:Token: None 2023-09-29T22:34:36.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM= 2023-09-29T22:34:36.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM=', 'Content-Length': '0'} 2023-09-29T22:34:36.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009719e5916df83717-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.361 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.361 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.361 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.361 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.361 DEBUG:boto:Method: GET 2023-09-29T22:34:36.361 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.362 DEBUG:boto:Data: 2023-09-29T22:34:36.362 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.362 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.362 DEBUG:boto:Params: {} 2023-09-29T22:34:36.362 DEBUG:boto:Token: None 2023-09-29T22:34:36.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271?versionId=null 2023-09-29T22:34:36.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM= 2023-09-29T22:34:36.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MkVePUwG83EBwK530Fpbk+qESKM=', 'Content-Length': '0'} 2023-09-29T22:34:36.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000965bf3b2044aed39-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.410 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.411 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.411 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.411 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.411 DEBUG:boto:Method: GET 2023-09-29T22:34:36.411 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.411 DEBUG:boto:Data: 2023-09-29T22:34:36.411 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.412 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.412 DEBUG:boto:Params: {} 2023-09-29T22:34:36.412 DEBUG:boto:Token: None 2023-09-29T22:34:36.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013809338ee810f21-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.417 DEBUG:boto:b'hcnphz-58obj000000000000000002711000falseobj000000000000000002712023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.417 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.418 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.418 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.418 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.418 DEBUG:boto:Method: GET 2023-09-29T22:34:36.418 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.418 DEBUG:boto:Data: 2023-09-29T22:34:36.419 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.419 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.419 DEBUG:boto:Params: {} 2023-09-29T22:34:36.419 DEBUG:boto:Token: None 2023-09-29T22:34:36.419 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb3cfba0d466ecc9-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.428 DEBUG:boto:b'hcnphz-58obj000000000000000002711000falseobj000000000000000002712023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.429 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.429 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.429 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.429 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.429 DEBUG:boto:Data: 2023-09-29T22:34:36.429 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.430 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.430 DEBUG:boto:Params: {} 2023-09-29T22:34:36.430 DEBUG:boto:Token: None 2023-09-29T22:34:36.430 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.430 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wWZJXwTZJKG0LXBWbloSfs44wws= 2023-09-29T22:34:36.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wWZJXwTZJKG0LXBWbloSfs44wws=', 'Content-Length': '0'} 2023-09-29T22:34:36.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060fe4237a5633646-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.433 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.433 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.433 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.433 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.433 DEBUG:boto:Data: 2023-09-29T22:34:36.434 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.434 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.434 DEBUG:boto:Params: {} 2023-09-29T22:34:36.434 DEBUG:boto:Token: None 2023-09-29T22:34:36.434 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wWZJXwTZJKG0LXBWbloSfs44wws= 2023-09-29T22:34:36.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wWZJXwTZJKG0LXBWbloSfs44wws=', 'Content-Length': '0'} 2023-09-29T22:34:36.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd0e688fa5d8d77-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000271 2023-09-29T22:34:36.437 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.437 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.437 DEBUG:boto:Method: GET 2023-09-29T22:34:36.437 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.438 DEBUG:boto:Data: 2023-09-29T22:34:36.438 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.438 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.438 DEBUG:boto:Params: {} 2023-09-29T22:34:36.438 DEBUG:boto:Token: None 2023-09-29T22:34:36.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.439 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X5dXKrjTJwQOXGAn+WqbNhLRQtQ= 2023-09-29T22:34:36.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X5dXKrjTJwQOXGAn+WqbNhLRQtQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e54cc4e290f48e4f-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.482 DEBUG:boto:path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.483 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.483 DEBUG:boto:Method: GET 2023-09-29T22:34:36.483 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.483 DEBUG:boto:Data: 2023-09-29T22:34:36.483 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.483 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.483 DEBUG:boto:Params: {} 2023-09-29T22:34:36.483 DEBUG:boto:Token: None 2023-09-29T22:34:36.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000271 2023-09-29T22:34:36.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X5dXKrjTJwQOXGAn+WqbNhLRQtQ= 2023-09-29T22:34:36.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X5dXKrjTJwQOXGAn+WqbNhLRQtQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd9c37f83fd02d9b-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.487 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.487 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.487 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.487 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.488 DEBUG:boto:Method: GET 2023-09-29T22:34:36.488 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.488 DEBUG:boto:Data: 2023-09-29T22:34:36.488 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.488 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.488 DEBUG:boto:Params: {} 2023-09-29T22:34:36.489 DEBUG:boto:Token: None 2023-09-29T22:34:36.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff704368c8f60f80-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.493 DEBUG:boto:b'hcnphz-58obj000000000000000002711000falseobj000000000000000002712023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.494 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.494 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.494 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.494 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.494 DEBUG:boto:Method: GET 2023-09-29T22:34:36.494 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000271 2023-09-29T22:34:36.494 DEBUG:boto:Data: 2023-09-29T22:34:36.495 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.495 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.495 DEBUG:boto:Params: {} 2023-09-29T22:34:36.495 DEBUG:boto:Token: None 2023-09-29T22:34:36.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.495 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.503 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e66a492cfe432eb-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.546 DEBUG:boto:b'hcnphz-58obj000000000000000002711000falseobj000000000000000002712023-09-29T22:34:14.133Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-09-29T22:34:36.547 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.548 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.548 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.548 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.548 DEBUG:boto:Method: GET 2023-09-29T22:34:36.548 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.549 DEBUG:boto:Data: 2023-09-29T22:34:36.549 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.549 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.549 DEBUG:boto:Params: {} 2023-09-29T22:34:36.549 DEBUG:boto:Token: None 2023-09-29T22:34:36.550 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.550 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ= 2023-09-29T22:34:36.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000377eeef9fefbbe7c-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.554 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.554 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.554 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.554 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.555 DEBUG:boto:Method: GET 2023-09-29T22:34:36.555 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.555 DEBUG:boto:Data: 2023-09-29T22:34:36.555 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.555 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.556 DEBUG:boto:Params: {} 2023-09-29T22:34:36.556 DEBUG:boto:Token: None 2023-09-29T22:34:36.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ= 2023-09-29T22:34:36.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f292d1a4bfbbd3a6-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.559 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.560 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.561 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.561 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.561 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.561 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.561 DEBUG:boto:Data: 2023-09-29T22:34:36.561 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.562 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.562 DEBUG:boto:Params: {} 2023-09-29T22:34:36.562 DEBUG:boto:Token: None 2023-09-29T22:34:36.562 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QBGAR8oW5mMR+/jRv/rCs3Mhvqs= 2023-09-29T22:34:36.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QBGAR8oW5mMR+/jRv/rCs3Mhvqs=', 'Content-Length': '0'} 2023-09-29T22:34:36.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e6017b1abe6fe338-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.565 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.566 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.566 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.566 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.566 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.566 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.566 DEBUG:boto:Data: 2023-09-29T22:34:36.566 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.567 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.567 DEBUG:boto:Params: {} 2023-09-29T22:34:36.567 DEBUG:boto:Token: None 2023-09-29T22:34:36.567 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QBGAR8oW5mMR+/jRv/rCs3Mhvqs= 2023-09-29T22:34:36.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QBGAR8oW5mMR+/jRv/rCs3Mhvqs=', 'Content-Length': '0'} 2023-09-29T22:34:36.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c8099becbdaf629-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-09-29T22:34:36.570 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.571 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.571 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.571 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.571 DEBUG:boto:Method: GET 2023-09-29T22:34:36.571 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.571 DEBUG:boto:Data: 2023-09-29T22:34:36.571 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.571 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.571 DEBUG:boto:Params: {} 2023-09-29T22:34:36.572 DEBUG:boto:Token: None 2023-09-29T22:34:36.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ= 2023-09-29T22:34:36.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.575 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5e59ef074b8c0f8-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.575 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.575 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.575 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.575 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.576 DEBUG:boto:Method: GET 2023-09-29T22:34:36.576 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.576 DEBUG:boto:Data: 2023-09-29T22:34:36.576 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.576 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.576 DEBUG:boto:Params: {} 2023-09-29T22:34:36.576 DEBUG:boto:Token: None 2023-09-29T22:34:36.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283?versionId=null 2023-09-29T22:34:36.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ= 2023-09-29T22:34:36.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GlA4M4bSllhzetBI4mHsD0gfZSQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061083234be110c72-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.580 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.580 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.580 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.580 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.580 DEBUG:boto:Method: GET 2023-09-29T22:34:36.580 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.580 DEBUG:boto:Data: 2023-09-29T22:34:36.580 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.581 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.581 DEBUG:boto:Params: {} 2023-09-29T22:34:36.581 DEBUG:boto:Token: None 2023-09-29T22:34:36.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.586 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd0f2a01e8e50049-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.586 DEBUG:boto:b'hcnphz-58obj000000000000000002831000falseobj000000000000000002832023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.586 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.586 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.587 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.587 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.587 DEBUG:boto:Method: GET 2023-09-29T22:34:36.587 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.587 DEBUG:boto:Data: 2023-09-29T22:34:36.587 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.587 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.587 DEBUG:boto:Params: {} 2023-09-29T22:34:36.587 DEBUG:boto:Token: None 2023-09-29T22:34:36.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d2bb2daeb2875ef-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.642 DEBUG:boto:b'hcnphz-58obj000000000000000002831000falseobj000000000000000002832023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.643 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.643 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.643 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.643 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.643 DEBUG:boto:Data: 2023-09-29T22:34:36.644 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.644 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.644 DEBUG:boto:Params: {} 2023-09-29T22:34:36.644 DEBUG:boto:Token: None 2023-09-29T22:34:36.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0VN1yISFeRIUCbCo09ZlDakSaO0= 2023-09-29T22:34:36.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0VN1yISFeRIUCbCo09ZlDakSaO0=', 'Content-Length': '0'} 2023-09-29T22:34:36.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6e2d15deeb667f4-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.647 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.648 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.648 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.648 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.648 DEBUG:boto:Data: 2023-09-29T22:34:36.648 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.648 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.648 DEBUG:boto:Params: {} 2023-09-29T22:34:36.649 DEBUG:boto:Token: None 2023-09-29T22:34:36.649 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0VN1yISFeRIUCbCo09ZlDakSaO0= 2023-09-29T22:34:36.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0VN1yISFeRIUCbCo09ZlDakSaO0=', 'Content-Length': '0'} 2023-09-29T22:34:36.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024193922d787d3cb-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000283 2023-09-29T22:34:36.652 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.652 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.652 DEBUG:boto:Method: GET 2023-09-29T22:34:36.653 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.653 DEBUG:boto:Data: 2023-09-29T22:34:36.653 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.653 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.653 DEBUG:boto:Params: {} 2023-09-29T22:34:36.653 DEBUG:boto:Token: None 2023-09-29T22:34:36.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Sk0Y2R/TU864j7X/81vWPndsYE= 2023-09-29T22:34:36.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Sk0Y2R/TU864j7X/81vWPndsYE=', 'Content-Length': '0'} 2023-09-29T22:34:36.656 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecb24b352809a13e-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.698 DEBUG:boto:path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.699 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.699 DEBUG:boto:Method: GET 2023-09-29T22:34:36.699 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.699 DEBUG:boto:Data: 2023-09-29T22:34:36.699 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.699 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.699 DEBUG:boto:Params: {} 2023-09-29T22:34:36.700 DEBUG:boto:Token: None 2023-09-29T22:34:36.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000283 2023-09-29T22:34:36.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Sk0Y2R/TU864j7X/81vWPndsYE= 2023-09-29T22:34:36.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Sk0Y2R/TU864j7X/81vWPndsYE=', 'Content-Length': '0'} 2023-09-29T22:34:36.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cfe3f4ef3e3e7fd0-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.704 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.704 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.704 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.704 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.704 DEBUG:boto:Method: GET 2023-09-29T22:34:36.704 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.705 DEBUG:boto:Data: 2023-09-29T22:34:36.705 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.705 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.705 DEBUG:boto:Params: {} 2023-09-29T22:34:36.705 DEBUG:boto:Token: None 2023-09-29T22:34:36.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003acd356101a2d0a6-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.710 DEBUG:boto:b'hcnphz-58obj000000000000000002831000falseobj000000000000000002832023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.710 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.711 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.711 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.711 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.711 DEBUG:boto:Method: GET 2023-09-29T22:34:36.711 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000283 2023-09-29T22:34:36.711 DEBUG:boto:Data: 2023-09-29T22:34:36.711 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.712 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.712 DEBUG:boto:Params: {} 2023-09-29T22:34:36.712 DEBUG:boto:Token: None 2023-09-29T22:34:36.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071b64a7e26d1bf4d-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.721 DEBUG:boto:b'hcnphz-58obj000000000000000002831000falseobj000000000000000002832023-09-29T22:34:14.455Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-09-29T22:34:36.721 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.721 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.721 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.721 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.722 DEBUG:boto:Method: GET 2023-09-29T22:34:36.722 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.722 DEBUG:boto:Data: 2023-09-29T22:34:36.722 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.722 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.722 DEBUG:boto:Params: {} 2023-09-29T22:34:36.722 DEBUG:boto:Token: None 2023-09-29T22:34:36.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ= 2023-09-29T22:34:36.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009709a7a09993fc5c-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.725 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.725 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.725 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.726 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.726 DEBUG:boto:Method: GET 2023-09-29T22:34:36.726 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.726 DEBUG:boto:Data: 2023-09-29T22:34:36.726 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.726 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.726 DEBUG:boto:Params: {} 2023-09-29T22:34:36.726 DEBUG:boto:Token: None 2023-09-29T22:34:36.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ= 2023-09-29T22:34:36.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064201b4adf40b1fa-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.770 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.770 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.771 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.771 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.771 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.771 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.771 DEBUG:boto:Data: 2023-09-29T22:34:36.771 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.771 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.772 DEBUG:boto:Params: {} 2023-09-29T22:34:36.772 DEBUG:boto:Token: None 2023-09-29T22:34:36.772 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.772 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ww/brW24gaUeHAHZt1E6+ghMh4= 2023-09-29T22:34:36.772 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ww/brW24gaUeHAHZt1E6+ghMh4=', 'Content-Length': '0'} 2023-09-29T22:34:36.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003796c302375f628a-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.775 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.775 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.775 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.776 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.776 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.776 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.776 DEBUG:boto:Data: 2023-09-29T22:34:36.776 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.776 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.776 DEBUG:boto:Params: {} 2023-09-29T22:34:36.776 DEBUG:boto:Token: None 2023-09-29T22:34:36.777 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ww/brW24gaUeHAHZt1E6+ghMh4= 2023-09-29T22:34:36.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ww/brW24gaUeHAHZt1E6+ghMh4=', 'Content-Length': '0'} 2023-09-29T22:34:36.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000853d370ed6e0a9e6-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-09-29T22:34:36.780 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.780 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.780 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.780 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.780 DEBUG:boto:Method: GET 2023-09-29T22:34:36.781 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.781 DEBUG:boto:Data: 2023-09-29T22:34:36.781 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.781 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.781 DEBUG:boto:Params: {} 2023-09-29T22:34:36.781 DEBUG:boto:Token: None 2023-09-29T22:34:36.781 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ= 2023-09-29T22:34:36.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.784 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df632d31d2cf1c14-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.826 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.826 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.827 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.827 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.827 DEBUG:boto:Method: GET 2023-09-29T22:34:36.827 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.827 DEBUG:boto:Data: 2023-09-29T22:34:36.827 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.827 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.827 DEBUG:boto:Params: {} 2023-09-29T22:34:36.828 DEBUG:boto:Token: None 2023-09-29T22:34:36.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317?versionId=null 2023-09-29T22:34:36.828 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ= 2023-09-29T22:34:36.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WlxLmTm7opc6DhcvRJQBHvE+UPQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000205f885bbe64e173-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.831 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.831 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.831 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.832 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.832 DEBUG:boto:Method: GET 2023-09-29T22:34:36.832 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.832 DEBUG:boto:Data: 2023-09-29T22:34:36.832 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.832 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.832 DEBUG:boto:Params: {} 2023-09-29T22:34:36.833 DEBUG:boto:Token: None 2023-09-29T22:34:36.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b026340277c2681-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.837 DEBUG:boto:b'hcnphz-58obj000000000000000003171000falseobj000000000000000003172023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.838 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.838 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.838 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.838 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.838 DEBUG:boto:Method: GET 2023-09-29T22:34:36.839 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.839 DEBUG:boto:Data: 2023-09-29T22:34:36.839 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.839 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.839 DEBUG:boto:Params: {} 2023-09-29T22:34:36.839 DEBUG:boto:Token: None 2023-09-29T22:34:36.839 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.839 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee5aceee5e27a311-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.847 DEBUG:boto:b'hcnphz-58obj000000000000000003171000falseobj000000000000000003172023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.848 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.848 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.848 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.848 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.848 DEBUG:boto:Data: 2023-09-29T22:34:36.848 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.849 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.849 DEBUG:boto:Params: {} 2023-09-29T22:34:36.849 DEBUG:boto:Token: None 2023-09-29T22:34:36.849 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nWdcXT+8P5OtQbW9dpYxOrQEkfg= 2023-09-29T22:34:36.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nWdcXT+8P5OtQbW9dpYxOrQEkfg=', 'Content-Length': '0'} 2023-09-29T22:34:36.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000659fdc5d2dfafa5d-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.851 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.851 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.852 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.852 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.852 DEBUG:boto:Data: 2023-09-29T22:34:36.852 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.852 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.852 DEBUG:boto:Params: {} 2023-09-29T22:34:36.852 DEBUG:boto:Token: None 2023-09-29T22:34:36.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.853 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nWdcXT+8P5OtQbW9dpYxOrQEkfg= 2023-09-29T22:34:36.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nWdcXT+8P5OtQbW9dpYxOrQEkfg=', 'Content-Length': '0'} 2023-09-29T22:34:36.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034cd4b1650c4f176-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000317 2023-09-29T22:34:36.855 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.855 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.856 DEBUG:boto:Method: GET 2023-09-29T22:34:36.856 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.856 DEBUG:boto:Data: 2023-09-29T22:34:36.856 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.856 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.856 DEBUG:boto:Params: {} 2023-09-29T22:34:36.856 DEBUG:boto:Token: None 2023-09-29T22:34:36.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QvJJqDPM7MFmNq9XLkjQ0N2GAgc= 2023-09-29T22:34:36.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QvJJqDPM7MFmNq9XLkjQ0N2GAgc=', 'Content-Length': '0'} 2023-09-29T22:34:36.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c794dc0b5da49962-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.902 DEBUG:boto:path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.903 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.903 DEBUG:boto:Method: GET 2023-09-29T22:34:36.903 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.903 DEBUG:boto:Data: 2023-09-29T22:34:36.903 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.903 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.903 DEBUG:boto:Params: {} 2023-09-29T22:34:36.904 DEBUG:boto:Token: None 2023-09-29T22:34:36.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000317 2023-09-29T22:34:36.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QvJJqDPM7MFmNq9XLkjQ0N2GAgc= 2023-09-29T22:34:36.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QvJJqDPM7MFmNq9XLkjQ0N2GAgc=', 'Content-Length': '0'} 2023-09-29T22:34:36.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000966268e45421a559-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.908 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.908 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.908 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.908 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.908 DEBUG:boto:Method: GET 2023-09-29T22:34:36.908 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.908 DEBUG:boto:Data: 2023-09-29T22:34:36.909 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.909 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.909 DEBUG:boto:Params: {} 2023-09-29T22:34:36.909 DEBUG:boto:Token: None 2023-09-29T22:34:36.909 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.909 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.914 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063eff483ad7da743-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.914 DEBUG:boto:b'hcnphz-58obj000000000000000003171000falseobj000000000000000003172023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.915 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.915 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.915 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.915 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.915 DEBUG:boto:Method: GET 2023-09-29T22:34:36.916 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000317 2023-09-29T22:34:36.916 DEBUG:boto:Data: 2023-09-29T22:34:36.916 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.916 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.916 DEBUG:boto:Params: {} 2023-09-29T22:34:36.916 DEBUG:boto:Token: None 2023-09-29T22:34:36.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.916 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.924 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000032a6164ba16bcb92-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.924 DEBUG:boto:b'hcnphz-58obj000000000000000003171000falseobj000000000000000003172023-09-29T22:34:13.992Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-09-29T22:34:36.925 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.925 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.925 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.925 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.925 DEBUG:boto:Method: GET 2023-09-29T22:34:36.925 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.926 DEBUG:boto:Data: 2023-09-29T22:34:36.926 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.926 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.926 DEBUG:boto:Params: {} 2023-09-29T22:34:36.926 DEBUG:boto:Token: None 2023-09-29T22:34:36.926 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.926 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI= 2023-09-29T22:34:36.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI=', 'Content-Length': '0'} 2023-09-29T22:34:36.929 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbc39e9e00ba152b-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.929 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.929 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.929 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.929 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.930 DEBUG:boto:Method: GET 2023-09-29T22:34:36.930 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.930 DEBUG:boto:Data: 2023-09-29T22:34:36.930 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.930 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.930 DEBUG:boto:Params: {} 2023-09-29T22:34:36.930 DEBUG:boto:Token: None 2023-09-29T22:34:36.930 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI= 2023-09-29T22:34:36.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI=', 'Content-Length': '0'} 2023-09-29T22:34:36.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0d68a6c66dc4b71-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.933 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.933 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.933 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.933 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.933 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.933 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.933 DEBUG:boto:Data: 2023-09-29T22:34:36.934 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.934 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.934 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.934 DEBUG:boto:Params: {} 2023-09-29T22:34:36.934 DEBUG:boto:Token: None 2023-09-29T22:34:36.934 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2056KjYPejnVy8XZIcU2WTCeLw= 2023-09-29T22:34:36.935 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2056KjYPejnVy8XZIcU2WTCeLw=', 'Content-Length': '0'} 2023-09-29T22:34:36.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f68eeba167e79039-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.937 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.937 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.937 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.937 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.937 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.937 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.937 DEBUG:boto:Data: 2023-09-29T22:34:36.937 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.938 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.938 DEBUG:boto:Params: {} 2023-09-29T22:34:36.938 DEBUG:boto:Token: None 2023-09-29T22:34:36.938 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.938 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X2056KjYPejnVy8XZIcU2WTCeLw= 2023-09-29T22:34:36.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X2056KjYPejnVy8XZIcU2WTCeLw=', 'Content-Length': '0'} 2023-09-29T22:34:36.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020486b6be3e7b3fb-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-09-29T22:34:36.941 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.941 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.941 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.941 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.941 DEBUG:boto:Method: GET 2023-09-29T22:34:36.942 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.942 DEBUG:boto:Data: 2023-09-29T22:34:36.942 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.942 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.942 DEBUG:boto:Params: {} 2023-09-29T22:34:36.943 DEBUG:boto:Token: None 2023-09-29T22:34:36.943 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI= 2023-09-29T22:34:36.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI=', 'Content-Length': '0'} 2023-09-29T22:34:36.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675cb72a52892267-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.946 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.946 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.946 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.946 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.946 DEBUG:boto:Method: GET 2023-09-29T22:34:36.946 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.946 DEBUG:boto:Data: 2023-09-29T22:34:36.946 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.947 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.947 DEBUG:boto:Params: {} 2023-09-29T22:34:36.947 DEBUG:boto:Token: None 2023-09-29T22:34:36.947 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423?versionId=null 2023-09-29T22:34:36.947 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI= 2023-09-29T22:34:36.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qL+NkGswOAUIw35uFhYw1oaLkKI=', 'Content-Length': '0'} 2023-09-29T22:34:36.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012b7619bbff55ff1-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.950 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.950 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.950 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:36.950 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:36.951 DEBUG:boto:Method: GET 2023-09-29T22:34:36.951 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:36.951 DEBUG:boto:Data: 2023-09-29T22:34:36.951 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.951 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.951 DEBUG:boto:Params: {} 2023-09-29T22:34:36.952 DEBUG:boto:Token: None 2023-09-29T22:34:36.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.956 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f48f17786a633f52-00651750fc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.956 DEBUG:boto:b'hcnphz-58obj000000000000000004231000falseobj000000000000000004232023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.957 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:36.957 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:36.957 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:36.957 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:36.957 DEBUG:boto:Method: GET 2023-09-29T22:34:36.957 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:36.957 DEBUG:boto:Data: 2023-09-29T22:34:36.957 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.958 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.958 DEBUG:boto:Params: {} 2023-09-29T22:34:36.958 DEBUG:boto:Token: None 2023-09-29T22:34:36.958 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/ 2023-09-29T22:34:36.958 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs= 2023-09-29T22:34:36.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rW8ep3eTzMG4Nr4MnRuFu4T20xs=', 'Content-Length': '0'} 2023-09-29T22:34:36.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb4a0d8da85ab469-00651750fc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.966 DEBUG:boto:b'hcnphz-58obj000000000000000004231000falseobj000000000000000004232023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:36.966 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.967 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.967 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.967 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.967 DEBUG:boto:Data: 2023-09-29T22:34:36.967 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.967 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.967 DEBUG:boto:Params: {} 2023-09-29T22:34:36.967 DEBUG:boto:Token: None 2023-09-29T22:34:36.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WtGG3FJOaJOuJEGn+YCizklFbVQ= 2023-09-29T22:34:36.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WtGG3FJOaJOuJEGn+YCizklFbVQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005762f579e639c37f-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.970 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.970 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.970 DEBUG:boto:Method: HEAD 2023-09-29T22:34:36.970 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.970 DEBUG:boto:Data: 2023-09-29T22:34:36.971 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:36.971 DEBUG:boto:Port: 8000 2023-09-29T22:34:36.971 DEBUG:boto:Params: {} 2023-09-29T22:34:36.971 DEBUG:boto:Token: None 2023-09-29T22:34:36.971 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.971 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WtGG3FJOaJOuJEGn+YCizklFbVQ= 2023-09-29T22:34:36.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WtGG3FJOaJOuJEGn+YCizklFbVQ=', 'Content-Length': '0'} 2023-09-29T22:34:36.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc418ef7f7587b3-00651750fc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:36.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000423 2023-09-29T22:34:36.974 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.974 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.974 DEBUG:boto:Method: GET 2023-09-29T22:34:36.974 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.974 DEBUG:boto:Data: 2023-09-29T22:34:36.975 DEBUG:boto:Headers: {} 2023-09-29T22:34:36.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:36.975 DEBUG:boto:Port: 8001 2023-09-29T22:34:36.975 DEBUG:boto:Params: {} 2023-09-29T22:34:36.975 DEBUG:boto:Token: None 2023-09-29T22:34:36.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:36 GMT /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:36.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mKubOT0nhlndliJ1lsfbZD2kmb0= 2023-09-29T22:34:36.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mKubOT0nhlndliJ1lsfbZD2kmb0=', 'Content-Length': '0'} 2023-09-29T22:34:36.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cdb9c5f18801f4c-00651750fc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.018 DEBUG:boto:path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:37.018 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000423 2023-09-29T22:34:37.019 DEBUG:boto:Method: GET 2023-09-29T22:34:37.019 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:37.019 DEBUG:boto:Data: 2023-09-29T22:34:37.019 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.019 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.019 DEBUG:boto:Params: {} 2023-09-29T22:34:37.019 DEBUG:boto:Token: None 2023-09-29T22:34:37.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000423 2023-09-29T22:34:37.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xXNNXsh3QDjNiN0bjAJReUHkEEc= 2023-09-29T22:34:37.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xXNNXsh3QDjNiN0bjAJReUHkEEc=', 'Content-Length': '0'} 2023-09-29T22:34:37.022 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db3da07b68f30792-00651750fd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.023 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:37.023 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:37.023 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:37.023 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:37.023 DEBUG:boto:Method: GET 2023-09-29T22:34:37.023 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:37.023 DEBUG:boto:Data: 2023-09-29T22:34:37.023 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.024 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.024 DEBUG:boto:Params: {} 2023-09-29T22:34:37.024 DEBUG:boto:Token: None 2023-09-29T22:34:37.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/ 2023-09-29T22:34:37.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y= 2023-09-29T22:34:37.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y=', 'Content-Length': '0'} 2023-09-29T22:34:37.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd8771e398716781-00651750fd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.029 DEBUG:boto:b'hcnphz-58obj000000000000000004231000falseobj000000000000000004232023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:37.029 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:37.029 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:37.030 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:37.030 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:37.030 DEBUG:boto:Method: GET 2023-09-29T22:34:37.030 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000423 2023-09-29T22:34:37.030 DEBUG:boto:Data: 2023-09-29T22:34:37.030 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.030 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.030 DEBUG:boto:Params: {} 2023-09-29T22:34:37.031 DEBUG:boto:Token: None 2023-09-29T22:34:37.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/ 2023-09-29T22:34:37.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y= 2023-09-29T22:34:37.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y=', 'Content-Length': '0'} 2023-09-29T22:34:37.039 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bae899c60def9224-00651750fd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.039 DEBUG:boto:b'hcnphz-58obj000000000000000004231000falseobj000000000000000004232023-09-29T22:34:14.812Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:37.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-09-29T22:34:37.039 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.039 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.040 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.040 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.040 DEBUG:boto:Method: GET 2023-09-29T22:34:37.040 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.040 DEBUG:boto:Data: 2023-09-29T22:34:37.040 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.040 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.040 DEBUG:boto:Params: {} 2023-09-29T22:34:37.041 DEBUG:boto:Token: None 2023-09-29T22:34:37.041 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts= 2023-09-29T22:34:37.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts=', 'Content-Length': '0'} 2023-09-29T22:34:37.043 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a490a48015ffc2fd-00651750fd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.044 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.044 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.044 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.044 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.044 DEBUG:boto:Method: GET 2023-09-29T22:34:37.044 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.044 DEBUG:boto:Data: 2023-09-29T22:34:37.044 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.044 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.044 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.045 DEBUG:boto:Params: {} 2023-09-29T22:34:37.045 DEBUG:boto:Token: None 2023-09-29T22:34:37.045 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.045 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts= 2023-09-29T22:34:37.045 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts=', 'Content-Length': '0'} 2023-09-29T22:34:37.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000681bf4fd66743ccf-00651750fd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.048 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.048 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.048 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.048 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.048 DEBUG:boto:Method: HEAD 2023-09-29T22:34:37.048 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.048 DEBUG:boto:Data: 2023-09-29T22:34:37.049 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.049 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.049 DEBUG:boto:Params: {} 2023-09-29T22:34:37.049 DEBUG:boto:Token: None 2023-09-29T22:34:37.049 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YqHhGdivZS+lfaWydUK4V0hzDew= 2023-09-29T22:34:37.050 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YqHhGdivZS+lfaWydUK4V0hzDew=', 'Content-Length': '0'} 2023-09-29T22:34:37.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3bb3ad6a3571f7-00651750fd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.052 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.052 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.052 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.052 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.052 DEBUG:boto:Method: HEAD 2023-09-29T22:34:37.052 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.053 DEBUG:boto:Data: 2023-09-29T22:34:37.053 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.053 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.053 DEBUG:boto:Params: {} 2023-09-29T22:34:37.053 DEBUG:boto:Token: None 2023-09-29T22:34:37.053 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.053 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YqHhGdivZS+lfaWydUK4V0hzDew= 2023-09-29T22:34:37.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YqHhGdivZS+lfaWydUK4V0hzDew=', 'Content-Length': '0'} 2023-09-29T22:34:37.056 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd0af273df54e21-00651750fd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-09-29T22:34:37.056 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.056 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.056 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.056 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.056 DEBUG:boto:Method: GET 2023-09-29T22:34:37.057 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.057 DEBUG:boto:Data: 2023-09-29T22:34:37.057 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.057 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.057 DEBUG:boto:Params: {} 2023-09-29T22:34:37.057 DEBUG:boto:Token: None 2023-09-29T22:34:37.058 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.058 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts= 2023-09-29T22:34:37.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts=', 'Content-Length': '0'} 2023-09-29T22:34:37.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e8c0220b99043fe-00651750fd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.102 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.102 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.103 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.103 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.103 DEBUG:boto:Method: GET 2023-09-29T22:34:37.103 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.104 DEBUG:boto:Data: 2023-09-29T22:34:37.104 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.104 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.104 DEBUG:boto:Params: {} 2023-09-29T22:34:37.104 DEBUG:boto:Token: None 2023-09-29T22:34:37.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467?versionId=null 2023-09-29T22:34:37.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts= 2023-09-29T22:34:37.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0XHHmFbeAKVdS5w3A+vEjnuAhts=', 'Content-Length': '0'} 2023-09-29T22:34:37.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447fdf79bc53127a-00651750fd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.107 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:37.108 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:37.108 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.108 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.108 DEBUG:boto:Method: GET 2023-09-29T22:34:37.108 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.108 DEBUG:boto:Data: 2023-09-29T22:34:37.108 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.108 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.109 DEBUG:boto:Params: {} 2023-09-29T22:34:37.109 DEBUG:boto:Token: None 2023-09-29T22:34:37.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/ 2023-09-29T22:34:37.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y= 2023-09-29T22:34:37.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y=', 'Content-Length': '0'} 2023-09-29T22:34:37.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf7ddaeec16ffec-00651750fd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.114 DEBUG:boto:b'hcnphz-58obj000000000000000004671000falseobj000000000000000004672023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:37.114 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:37.114 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:37.114 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.115 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.115 DEBUG:boto:Method: GET 2023-09-29T22:34:37.115 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.115 DEBUG:boto:Data: 2023-09-29T22:34:37.115 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.115 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.115 DEBUG:boto:Params: {} 2023-09-29T22:34:37.115 DEBUG:boto:Token: None 2023-09-29T22:34:37.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/ 2023-09-29T22:34:37.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y= 2023-09-29T22:34:37.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y=', 'Content-Length': '0'} 2023-09-29T22:34:37.124 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fcdfd9e5a5536a7-00651750fd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.124 DEBUG:boto:b'hcnphz-58obj000000000000000004671000falseobj000000000000000004672023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:37.124 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.124 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.124 DEBUG:boto:Method: HEAD 2023-09-29T22:34:37.125 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.125 DEBUG:boto:Data: 2023-09-29T22:34:37.125 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.125 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.125 DEBUG:boto:Params: {} 2023-09-29T22:34:37.125 DEBUG:boto:Token: None 2023-09-29T22:34:37.125 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ee906fd3vr5QSE6HvPZBA8Eooj0= 2023-09-29T22:34:37.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ee906fd3vr5QSE6HvPZBA8Eooj0=', 'Content-Length': '0'} 2023-09-29T22:34:37.128 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ef1866a89b2dd3-00651750fd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.128 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.128 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.128 DEBUG:boto:Method: HEAD 2023-09-29T22:34:37.129 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.129 DEBUG:boto:Data: 2023-09-29T22:34:37.129 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.129 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.129 DEBUG:boto:Params: {} 2023-09-29T22:34:37.129 DEBUG:boto:Token: None 2023-09-29T22:34:37.129 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ee906fd3vr5QSE6HvPZBA8Eooj0= 2023-09-29T22:34:37.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ee906fd3vr5QSE6HvPZBA8Eooj0=', 'Content-Length': '0'} 2023-09-29T22:34:37.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031b2294a08a183a9-00651750fd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj00000000000000000467 2023-09-29T22:34:37.132 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.132 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.132 DEBUG:boto:Method: GET 2023-09-29T22:34:37.132 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.132 DEBUG:boto:Data: 2023-09-29T22:34:37.133 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.133 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.133 DEBUG:boto:Params: {} 2023-09-29T22:34:37.133 DEBUG:boto:Token: None 2023-09-29T22:34:37.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UhmSkAj99auQ7xTbP4bxAmhIiRQ= 2023-09-29T22:34:37.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UhmSkAj99auQ7xTbP4bxAmhIiRQ=', 'Content-Length': '0'} 2023-09-29T22:34:37.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-58:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.23'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026614fba63ede46d-00651750fd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.136 DEBUG:boto:path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.136 DEBUG:boto:auth_path=/hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.136 DEBUG:boto:Method: GET 2023-09-29T22:34:37.136 DEBUG:boto:Path: /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.136 DEBUG:boto:Data: 2023-09-29T22:34:37.136 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.137 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.137 DEBUG:boto:Params: {} 2023-09-29T22:34:37.137 DEBUG:boto:Token: None 2023-09-29T22:34:37.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/obj00000000000000000467 2023-09-29T22:34:37.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UhmSkAj99auQ7xTbP4bxAmhIiRQ= 2023-09-29T22:34:37.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UhmSkAj99auQ7xTbP4bxAmhIiRQ=', 'Content-Length': '0'} 2023-09-29T22:34:37.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:34:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec762476cef6d93-00651750fd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.182 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:37.182 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:37.183 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.183 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.183 DEBUG:boto:Method: GET 2023-09-29T22:34:37.183 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.183 DEBUG:boto:Data: 2023-09-29T22:34:37.183 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:34:37.183 DEBUG:boto:Port: 8001 2023-09-29T22:34:37.183 DEBUG:boto:Params: {} 2023-09-29T22:34:37.183 DEBUG:boto:Token: None 2023-09-29T22:34:37.184 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/ 2023-09-29T22:34:37.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y= 2023-09-29T22:34:37.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y=', 'Content-Length': '0'} 2023-09-29T22:34:37.188 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db58ead967e4a0da-00651750fd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.188 DEBUG:boto:b'hcnphz-58obj000000000000000004671000falseobj000000000000000004672023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:37.189 DEBUG:boto:path=/hcnphz-58/ 2023-09-29T22:34:37.189 DEBUG:boto:auth_path=/hcnphz-58/ 2023-09-29T22:34:37.189 DEBUG:boto:path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.189 DEBUG:boto:auth_path=/hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.189 DEBUG:boto:Method: GET 2023-09-29T22:34:37.189 DEBUG:boto:Path: /hcnphz-58/?prefix=obj00000000000000000467 2023-09-29T22:34:37.189 DEBUG:boto:Data: 2023-09-29T22:34:37.189 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.189 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.190 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.190 DEBUG:boto:Params: {} 2023-09-29T22:34:37.190 DEBUG:boto:Token: None 2023-09-29T22:34:37.190 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-58/ 2023-09-29T22:34:37.190 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y= 2023-09-29T22:34:37.190 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+40NZI4j2L0RDS+y+5DgmGwP0Y=', 'Content-Length': '0'} 2023-09-29T22:34:37.199 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001634492bac8f0325-00651750fd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.199 DEBUG:boto:b'hcnphz-58obj000000000000000004671000falseobj000000000000000004672023-09-29T22:34:14.584Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:34:37.199 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-58 zones={a2, a1} 2023-09-29T22:34:37.200 INFO:tasks.rgw_multisite_tests:Create several generations of objects, then run bucket sync ... ok 2023-09-29T22:34:37.202 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:34:37.202 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:34:37.203 DEBUG:boto:Method: PUT 2023-09-29T22:34:37.203 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:34:37.203 DEBUG:boto:Data: 2023-09-29T22:34:37.203 DEBUG:boto:Headers: {} 2023-09-29T22:34:37.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:34:37.203 DEBUG:boto:Port: 8000 2023-09-29T22:34:37.203 DEBUG:boto:Params: {} 2023-09-29T22:34:37.203 DEBUG:boto:Token: None 2023-09-29T22:34:37.203 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:34:37 GMT /hcnphz-59/ 2023-09-29T22:34:37.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TL8UeAs+BVESGUJhBwbqR5MCJ0o= 2023-09-29T22:34:37.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:34:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TL8UeAs+BVESGUJhBwbqR5MCJ0o=', 'Content-Length': '0'} 2023-09-29T22:34:37.233 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fb2e76de62460b99-00651750fd-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:34:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:34:37.234 DEBUG:rgw_multi.tests:created bucket=hcnphz-59 2023-09-29T22:34:37.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:37.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:37.235 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:37.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:37.277 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:37.336 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:34:37.365 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:34:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:34:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T22:34:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026877.230172_504.1", 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:34:37.230172Z" 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:34:37.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:37.367 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:34:37.368 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:34:37.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:34:37.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:37.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:37.369 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:37.413 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:37.413 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:37.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:37.467+0000 7fd0ce5fbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:37.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:37.467+0000 7fd0ce5fbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:37.497 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:34:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:34:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:34:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:34:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:34:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:34:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:34:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:34:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:34:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:34:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:34:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:34:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:34:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:34:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:34:37Z" 2023-09-29T22:34:37.507 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:37.508 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:37Z" } 2023-09-29T22:34:37.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:34:37.509 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:34:37.509 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:34:37.509 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:34:42.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:42.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:42.510 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:42.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:42.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:42.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:42.611+0000 7f740edd5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:42.614 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:42.611+0000 7f740edd5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:42.640 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:34:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:34:42.640 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:42.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:42.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:34:42.643 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:42.644 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:42.645 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:34:42.646 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:34:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:34:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:42.647 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:34:42Z" 2023-09-29T22:34:42.647 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:42.648 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:42Z" } 2023-09-29T22:34:42.648 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:34:42.648 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:34:42.648 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:34:42.649 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:34:47.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:47.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:47.650 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:47.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:47.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:47.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:47.751+0000 7fd77d632f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:47.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:47.751+0000 7fd77d632f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:47.781 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:47.782 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:34:47.783 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:47.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:47.785 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:47.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:47.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:34:47Z" 2023-09-29T22:34:47.788 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:47.789 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:47Z" } 2023-09-29T22:34:47.789 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:34:47.789 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:34:47.790 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:34:47.790 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:34:52.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:52.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:52.791 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:52.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:52.838 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:52.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:52.891+0000 7f71c886af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:52.895 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:52.891+0000 7f71c886af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:52.920 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:52.920 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:34:52.920 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:34:52.920 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:34:52.920 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:34:52.921 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:34:52.922 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:52.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:52.924 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:52.925 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:34:52.926 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:34:52.927 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:34:52.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:52.927 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:34:52Z" 2023-09-29T22:34:52.927 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:52.928 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:52Z" } 2023-09-29T22:34:52.928 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:34:52.928 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:34:52.928 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:34:52.928 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:34:57.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:57.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:34:57.930 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:34:57.976 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:34:57.976 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:34:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:58.027+0000 7f6b14c17f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:58.033 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:34:58.031+0000 7f6b14c17f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:34:58.059 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:34:58.059 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:34:58.060 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:58.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:34:58.062 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:58.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:34:58.064 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:34:58.065 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:34:58.066 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:34:58.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:34:58.066 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:34:58Z" 2023-09-29T22:34:58.066 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:34:58.067 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:34:58Z" } 2023-09-29T22:34:58.067 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:34:58.067 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:34:58.067 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:34:58.067 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:03.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:03.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:03.069 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:03.114 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:03.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:03.174 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:03.171+0000 7f9266584f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:03.175 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:03.171+0000 7f9266584f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:03.202 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:03.203 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:03.204 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:03.205 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:03.206 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:03.207 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:03.208 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:03.209 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:03.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:03.209 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:03Z" 2023-09-29T22:35:03.209 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:03.210 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:03Z" } 2023-09-29T22:35:03.210 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:03.210 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:03.210 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:03.210 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:08.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:08.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:08.212 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:08.258 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:08.258 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:08.315 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:08.310+0000 7f014b188f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:08.316 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:08.314+0000 7f014b188f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:08.341 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:08.341 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:08.341 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:08.342 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:08.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:08.344 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:08.345 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:08.346 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:08.347 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:08.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:08.348 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:08Z" 2023-09-29T22:35:08.348 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:08.349 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:08Z" } 2023-09-29T22:35:08.349 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:08.349 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:08.349 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:08.349 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:13.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:13.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:13.351 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:13.395 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:13.396 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:13.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:13.446+0000 7f8a5b75cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:13.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:13.446+0000 7f8a5b75cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:13.478 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:13.478 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:13.479 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:13.480 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:13.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:13.482 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:13.483 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:13.484 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:13Z" 2023-09-29T22:35:13.485 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:13.486 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:13Z" } 2023-09-29T22:35:13.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:13.487 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:13.487 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:13.487 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:18.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:18.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:18.489 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:18.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:18.535 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:18.590 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:18.586+0000 7fd1d3141f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:18.591 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:18.586+0000 7fd1d3141f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:18.617 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:18.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:18.619 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:18.620 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:18.621 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:18.622 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:18.623 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:18Z" 2023-09-29T22:35:18.624 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:18.625 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:18Z" } 2023-09-29T22:35:18.625 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:18.626 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:18.626 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:18.626 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:23.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:23.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:23.628 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:23.677 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:23.677 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:23.731 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:23.726+0000 7f99cfed8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:23.733 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:23.730+0000 7f99cfed8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:23.757 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:23.757 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:23.758 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:23.759 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:23.760 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:23.761 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:23.762 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:23.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:23Z" 2023-09-29T22:35:23.764 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:23.765 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:23Z" } 2023-09-29T22:35:23.765 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:23.765 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:23.766 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:23.766 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:28.766 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:28.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:28.767 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:28.814 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:28.814 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:28.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:28.866+0000 7f01e100ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:28.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:28.870+0000 7f01e100ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:28.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:28.901 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:28.902 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:28.903 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:28.904 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:28.905 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:28Z" 2023-09-29T22:35:28.906 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:28.907 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:28Z" } 2023-09-29T22:35:28.907 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:28.908 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:28.908 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:28.908 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:33.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:33.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:33.909 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:33.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:33.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:34.010 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:34.006+0000 7f05d3359f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:34.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:34.006+0000 7f05d3359f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:34.038 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:34.039 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:34.040 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026819.471267_466.1", 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:33:39.471267Z", 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:34.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:34.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:34.043 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:34.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:34.045 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:34Z" 2023-09-29T22:35:34.045 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:34.046 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026819.471267_466.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:33:39.471267Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:34Z" } 2023-09-29T22:35:34.046 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026819.471267_466.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:33:39.471267Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:34.046 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:34.046 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026819.471267_466.1', 2: '', 3: ''} 2023-09-29T22:35:34.046 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026819.471267_466.1 2023-09-29T22:35:39.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:39.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:39.049 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:39.097 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:39.097 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:39.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:39.150+0000 7f71d0b8cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:39.154 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:39.150+0000 7f71d0b8cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:39.180 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:39.180 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:39.181 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:39.182 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026849.014127_476.1", 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:34:09.014127Z", 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:39.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:39.184 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:39.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:39.186 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:39.187 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:39.187 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:39Z" 2023-09-29T22:35:39.187 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:39.188 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026849.014127_476.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:34:09.014127Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:39Z" } 2023-09-29T22:35:39.188 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026849.014127_476.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:34:09.014127Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:39.188 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:39.188 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026849.014127_476.1', 2: '', 3: ''} 2023-09-29T22:35:39.189 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696026877.230172_504.1 target=1_1696026849.014127_476.1 2023-09-29T22:35:44.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:44.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:35:44.190 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:35:44.235 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:44.235 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:44.292 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:44.290+0000 7f656f9e6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:44.294 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:44.290+0000 7f656f9e6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:35:44.321 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:35:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:35:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:35:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:35:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:35:44.321 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:44.322 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:44.323 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696026877.230172_504.1", 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:34:37.230172Z", 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:44.324 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:44.325 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:35:44.326 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:35:44Z" 2023-09-29T22:35:44.327 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:35:44.328 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696026877.230172_504.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:34:37.230172Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:35:44Z" } 2023-09-29T22:35:44.329 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026877.230172_504.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:34:37.230172Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:35:44.329 DEBUG:rgw_multi.tests:log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:44.329 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026877.230172_504.1', 2: '', 3: ''} 2023-09-29T22:35:44.329 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:35:44.329 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:35:44.329 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:35:44.330 DEBUG:boto:Method: HEAD 2023-09-29T22:35:44.330 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:35:44.330 DEBUG:boto:Data: 2023-09-29T22:35:44.330 DEBUG:boto:Headers: {} 2023-09-29T22:35:44.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.330 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.330 DEBUG:boto:Params: {} 2023-09-29T22:35:44.330 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:35:44.331 DEBUG:boto:Token: None 2023-09-29T22:35:44.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/ 2023-09-29T22:35:44.331 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8= 2023-09-29T22:35:44.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8=', 'Content-Length': '0'} 2023-09-29T22:35:44.336 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000139027dcec11f299-0065175140-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.337 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:35:44.337 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:35:44.337 DEBUG:boto:Method: PUT 2023-09-29T22:35:44.337 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:35:44.337 DEBUG:boto:Data: 2023-09-29T22:35:44.337 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:35:44.337 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.338 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.338 DEBUG:boto:Params: {} 2023-09-29T22:35:44.338 DEBUG:boto:Token: None 2023-09-29T22:35:44.338 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/obj11 2023-09-29T22:35:44.338 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/HhdSSAN0jeSWbdKoUqW2trjOPA= 2023-09-29T22:35:44.338 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/HhdSSAN0jeSWbdKoUqW2trjOPA='} 2023-09-29T22:35:44.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fc98ffe54f303e13-0065175140-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.350 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:35:44.350 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:35:44.350 DEBUG:boto:Method: HEAD 2023-09-29T22:35:44.350 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:35:44.350 DEBUG:boto:Data: 2023-09-29T22:35:44.351 DEBUG:boto:Headers: {} 2023-09-29T22:35:44.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.351 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.351 DEBUG:boto:Params: {} 2023-09-29T22:35:44.351 DEBUG:boto:Token: None 2023-09-29T22:35:44.351 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/ 2023-09-29T22:35:44.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8= 2023-09-29T22:35:44.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8=', 'Content-Length': '0'} 2023-09-29T22:35:44.354 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057a6cc35b0b80a1d-0065175140-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.355 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:35:44.355 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:35:44.355 DEBUG:boto:Method: PUT 2023-09-29T22:35:44.355 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:35:44.355 DEBUG:boto:Data: 2023-09-29T22:35:44.355 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:35:44.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.355 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.355 DEBUG:boto:Params: {} 2023-09-29T22:35:44.356 DEBUG:boto:Token: None 2023-09-29T22:35:44.356 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/obj22 2023-09-29T22:35:44.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HSh0sOsOEOkVVcIrnelhymykL+E= 2023-09-29T22:35:44.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HSh0sOsOEOkVVcIrnelhymykL+E='} 2023-09-29T22:35:44.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000020ed7e108c298757-0065175140-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.399 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:35:44.399 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:35:44.399 DEBUG:boto:Method: HEAD 2023-09-29T22:35:44.399 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:35:44.399 DEBUG:boto:Data: 2023-09-29T22:35:44.400 DEBUG:boto:Headers: {} 2023-09-29T22:35:44.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.400 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.400 DEBUG:boto:Params: {} 2023-09-29T22:35:44.400 DEBUG:boto:Token: None 2023-09-29T22:35:44.400 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/ 2023-09-29T22:35:44.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8= 2023-09-29T22:35:44.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8=', 'Content-Length': '0'} 2023-09-29T22:35:44.404 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bf5f98673cb594d-0065175140-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.405 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:35:44.405 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:35:44.405 DEBUG:boto:Method: PUT 2023-09-29T22:35:44.405 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:35:44.405 DEBUG:boto:Data: 2023-09-29T22:35:44.405 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:35:44.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.405 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.405 DEBUG:boto:Params: {} 2023-09-29T22:35:44.406 DEBUG:boto:Token: None 2023-09-29T22:35:44.406 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/obj33 2023-09-29T22:35:44.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RFdseFD2wHXA7RKCqTPcs622He0= 2023-09-29T22:35:44.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RFdseFD2wHXA7RKCqTPcs622He0='} 2023-09-29T22:35:44.417 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000e17027958eb33b9-0065175140-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.417 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:35:44.417 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:35:44.418 DEBUG:boto:Method: HEAD 2023-09-29T22:35:44.418 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:35:44.418 DEBUG:boto:Data: 2023-09-29T22:35:44.418 DEBUG:boto:Headers: {} 2023-09-29T22:35:44.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.418 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.418 DEBUG:boto:Params: {} 2023-09-29T22:35:44.418 DEBUG:boto:Token: None 2023-09-29T22:35:44.419 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/ 2023-09-29T22:35:44.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8= 2023-09-29T22:35:44.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8=', 'Content-Length': '0'} 2023-09-29T22:35:44.422 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e9e2c44f2604ede0-0065175140-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.422 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:35:44.423 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:35:44.423 DEBUG:boto:Method: PUT 2023-09-29T22:35:44.423 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:35:44.423 DEBUG:boto:Data: 2023-09-29T22:35:44.423 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:35:44.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.423 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.423 DEBUG:boto:Params: {} 2023-09-29T22:35:44.423 DEBUG:boto:Token: None 2023-09-29T22:35:44.424 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/obj44 2023-09-29T22:35:44.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Zdh9C069RVwXpxbxa99pFNHqzE= 2023-09-29T22:35:44.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Zdh9C069RVwXpxbxa99pFNHqzE='} 2023-09-29T22:35:44.466 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000041fdc12fed39c5f-0065175140-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.467 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:35:44.467 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:35:44.467 DEBUG:boto:Method: HEAD 2023-09-29T22:35:44.467 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:35:44.467 DEBUG:boto:Data: 2023-09-29T22:35:44.467 DEBUG:boto:Headers: {} 2023-09-29T22:35:44.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.467 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.468 DEBUG:boto:Params: {} 2023-09-29T22:35:44.468 DEBUG:boto:Token: None 2023-09-29T22:35:44.468 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/ 2023-09-29T22:35:44.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8= 2023-09-29T22:35:44.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DZ4qKXWFfUz/2By2eDH99Xoa8f8=', 'Content-Length': '0'} 2023-09-29T22:35:44.472 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2238d00ef71ad53-0065175140-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.473 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:35:44.473 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:35:44.473 DEBUG:boto:Method: PUT 2023-09-29T22:35:44.473 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:35:44.473 DEBUG:boto:Data: 2023-09-29T22:35:44.473 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:35:44.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:35:44.473 DEBUG:boto:Port: 8000 2023-09-29T22:35:44.474 DEBUG:boto:Params: {} 2023-09-29T22:35:44.474 DEBUG:boto:Token: None 2023-09-29T22:35:44.474 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:35:44 GMT /hcnphz-59/obj55 2023-09-29T22:35:44.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8SThX0pXw5zTSrmEYMQRJ3gGhMw= 2023-09-29T22:35:44.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:35:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8SThX0pXw5zTSrmEYMQRJ3gGhMw='} 2023-09-29T22:35:44.483 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000836a2d7151f44800-0065175140-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:35:44.483 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:35:44.483 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:35:44.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:35:44.484 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:35:44.536 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:44.536 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:44.599 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:44.594+0000 7f6ad9310f40 1 waiting to reach incremental sync.. 2023-09-29T22:35:49.600 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:49.597+0000 7f6ad9310f40 1 waiting to reach incremental sync.. 2023-09-29T22:35:54.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:54.597+0000 7f6ad9310f40 1 bucket sync caught up with source: 2023-09-29T22:35:54.604 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.63.6, 00000000002.84.6, 00000000002.74.6] 2023-09-29T22:35:54.604 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.63.6, 00000000002.84.6, 00000000002.74.6] 2023-09-29T22:35:54.604 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:54.597+0000 7f6ad9310f40 0 bucket checkpoint complete 2023-09-29T22:35:54.634 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:35:54.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:35:54.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:35:54.635 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:35:54.685 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:35:54.685 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:35:54.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:54.749+0000 7fd3f642ff40 1 waiting to reach incremental sync.. 2023-09-29T22:35:59.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:35:59.753+0000 7fd3f642ff40 1 waiting to reach incremental sync.. 2023-09-29T22:36:04.757 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:04.757+0000 7fd3f642ff40 1 bucket sync caught up with source: 2023-09-29T22:36:04.758 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.260.6, 00000000002.169.6, 00000000002.300.6] 2023-09-29T22:36:04.758 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.260.6, 00000000002.169.6, 00000000002.300.6] 2023-09-29T22:36:04.758 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:04.757+0000 7fd3f642ff40 0 bucket checkpoint complete 2023-09-29T22:36:04.788 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:36:04.788 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.788 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.788 DEBUG:boto:Method: HEAD 2023-09-29T22:36:04.789 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:04.789 DEBUG:boto:Data: 2023-09-29T22:36:04.789 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.789 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.789 DEBUG:boto:Params: {} 2023-09-29T22:36:04.789 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:36:04.790 DEBUG:boto:Token: None 2023-09-29T22:36:04.790 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/ 2023-09-29T22:36:04.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Jsbdves/DdPpFfWnD2+JHWzO+c= 2023-09-29T22:36:04.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Jsbdves/DdPpFfWnD2+JHWzO+c=', 'Content-Length': '0'} 2023-09-29T22:36:04.795 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a67bc09ff35a4635-0065175154-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.795 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.795 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.796 DEBUG:boto:Method: HEAD 2023-09-29T22:36:04.796 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:04.796 DEBUG:boto:Data: 2023-09-29T22:36:04.796 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.796 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.796 DEBUG:boto:Params: {} 2023-09-29T22:36:04.796 DEBUG:boto:Token: None 2023-09-29T22:36:04.796 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/ 2023-09-29T22:36:04.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Jsbdves/DdPpFfWnD2+JHWzO+c= 2023-09-29T22:36:04.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Jsbdves/DdPpFfWnD2+JHWzO+c=', 'Content-Length': '0'} 2023-09-29T22:36:04.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000031686be7b7675d64-0065175154-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:36:04.801 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.801 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.801 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.801 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.801 DEBUG:boto:Method: GET 2023-09-29T22:36:04.801 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.802 DEBUG:boto:Data: 2023-09-29T22:36:04.802 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.802 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.802 DEBUG:boto:Params: {} 2023-09-29T22:36:04.802 DEBUG:boto:Token: None 2023-09-29T22:36:04.802 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/?versions 2023-09-29T22:36:04.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg= 2023-09-29T22:36:04.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg=', 'Content-Length': '0'} 2023-09-29T22:36:04.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b65c67277f81d029-0065175154-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.807 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:04.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:36:04.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:36:04.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:36:04.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:36:04.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:36:04.808 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:36:04.808 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.808 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.808 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.808 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.808 DEBUG:boto:Method: GET 2023-09-29T22:36:04.808 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.808 DEBUG:boto:Data: 2023-09-29T22:36:04.809 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.809 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.809 DEBUG:boto:Params: {} 2023-09-29T22:36:04.809 DEBUG:boto:Token: None 2023-09-29T22:36:04.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/?versions 2023-09-29T22:36:04.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg= 2023-09-29T22:36:04.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg=', 'Content-Length': '0'} 2023-09-29T22:36:04.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9cce5c46ed1ba41-0065175154-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.815 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:04.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:36:04.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:36:04.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:36:04.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:36:04.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:36:04.816 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.816 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.816 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.816 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.816 DEBUG:boto:Method: GET 2023-09-29T22:36:04.816 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.816 DEBUG:boto:Data: 2023-09-29T22:36:04.816 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.816 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.817 DEBUG:boto:Params: {} 2023-09-29T22:36:04.817 DEBUG:boto:Token: None 2023-09-29T22:36:04.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/?versions 2023-09-29T22:36:04.817 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg= 2023-09-29T22:36:04.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg=', 'Content-Length': '0'} 2023-09-29T22:36:04.820 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aee8464f6b21c214-0065175154-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.820 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:04.821 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.821 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.821 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.821 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.821 DEBUG:boto:Method: GET 2023-09-29T22:36:04.822 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:36:04.822 DEBUG:boto:Data: 2023-09-29T22:36:04.822 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.822 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.822 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.822 DEBUG:boto:Params: {} 2023-09-29T22:36:04.822 DEBUG:boto:Token: None 2023-09-29T22:36:04.822 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/?versions 2023-09-29T22:36:04.822 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg= 2023-09-29T22:36:04.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n+kzmqJFJMhXeiXvvh9kwJBdoQg=', 'Content-Length': '0'} 2023-09-29T22:36:04.826 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000738e718acb48b1f4-0065175154-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.826 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:04.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:36:04.827 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.827 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.827 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.827 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.828 DEBUG:boto:Method: GET 2023-09-29T22:36:04.828 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.828 DEBUG:boto:Data: 2023-09-29T22:36:04.828 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.828 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.828 DEBUG:boto:Params: {} 2023-09-29T22:36:04.828 DEBUG:boto:Token: None 2023-09-29T22:36:04.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M= 2023-09-29T22:36:04.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M=', 'Content-Length': '0'} 2023-09-29T22:36:04.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8f0534d701b87ca-0065175154-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.874 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.875 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.875 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.875 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.875 DEBUG:boto:Method: GET 2023-09-29T22:36:04.875 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.875 DEBUG:boto:Data: 2023-09-29T22:36:04.875 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.875 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.875 DEBUG:boto:Params: {} 2023-09-29T22:36:04.876 DEBUG:boto:Token: None 2023-09-29T22:36:04.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M= 2023-09-29T22:36:04.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M=', 'Content-Length': '0'} 2023-09-29T22:36:04.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008b7eddc23e2a02f-0065175154-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.879 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.879 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.880 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.880 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.880 DEBUG:boto:Method: HEAD 2023-09-29T22:36:04.880 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.880 DEBUG:boto:Data: 2023-09-29T22:36:04.880 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.880 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.880 DEBUG:boto:Params: {} 2023-09-29T22:36:04.880 DEBUG:boto:Token: None 2023-09-29T22:36:04.881 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YX7wwWN1uDCWQ7Aqjc+Qce3w5ac= 2023-09-29T22:36:04.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YX7wwWN1uDCWQ7Aqjc+Qce3w5ac=', 'Content-Length': '0'} 2023-09-29T22:36:04.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5432ed92668d65-0065175154-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.883 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.884 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.884 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.884 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.884 DEBUG:boto:Method: HEAD 2023-09-29T22:36:04.884 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.884 DEBUG:boto:Data: 2023-09-29T22:36:04.884 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.884 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.884 DEBUG:boto:Params: {} 2023-09-29T22:36:04.885 DEBUG:boto:Token: None 2023-09-29T22:36:04.885 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YX7wwWN1uDCWQ7Aqjc+Qce3w5ac= 2023-09-29T22:36:04.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YX7wwWN1uDCWQ7Aqjc+Qce3w5ac=', 'Content-Length': '0'} 2023-09-29T22:36:04.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000efae0915d6f8aa58-0065175154-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:36:04.888 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.888 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.888 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.888 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.888 DEBUG:boto:Method: GET 2023-09-29T22:36:04.888 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.888 DEBUG:boto:Data: 2023-09-29T22:36:04.888 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.889 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.889 DEBUG:boto:Params: {} 2023-09-29T22:36:04.889 DEBUG:boto:Token: None 2023-09-29T22:36:04.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M= 2023-09-29T22:36:04.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M=', 'Content-Length': '0'} 2023-09-29T22:36:04.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ab0c3ceeb4c4655-0065175154-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.892 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.892 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.892 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.892 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.892 DEBUG:boto:Method: GET 2023-09-29T22:36:04.892 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.892 DEBUG:boto:Data: 2023-09-29T22:36:04.892 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.893 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.893 DEBUG:boto:Params: {} 2023-09-29T22:36:04.893 DEBUG:boto:Token: None 2023-09-29T22:36:04.893 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:36:04.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M= 2023-09-29T22:36:04.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faltoubz044h5cJkqB8Wdl+Jk4M=', 'Content-Length': '0'} 2023-09-29T22:36:04.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ba17654dc6cb72-0065175154-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.896 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.896 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.896 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:04.896 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:04.896 DEBUG:boto:Method: GET 2023-09-29T22:36:04.896 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:36:04.896 DEBUG:boto:Data: 2023-09-29T22:36:04.896 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.897 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.897 DEBUG:boto:Params: {} 2023-09-29T22:36:04.897 DEBUG:boto:Token: None 2023-09-29T22:36:04.897 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/ 2023-09-29T22:36:04.897 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KgOncmrjB8d9/lmEFvj1WsZcSoM= 2023-09-29T22:36:04.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KgOncmrjB8d9/lmEFvj1WsZcSoM=', 'Content-Length': '0'} 2023-09-29T22:36:04.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001792863b4b0b9987-0065175154-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.942 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:04.943 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:04.943 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:04.943 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:04.943 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:04.943 DEBUG:boto:Method: GET 2023-09-29T22:36:04.943 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:36:04.943 DEBUG:boto:Data: 2023-09-29T22:36:04.944 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.944 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.944 DEBUG:boto:Params: {} 2023-09-29T22:36:04.944 DEBUG:boto:Token: None 2023-09-29T22:36:04.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/ 2023-09-29T22:36:04.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KgOncmrjB8d9/lmEFvj1WsZcSoM= 2023-09-29T22:36:04.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KgOncmrjB8d9/lmEFvj1WsZcSoM=', 'Content-Length': '0'} 2023-09-29T22:36:04.949 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f21d2904e883772-0065175154-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.949 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:04.950 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.950 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.950 DEBUG:boto:Method: HEAD 2023-09-29T22:36:04.950 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:36:04.950 DEBUG:boto:Data: 2023-09-29T22:36:04.951 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.951 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.951 DEBUG:boto:Params: {} 2023-09-29T22:36:04.951 DEBUG:boto:Token: None 2023-09-29T22:36:04.951 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11 2023-09-29T22:36:04.951 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t3pWKUUsvq48pcIYSUSL5hIK51Q= 2023-09-29T22:36:04.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t3pWKUUsvq48pcIYSUSL5hIK51Q=', 'Content-Length': '0'} 2023-09-29T22:36:04.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf50560409404e4b-0065175154-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.954 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.955 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.955 DEBUG:boto:Method: HEAD 2023-09-29T22:36:04.955 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:36:04.955 DEBUG:boto:Data: 2023-09-29T22:36:04.955 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.955 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.955 DEBUG:boto:Params: {} 2023-09-29T22:36:04.955 DEBUG:boto:Token: None 2023-09-29T22:36:04.955 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11 2023-09-29T22:36:04.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t3pWKUUsvq48pcIYSUSL5hIK51Q= 2023-09-29T22:36:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t3pWKUUsvq48pcIYSUSL5hIK51Q=', 'Content-Length': '0'} 2023-09-29T22:36:04.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde0bd43e3d3519c-0065175154-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:36:04.959 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.959 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.959 DEBUG:boto:Method: GET 2023-09-29T22:36:04.959 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:36:04.959 DEBUG:boto:Data: 2023-09-29T22:36:04.959 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:04.959 DEBUG:boto:Port: 8001 2023-09-29T22:36:04.959 DEBUG:boto:Params: {} 2023-09-29T22:36:04.960 DEBUG:boto:Token: None 2023-09-29T22:36:04.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11 2023-09-29T22:36:04.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VmnndnDgQieZvCXSLSwoUexdXRA= 2023-09-29T22:36:04.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VmnndnDgQieZvCXSLSwoUexdXRA=', 'Content-Length': '0'} 2023-09-29T22:36:04.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65194ceb2be5d29-0065175154-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:04.963 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:36:04.963 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:36:04.963 DEBUG:boto:Method: GET 2023-09-29T22:36:04.963 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:36:04.963 DEBUG:boto:Data: 2023-09-29T22:36:04.964 DEBUG:boto:Headers: {} 2023-09-29T22:36:04.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:04.964 DEBUG:boto:Port: 8000 2023-09-29T22:36:04.964 DEBUG:boto:Params: {} 2023-09-29T22:36:04.964 DEBUG:boto:Token: None 2023-09-29T22:36:04.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:04 GMT /hcnphz-59/obj11 2023-09-29T22:36:04.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VmnndnDgQieZvCXSLSwoUexdXRA= 2023-09-29T22:36:04.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VmnndnDgQieZvCXSLSwoUexdXRA=', 'Content-Length': '0'} 2023-09-29T22:36:04.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046c7111fd34f60cc-0065175154-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.010 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.011 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.011 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:05.011 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:05.011 DEBUG:boto:Method: GET 2023-09-29T22:36:05.011 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:36:05.012 DEBUG:boto:Data: 2023-09-29T22:36:05.012 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.012 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.012 DEBUG:boto:Params: {} 2023-09-29T22:36:05.012 DEBUG:boto:Token: None 2023-09-29T22:36:05.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.017 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055335f8eeac12a53-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.017 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.018 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.018 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.018 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:05.018 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:36:05.018 DEBUG:boto:Method: GET 2023-09-29T22:36:05.018 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:36:05.018 DEBUG:boto:Data: 2023-09-29T22:36:05.019 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.019 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.019 DEBUG:boto:Params: {} 2023-09-29T22:36:05.019 DEBUG:boto:Token: None 2023-09-29T22:36:05.019 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af897a19173ad778-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.024 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:36:05.025 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.025 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.025 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.025 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.025 DEBUG:boto:Method: GET 2023-09-29T22:36:05.025 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.025 DEBUG:boto:Data: 2023-09-29T22:36:05.025 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.026 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.026 DEBUG:boto:Params: {} 2023-09-29T22:36:05.026 DEBUG:boto:Token: None 2023-09-29T22:36:05.026 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ= 2023-09-29T22:36:05.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cfb49adf758bc44-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.029 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.029 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.029 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.029 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.029 DEBUG:boto:Method: GET 2023-09-29T22:36:05.029 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.029 DEBUG:boto:Data: 2023-09-29T22:36:05.029 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.030 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.030 DEBUG:boto:Params: {} 2023-09-29T22:36:05.030 DEBUG:boto:Token: None 2023-09-29T22:36:05.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ= 2023-09-29T22:36:05.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfcc3fd075b4f5b1-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.033 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.033 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.033 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.033 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.033 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.033 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.034 DEBUG:boto:Data: 2023-09-29T22:36:05.034 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.034 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.034 DEBUG:boto:Params: {} 2023-09-29T22:36:05.034 DEBUG:boto:Token: None 2023-09-29T22:36:05.034 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIe+tNOX32mKbqoe7Dn2DgSSShY= 2023-09-29T22:36:05.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIe+tNOX32mKbqoe7Dn2DgSSShY=', 'Content-Length': '0'} 2023-09-29T22:36:05.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa7be194ac7d31d-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.037 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.037 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.037 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.037 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.037 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.037 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.038 DEBUG:boto:Data: 2023-09-29T22:36:05.038 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.038 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.038 DEBUG:boto:Params: {} 2023-09-29T22:36:05.038 DEBUG:boto:Token: None 2023-09-29T22:36:05.038 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIe+tNOX32mKbqoe7Dn2DgSSShY= 2023-09-29T22:36:05.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIe+tNOX32mKbqoe7Dn2DgSSShY=', 'Content-Length': '0'} 2023-09-29T22:36:05.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004daff0d57d1a347f-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:36:05.041 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.041 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.042 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.042 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.042 DEBUG:boto:Method: GET 2023-09-29T22:36:05.042 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.042 DEBUG:boto:Data: 2023-09-29T22:36:05.042 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.042 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.042 DEBUG:boto:Params: {} 2023-09-29T22:36:05.043 DEBUG:boto:Token: None 2023-09-29T22:36:05.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.043 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ= 2023-09-29T22:36:05.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4992b07954f13b2-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.046 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.046 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.046 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.046 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.046 DEBUG:boto:Method: GET 2023-09-29T22:36:05.047 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.047 DEBUG:boto:Data: 2023-09-29T22:36:05.047 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.047 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.047 DEBUG:boto:Params: {} 2023-09-29T22:36:05.047 DEBUG:boto:Token: None 2023-09-29T22:36:05.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:36:05.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ= 2023-09-29T22:36:05.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ay2Ys+L1XGXynYe2L+584lxUxeQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293718d69ba39f13-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.050 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.051 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.051 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.051 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.051 DEBUG:boto:Method: GET 2023-09-29T22:36:05.051 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.051 DEBUG:boto:Data: 2023-09-29T22:36:05.051 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.051 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.052 DEBUG:boto:Params: {} 2023-09-29T22:36:05.052 DEBUG:boto:Token: None 2023-09-29T22:36:05.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.056 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024904b7f68587882-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.057 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.057 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.057 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.057 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.057 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.058 DEBUG:boto:Method: GET 2023-09-29T22:36:05.058 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.058 DEBUG:boto:Data: 2023-09-29T22:36:05.058 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.058 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.058 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.058 DEBUG:boto:Params: {} 2023-09-29T22:36:05.058 DEBUG:boto:Token: None 2023-09-29T22:36:05.058 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.059 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.059 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb8079c871779af9-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.106 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.107 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.107 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.107 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.107 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:36:05.107 DEBUG:boto:Data: 2023-09-29T22:36:05.107 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.108 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.108 DEBUG:boto:Params: {} 2023-09-29T22:36:05.108 DEBUG:boto:Token: None 2023-09-29T22:36:05.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22 2023-09-29T22:36:05.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cLyDEHUkpjV2XqCe0bBhTTqUFtA= 2023-09-29T22:36:05.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cLyDEHUkpjV2XqCe0bBhTTqUFtA=', 'Content-Length': '0'} 2023-09-29T22:36:05.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009492714d6251ccd7-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.111 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.111 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.111 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.111 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:36:05.111 DEBUG:boto:Data: 2023-09-29T22:36:05.112 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.112 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.112 DEBUG:boto:Params: {} 2023-09-29T22:36:05.112 DEBUG:boto:Token: None 2023-09-29T22:36:05.112 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22 2023-09-29T22:36:05.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cLyDEHUkpjV2XqCe0bBhTTqUFtA= 2023-09-29T22:36:05.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cLyDEHUkpjV2XqCe0bBhTTqUFtA=', 'Content-Length': '0'} 2023-09-29T22:36:05.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab86b65a8443c2ca-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:36:05.116 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.116 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.116 DEBUG:boto:Method: GET 2023-09-29T22:36:05.116 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:36:05.116 DEBUG:boto:Data: 2023-09-29T22:36:05.116 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.116 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.116 DEBUG:boto:Params: {} 2023-09-29T22:36:05.117 DEBUG:boto:Token: None 2023-09-29T22:36:05.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22 2023-09-29T22:36:05.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oc2+vOsw54Oc9AxayPgiIwRRkAE= 2023-09-29T22:36:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oc2+vOsw54Oc9AxayPgiIwRRkAE=', 'Content-Length': '0'} 2023-09-29T22:36:05.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000480a9c2bc0fcb790-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.166 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:36:05.167 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:36:05.167 DEBUG:boto:Method: GET 2023-09-29T22:36:05.167 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:36:05.167 DEBUG:boto:Data: 2023-09-29T22:36:05.167 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.167 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.167 DEBUG:boto:Params: {} 2023-09-29T22:36:05.168 DEBUG:boto:Token: None 2023-09-29T22:36:05.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj22 2023-09-29T22:36:05.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oc2+vOsw54Oc9AxayPgiIwRRkAE= 2023-09-29T22:36:05.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oc2+vOsw54Oc9AxayPgiIwRRkAE=', 'Content-Length': '0'} 2023-09-29T22:36:05.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005526843b52ce09a3-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.171 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.171 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.171 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.171 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.171 DEBUG:boto:Method: GET 2023-09-29T22:36:05.172 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.172 DEBUG:boto:Data: 2023-09-29T22:36:05.172 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.172 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.172 DEBUG:boto:Params: {} 2023-09-29T22:36:05.172 DEBUG:boto:Token: None 2023-09-29T22:36:05.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.177 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007cc27e8096f65c81-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.177 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.178 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.178 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.178 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.178 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.178 DEBUG:boto:Method: GET 2023-09-29T22:36:05.178 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:36:05.178 DEBUG:boto:Data: 2023-09-29T22:36:05.179 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.179 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.179 DEBUG:boto:Params: {} 2023-09-29T22:36:05.179 DEBUG:boto:Token: None 2023-09-29T22:36:05.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.179 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.185 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a47a5adba7df4bd9-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.185 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:36:05.186 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.186 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.186 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.186 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.186 DEBUG:boto:Method: GET 2023-09-29T22:36:05.186 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.186 DEBUG:boto:Data: 2023-09-29T22:36:05.186 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.187 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.187 DEBUG:boto:Params: {} 2023-09-29T22:36:05.187 DEBUG:boto:Token: None 2023-09-29T22:36:05.187 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w= 2023-09-29T22:36:05.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w=', 'Content-Length': '0'} 2023-09-29T22:36:05.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063dc0a6ef6f37fc2-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.190 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.190 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.190 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.190 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.190 DEBUG:boto:Method: GET 2023-09-29T22:36:05.191 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.191 DEBUG:boto:Data: 2023-09-29T22:36:05.191 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.191 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.191 DEBUG:boto:Params: {} 2023-09-29T22:36:05.191 DEBUG:boto:Token: None 2023-09-29T22:36:05.191 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w= 2023-09-29T22:36:05.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w=', 'Content-Length': '0'} 2023-09-29T22:36:05.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061170760d9702f39-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.238 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.239 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.239 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.239 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.239 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.239 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.239 DEBUG:boto:Data: 2023-09-29T22:36:05.239 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.240 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.240 DEBUG:boto:Params: {} 2023-09-29T22:36:05.240 DEBUG:boto:Token: None 2023-09-29T22:36:05.240 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.240 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgmF7VYsFfKknLvb3IMV13UQLgM= 2023-09-29T22:36:05.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgmF7VYsFfKknLvb3IMV13UQLgM=', 'Content-Length': '0'} 2023-09-29T22:36:05.243 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004464f65aef41140e-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.244 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.244 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.244 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.244 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.245 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.245 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.245 DEBUG:boto:Data: 2023-09-29T22:36:05.245 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.245 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.245 DEBUG:boto:Params: {} 2023-09-29T22:36:05.245 DEBUG:boto:Token: None 2023-09-29T22:36:05.245 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jgmF7VYsFfKknLvb3IMV13UQLgM= 2023-09-29T22:36:05.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jgmF7VYsFfKknLvb3IMV13UQLgM=', 'Content-Length': '0'} 2023-09-29T22:36:05.248 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da15fc544a7d9e1d-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:36:05.249 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.249 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.249 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.249 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.249 DEBUG:boto:Method: GET 2023-09-29T22:36:05.249 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.249 DEBUG:boto:Data: 2023-09-29T22:36:05.250 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.250 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.250 DEBUG:boto:Params: {} 2023-09-29T22:36:05.250 DEBUG:boto:Token: None 2023-09-29T22:36:05.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w= 2023-09-29T22:36:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w=', 'Content-Length': '0'} 2023-09-29T22:36:05.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001240c8d8ed816e19-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.294 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.295 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.295 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.295 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.295 DEBUG:boto:Method: GET 2023-09-29T22:36:05.295 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.295 DEBUG:boto:Data: 2023-09-29T22:36:05.295 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.295 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.296 DEBUG:boto:Params: {} 2023-09-29T22:36:05.296 DEBUG:boto:Token: None 2023-09-29T22:36:05.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:36:05.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w= 2023-09-29T22:36:05.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7zD7mnoA28/S8x9PqiZ/4aqye3w=', 'Content-Length': '0'} 2023-09-29T22:36:05.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e86d46b11e5560-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.299 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.299 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.299 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.300 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.300 DEBUG:boto:Method: GET 2023-09-29T22:36:05.300 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.300 DEBUG:boto:Data: 2023-09-29T22:36:05.300 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.300 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.300 DEBUG:boto:Params: {} 2023-09-29T22:36:05.300 DEBUG:boto:Token: None 2023-09-29T22:36:05.301 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.301 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007119fbda7d6558ea-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.306 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.306 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.306 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.307 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.307 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.307 DEBUG:boto:Method: GET 2023-09-29T22:36:05.307 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.307 DEBUG:boto:Data: 2023-09-29T22:36:05.307 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.307 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.307 DEBUG:boto:Params: {} 2023-09-29T22:36:05.308 DEBUG:boto:Token: None 2023-09-29T22:36:05.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.308 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000017f8093fda6ec6bc-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.313 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.313 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.313 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.314 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.314 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:36:05.314 DEBUG:boto:Data: 2023-09-29T22:36:05.314 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.314 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.314 DEBUG:boto:Params: {} 2023-09-29T22:36:05.314 DEBUG:boto:Token: None 2023-09-29T22:36:05.314 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33 2023-09-29T22:36:05.315 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6vIbddLIH/CJY8LTDAJe4scW5SY= 2023-09-29T22:36:05.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6vIbddLIH/CJY8LTDAJe4scW5SY=', 'Content-Length': '0'} 2023-09-29T22:36:05.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24230fe4636d70b-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.317 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.317 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.318 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.318 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:36:05.318 DEBUG:boto:Data: 2023-09-29T22:36:05.318 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.318 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.318 DEBUG:boto:Params: {} 2023-09-29T22:36:05.318 DEBUG:boto:Token: None 2023-09-29T22:36:05.318 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33 2023-09-29T22:36:05.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6vIbddLIH/CJY8LTDAJe4scW5SY= 2023-09-29T22:36:05.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6vIbddLIH/CJY8LTDAJe4scW5SY=', 'Content-Length': '0'} 2023-09-29T22:36:05.321 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009299afee057606b6-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:36:05.321 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.321 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.322 DEBUG:boto:Method: GET 2023-09-29T22:36:05.322 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:36:05.322 DEBUG:boto:Data: 2023-09-29T22:36:05.322 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.322 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.322 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.322 DEBUG:boto:Params: {} 2023-09-29T22:36:05.322 DEBUG:boto:Token: None 2023-09-29T22:36:05.322 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33 2023-09-29T22:36:05.323 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hHPnweIZ5GDwjKRtsZUCUhyhUgI= 2023-09-29T22:36:05.323 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hHPnweIZ5GDwjKRtsZUCUhyhUgI=', 'Content-Length': '0'} 2023-09-29T22:36:05.325 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ad751c0f128043-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.325 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:36:05.325 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:36:05.325 DEBUG:boto:Method: GET 2023-09-29T22:36:05.326 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:36:05.326 DEBUG:boto:Data: 2023-09-29T22:36:05.326 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.326 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.326 DEBUG:boto:Params: {} 2023-09-29T22:36:05.326 DEBUG:boto:Token: None 2023-09-29T22:36:05.326 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj33 2023-09-29T22:36:05.326 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hHPnweIZ5GDwjKRtsZUCUhyhUgI= 2023-09-29T22:36:05.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hHPnweIZ5GDwjKRtsZUCUhyhUgI=', 'Content-Length': '0'} 2023-09-29T22:36:05.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e47753b11a71f79-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.371 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.371 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.371 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.371 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.371 DEBUG:boto:Method: GET 2023-09-29T22:36:05.371 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.371 DEBUG:boto:Data: 2023-09-29T22:36:05.372 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.372 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.372 DEBUG:boto:Params: {} 2023-09-29T22:36:05.372 DEBUG:boto:Token: None 2023-09-29T22:36:05.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.376 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000617d9cd37590fa3b-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.377 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.377 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.378 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.378 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.378 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.378 DEBUG:boto:Method: GET 2023-09-29T22:36:05.378 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:36:05.378 DEBUG:boto:Data: 2023-09-29T22:36:05.378 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.378 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.379 DEBUG:boto:Params: {} 2023-09-29T22:36:05.379 DEBUG:boto:Token: None 2023-09-29T22:36:05.379 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.379 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcdefa5ff296f2cb-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.384 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:36:05.384 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.385 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.385 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.385 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.385 DEBUG:boto:Method: GET 2023-09-29T22:36:05.385 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.385 DEBUG:boto:Data: 2023-09-29T22:36:05.385 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.385 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.385 DEBUG:boto:Params: {} 2023-09-29T22:36:05.386 DEBUG:boto:Token: None 2023-09-29T22:36:05.386 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA= 2023-09-29T22:36:05.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA=', 'Content-Length': '0'} 2023-09-29T22:36:05.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053425efb87cb8332-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.434 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.435 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.435 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.435 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.435 DEBUG:boto:Method: GET 2023-09-29T22:36:05.435 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.435 DEBUG:boto:Data: 2023-09-29T22:36:05.435 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.436 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.436 DEBUG:boto:Params: {} 2023-09-29T22:36:05.436 DEBUG:boto:Token: None 2023-09-29T22:36:05.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA= 2023-09-29T22:36:05.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA=', 'Content-Length': '0'} 2023-09-29T22:36:05.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f285ed88a48c78-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.439 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.439 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.440 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.440 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.440 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.440 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.440 DEBUG:boto:Data: 2023-09-29T22:36:05.440 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.440 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.440 DEBUG:boto:Params: {} 2023-09-29T22:36:05.440 DEBUG:boto:Token: None 2023-09-29T22:36:05.441 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.441 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rboth4lV2Y4TPW3ncksBpIWp+1Q= 2023-09-29T22:36:05.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rboth4lV2Y4TPW3ncksBpIWp+1Q=', 'Content-Length': '0'} 2023-09-29T22:36:05.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009baedb77fadc43eb-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.444 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.444 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.444 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.444 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.445 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.445 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.445 DEBUG:boto:Data: 2023-09-29T22:36:05.445 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.445 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.445 DEBUG:boto:Params: {} 2023-09-29T22:36:05.445 DEBUG:boto:Token: None 2023-09-29T22:36:05.445 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.446 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rboth4lV2Y4TPW3ncksBpIWp+1Q= 2023-09-29T22:36:05.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rboth4lV2Y4TPW3ncksBpIWp+1Q=', 'Content-Length': '0'} 2023-09-29T22:36:05.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000405879cf5fe96b6e-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:36:05.448 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.449 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.449 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.449 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.449 DEBUG:boto:Method: GET 2023-09-29T22:36:05.449 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.449 DEBUG:boto:Data: 2023-09-29T22:36:05.449 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.449 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.450 DEBUG:boto:Params: {} 2023-09-29T22:36:05.450 DEBUG:boto:Token: None 2023-09-29T22:36:05.450 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.450 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA= 2023-09-29T22:36:05.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA=', 'Content-Length': '0'} 2023-09-29T22:36:05.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e4b1816af0a22f7-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.453 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.453 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.453 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.453 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.454 DEBUG:boto:Method: GET 2023-09-29T22:36:05.454 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.454 DEBUG:boto:Data: 2023-09-29T22:36:05.454 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.454 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.454 DEBUG:boto:Params: {} 2023-09-29T22:36:05.455 DEBUG:boto:Token: None 2023-09-29T22:36:05.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:36:05.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA= 2023-09-29T22:36:05.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vv74ZCTX0i4KGz3KAu7kYDNDYFA=', 'Content-Length': '0'} 2023-09-29T22:36:05.457 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027061515e48d05af-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.458 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.458 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.458 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.458 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.458 DEBUG:boto:Method: GET 2023-09-29T22:36:05.458 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.458 DEBUG:boto:Data: 2023-09-29T22:36:05.458 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.459 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.459 DEBUG:boto:Params: {} 2023-09-29T22:36:05.459 DEBUG:boto:Token: None 2023-09-29T22:36:05.459 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a954b4a94af3ff3-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.464 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.465 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.465 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.465 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.465 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.465 DEBUG:boto:Method: GET 2023-09-29T22:36:05.465 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.465 DEBUG:boto:Data: 2023-09-29T22:36:05.465 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.466 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.466 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.466 DEBUG:boto:Params: {} 2023-09-29T22:36:05.466 DEBUG:boto:Token: None 2023-09-29T22:36:05.466 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.466 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.471 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c7a9c07d94a875b-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.471 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.472 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.472 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.472 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.472 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:36:05.472 DEBUG:boto:Data: 2023-09-29T22:36:05.472 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.472 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.473 DEBUG:boto:Params: {} 2023-09-29T22:36:05.473 DEBUG:boto:Token: None 2023-09-29T22:36:05.473 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44 2023-09-29T22:36:05.473 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iH+i+2FjxuQmkOGeTJEYLvSVq1E= 2023-09-29T22:36:05.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iH+i+2FjxuQmkOGeTJEYLvSVq1E=', 'Content-Length': '0'} 2023-09-29T22:36:05.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc33baa95f7b7fbb-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.476 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.476 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.476 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.476 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:36:05.476 DEBUG:boto:Data: 2023-09-29T22:36:05.476 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.476 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.476 DEBUG:boto:Params: {} 2023-09-29T22:36:05.477 DEBUG:boto:Token: None 2023-09-29T22:36:05.477 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44 2023-09-29T22:36:05.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iH+i+2FjxuQmkOGeTJEYLvSVq1E= 2023-09-29T22:36:05.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iH+i+2FjxuQmkOGeTJEYLvSVq1E=', 'Content-Length': '0'} 2023-09-29T22:36:05.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009010bb50b4b2b615-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:36:05.480 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.480 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.480 DEBUG:boto:Method: GET 2023-09-29T22:36:05.480 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:36:05.480 DEBUG:boto:Data: 2023-09-29T22:36:05.480 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.481 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.481 DEBUG:boto:Params: {} 2023-09-29T22:36:05.481 DEBUG:boto:Token: None 2023-09-29T22:36:05.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44 2023-09-29T22:36:05.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v2QQB3nW5cxDXwjiIawoG86BajA= 2023-09-29T22:36:05.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v2QQB3nW5cxDXwjiIawoG86BajA=', 'Content-Length': '0'} 2023-09-29T22:36:05.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023e708c19a44348e-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.526 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:36:05.527 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:36:05.527 DEBUG:boto:Method: GET 2023-09-29T22:36:05.527 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:36:05.527 DEBUG:boto:Data: 2023-09-29T22:36:05.527 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.527 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.527 DEBUG:boto:Params: {} 2023-09-29T22:36:05.528 DEBUG:boto:Token: None 2023-09-29T22:36:05.528 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj44 2023-09-29T22:36:05.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v2QQB3nW5cxDXwjiIawoG86BajA= 2023-09-29T22:36:05.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v2QQB3nW5cxDXwjiIawoG86BajA=', 'Content-Length': '0'} 2023-09-29T22:36:05.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725e1463b3a28cd0-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.532 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.532 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.532 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.532 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.532 DEBUG:boto:Method: GET 2023-09-29T22:36:05.532 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.532 DEBUG:boto:Data: 2023-09-29T22:36:05.532 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.533 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.533 DEBUG:boto:Params: {} 2023-09-29T22:36:05.533 DEBUG:boto:Token: None 2023-09-29T22:36:05.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1075920cebf1dd9-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.539 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.539 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.539 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.539 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.540 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.540 DEBUG:boto:Method: GET 2023-09-29T22:36:05.540 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:36:05.540 DEBUG:boto:Data: 2023-09-29T22:36:05.540 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.540 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.540 DEBUG:boto:Params: {} 2023-09-29T22:36:05.540 DEBUG:boto:Token: None 2023-09-29T22:36:05.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.545 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068d7aca2abb0c241-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.546 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:36:05.546 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.546 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.547 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.547 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.547 DEBUG:boto:Method: GET 2023-09-29T22:36:05.547 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.547 DEBUG:boto:Data: 2023-09-29T22:36:05.547 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.547 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.547 DEBUG:boto:Params: {} 2023-09-29T22:36:05.547 DEBUG:boto:Token: None 2023-09-29T22:36:05.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE= 2023-09-29T22:36:05.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE=', 'Content-Length': '0'} 2023-09-29T22:36:05.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7d6f2afe255a4c2-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.551 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.551 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.551 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.551 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.551 DEBUG:boto:Method: GET 2023-09-29T22:36:05.551 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.551 DEBUG:boto:Data: 2023-09-29T22:36:05.551 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.552 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.552 DEBUG:boto:Params: {} 2023-09-29T22:36:05.552 DEBUG:boto:Token: None 2023-09-29T22:36:05.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE= 2023-09-29T22:36:05.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE=', 'Content-Length': '0'} 2023-09-29T22:36:05.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6477bd5250a8251-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.555 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.555 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.556 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.556 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.556 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.556 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.556 DEBUG:boto:Data: 2023-09-29T22:36:05.556 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.556 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.556 DEBUG:boto:Params: {} 2023-09-29T22:36:05.556 DEBUG:boto:Token: None 2023-09-29T22:36:05.557 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fn/DcZYXSAIoL6g5w9j+W/fXKOw= 2023-09-29T22:36:05.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fn/DcZYXSAIoL6g5w9j+W/fXKOw=', 'Content-Length': '0'} 2023-09-29T22:36:05.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b63c799b9fd4753-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.559 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.559 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.560 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.560 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.560 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.560 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.560 DEBUG:boto:Data: 2023-09-29T22:36:05.560 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.560 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.560 DEBUG:boto:Params: {} 2023-09-29T22:36:05.561 DEBUG:boto:Token: None 2023-09-29T22:36:05.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fn/DcZYXSAIoL6g5w9j+W/fXKOw= 2023-09-29T22:36:05.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fn/DcZYXSAIoL6g5w9j+W/fXKOw=', 'Content-Length': '0'} 2023-09-29T22:36:05.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f507d27354f7bd5c-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:36:05.563 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.564 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.564 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.564 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.564 DEBUG:boto:Method: GET 2023-09-29T22:36:05.564 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.564 DEBUG:boto:Data: 2023-09-29T22:36:05.564 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.564 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.564 DEBUG:boto:Params: {} 2023-09-29T22:36:05.565 DEBUG:boto:Token: None 2023-09-29T22:36:05.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE= 2023-09-29T22:36:05.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE=', 'Content-Length': '0'} 2023-09-29T22:36:05.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e036d212dd5be8f7-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.615 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.615 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.615 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.615 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.616 DEBUG:boto:Method: GET 2023-09-29T22:36:05.616 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.616 DEBUG:boto:Data: 2023-09-29T22:36:05.616 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.616 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.616 DEBUG:boto:Params: {} 2023-09-29T22:36:05.616 DEBUG:boto:Token: None 2023-09-29T22:36:05.617 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:36:05.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE= 2023-09-29T22:36:05.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X612LF5Hz4DpgZEXq6sJDQ8qFVE=', 'Content-Length': '0'} 2023-09-29T22:36:05.620 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3450f1e48b61174-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.620 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.620 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.620 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.620 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.620 DEBUG:boto:Method: GET 2023-09-29T22:36:05.621 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.621 DEBUG:boto:Data: 2023-09-29T22:36:05.621 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.621 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.621 DEBUG:boto:Params: {} 2023-09-29T22:36:05.621 DEBUG:boto:Token: None 2023-09-29T22:36:05.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.621 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.626 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3848c73b832badc-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.626 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.627 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.627 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.627 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.627 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.627 DEBUG:boto:Method: GET 2023-09-29T22:36:05.627 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.627 DEBUG:boto:Data: 2023-09-29T22:36:05.628 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.628 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.628 DEBUG:boto:Params: {} 2023-09-29T22:36:05.628 DEBUG:boto:Token: None 2023-09-29T22:36:05.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b3bee18ca2a5e48-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.633 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.633 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.633 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.634 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.634 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:36:05.634 DEBUG:boto:Data: 2023-09-29T22:36:05.634 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.634 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.634 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.634 DEBUG:boto:Params: {} 2023-09-29T22:36:05.634 DEBUG:boto:Token: None 2023-09-29T22:36:05.634 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55 2023-09-29T22:36:05.635 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5CypY9dUh81948yjpBvBMiJ+FqQ= 2023-09-29T22:36:05.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5CypY9dUh81948yjpBvBMiJ+FqQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.637 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef52ac2bfc041056-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.637 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.637 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.637 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.637 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:36:05.638 DEBUG:boto:Data: 2023-09-29T22:36:05.638 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.638 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.638 DEBUG:boto:Params: {} 2023-09-29T22:36:05.638 DEBUG:boto:Token: None 2023-09-29T22:36:05.638 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55 2023-09-29T22:36:05.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5CypY9dUh81948yjpBvBMiJ+FqQ= 2023-09-29T22:36:05.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5CypY9dUh81948yjpBvBMiJ+FqQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ba59ec49e1fc67d-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:36:05.641 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.641 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.641 DEBUG:boto:Method: GET 2023-09-29T22:36:05.641 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:36:05.641 DEBUG:boto:Data: 2023-09-29T22:36:05.642 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.642 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.642 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.642 DEBUG:boto:Params: {} 2023-09-29T22:36:05.642 DEBUG:boto:Token: None 2023-09-29T22:36:05.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55 2023-09-29T22:36:05.643 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:br9JrVN871ftVwUZMNM6/irzqlQ= 2023-09-29T22:36:05.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:br9JrVN871ftVwUZMNM6/irzqlQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054f54d0115512307-0065175155-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.645 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:36:05.646 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:36:05.646 DEBUG:boto:Method: GET 2023-09-29T22:36:05.646 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:36:05.646 DEBUG:boto:Data: 2023-09-29T22:36:05.646 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.646 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.646 DEBUG:boto:Params: {} 2023-09-29T22:36:05.646 DEBUG:boto:Token: None 2023-09-29T22:36:05.647 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj55 2023-09-29T22:36:05.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:br9JrVN871ftVwUZMNM6/irzqlQ= 2023-09-29T22:36:05.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:br9JrVN871ftVwUZMNM6/irzqlQ=', 'Content-Length': '0'} 2023-09-29T22:36:05.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032c0aa8642f4d5bc-0065175155-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.650 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.650 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.650 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.650 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.650 DEBUG:boto:Method: GET 2023-09-29T22:36:05.650 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.650 DEBUG:boto:Data: 2023-09-29T22:36:05.650 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:36:05.651 DEBUG:boto:Port: 8001 2023-09-29T22:36:05.651 DEBUG:boto:Params: {} 2023-09-29T22:36:05.651 DEBUG:boto:Token: None 2023-09-29T22:36:05.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.655 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e83654a9ed0b3777-0065175155-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.656 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.656 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.656 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.656 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.656 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.656 DEBUG:boto:Method: GET 2023-09-29T22:36:05.657 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:36:05.657 DEBUG:boto:Data: 2023-09-29T22:36:05.657 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.657 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.657 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.657 DEBUG:boto:Params: {} 2023-09-29T22:36:05.657 DEBUG:boto:Token: None 2023-09-29T22:36:05.657 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w= 2023-09-29T22:36:05.658 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJovVvO0vibxJeb5x4px9v/Z2+w=', 'Content-Length': '0'} 2023-09-29T22:36:05.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000474da926da02e85c-0065175155-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.662 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:36:05.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:36:05.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:36:05.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:36:05.663 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:36:05.710 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:05.711 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:05.788 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:36:05.788 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-59 2023-09-29T22:36:05.796 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-09-29T22:36:05.871 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.871 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.872 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.872 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:05.872 DEBUG:boto:Data: 2023-09-29T22:36:05.872 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.872 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.872 DEBUG:boto:Params: {} 2023-09-29T22:36:05.872 DEBUG:boto:Token: None 2023-09-29T22:36:05.873 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.873 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4= 2023-09-29T22:36:05.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4=', 'Content-Length': '0'} 2023-09-29T22:36:05.878 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ab526fe3b5187f2e-0065175155-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.878 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:36:05.878 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:36:05.878 DEBUG:boto:Method: PUT 2023-09-29T22:36:05.879 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:36:05.879 DEBUG:boto:Data: 2023-09-29T22:36:05.879 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:36:05.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.879 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.879 DEBUG:boto:Params: {} 2023-09-29T22:36:05.879 DEBUG:boto:Token: None 2023-09-29T22:36:05.879 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj17 2023-09-29T22:36:05.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VjbYYFPEdyBBueRiSi6m5wNpSHY= 2023-09-29T22:36:05.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VjbYYFPEdyBBueRiSi6m5wNpSHY='} 2023-09-29T22:36:05.922 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4c73f80971ad972-0065175155-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.923 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.923 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.923 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.923 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:05.923 DEBUG:boto:Data: 2023-09-29T22:36:05.923 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.924 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.924 DEBUG:boto:Params: {} 2023-09-29T22:36:05.924 DEBUG:boto:Token: None 2023-09-29T22:36:05.924 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4= 2023-09-29T22:36:05.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4=', 'Content-Length': '0'} 2023-09-29T22:36:05.929 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000036cdb91b1d310e50-0065175155-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.930 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:36:05.930 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:36:05.930 DEBUG:boto:Method: PUT 2023-09-29T22:36:05.930 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:36:05.930 DEBUG:boto:Data: 2023-09-29T22:36:05.931 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:36:05.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.931 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.931 DEBUG:boto:Params: {} 2023-09-29T22:36:05.931 DEBUG:boto:Token: None 2023-09-29T22:36:05.931 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj34 2023-09-29T22:36:05.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sfiwgsy42ELA9GkeUSrg2A+qIKQ= 2023-09-29T22:36:05.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sfiwgsy42ELA9GkeUSrg2A+qIKQ='} 2023-09-29T22:36:05.942 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008fb061abcaf99ea6-0065175155-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.942 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.942 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.943 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.943 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:05.943 DEBUG:boto:Data: 2023-09-29T22:36:05.943 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.943 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.943 DEBUG:boto:Params: {} 2023-09-29T22:36:05.943 DEBUG:boto:Token: None 2023-09-29T22:36:05.943 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4= 2023-09-29T22:36:05.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4=', 'Content-Length': '0'} 2023-09-29T22:36:05.947 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf75f210b2f97d7e-0065175155-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.948 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:36:05.948 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:36:05.948 DEBUG:boto:Method: PUT 2023-09-29T22:36:05.948 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:36:05.948 DEBUG:boto:Data: 2023-09-29T22:36:05.948 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:36:05.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.948 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.948 DEBUG:boto:Params: {} 2023-09-29T22:36:05.949 DEBUG:boto:Token: None 2023-09-29T22:36:05.949 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/obj51 2023-09-29T22:36:05.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XClcxTu6XxSTKVu1P1BV8zOb5MU= 2023-09-29T22:36:05.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XClcxTu6XxSTKVu1P1BV8zOb5MU='} 2023-09-29T22:36:05.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030432ec1d4597ee1-0065175155-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:05.995 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:05.995 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:05.995 DEBUG:boto:Method: HEAD 2023-09-29T22:36:05.995 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:05.995 DEBUG:boto:Data: 2023-09-29T22:36:05.995 DEBUG:boto:Headers: {} 2023-09-29T22:36:05.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:05.995 DEBUG:boto:Port: 8000 2023-09-29T22:36:05.996 DEBUG:boto:Params: {} 2023-09-29T22:36:05.996 DEBUG:boto:Token: None 2023-09-29T22:36:05.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:05 GMT /hcnphz-59/ 2023-09-29T22:36:05.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4= 2023-09-29T22:36:05.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:loMjtmDVOXTj1nldO/wz1w662K4=', 'Content-Length': '0'} 2023-09-29T22:36:06.001 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000195b30af7e463ce3-0065175155-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:06.001 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:36:06.001 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:36:06.001 DEBUG:boto:Method: PUT 2023-09-29T22:36:06.001 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:36:06.001 DEBUG:boto:Data: 2023-09-29T22:36:06.002 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:36:06.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:06.002 DEBUG:boto:Port: 8000 2023-09-29T22:36:06.002 DEBUG:boto:Params: {} 2023-09-29T22:36:06.002 DEBUG:boto:Token: None 2023-09-29T22:36:06.002 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:36:06 GMT /hcnphz-59/obj68 2023-09-29T22:36:06.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fUuTiHTzs5ON+rm33KyejHX9qe0= 2023-09-29T22:36:06.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:36:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fUuTiHTzs5ON+rm33KyejHX9qe0='} 2023-09-29T22:36:06.012 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070f3f1767b152bdb-0065175156-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:06.012 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:36:06.013 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:36:06.013 DEBUG:boto:Method: HEAD 2023-09-29T22:36:06.013 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:36:06.013 DEBUG:boto:Data: 2023-09-29T22:36:06.013 DEBUG:boto:Headers: {} 2023-09-29T22:36:06.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:06.013 DEBUG:boto:Port: 8000 2023-09-29T22:36:06.013 DEBUG:boto:Params: {} 2023-09-29T22:36:06.013 DEBUG:boto:Token: None 2023-09-29T22:36:06.013 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:36:06 GMT /hcnphz-59/ 2023-09-29T22:36:06.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YnYRkJCbkzv7HGi3i/strLkrIUc= 2023-09-29T22:36:06.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:36:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YnYRkJCbkzv7HGi3i/strLkrIUc=', 'Content-Length': '0'} 2023-09-29T22:36:06.017 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000247527ec7d4c388e-0065175156-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:06.017 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:36:06.018 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:36:06.018 DEBUG:boto:Method: PUT 2023-09-29T22:36:06.018 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:36:06.018 DEBUG:boto:Data: 2023-09-29T22:36:06.018 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:36:06.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:36:06.019 DEBUG:boto:Port: 8000 2023-09-29T22:36:06.019 DEBUG:boto:Params: {} 2023-09-29T22:36:06.019 DEBUG:boto:Token: None 2023-09-29T22:36:06.019 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:36:06 GMT /hcnphz-59/obj85 2023-09-29T22:36:06.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jh9QOfJZ43svZeRRN5U0OHxMMJs= 2023-09-29T22:36:06.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:36:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jh9QOfJZ43svZeRRN5U0OHxMMJs='} 2023-09-29T22:36:06.063 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001c8ada7ccdf17e0f-0065175156-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:36:06.063 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:36:06.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:36:06.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:36:06.063 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:36:06.124 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:36:06.124 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:36:06.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:06.177+0000 7f866f960f40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T22:36:11.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:11.177+0000 7f866f960f40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T22:36:16.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:16.177+0000 7f866f960f40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T22:36:21.181 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:21.181+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:21.181 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-09-29T22:36:21.181 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:26.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:26.180+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:26.183 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-09-29T22:36:26.183 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:31.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:31.184+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:31.186 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-09-29T22:36:31.186 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:36.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:36.188+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:36.188 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-09-29T22:36:36.189 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:41.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:41.188+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:41.191 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-09-29T22:36:41.191 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:46.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:46.192+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:46.193 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , ] 2023-09-29T22:36:46.194 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:51.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:51.196+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:51.197 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:36:51.197 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:36:56.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:36:56.196+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:36:56.200 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:36:56.200 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:01.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:01.200+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:01.203 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:37:01.203 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:06.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:06.203+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:06.205 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:37:06.206 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:10.227 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:37:10.148+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:37:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:11.207+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:11.208 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:37:11.209 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:16.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:16.207+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:16.212 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:37:16.212 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:21.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:21.211+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:21.214 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:37:21.214 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:26.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:26.215+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:26.217 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000001.53.6, , , , , , , , , , ] 2023-09-29T22:37:26.217 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:31.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:31.215+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:31.220 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:37:31.220 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:36.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:36.219+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:36.223 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:37:36.223 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:41.225 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:41.222+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:41.225 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:37:41.226 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:46.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:46.226+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:46.229 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:37:46.229 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:51.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:51.226+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:51.231 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:37:51.232 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:37:56.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:37:56.230+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:37:56.234 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:37:56.234 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:01.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:01.234+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:01.237 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:38:01.237 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:06.240 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:06.234+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:06.240 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:38:06.240 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:11.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:11.238+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:11.243 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:38:11.244 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:16.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:16.242+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:16.246 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:38:16.246 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:21.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:21.245+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:21.249 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:38:21.249 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:26.251 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:26.245+0000 7f866f960f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:26.252 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, , , , ] 2023-09-29T22:38:26.252 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:31.254 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:31.249+0000 7f866f960f40 1 bucket sync caught up with source: 2023-09-29T22:38:31.255 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:31.255 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000001.68.6, , , , , , 00000000001.53.6, , 00000000001.53.6, , , , 00000000001.33.6, 00000000001.177.6, , , ] 2023-09-29T22:38:31.255 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:31.249+0000 7f866f960f40 0 bucket checkpoint complete 2023-09-29T22:38:31.283 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:38:31.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:38:31.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:38:31.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:38:31.336 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:31.336 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:31.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:31.393+0000 7f133c5a6f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:31.401 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.257.6, , , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:31.402 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.257.6, 00000000001.580.6, , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:36.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:36.401+0000 7f133c5a6f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:36.402 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.257.6, , , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:36.402 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.257.6, 00000000001.580.6, , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:41.403 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:41.401+0000 7f133c5a6f40 1 waiting for incremental sync to catch up: 2023-09-29T22:38:41.404 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.257.6, , , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:41.404 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.257.6, 00000000001.580.6, , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:46.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:46.405+0000 7f133c5a6f40 1 bucket sync caught up with source: 2023-09-29T22:38:46.406 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000001.257.6, 00000000001.580.6, , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:46.406 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000001.257.6, 00000000001.580.6, , , 00000000001.222.6, , , 00000000001.260.6, 00000000003.175.6, 00000000003.306.6] 2023-09-29T22:38:46.407 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:46.405+0000 7f133c5a6f40 0 bucket checkpoint complete 2023-09-29T22:38:46.437 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:38:46.437 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.437 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.438 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.438 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:46.438 DEBUG:boto:Data: 2023-09-29T22:38:46.438 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.438 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.438 DEBUG:boto:Params: {} 2023-09-29T22:38:46.439 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:38:46.439 DEBUG:boto:Token: None 2023-09-29T22:38:46.439 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.439 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a3k6UwVjoefxw582SZwZQD5UyHE= 2023-09-29T22:38:46.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a3k6UwVjoefxw582SZwZQD5UyHE=', 'Content-Length': '0'} 2023-09-29T22:38:46.445 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000068903afc87e1f5fc-00651751f6-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.445 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.445 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.446 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.446 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:46.446 DEBUG:boto:Data: 2023-09-29T22:38:46.446 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.446 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.446 DEBUG:boto:Params: {} 2023-09-29T22:38:46.447 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:38:46.447 DEBUG:boto:Token: None 2023-09-29T22:38:46.447 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a3k6UwVjoefxw582SZwZQD5UyHE= 2023-09-29T22:38:46.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a3k6UwVjoefxw582SZwZQD5UyHE=', 'Content-Length': '0'} 2023-09-29T22:38:46.453 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ebe09e822034514-00651751f6-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:38:46.453 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.453 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.454 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.454 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.454 DEBUG:boto:Method: GET 2023-09-29T22:38:46.454 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.454 DEBUG:boto:Data: 2023-09-29T22:38:46.454 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.454 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.455 DEBUG:boto:Params: {} 2023-09-29T22:38:46.455 DEBUG:boto:Token: None 2023-09-29T22:38:46.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/?versions 2023-09-29T22:38:46.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac= 2023-09-29T22:38:46.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac=', 'Content-Length': '0'} 2023-09-29T22:38:46.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c60bdff5530c7773-00651751f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.459 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:38:46.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:38:46.462 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:38:46.462 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.462 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.462 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.462 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.462 DEBUG:boto:Method: GET 2023-09-29T22:38:46.462 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.462 DEBUG:boto:Data: 2023-09-29T22:38:46.462 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.463 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.463 DEBUG:boto:Params: {} 2023-09-29T22:38:46.463 DEBUG:boto:Token: None 2023-09-29T22:38:46.463 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/?versions 2023-09-29T22:38:46.463 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac= 2023-09-29T22:38:46.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac=', 'Content-Length': '0'} 2023-09-29T22:38:46.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a923879792aa72f9-00651751f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.468 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:38:46.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:38:46.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:38:46.470 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.470 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.470 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.470 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.470 DEBUG:boto:Method: GET 2023-09-29T22:38:46.470 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.470 DEBUG:boto:Data: 2023-09-29T22:38:46.470 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.471 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.471 DEBUG:boto:Params: {} 2023-09-29T22:38:46.471 DEBUG:boto:Token: None 2023-09-29T22:38:46.471 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/?versions 2023-09-29T22:38:46.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac= 2023-09-29T22:38:46.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac=', 'Content-Length': '0'} 2023-09-29T22:38:46.475 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc4fae85755a358f-00651751f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.475 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.476 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.476 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.476 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.476 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.476 DEBUG:boto:Method: GET 2023-09-29T22:38:46.476 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:38:46.476 DEBUG:boto:Data: 2023-09-29T22:38:46.476 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.477 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.477 DEBUG:boto:Params: {} 2023-09-29T22:38:46.477 DEBUG:boto:Token: None 2023-09-29T22:38:46.477 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/?versions 2023-09-29T22:38:46.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac= 2023-09-29T22:38:46.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:orWnb9JM7rlS3rEcDvoT0V+rIac=', 'Content-Length': '0'} 2023-09-29T22:38:46.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c58e936990a8f420-00651751f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.482 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:38:46.483 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.483 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.483 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.484 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.484 DEBUG:boto:Method: GET 2023-09-29T22:38:46.484 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.484 DEBUG:boto:Data: 2023-09-29T22:38:46.484 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.484 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.484 DEBUG:boto:Params: {} 2023-09-29T22:38:46.484 DEBUG:boto:Token: None 2023-09-29T22:38:46.485 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY= 2023-09-29T22:38:46.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY=', 'Content-Length': '0'} 2023-09-29T22:38:46.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b28b4fd4e6277b2-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.534 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.535 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.535 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.535 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.535 DEBUG:boto:Method: GET 2023-09-29T22:38:46.535 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.535 DEBUG:boto:Data: 2023-09-29T22:38:46.535 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.535 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.536 DEBUG:boto:Params: {} 2023-09-29T22:38:46.536 DEBUG:boto:Token: None 2023-09-29T22:38:46.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY= 2023-09-29T22:38:46.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY=', 'Content-Length': '0'} 2023-09-29T22:38:46.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dbb5205531ed5d3-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.539 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.539 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.540 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.540 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.540 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.540 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.540 DEBUG:boto:Data: 2023-09-29T22:38:46.540 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.540 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.540 DEBUG:boto:Params: {} 2023-09-29T22:38:46.541 DEBUG:boto:Token: None 2023-09-29T22:38:46.541 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c5wmyFZblYpwYk3AzAkiEHQ1KCY= 2023-09-29T22:38:46.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c5wmyFZblYpwYk3AzAkiEHQ1KCY=', 'Content-Length': '0'} 2023-09-29T22:38:46.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000029c4abe3b6d193f-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.544 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.544 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.544 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.544 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.544 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.544 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.544 DEBUG:boto:Data: 2023-09-29T22:38:46.545 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.545 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.545 DEBUG:boto:Params: {} 2023-09-29T22:38:46.545 DEBUG:boto:Token: None 2023-09-29T22:38:46.545 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c5wmyFZblYpwYk3AzAkiEHQ1KCY= 2023-09-29T22:38:46.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c5wmyFZblYpwYk3AzAkiEHQ1KCY=', 'Content-Length': '0'} 2023-09-29T22:38:46.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010e140942528b9be-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:38:46.549 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.549 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.549 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.549 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.549 DEBUG:boto:Method: GET 2023-09-29T22:38:46.549 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.549 DEBUG:boto:Data: 2023-09-29T22:38:46.549 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.550 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.550 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.550 DEBUG:boto:Params: {} 2023-09-29T22:38:46.550 DEBUG:boto:Token: None 2023-09-29T22:38:46.550 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.550 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY= 2023-09-29T22:38:46.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY=', 'Content-Length': '0'} 2023-09-29T22:38:46.553 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000427a48254a45147a-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.553 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.553 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.553 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.553 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.553 DEBUG:boto:Method: GET 2023-09-29T22:38:46.554 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.554 DEBUG:boto:Data: 2023-09-29T22:38:46.554 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.554 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.554 DEBUG:boto:Params: {} 2023-09-29T22:38:46.554 DEBUG:boto:Token: None 2023-09-29T22:38:46.554 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:38:46.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY= 2023-09-29T22:38:46.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iQcvqYEiGOhuvuj5FN1N4puCxGY=', 'Content-Length': '0'} 2023-09-29T22:38:46.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025c0fa3329d115c3-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.598 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.599 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.599 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.599 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.599 DEBUG:boto:Method: GET 2023-09-29T22:38:46.599 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.599 DEBUG:boto:Data: 2023-09-29T22:38:46.599 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.600 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.600 DEBUG:boto:Params: {} 2023-09-29T22:38:46.600 DEBUG:boto:Token: None 2023-09-29T22:38:46.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8e46cd155b973e6-00651751f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.606 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.610 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.610 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.610 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.610 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.610 DEBUG:boto:Method: GET 2023-09-29T22:38:46.610 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.610 DEBUG:boto:Data: 2023-09-29T22:38:46.610 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.611 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.611 DEBUG:boto:Params: {} 2023-09-29T22:38:46.611 DEBUG:boto:Token: None 2023-09-29T22:38:46.611 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000759c5a31015ae60b-00651751f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.617 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.617 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.618 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.618 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.618 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:38:46.618 DEBUG:boto:Data: 2023-09-29T22:38:46.618 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.618 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.618 DEBUG:boto:Params: {} 2023-09-29T22:38:46.619 DEBUG:boto:Token: None 2023-09-29T22:38:46.619 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11 2023-09-29T22:38:46.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uT7gv2km3GgVDMbckyEHLe7Etbs= 2023-09-29T22:38:46.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uT7gv2km3GgVDMbckyEHLe7Etbs=', 'Content-Length': '0'} 2023-09-29T22:38:46.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70aa7803ccda387-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.622 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.622 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.622 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.622 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:38:46.622 DEBUG:boto:Data: 2023-09-29T22:38:46.622 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.622 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.622 DEBUG:boto:Params: {} 2023-09-29T22:38:46.623 DEBUG:boto:Token: None 2023-09-29T22:38:46.623 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11 2023-09-29T22:38:46.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uT7gv2km3GgVDMbckyEHLe7Etbs= 2023-09-29T22:38:46.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uT7gv2km3GgVDMbckyEHLe7Etbs=', 'Content-Length': '0'} 2023-09-29T22:38:46.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d7a4be37a30b054-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:38:46.626 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.626 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.626 DEBUG:boto:Method: GET 2023-09-29T22:38:46.626 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:38:46.626 DEBUG:boto:Data: 2023-09-29T22:38:46.626 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.626 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.626 DEBUG:boto:Params: {} 2023-09-29T22:38:46.627 DEBUG:boto:Token: None 2023-09-29T22:38:46.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11 2023-09-29T22:38:46.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zux9gwJksM0VdIZ42AxX3vwB13M= 2023-09-29T22:38:46.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zux9gwJksM0VdIZ42AxX3vwB13M=', 'Content-Length': '0'} 2023-09-29T22:38:46.630 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a28850a9bcb0c166-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.674 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:38:46.675 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:38:46.675 DEBUG:boto:Method: GET 2023-09-29T22:38:46.675 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:38:46.675 DEBUG:boto:Data: 2023-09-29T22:38:46.675 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.675 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.675 DEBUG:boto:Params: {} 2023-09-29T22:38:46.676 DEBUG:boto:Token: None 2023-09-29T22:38:46.676 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj11 2023-09-29T22:38:46.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zux9gwJksM0VdIZ42AxX3vwB13M= 2023-09-29T22:38:46.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zux9gwJksM0VdIZ42AxX3vwB13M=', 'Content-Length': '0'} 2023-09-29T22:38:46.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000daeb369424ef3578-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.679 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.680 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.680 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.680 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.680 DEBUG:boto:Method: GET 2023-09-29T22:38:46.680 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.680 DEBUG:boto:Data: 2023-09-29T22:38:46.680 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.680 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.681 DEBUG:boto:Params: {} 2023-09-29T22:38:46.681 DEBUG:boto:Token: None 2023-09-29T22:38:46.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.685 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083deb85b16ba2e1c-00651751f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.686 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.686 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.686 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.686 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.686 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.687 DEBUG:boto:Method: GET 2023-09-29T22:38:46.687 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:38:46.687 DEBUG:boto:Data: 2023-09-29T22:38:46.687 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.687 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.687 DEBUG:boto:Params: {} 2023-09-29T22:38:46.687 DEBUG:boto:Token: None 2023-09-29T22:38:46.687 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000998f1d921f82ece7-00651751f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.738 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.739 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:38:46.739 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.739 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.739 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.739 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.739 DEBUG:boto:Method: GET 2023-09-29T22:38:46.740 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.740 DEBUG:boto:Data: 2023-09-29T22:38:46.740 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.740 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.740 DEBUG:boto:Params: {} 2023-09-29T22:38:46.740 DEBUG:boto:Token: None 2023-09-29T22:38:46.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY= 2023-09-29T22:38:46.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY=', 'Content-Length': '0'} 2023-09-29T22:38:46.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e53199d0220fa35a-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.744 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.744 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.744 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.744 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.744 DEBUG:boto:Method: GET 2023-09-29T22:38:46.745 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.745 DEBUG:boto:Data: 2023-09-29T22:38:46.745 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.745 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.745 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.745 DEBUG:boto:Params: {} 2023-09-29T22:38:46.745 DEBUG:boto:Token: None 2023-09-29T22:38:46.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY= 2023-09-29T22:38:46.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY=', 'Content-Length': '0'} 2023-09-29T22:38:46.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002617044449d6b205-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.749 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.749 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.749 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.749 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.749 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.749 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.749 DEBUG:boto:Data: 2023-09-29T22:38:46.749 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.749 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.749 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.750 DEBUG:boto:Params: {} 2023-09-29T22:38:46.750 DEBUG:boto:Token: None 2023-09-29T22:38:46.750 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.750 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gukjF9/xrL/tdImGj5bo2NVGhrM= 2023-09-29T22:38:46.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gukjF9/xrL/tdImGj5bo2NVGhrM=', 'Content-Length': '0'} 2023-09-29T22:38:46.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000019370ba674e0707-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.753 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.753 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.753 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.753 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.753 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.754 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.754 DEBUG:boto:Data: 2023-09-29T22:38:46.754 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.754 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.754 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.754 DEBUG:boto:Params: {} 2023-09-29T22:38:46.754 DEBUG:boto:Token: None 2023-09-29T22:38:46.754 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.755 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gukjF9/xrL/tdImGj5bo2NVGhrM= 2023-09-29T22:38:46.755 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gukjF9/xrL/tdImGj5bo2NVGhrM=', 'Content-Length': '0'} 2023-09-29T22:38:46.757 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab74e1045f0c863-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:38:46.758 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.758 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.758 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.758 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.758 DEBUG:boto:Method: GET 2023-09-29T22:38:46.758 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.758 DEBUG:boto:Data: 2023-09-29T22:38:46.758 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.759 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.759 DEBUG:boto:Params: {} 2023-09-29T22:38:46.759 DEBUG:boto:Token: None 2023-09-29T22:38:46.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.759 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY= 2023-09-29T22:38:46.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY=', 'Content-Length': '0'} 2023-09-29T22:38:46.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978d73e3b13392e7-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.762 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.762 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.763 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.763 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.763 DEBUG:boto:Method: GET 2023-09-29T22:38:46.763 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.763 DEBUG:boto:Data: 2023-09-29T22:38:46.763 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.763 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.763 DEBUG:boto:Params: {} 2023-09-29T22:38:46.763 DEBUG:boto:Token: None 2023-09-29T22:38:46.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:38:46.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY= 2023-09-29T22:38:46.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9WCBDUwMGN/ATQRbV1IY1I2YvfY=', 'Content-Length': '0'} 2023-09-29T22:38:46.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009180df4b01864d78-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.767 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.767 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.767 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.767 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.767 DEBUG:boto:Method: GET 2023-09-29T22:38:46.767 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.768 DEBUG:boto:Data: 2023-09-29T22:38:46.768 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.768 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.768 DEBUG:boto:Params: {} 2023-09-29T22:38:46.768 DEBUG:boto:Token: None 2023-09-29T22:38:46.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0f3bcd0c016fea5-00651751f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.773 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.774 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.774 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.774 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.774 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.774 DEBUG:boto:Method: GET 2023-09-29T22:38:46.774 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.774 DEBUG:boto:Data: 2023-09-29T22:38:46.775 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.775 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.775 DEBUG:boto:Params: {} 2023-09-29T22:38:46.775 DEBUG:boto:Token: None 2023-09-29T22:38:46.775 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.775 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.775 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5d63b3d10411485-00651751f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.822 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.823 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.823 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.824 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.824 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:38:46.824 DEBUG:boto:Data: 2023-09-29T22:38:46.824 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.824 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.825 DEBUG:boto:Params: {} 2023-09-29T22:38:46.825 DEBUG:boto:Token: None 2023-09-29T22:38:46.825 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17 2023-09-29T22:38:46.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3deLw88Bsdo3bRzUfyq69ealGzc= 2023-09-29T22:38:46.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3deLw88Bsdo3bRzUfyq69ealGzc=', 'Content-Length': '0'} 2023-09-29T22:38:46.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4f1ffa983cb6078-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.829 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.829 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.829 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.829 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:38:46.829 DEBUG:boto:Data: 2023-09-29T22:38:46.829 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.830 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.830 DEBUG:boto:Params: {} 2023-09-29T22:38:46.830 DEBUG:boto:Token: None 2023-09-29T22:38:46.830 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17 2023-09-29T22:38:46.831 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3deLw88Bsdo3bRzUfyq69ealGzc= 2023-09-29T22:38:46.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3deLw88Bsdo3bRzUfyq69ealGzc=', 'Content-Length': '0'} 2023-09-29T22:38:46.834 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7aadcfc067b9e6d-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:38:46.835 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.835 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.835 DEBUG:boto:Method: GET 2023-09-29T22:38:46.836 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:38:46.836 DEBUG:boto:Data: 2023-09-29T22:38:46.836 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.836 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.837 DEBUG:boto:Params: {} 2023-09-29T22:38:46.837 DEBUG:boto:Token: None 2023-09-29T22:38:46.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17 2023-09-29T22:38:46.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NMy6si7xVYxO3cEEod+nQC6SsYw= 2023-09-29T22:38:46.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NMy6si7xVYxO3cEEod+nQC6SsYw=', 'Content-Length': '0'} 2023-09-29T22:38:46.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d91d46f8bdcc64c-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.882 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:38:46.883 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:38:46.883 DEBUG:boto:Method: GET 2023-09-29T22:38:46.883 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:38:46.883 DEBUG:boto:Data: 2023-09-29T22:38:46.883 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.883 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.883 DEBUG:boto:Params: {} 2023-09-29T22:38:46.884 DEBUG:boto:Token: None 2023-09-29T22:38:46.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj17 2023-09-29T22:38:46.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NMy6si7xVYxO3cEEod+nQC6SsYw= 2023-09-29T22:38:46.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NMy6si7xVYxO3cEEod+nQC6SsYw=', 'Content-Length': '0'} 2023-09-29T22:38:46.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f543ef92293ec3ea-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.887 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.887 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.888 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.888 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.888 DEBUG:boto:Method: GET 2023-09-29T22:38:46.888 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.888 DEBUG:boto:Data: 2023-09-29T22:38:46.888 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.888 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.888 DEBUG:boto:Params: {} 2023-09-29T22:38:46.889 DEBUG:boto:Token: None 2023-09-29T22:38:46.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076354951efede932-00651751f6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.893 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.894 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:46.894 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:46.894 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.894 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.894 DEBUG:boto:Method: GET 2023-09-29T22:38:46.894 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:38:46.895 DEBUG:boto:Data: 2023-09-29T22:38:46.895 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.895 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.895 DEBUG:boto:Params: {} 2023-09-29T22:38:46.895 DEBUG:boto:Token: None 2023-09-29T22:38:46.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/ 2023-09-29T22:38:46.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A= 2023-09-29T22:38:46.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqWCLqLP5l5DCwpxVPr5wSFXC0A=', 'Content-Length': '0'} 2023-09-29T22:38:46.901 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3e904eb9ee2a3e4-00651751f6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.902 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:46.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:38:46.902 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:46.902 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:46.902 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.903 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.903 DEBUG:boto:Method: GET 2023-09-29T22:38:46.903 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.903 DEBUG:boto:Data: 2023-09-29T22:38:46.903 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.903 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.903 DEBUG:boto:Params: {} 2023-09-29T22:38:46.903 DEBUG:boto:Token: None 2023-09-29T22:38:46.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HkboqLu3UltHqCAPKLWt5jrA6Co= 2023-09-29T22:38:46.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HkboqLu3UltHqCAPKLWt5jrA6Co=', 'Content-Length': '0'} 2023-09-29T22:38:46.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2ebe2e0cd4b2ab2-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.907 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:46.907 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:46.907 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.907 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.907 DEBUG:boto:Method: GET 2023-09-29T22:38:46.908 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.908 DEBUG:boto:Data: 2023-09-29T22:38:46.908 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.908 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.908 DEBUG:boto:Params: {} 2023-09-29T22:38:46.908 DEBUG:boto:Token: None 2023-09-29T22:38:46.908 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HkboqLu3UltHqCAPKLWt5jrA6Co= 2023-09-29T22:38:46.909 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HkboqLu3UltHqCAPKLWt5jrA6Co=', 'Content-Length': '0'} 2023-09-29T22:38:46.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82be39012253150-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.958 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:46.959 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:46.959 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.959 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.959 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.959 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.959 DEBUG:boto:Data: 2023-09-29T22:38:46.959 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.959 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.960 DEBUG:boto:Params: {} 2023-09-29T22:38:46.960 DEBUG:boto:Token: None 2023-09-29T22:38:46.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RrqsUctsPn15vmezbcqpeJHBYBc= 2023-09-29T22:38:46.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RrqsUctsPn15vmezbcqpeJHBYBc=', 'Content-Length': '0'} 2023-09-29T22:38:46.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000296e86240897e033-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.964 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:46.964 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:46.964 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.964 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.964 DEBUG:boto:Method: HEAD 2023-09-29T22:38:46.964 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.964 DEBUG:boto:Data: 2023-09-29T22:38:46.964 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:46.965 DEBUG:boto:Port: 8000 2023-09-29T22:38:46.965 DEBUG:boto:Params: {} 2023-09-29T22:38:46.965 DEBUG:boto:Token: None 2023-09-29T22:38:46.965 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.965 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RrqsUctsPn15vmezbcqpeJHBYBc= 2023-09-29T22:38:46.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RrqsUctsPn15vmezbcqpeJHBYBc=', 'Content-Length': '0'} 2023-09-29T22:38:46.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004eec468ab8669b15-00651751f6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:46.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:38:46.969 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:46.970 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:46.970 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.971 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.971 DEBUG:boto:Method: GET 2023-09-29T22:38:46.971 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.971 DEBUG:boto:Data: 2023-09-29T22:38:46.972 DEBUG:boto:Headers: {} 2023-09-29T22:38:46.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:46.972 DEBUG:boto:Port: 8001 2023-09-29T22:38:46.972 DEBUG:boto:Params: {} 2023-09-29T22:38:46.973 DEBUG:boto:Token: None 2023-09-29T22:38:46.973 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:46 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:46.973 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HkboqLu3UltHqCAPKLWt5jrA6Co= 2023-09-29T22:38:46.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HkboqLu3UltHqCAPKLWt5jrA6Co=', 'Content-Length': '0'} 2023-09-29T22:38:46.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050109b71a4595c07-00651751f6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.018 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:47.019 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:47.019 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:47.019 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:38:47.019 DEBUG:boto:Method: GET 2023-09-29T22:38:47.020 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:47.020 DEBUG:boto:Data: 2023-09-29T22:38:47.020 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.021 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.021 DEBUG:boto:Params: {} 2023-09-29T22:38:47.021 DEBUG:boto:Token: None 2023-09-29T22:38:47.022 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:38:47.022 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y7gbnmQd6VnF6qQHCQ4anIBFXBM= 2023-09-29T22:38:47.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y7gbnmQd6VnF6qQHCQ4anIBFXBM=', 'Content-Length': '0'} 2023-09-29T22:38:47.025 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b8179bb94dbd3a-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.026 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.026 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.026 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.026 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.027 DEBUG:boto:Method: GET 2023-09-29T22:38:47.027 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.027 DEBUG:boto:Data: 2023-09-29T22:38:47.027 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.027 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.028 DEBUG:boto:Params: {} 2023-09-29T22:38:47.028 DEBUG:boto:Token: None 2023-09-29T22:38:47.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aac4b54605a1f496-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.034 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.035 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.035 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.035 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.035 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.035 DEBUG:boto:Method: GET 2023-09-29T22:38:47.035 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.036 DEBUG:boto:Data: 2023-09-29T22:38:47.036 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.036 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.036 DEBUG:boto:Params: {} 2023-09-29T22:38:47.037 DEBUG:boto:Token: None 2023-09-29T22:38:47.037 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.037 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf68c8ed731e6ead-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.043 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.044 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:47.044 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:47.044 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.044 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:38:47.045 DEBUG:boto:Data: 2023-09-29T22:38:47.045 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.045 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.045 DEBUG:boto:Params: {} 2023-09-29T22:38:47.045 DEBUG:boto:Token: None 2023-09-29T22:38:47.046 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj22 2023-09-29T22:38:47.046 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FoiO0bEe00iTkt6WMQN8p0vNANE= 2023-09-29T22:38:47.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FoiO0bEe00iTkt6WMQN8p0vNANE=', 'Content-Length': '0'} 2023-09-29T22:38:47.048 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f747394cecf5c0b-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.049 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:47.049 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:47.049 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.049 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:38:47.049 DEBUG:boto:Data: 2023-09-29T22:38:47.050 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.050 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.050 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.050 DEBUG:boto:Params: {} 2023-09-29T22:38:47.050 DEBUG:boto:Token: None 2023-09-29T22:38:47.050 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj22 2023-09-29T22:38:47.051 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FoiO0bEe00iTkt6WMQN8p0vNANE= 2023-09-29T22:38:47.051 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FoiO0bEe00iTkt6WMQN8p0vNANE=', 'Content-Length': '0'} 2023-09-29T22:38:47.053 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bd1c552229d0201-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:38:47.054 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:47.054 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:47.054 DEBUG:boto:Method: GET 2023-09-29T22:38:47.054 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:38:47.054 DEBUG:boto:Data: 2023-09-29T22:38:47.055 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.055 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.055 DEBUG:boto:Params: {} 2023-09-29T22:38:47.055 DEBUG:boto:Token: None 2023-09-29T22:38:47.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj22 2023-09-29T22:38:47.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aI/UAyosewAcIWHxWhXmrnlJXas= 2023-09-29T22:38:47.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aI/UAyosewAcIWHxWhXmrnlJXas=', 'Content-Length': '0'} 2023-09-29T22:38:47.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009001ec511c7e21ab-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.060 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:38:47.060 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:38:47.060 DEBUG:boto:Method: GET 2023-09-29T22:38:47.060 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:38:47.060 DEBUG:boto:Data: 2023-09-29T22:38:47.061 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.061 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.061 DEBUG:boto:Params: {} 2023-09-29T22:38:47.061 DEBUG:boto:Token: None 2023-09-29T22:38:47.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj22 2023-09-29T22:38:47.062 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aI/UAyosewAcIWHxWhXmrnlJXas= 2023-09-29T22:38:47.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aI/UAyosewAcIWHxWhXmrnlJXas=', 'Content-Length': '0'} 2023-09-29T22:38:47.065 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000176a7407be25fbfb-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.065 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.065 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.065 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.066 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.066 DEBUG:boto:Method: GET 2023-09-29T22:38:47.066 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.066 DEBUG:boto:Data: 2023-09-29T22:38:47.066 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.067 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.067 DEBUG:boto:Params: {} 2023-09-29T22:38:47.067 DEBUG:boto:Token: None 2023-09-29T22:38:47.067 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.067 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099959a42733107cd-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.114 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.115 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.115 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.116 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.116 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.116 DEBUG:boto:Method: GET 2023-09-29T22:38:47.116 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:38:47.116 DEBUG:boto:Data: 2023-09-29T22:38:47.116 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.116 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.116 DEBUG:boto:Params: {} 2023-09-29T22:38:47.117 DEBUG:boto:Token: None 2023-09-29T22:38:47.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ece7aad309eb17-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.123 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:38:47.124 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.124 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.124 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.124 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.125 DEBUG:boto:Method: GET 2023-09-29T22:38:47.125 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.125 DEBUG:boto:Data: 2023-09-29T22:38:47.125 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.126 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.126 DEBUG:boto:Params: {} 2023-09-29T22:38:47.126 DEBUG:boto:Token: None 2023-09-29T22:38:47.126 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.127 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI= 2023-09-29T22:38:47.127 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI=', 'Content-Length': '0'} 2023-09-29T22:38:47.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096d2d7a97d5079be-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.130 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.130 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.131 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.131 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.131 DEBUG:boto:Method: GET 2023-09-29T22:38:47.131 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.131 DEBUG:boto:Data: 2023-09-29T22:38:47.131 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.132 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.132 DEBUG:boto:Params: {} 2023-09-29T22:38:47.132 DEBUG:boto:Token: None 2023-09-29T22:38:47.132 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI= 2023-09-29T22:38:47.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI=', 'Content-Length': '0'} 2023-09-29T22:38:47.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000662765b0b06e678b-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.182 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.183 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.183 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.183 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.183 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.183 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.183 DEBUG:boto:Data: 2023-09-29T22:38:47.183 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.184 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.184 DEBUG:boto:Params: {} 2023-09-29T22:38:47.184 DEBUG:boto:Token: None 2023-09-29T22:38:47.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylVyE5k/a9OFaPcFsvmMscnDhWQ= 2023-09-29T22:38:47.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylVyE5k/a9OFaPcFsvmMscnDhWQ=', 'Content-Length': '0'} 2023-09-29T22:38:47.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b33788d3b914ce4d-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.187 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.188 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.188 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.188 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.188 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.188 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.188 DEBUG:boto:Data: 2023-09-29T22:38:47.188 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.188 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.189 DEBUG:boto:Params: {} 2023-09-29T22:38:47.189 DEBUG:boto:Token: None 2023-09-29T22:38:47.189 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.189 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ylVyE5k/a9OFaPcFsvmMscnDhWQ= 2023-09-29T22:38:47.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ylVyE5k/a9OFaPcFsvmMscnDhWQ=', 'Content-Length': '0'} 2023-09-29T22:38:47.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020bbfefb29d2ca6d-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.192 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:38:47.192 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.192 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.193 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.193 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.193 DEBUG:boto:Method: GET 2023-09-29T22:38:47.193 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.193 DEBUG:boto:Data: 2023-09-29T22:38:47.193 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.193 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.194 DEBUG:boto:Params: {} 2023-09-29T22:38:47.194 DEBUG:boto:Token: None 2023-09-29T22:38:47.194 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI= 2023-09-29T22:38:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI=', 'Content-Length': '0'} 2023-09-29T22:38:47.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ba35c2995628024-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.198 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.198 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.198 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.198 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.198 DEBUG:boto:Method: GET 2023-09-29T22:38:47.198 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.198 DEBUG:boto:Data: 2023-09-29T22:38:47.199 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.199 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.199 DEBUG:boto:Params: {} 2023-09-29T22:38:47.199 DEBUG:boto:Token: None 2023-09-29T22:38:47.199 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:38:47.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI= 2023-09-29T22:38:47.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PSNAqy0e9oXjeJLB6NsavP+uiwI=', 'Content-Length': '0'} 2023-09-29T22:38:47.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a328f0131cb1d30a-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.203 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.203 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.203 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.203 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.203 DEBUG:boto:Method: GET 2023-09-29T22:38:47.203 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.203 DEBUG:boto:Data: 2023-09-29T22:38:47.203 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.203 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.203 DEBUG:boto:Params: {} 2023-09-29T22:38:47.204 DEBUG:boto:Token: None 2023-09-29T22:38:47.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a84933701a7de82e-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.209 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.210 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.210 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.210 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.210 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.210 DEBUG:boto:Method: GET 2023-09-29T22:38:47.210 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.210 DEBUG:boto:Data: 2023-09-29T22:38:47.210 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.211 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.211 DEBUG:boto:Params: {} 2023-09-29T22:38:47.211 DEBUG:boto:Token: None 2023-09-29T22:38:47.211 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.216 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000413eb3aef2138575-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.262 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.263 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.263 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.263 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.263 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:38:47.264 DEBUG:boto:Data: 2023-09-29T22:38:47.264 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.264 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.264 DEBUG:boto:Params: {} 2023-09-29T22:38:47.265 DEBUG:boto:Token: None 2023-09-29T22:38:47.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33 2023-09-29T22:38:47.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UYq6CEvz7vdx8G7XRAMqjIVaMSg= 2023-09-29T22:38:47.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UYq6CEvz7vdx8G7XRAMqjIVaMSg=', 'Content-Length': '0'} 2023-09-29T22:38:47.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce5442e2c2bed6b-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.268 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.268 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.268 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.268 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:38:47.269 DEBUG:boto:Data: 2023-09-29T22:38:47.269 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.269 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.269 DEBUG:boto:Params: {} 2023-09-29T22:38:47.269 DEBUG:boto:Token: None 2023-09-29T22:38:47.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33 2023-09-29T22:38:47.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UYq6CEvz7vdx8G7XRAMqjIVaMSg= 2023-09-29T22:38:47.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UYq6CEvz7vdx8G7XRAMqjIVaMSg=', 'Content-Length': '0'} 2023-09-29T22:38:47.272 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2bd64e1956258ce-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:38:47.272 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.273 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.273 DEBUG:boto:Method: GET 2023-09-29T22:38:47.273 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:38:47.273 DEBUG:boto:Data: 2023-09-29T22:38:47.273 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.273 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.273 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.273 DEBUG:boto:Params: {} 2023-09-29T22:38:47.274 DEBUG:boto:Token: None 2023-09-29T22:38:47.274 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33 2023-09-29T22:38:47.274 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x2RZciL2yjGDCJmaqqeX3AALnX8= 2023-09-29T22:38:47.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x2RZciL2yjGDCJmaqqeX3AALnX8=', 'Content-Length': '0'} 2023-09-29T22:38:47.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e7f5c675d1b0388-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.322 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:38:47.323 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:38:47.323 DEBUG:boto:Method: GET 2023-09-29T22:38:47.323 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:38:47.323 DEBUG:boto:Data: 2023-09-29T22:38:47.323 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.323 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.323 DEBUG:boto:Params: {} 2023-09-29T22:38:47.324 DEBUG:boto:Token: None 2023-09-29T22:38:47.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj33 2023-09-29T22:38:47.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x2RZciL2yjGDCJmaqqeX3AALnX8= 2023-09-29T22:38:47.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x2RZciL2yjGDCJmaqqeX3AALnX8=', 'Content-Length': '0'} 2023-09-29T22:38:47.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c44c846965e35b30-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.327 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.328 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.328 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.328 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.328 DEBUG:boto:Method: GET 2023-09-29T22:38:47.328 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.328 DEBUG:boto:Data: 2023-09-29T22:38:47.328 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.328 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.329 DEBUG:boto:Params: {} 2023-09-29T22:38:47.329 DEBUG:boto:Token: None 2023-09-29T22:38:47.329 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b7f57d0bcd4893-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.334 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.334 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.334 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.335 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.335 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.335 DEBUG:boto:Method: GET 2023-09-29T22:38:47.335 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:38:47.335 DEBUG:boto:Data: 2023-09-29T22:38:47.335 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.335 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.335 DEBUG:boto:Params: {} 2023-09-29T22:38:47.335 DEBUG:boto:Token: None 2023-09-29T22:38:47.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a06da37dfa5d2c85-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.341 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:38:47.342 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.342 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.342 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.342 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.342 DEBUG:boto:Method: GET 2023-09-29T22:38:47.342 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.342 DEBUG:boto:Data: 2023-09-29T22:38:47.342 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.343 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.343 DEBUG:boto:Params: {} 2023-09-29T22:38:47.343 DEBUG:boto:Token: None 2023-09-29T22:38:47.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw= 2023-09-29T22:38:47.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw=', 'Content-Length': '0'} 2023-09-29T22:38:47.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013fde225b21d2724-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.346 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.347 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.347 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.347 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.347 DEBUG:boto:Method: GET 2023-09-29T22:38:47.347 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.347 DEBUG:boto:Data: 2023-09-29T22:38:47.347 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.347 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.348 DEBUG:boto:Params: {} 2023-09-29T22:38:47.348 DEBUG:boto:Token: None 2023-09-29T22:38:47.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw= 2023-09-29T22:38:47.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw=', 'Content-Length': '0'} 2023-09-29T22:38:47.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e98785a3da934f15-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.394 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.395 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.395 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.395 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.395 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.395 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.395 DEBUG:boto:Data: 2023-09-29T22:38:47.395 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.395 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.396 DEBUG:boto:Params: {} 2023-09-29T22:38:47.396 DEBUG:boto:Token: None 2023-09-29T22:38:47.396 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:46H3yt3kxXZh4XykClLktGQYbgI= 2023-09-29T22:38:47.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:46H3yt3kxXZh4XykClLktGQYbgI=', 'Content-Length': '0'} 2023-09-29T22:38:47.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8a11f13b17db6e3-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.399 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.399 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.399 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.400 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.400 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.400 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.400 DEBUG:boto:Data: 2023-09-29T22:38:47.400 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.400 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.400 DEBUG:boto:Params: {} 2023-09-29T22:38:47.400 DEBUG:boto:Token: None 2023-09-29T22:38:47.401 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:46H3yt3kxXZh4XykClLktGQYbgI= 2023-09-29T22:38:47.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:46H3yt3kxXZh4XykClLktGQYbgI=', 'Content-Length': '0'} 2023-09-29T22:38:47.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001273d1ee99e8925-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:38:47.404 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.404 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.404 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.405 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.405 DEBUG:boto:Method: GET 2023-09-29T22:38:47.405 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.405 DEBUG:boto:Data: 2023-09-29T22:38:47.405 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.405 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.405 DEBUG:boto:Params: {} 2023-09-29T22:38:47.405 DEBUG:boto:Token: None 2023-09-29T22:38:47.406 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw= 2023-09-29T22:38:47.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw=', 'Content-Length': '0'} 2023-09-29T22:38:47.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000616e196580d3f405-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.450 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.451 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.451 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.451 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.451 DEBUG:boto:Method: GET 2023-09-29T22:38:47.451 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.451 DEBUG:boto:Data: 2023-09-29T22:38:47.451 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.451 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.452 DEBUG:boto:Params: {} 2023-09-29T22:38:47.452 DEBUG:boto:Token: None 2023-09-29T22:38:47.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:38:47.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw= 2023-09-29T22:38:47.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uZXnSJ/swMLrONrejku9NYA2Pnw=', 'Content-Length': '0'} 2023-09-29T22:38:47.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc6907883539d568-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.456 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.456 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.456 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.456 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.456 DEBUG:boto:Method: GET 2023-09-29T22:38:47.456 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.456 DEBUG:boto:Data: 2023-09-29T22:38:47.456 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.456 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.457 DEBUG:boto:Params: {} 2023-09-29T22:38:47.457 DEBUG:boto:Token: None 2023-09-29T22:38:47.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e1647bb39272637-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.462 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.462 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.463 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.463 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.463 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.463 DEBUG:boto:Method: GET 2023-09-29T22:38:47.463 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.463 DEBUG:boto:Data: 2023-09-29T22:38:47.463 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.463 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.463 DEBUG:boto:Params: {} 2023-09-29T22:38:47.464 DEBUG:boto:Token: None 2023-09-29T22:38:47.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.469 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b9ade0e7a665c20-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.469 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.470 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.470 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.470 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.470 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:38:47.470 DEBUG:boto:Data: 2023-09-29T22:38:47.470 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.471 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.471 DEBUG:boto:Params: {} 2023-09-29T22:38:47.471 DEBUG:boto:Token: None 2023-09-29T22:38:47.471 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34 2023-09-29T22:38:47.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ukkDQ56YESX6HjIpvx1t7Dhks2A= 2023-09-29T22:38:47.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ukkDQ56YESX6HjIpvx1t7Dhks2A=', 'Content-Length': '0'} 2023-09-29T22:38:47.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720b77e2d5ab8dd7-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.474 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.474 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.474 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.474 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:38:47.474 DEBUG:boto:Data: 2023-09-29T22:38:47.474 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.475 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.475 DEBUG:boto:Params: {} 2023-09-29T22:38:47.475 DEBUG:boto:Token: None 2023-09-29T22:38:47.475 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34 2023-09-29T22:38:47.475 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ukkDQ56YESX6HjIpvx1t7Dhks2A= 2023-09-29T22:38:47.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ukkDQ56YESX6HjIpvx1t7Dhks2A=', 'Content-Length': '0'} 2023-09-29T22:38:47.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4d6345d42b97026-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:38:47.478 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.478 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.478 DEBUG:boto:Method: GET 2023-09-29T22:38:47.478 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:38:47.478 DEBUG:boto:Data: 2023-09-29T22:38:47.478 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.478 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.478 DEBUG:boto:Params: {} 2023-09-29T22:38:47.479 DEBUG:boto:Token: None 2023-09-29T22:38:47.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34 2023-09-29T22:38:47.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y/EsMrBKgMAL07LwfT6vLG35ctU= 2023-09-29T22:38:47.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y/EsMrBKgMAL07LwfT6vLG35ctU=', 'Content-Length': '0'} 2023-09-29T22:38:47.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fc6de8bd35f50e5-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.481 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:38:47.482 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:38:47.482 DEBUG:boto:Method: GET 2023-09-29T22:38:47.482 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:38:47.482 DEBUG:boto:Data: 2023-09-29T22:38:47.482 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.482 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.482 DEBUG:boto:Params: {} 2023-09-29T22:38:47.483 DEBUG:boto:Token: None 2023-09-29T22:38:47.483 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj34 2023-09-29T22:38:47.483 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y/EsMrBKgMAL07LwfT6vLG35ctU= 2023-09-29T22:38:47.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y/EsMrBKgMAL07LwfT6vLG35ctU=', 'Content-Length': '0'} 2023-09-29T22:38:47.485 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e372c003778c35a-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.526 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.527 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.527 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.527 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.527 DEBUG:boto:Method: GET 2023-09-29T22:38:47.527 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.527 DEBUG:boto:Data: 2023-09-29T22:38:47.527 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.527 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.527 DEBUG:boto:Params: {} 2023-09-29T22:38:47.528 DEBUG:boto:Token: None 2023-09-29T22:38:47.528 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045ff3bfed77be175-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.533 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.533 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.533 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.533 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.534 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.534 DEBUG:boto:Method: GET 2023-09-29T22:38:47.534 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:38:47.534 DEBUG:boto:Data: 2023-09-29T22:38:47.534 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.534 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.534 DEBUG:boto:Params: {} 2023-09-29T22:38:47.534 DEBUG:boto:Token: None 2023-09-29T22:38:47.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000049e1296393467be-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.541 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:38:47.541 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.541 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.541 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.541 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.542 DEBUG:boto:Method: GET 2023-09-29T22:38:47.542 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.542 DEBUG:boto:Data: 2023-09-29T22:38:47.542 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.542 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.542 DEBUG:boto:Params: {} 2023-09-29T22:38:47.542 DEBUG:boto:Token: None 2023-09-29T22:38:47.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.543 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg= 2023-09-29T22:38:47.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg=', 'Content-Length': '0'} 2023-09-29T22:38:47.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a717ea3809c714c5-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.586 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.587 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.587 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.587 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.587 DEBUG:boto:Method: GET 2023-09-29T22:38:47.587 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.587 DEBUG:boto:Data: 2023-09-29T22:38:47.587 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.587 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.588 DEBUG:boto:Params: {} 2023-09-29T22:38:47.588 DEBUG:boto:Token: None 2023-09-29T22:38:47.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg= 2023-09-29T22:38:47.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg=', 'Content-Length': '0'} 2023-09-29T22:38:47.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fe108ee9ec6a17-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.593 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.594 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.594 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.594 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.594 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.594 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.594 DEBUG:boto:Data: 2023-09-29T22:38:47.594 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.595 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.595 DEBUG:boto:Params: {} 2023-09-29T22:38:47.595 DEBUG:boto:Token: None 2023-09-29T22:38:47.595 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.595 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XGUePm80GQa5uiDwTNX5l7OHvrc= 2023-09-29T22:38:47.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XGUePm80GQa5uiDwTNX5l7OHvrc=', 'Content-Length': '0'} 2023-09-29T22:38:47.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faf72f4c08ab6387-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.598 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.599 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.599 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.599 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.599 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.599 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.599 DEBUG:boto:Data: 2023-09-29T22:38:47.599 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.600 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.600 DEBUG:boto:Params: {} 2023-09-29T22:38:47.600 DEBUG:boto:Token: None 2023-09-29T22:38:47.600 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XGUePm80GQa5uiDwTNX5l7OHvrc= 2023-09-29T22:38:47.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XGUePm80GQa5uiDwTNX5l7OHvrc=', 'Content-Length': '0'} 2023-09-29T22:38:47.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060205cd3d8dcba7a-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:38:47.603 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.603 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.604 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.604 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.604 DEBUG:boto:Method: GET 2023-09-29T22:38:47.604 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.604 DEBUG:boto:Data: 2023-09-29T22:38:47.604 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.604 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.604 DEBUG:boto:Params: {} 2023-09-29T22:38:47.604 DEBUG:boto:Token: None 2023-09-29T22:38:47.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.605 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg= 2023-09-29T22:38:47.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg=', 'Content-Length': '0'} 2023-09-29T22:38:47.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5168a14012f806e-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.608 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.608 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.608 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.608 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.608 DEBUG:boto:Method: GET 2023-09-29T22:38:47.608 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.608 DEBUG:boto:Data: 2023-09-29T22:38:47.608 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.609 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.609 DEBUG:boto:Params: {} 2023-09-29T22:38:47.609 DEBUG:boto:Token: None 2023-09-29T22:38:47.609 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:38:47.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg= 2023-09-29T22:38:47.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:14s6wHIsIwtwmsJPKxDNt82FMAg=', 'Content-Length': '0'} 2023-09-29T22:38:47.612 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cce3512f8c5c857-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.654 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.655 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.655 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.655 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.655 DEBUG:boto:Method: GET 2023-09-29T22:38:47.655 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.655 DEBUG:boto:Data: 2023-09-29T22:38:47.655 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.655 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.656 DEBUG:boto:Params: {} 2023-09-29T22:38:47.656 DEBUG:boto:Token: None 2023-09-29T22:38:47.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c0c39e7dd2ad37b-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.661 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.661 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.661 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.661 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.661 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.662 DEBUG:boto:Method: GET 2023-09-29T22:38:47.662 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.662 DEBUG:boto:Data: 2023-09-29T22:38:47.662 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.662 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.662 DEBUG:boto:Params: {} 2023-09-29T22:38:47.662 DEBUG:boto:Token: None 2023-09-29T22:38:47.662 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.668 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c578f42479e89b18-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.668 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.668 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.668 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.669 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.669 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:38:47.669 DEBUG:boto:Data: 2023-09-29T22:38:47.669 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.669 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.669 DEBUG:boto:Params: {} 2023-09-29T22:38:47.669 DEBUG:boto:Token: None 2023-09-29T22:38:47.669 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44 2023-09-29T22:38:47.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mx+lrLU4X22ZUm8qSbJIfzUuUM4= 2023-09-29T22:38:47.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mx+lrLU4X22ZUm8qSbJIfzUuUM4=', 'Content-Length': '0'} 2023-09-29T22:38:47.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2841c24b49a9af1-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.672 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.672 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.672 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.672 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:38:47.672 DEBUG:boto:Data: 2023-09-29T22:38:47.673 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.673 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.673 DEBUG:boto:Params: {} 2023-09-29T22:38:47.673 DEBUG:boto:Token: None 2023-09-29T22:38:47.673 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44 2023-09-29T22:38:47.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mx+lrLU4X22ZUm8qSbJIfzUuUM4= 2023-09-29T22:38:47.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mx+lrLU4X22ZUm8qSbJIfzUuUM4=', 'Content-Length': '0'} 2023-09-29T22:38:47.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdfaf2d6f4d19aeb-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:38:47.675 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.676 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.676 DEBUG:boto:Method: GET 2023-09-29T22:38:47.676 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:38:47.676 DEBUG:boto:Data: 2023-09-29T22:38:47.676 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.676 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.676 DEBUG:boto:Params: {} 2023-09-29T22:38:47.676 DEBUG:boto:Token: None 2023-09-29T22:38:47.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44 2023-09-29T22:38:47.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZiCNPAB3UPwx1aAYRksSSVqYui4= 2023-09-29T22:38:47.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZiCNPAB3UPwx1aAYRksSSVqYui4=', 'Content-Length': '0'} 2023-09-29T22:38:47.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7be584aa4370268-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.679 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:38:47.680 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:38:47.680 DEBUG:boto:Method: GET 2023-09-29T22:38:47.680 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:38:47.680 DEBUG:boto:Data: 2023-09-29T22:38:47.680 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.680 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.680 DEBUG:boto:Params: {} 2023-09-29T22:38:47.681 DEBUG:boto:Token: None 2023-09-29T22:38:47.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj44 2023-09-29T22:38:47.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZiCNPAB3UPwx1aAYRksSSVqYui4= 2023-09-29T22:38:47.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZiCNPAB3UPwx1aAYRksSSVqYui4=', 'Content-Length': '0'} 2023-09-29T22:38:47.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025ff7477347139ab-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.730 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.731 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.731 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.731 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.731 DEBUG:boto:Method: GET 2023-09-29T22:38:47.731 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.731 DEBUG:boto:Data: 2023-09-29T22:38:47.731 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.732 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.732 DEBUG:boto:Params: {} 2023-09-29T22:38:47.732 DEBUG:boto:Token: None 2023-09-29T22:38:47.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d30f286dc41562c7-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.738 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.738 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.738 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.738 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.739 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.739 DEBUG:boto:Method: GET 2023-09-29T22:38:47.739 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:38:47.739 DEBUG:boto:Data: 2023-09-29T22:38:47.739 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.739 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.739 DEBUG:boto:Params: {} 2023-09-29T22:38:47.739 DEBUG:boto:Token: None 2023-09-29T22:38:47.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027168d2fd6c4c99e-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.746 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:38:47.746 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.746 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.746 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.746 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.747 DEBUG:boto:Method: GET 2023-09-29T22:38:47.747 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.747 DEBUG:boto:Data: 2023-09-29T22:38:47.747 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.747 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.747 DEBUG:boto:Params: {} 2023-09-29T22:38:47.747 DEBUG:boto:Token: None 2023-09-29T22:38:47.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc= 2023-09-29T22:38:47.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc=', 'Content-Length': '0'} 2023-09-29T22:38:47.750 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4dead343b02b564-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.790 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.791 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.791 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.791 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.791 DEBUG:boto:Method: GET 2023-09-29T22:38:47.791 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.791 DEBUG:boto:Data: 2023-09-29T22:38:47.791 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.791 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.792 DEBUG:boto:Params: {} 2023-09-29T22:38:47.792 DEBUG:boto:Token: None 2023-09-29T22:38:47.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc= 2023-09-29T22:38:47.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc=', 'Content-Length': '0'} 2023-09-29T22:38:47.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2a7c2537cf1ab14-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.795 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.797 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.797 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.797 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.797 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.797 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.798 DEBUG:boto:Data: 2023-09-29T22:38:47.798 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.798 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.798 DEBUG:boto:Params: {} 2023-09-29T22:38:47.798 DEBUG:boto:Token: None 2023-09-29T22:38:47.798 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faPJIQeLarhvseiP8c2ERBaG1fM= 2023-09-29T22:38:47.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faPJIQeLarhvseiP8c2ERBaG1fM=', 'Content-Length': '0'} 2023-09-29T22:38:47.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a44967b68d4d13f5-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.801 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.801 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.802 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.802 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.802 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.802 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.802 DEBUG:boto:Data: 2023-09-29T22:38:47.802 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.802 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.802 DEBUG:boto:Params: {} 2023-09-29T22:38:47.803 DEBUG:boto:Token: None 2023-09-29T22:38:47.803 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.803 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faPJIQeLarhvseiP8c2ERBaG1fM= 2023-09-29T22:38:47.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faPJIQeLarhvseiP8c2ERBaG1fM=', 'Content-Length': '0'} 2023-09-29T22:38:47.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cf4441b9d77cc82-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:38:47.806 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.806 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.806 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.806 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.806 DEBUG:boto:Method: GET 2023-09-29T22:38:47.807 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.807 DEBUG:boto:Data: 2023-09-29T22:38:47.807 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.807 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.807 DEBUG:boto:Params: {} 2023-09-29T22:38:47.807 DEBUG:boto:Token: None 2023-09-29T22:38:47.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc= 2023-09-29T22:38:47.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc=', 'Content-Length': '0'} 2023-09-29T22:38:47.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde02f9c9b2dea56-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.811 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.811 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.811 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.811 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.811 DEBUG:boto:Method: GET 2023-09-29T22:38:47.811 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.811 DEBUG:boto:Data: 2023-09-29T22:38:47.811 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.812 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.812 DEBUG:boto:Params: {} 2023-09-29T22:38:47.812 DEBUG:boto:Token: None 2023-09-29T22:38:47.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:38:47.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc= 2023-09-29T22:38:47.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CSIbGa3Qaohai12LiI3gkVE9Ttc=', 'Content-Length': '0'} 2023-09-29T22:38:47.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b8d8db966734f38-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.815 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.815 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.815 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.816 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.816 DEBUG:boto:Method: GET 2023-09-29T22:38:47.816 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.816 DEBUG:boto:Data: 2023-09-29T22:38:47.816 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.816 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.816 DEBUG:boto:Params: {} 2023-09-29T22:38:47.816 DEBUG:boto:Token: None 2023-09-29T22:38:47.817 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.817 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.822 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d0348a97156e01-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.822 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.822 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.822 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.823 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.823 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.823 DEBUG:boto:Method: GET 2023-09-29T22:38:47.823 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.823 DEBUG:boto:Data: 2023-09-29T22:38:47.823 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.823 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.823 DEBUG:boto:Params: {} 2023-09-29T22:38:47.824 DEBUG:boto:Token: None 2023-09-29T22:38:47.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db7c4f9f0eb0d295-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.870 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.871 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.871 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.871 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.871 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:38:47.871 DEBUG:boto:Data: 2023-09-29T22:38:47.872 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.872 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.872 DEBUG:boto:Params: {} 2023-09-29T22:38:47.872 DEBUG:boto:Token: None 2023-09-29T22:38:47.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51 2023-09-29T22:38:47.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tufWxeZopU3Q2XgfRTkcegWFoEs= 2023-09-29T22:38:47.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tufWxeZopU3Q2XgfRTkcegWFoEs=', 'Content-Length': '0'} 2023-09-29T22:38:47.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000035240e38cfb7151-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.875 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.876 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.876 DEBUG:boto:Method: HEAD 2023-09-29T22:38:47.876 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:38:47.876 DEBUG:boto:Data: 2023-09-29T22:38:47.876 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.876 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.876 DEBUG:boto:Params: {} 2023-09-29T22:38:47.876 DEBUG:boto:Token: None 2023-09-29T22:38:47.877 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51 2023-09-29T22:38:47.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tufWxeZopU3Q2XgfRTkcegWFoEs= 2023-09-29T22:38:47.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tufWxeZopU3Q2XgfRTkcegWFoEs=', 'Content-Length': '0'} 2023-09-29T22:38:47.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3536e6967b8b6a-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:38:47.880 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.880 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.880 DEBUG:boto:Method: GET 2023-09-29T22:38:47.880 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:38:47.880 DEBUG:boto:Data: 2023-09-29T22:38:47.880 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.880 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.880 DEBUG:boto:Params: {} 2023-09-29T22:38:47.881 DEBUG:boto:Token: None 2023-09-29T22:38:47.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51 2023-09-29T22:38:47.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I6b321aQrKY70ISYtNflmO0UwYA= 2023-09-29T22:38:47.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I6b321aQrKY70ISYtNflmO0UwYA=', 'Content-Length': '0'} 2023-09-29T22:38:47.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c23038a34eef1fca-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.930 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:38:47.931 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:38:47.931 DEBUG:boto:Method: GET 2023-09-29T22:38:47.931 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:38:47.931 DEBUG:boto:Data: 2023-09-29T22:38:47.931 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.931 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.931 DEBUG:boto:Params: {} 2023-09-29T22:38:47.931 DEBUG:boto:Token: None 2023-09-29T22:38:47.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj51 2023-09-29T22:38:47.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I6b321aQrKY70ISYtNflmO0UwYA= 2023-09-29T22:38:47.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I6b321aQrKY70ISYtNflmO0UwYA=', 'Content-Length': '0'} 2023-09-29T22:38:47.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000586473fd7465a10a-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.935 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.935 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.935 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.935 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.935 DEBUG:boto:Method: GET 2023-09-29T22:38:47.935 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.935 DEBUG:boto:Data: 2023-09-29T22:38:47.935 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.936 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.936 DEBUG:boto:Params: {} 2023-09-29T22:38:47.936 DEBUG:boto:Token: None 2023-09-29T22:38:47.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041105aa02d1b5b95-00651751f7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.941 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.941 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:47.941 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:47.941 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.941 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.942 DEBUG:boto:Method: GET 2023-09-29T22:38:47.942 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:38:47.942 DEBUG:boto:Data: 2023-09-29T22:38:47.942 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.942 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.942 DEBUG:boto:Params: {} 2023-09-29T22:38:47.942 DEBUG:boto:Token: None 2023-09-29T22:38:47.942 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/ 2023-09-29T22:38:47.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao= 2023-09-29T22:38:47.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFCKPnt6l8dyhJx7RHrtkszlNao=', 'Content-Length': '0'} 2023-09-29T22:38:47.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000622389630d950c86-00651751f7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.948 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:47.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:38:47.948 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:47.948 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:47.948 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.949 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.949 DEBUG:boto:Method: GET 2023-09-29T22:38:47.949 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.949 DEBUG:boto:Data: 2023-09-29T22:38:47.949 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:47.949 DEBUG:boto:Port: 8001 2023-09-29T22:38:47.949 DEBUG:boto:Params: {} 2023-09-29T22:38:47.949 DEBUG:boto:Token: None 2023-09-29T22:38:47.950 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.950 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oIephiCTwxrdXXKuciQeLcskzLQ= 2023-09-29T22:38:47.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oIephiCTwxrdXXKuciQeLcskzLQ=', 'Content-Length': '0'} 2023-09-29T22:38:47.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000258989b4bb39d579-00651751f7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:47.953 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:47.953 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:47.953 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.953 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.953 DEBUG:boto:Method: GET 2023-09-29T22:38:47.953 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.953 DEBUG:boto:Data: 2023-09-29T22:38:47.953 DEBUG:boto:Headers: {} 2023-09-29T22:38:47.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:47.954 DEBUG:boto:Port: 8000 2023-09-29T22:38:47.954 DEBUG:boto:Params: {} 2023-09-29T22:38:47.954 DEBUG:boto:Token: None 2023-09-29T22:38:47.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:47 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:47.954 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oIephiCTwxrdXXKuciQeLcskzLQ= 2023-09-29T22:38:47.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oIephiCTwxrdXXKuciQeLcskzLQ=', 'Content-Length': '0'} 2023-09-29T22:38:47.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ab0cabbceda4260-00651751f7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.002 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.003 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.003 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.003 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.003 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.003 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.003 DEBUG:boto:Data: 2023-09-29T22:38:48.003 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.004 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.004 DEBUG:boto:Params: {} 2023-09-29T22:38:48.004 DEBUG:boto:Token: None 2023-09-29T22:38:48.004 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qfNl+jf1BdPUnDBNNN9ef2/GSZQ= 2023-09-29T22:38:48.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qfNl+jf1BdPUnDBNNN9ef2/GSZQ=', 'Content-Length': '0'} 2023-09-29T22:38:48.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b54fc55dbb10ee57-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.007 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.008 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.008 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.008 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.008 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.008 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.008 DEBUG:boto:Data: 2023-09-29T22:38:48.008 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.008 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.008 DEBUG:boto:Params: {} 2023-09-29T22:38:48.009 DEBUG:boto:Token: None 2023-09-29T22:38:48.009 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qfNl+jf1BdPUnDBNNN9ef2/GSZQ= 2023-09-29T22:38:48.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qfNl+jf1BdPUnDBNNN9ef2/GSZQ=', 'Content-Length': '0'} 2023-09-29T22:38:48.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003876bd98955894cb-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:38:48.012 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.012 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.012 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.012 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.013 DEBUG:boto:Method: GET 2023-09-29T22:38:48.013 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.013 DEBUG:boto:Data: 2023-09-29T22:38:48.013 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.013 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.013 DEBUG:boto:Params: {} 2023-09-29T22:38:48.013 DEBUG:boto:Token: None 2023-09-29T22:38:48.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5KDFMIuPRSS0fXxmJ7q036hQQag= 2023-09-29T22:38:48.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5KDFMIuPRSS0fXxmJ7q036hQQag=', 'Content-Length': '0'} 2023-09-29T22:38:48.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e98d2b10c002a08-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.058 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.059 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.059 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.059 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.059 DEBUG:boto:Method: GET 2023-09-29T22:38:48.059 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.059 DEBUG:boto:Data: 2023-09-29T22:38:48.059 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.059 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.060 DEBUG:boto:Params: {} 2023-09-29T22:38:48.060 DEBUG:boto:Token: None 2023-09-29T22:38:48.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:38:48.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5KDFMIuPRSS0fXxmJ7q036hQQag= 2023-09-29T22:38:48.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5KDFMIuPRSS0fXxmJ7q036hQQag=', 'Content-Length': '0'} 2023-09-29T22:38:48.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecbea3347ca2cfa6-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.063 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.064 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.064 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.064 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.064 DEBUG:boto:Method: GET 2023-09-29T22:38:48.064 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.064 DEBUG:boto:Data: 2023-09-29T22:38:48.064 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.064 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.064 DEBUG:boto:Params: {} 2023-09-29T22:38:48.065 DEBUG:boto:Token: None 2023-09-29T22:38:48.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c3a338d1134be6b-00651751f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.070 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.071 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.071 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.071 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.071 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.071 DEBUG:boto:Method: GET 2023-09-29T22:38:48.071 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.071 DEBUG:boto:Data: 2023-09-29T22:38:48.072 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.072 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.072 DEBUG:boto:Params: {} 2023-09-29T22:38:48.072 DEBUG:boto:Token: None 2023-09-29T22:38:48.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000410e6881ada5c342-00651751f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.078 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.078 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.079 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.079 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.079 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:38:48.079 DEBUG:boto:Data: 2023-09-29T22:38:48.079 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.079 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.079 DEBUG:boto:Params: {} 2023-09-29T22:38:48.080 DEBUG:boto:Token: None 2023-09-29T22:38:48.080 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55 2023-09-29T22:38:48.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vsovfxSqqv3KhjMURONX/3f57Y4= 2023-09-29T22:38:48.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vsovfxSqqv3KhjMURONX/3f57Y4=', 'Content-Length': '0'} 2023-09-29T22:38:48.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc9d61894e6b0107-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.082 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.083 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.083 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.083 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:38:48.083 DEBUG:boto:Data: 2023-09-29T22:38:48.083 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.083 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.083 DEBUG:boto:Params: {} 2023-09-29T22:38:48.084 DEBUG:boto:Token: None 2023-09-29T22:38:48.084 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55 2023-09-29T22:38:48.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vsovfxSqqv3KhjMURONX/3f57Y4= 2023-09-29T22:38:48.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vsovfxSqqv3KhjMURONX/3f57Y4=', 'Content-Length': '0'} 2023-09-29T22:38:48.086 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e6dd72900db944e-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:38:48.087 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.087 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.087 DEBUG:boto:Method: GET 2023-09-29T22:38:48.087 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:38:48.087 DEBUG:boto:Data: 2023-09-29T22:38:48.087 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.088 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.088 DEBUG:boto:Params: {} 2023-09-29T22:38:48.088 DEBUG:boto:Token: None 2023-09-29T22:38:48.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55 2023-09-29T22:38:48.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tmroCVJJhapfKlpagjGv5FKjELo= 2023-09-29T22:38:48.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tmroCVJJhapfKlpagjGv5FKjELo=', 'Content-Length': '0'} 2023-09-29T22:38:48.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099621d6b0b6caacd-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.091 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:38:48.091 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:38:48.091 DEBUG:boto:Method: GET 2023-09-29T22:38:48.091 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:38:48.091 DEBUG:boto:Data: 2023-09-29T22:38:48.091 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.091 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.091 DEBUG:boto:Params: {} 2023-09-29T22:38:48.092 DEBUG:boto:Token: None 2023-09-29T22:38:48.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj55 2023-09-29T22:38:48.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tmroCVJJhapfKlpagjGv5FKjELo= 2023-09-29T22:38:48.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tmroCVJJhapfKlpagjGv5FKjELo=', 'Content-Length': '0'} 2023-09-29T22:38:48.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4390388e54d38b7-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.142 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.143 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.143 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.143 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.143 DEBUG:boto:Method: GET 2023-09-29T22:38:48.143 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.143 DEBUG:boto:Data: 2023-09-29T22:38:48.143 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.144 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.144 DEBUG:boto:Params: {} 2023-09-29T22:38:48.144 DEBUG:boto:Token: None 2023-09-29T22:38:48.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.148 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000abfd67c6e35d9221-00651751f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.149 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.149 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.149 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.149 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.149 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.150 DEBUG:boto:Method: GET 2023-09-29T22:38:48.150 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:38:48.150 DEBUG:boto:Data: 2023-09-29T22:38:48.150 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.150 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.150 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.150 DEBUG:boto:Params: {} 2023-09-29T22:38:48.150 DEBUG:boto:Token: None 2023-09-29T22:38:48.150 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.151 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.151 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.156 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d8c6bd085c49461-00651751f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.157 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:38:48.157 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.157 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.157 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.158 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.158 DEBUG:boto:Method: GET 2023-09-29T22:38:48.158 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.158 DEBUG:boto:Data: 2023-09-29T22:38:48.158 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.158 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.158 DEBUG:boto:Params: {} 2023-09-29T22:38:48.158 DEBUG:boto:Token: None 2023-09-29T22:38:48.159 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw= 2023-09-29T22:38:48.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw=', 'Content-Length': '0'} 2023-09-29T22:38:48.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b390f891171cd24a-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.202 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.202 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.203 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.203 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.203 DEBUG:boto:Method: GET 2023-09-29T22:38:48.203 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.203 DEBUG:boto:Data: 2023-09-29T22:38:48.203 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.203 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.203 DEBUG:boto:Params: {} 2023-09-29T22:38:48.203 DEBUG:boto:Token: None 2023-09-29T22:38:48.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw= 2023-09-29T22:38:48.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw=', 'Content-Length': '0'} 2023-09-29T22:38:48.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f2796cb439d2186-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.207 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.207 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.207 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.207 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.207 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.208 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.208 DEBUG:boto:Data: 2023-09-29T22:38:48.208 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.208 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.208 DEBUG:boto:Params: {} 2023-09-29T22:38:48.208 DEBUG:boto:Token: None 2023-09-29T22:38:48.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X3OY+ejHI6tmrsTDAmf6XQY0548= 2023-09-29T22:38:48.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X3OY+ejHI6tmrsTDAmf6XQY0548=', 'Content-Length': '0'} 2023-09-29T22:38:48.211 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7d1ba11f4d9c8f1-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.211 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.211 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.211 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.212 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.212 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.212 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.212 DEBUG:boto:Data: 2023-09-29T22:38:48.212 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.212 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.212 DEBUG:boto:Params: {} 2023-09-29T22:38:48.212 DEBUG:boto:Token: None 2023-09-29T22:38:48.213 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X3OY+ejHI6tmrsTDAmf6XQY0548= 2023-09-29T22:38:48.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X3OY+ejHI6tmrsTDAmf6XQY0548=', 'Content-Length': '0'} 2023-09-29T22:38:48.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ab5cf37819912a7-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:38:48.216 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.216 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.216 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.216 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.216 DEBUG:boto:Method: GET 2023-09-29T22:38:48.217 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.217 DEBUG:boto:Data: 2023-09-29T22:38:48.217 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.217 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.217 DEBUG:boto:Params: {} 2023-09-29T22:38:48.217 DEBUG:boto:Token: None 2023-09-29T22:38:48.217 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw= 2023-09-29T22:38:48.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw=', 'Content-Length': '0'} 2023-09-29T22:38:48.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043058a5ab5679900-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.220 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.220 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.221 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.221 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.221 DEBUG:boto:Method: GET 2023-09-29T22:38:48.221 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.221 DEBUG:boto:Data: 2023-09-29T22:38:48.221 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.221 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.221 DEBUG:boto:Params: {} 2023-09-29T22:38:48.222 DEBUG:boto:Token: None 2023-09-29T22:38:48.222 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:38:48.222 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw= 2023-09-29T22:38:48.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VwPgHTWrlU8PCXc6gyTIRoCvqyw=', 'Content-Length': '0'} 2023-09-29T22:38:48.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b149c8ea01434d4-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.266 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.266 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.267 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.267 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.267 DEBUG:boto:Method: GET 2023-09-29T22:38:48.267 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.267 DEBUG:boto:Data: 2023-09-29T22:38:48.267 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.267 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.267 DEBUG:boto:Params: {} 2023-09-29T22:38:48.268 DEBUG:boto:Token: None 2023-09-29T22:38:48.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b2d70d16bddf5248-00651751f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.273 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.273 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.273 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.274 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.274 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.274 DEBUG:boto:Method: GET 2023-09-29T22:38:48.274 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.274 DEBUG:boto:Data: 2023-09-29T22:38:48.274 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.274 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.274 DEBUG:boto:Params: {} 2023-09-29T22:38:48.275 DEBUG:boto:Token: None 2023-09-29T22:38:48.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000249968a314e450f4-00651751f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.281 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.281 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.281 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.282 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.282 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:38:48.282 DEBUG:boto:Data: 2023-09-29T22:38:48.282 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.282 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.282 DEBUG:boto:Params: {} 2023-09-29T22:38:48.282 DEBUG:boto:Token: None 2023-09-29T22:38:48.282 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68 2023-09-29T22:38:48.283 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v1O37tCNH3ZtjAhQ/C1V5ZS9s3g= 2023-09-29T22:38:48.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v1O37tCNH3ZtjAhQ/C1V5ZS9s3g=', 'Content-Length': '0'} 2023-09-29T22:38:48.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0e425d4c1515f25-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.285 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.285 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.286 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.286 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:38:48.286 DEBUG:boto:Data: 2023-09-29T22:38:48.286 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.286 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.286 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.286 DEBUG:boto:Params: {} 2023-09-29T22:38:48.286 DEBUG:boto:Token: None 2023-09-29T22:38:48.286 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68 2023-09-29T22:38:48.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v1O37tCNH3ZtjAhQ/C1V5ZS9s3g= 2023-09-29T22:38:48.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v1O37tCNH3ZtjAhQ/C1V5ZS9s3g=', 'Content-Length': '0'} 2023-09-29T22:38:48.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042d5cde13a28f130-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:38:48.289 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.289 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.289 DEBUG:boto:Method: GET 2023-09-29T22:38:48.290 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:38:48.290 DEBUG:boto:Data: 2023-09-29T22:38:48.290 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.290 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.290 DEBUG:boto:Params: {} 2023-09-29T22:38:48.290 DEBUG:boto:Token: None 2023-09-29T22:38:48.290 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68 2023-09-29T22:38:48.290 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F3abNewiJY248JcEtxT7OH7IurQ= 2023-09-29T22:38:48.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F3abNewiJY248JcEtxT7OH7IurQ=', 'Content-Length': '0'} 2023-09-29T22:38:48.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed4d1d9e0e762f37-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.334 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:38:48.334 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:38:48.335 DEBUG:boto:Method: GET 2023-09-29T22:38:48.335 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:38:48.335 DEBUG:boto:Data: 2023-09-29T22:38:48.335 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.335 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.335 DEBUG:boto:Params: {} 2023-09-29T22:38:48.335 DEBUG:boto:Token: None 2023-09-29T22:38:48.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj68 2023-09-29T22:38:48.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F3abNewiJY248JcEtxT7OH7IurQ= 2023-09-29T22:38:48.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F3abNewiJY248JcEtxT7OH7IurQ=', 'Content-Length': '0'} 2023-09-29T22:38:48.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6bb62b875b25f94-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.339 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.339 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.339 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.339 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.340 DEBUG:boto:Method: GET 2023-09-29T22:38:48.340 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.340 DEBUG:boto:Data: 2023-09-29T22:38:48.340 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.340 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.340 DEBUG:boto:Params: {} 2023-09-29T22:38:48.340 DEBUG:boto:Token: None 2023-09-29T22:38:48.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.345 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c12b0704db20778-00651751f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.345 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.345 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.346 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.346 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.346 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.346 DEBUG:boto:Method: GET 2023-09-29T22:38:48.346 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:38:48.346 DEBUG:boto:Data: 2023-09-29T22:38:48.346 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.347 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.347 DEBUG:boto:Params: {} 2023-09-29T22:38:48.347 DEBUG:boto:Token: None 2023-09-29T22:38:48.347 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d30916d6a75c3cd-00651751f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.394 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:38:48.395 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.395 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.395 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.395 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.395 DEBUG:boto:Method: GET 2023-09-29T22:38:48.395 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.395 DEBUG:boto:Data: 2023-09-29T22:38:48.396 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.396 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.396 DEBUG:boto:Params: {} 2023-09-29T22:38:48.396 DEBUG:boto:Token: None 2023-09-29T22:38:48.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y= 2023-09-29T22:38:48.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y=', 'Content-Length': '0'} 2023-09-29T22:38:48.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dc9381333dda758-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.400 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.400 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.400 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.400 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.400 DEBUG:boto:Method: GET 2023-09-29T22:38:48.400 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.400 DEBUG:boto:Data: 2023-09-29T22:38:48.401 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.401 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.401 DEBUG:boto:Params: {} 2023-09-29T22:38:48.401 DEBUG:boto:Token: None 2023-09-29T22:38:48.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y= 2023-09-29T22:38:48.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y=', 'Content-Length': '0'} 2023-09-29T22:38:48.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b951fdd006979967-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.405 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.405 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.405 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.405 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.405 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.405 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.405 DEBUG:boto:Data: 2023-09-29T22:38:48.405 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.406 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.406 DEBUG:boto:Params: {} 2023-09-29T22:38:48.406 DEBUG:boto:Token: None 2023-09-29T22:38:48.406 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.406 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZFsebD7by9g6fXUwT+21aHdmPFk= 2023-09-29T22:38:48.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZFsebD7by9g6fXUwT+21aHdmPFk=', 'Content-Length': '0'} 2023-09-29T22:38:48.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbf665a6eb41abdb-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.412 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.412 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.412 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.412 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.412 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.412 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.412 DEBUG:boto:Data: 2023-09-29T22:38:48.413 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.413 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.413 DEBUG:boto:Params: {} 2023-09-29T22:38:48.413 DEBUG:boto:Token: None 2023-09-29T22:38:48.413 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZFsebD7by9g6fXUwT+21aHdmPFk= 2023-09-29T22:38:48.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZFsebD7by9g6fXUwT+21aHdmPFk=', 'Content-Length': '0'} 2023-09-29T22:38:48.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000238b53a792158cac-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:38:48.417 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.417 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.417 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.417 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.417 DEBUG:boto:Method: GET 2023-09-29T22:38:48.417 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.417 DEBUG:boto:Data: 2023-09-29T22:38:48.417 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.417 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.418 DEBUG:boto:Params: {} 2023-09-29T22:38:48.418 DEBUG:boto:Token: None 2023-09-29T22:38:48.418 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y= 2023-09-29T22:38:48.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y=', 'Content-Length': '0'} 2023-09-29T22:38:48.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b4fa8868843d7a1-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.462 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.462 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.463 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.463 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.463 DEBUG:boto:Method: GET 2023-09-29T22:38:48.463 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.463 DEBUG:boto:Data: 2023-09-29T22:38:48.463 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.463 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.463 DEBUG:boto:Params: {} 2023-09-29T22:38:48.464 DEBUG:boto:Token: None 2023-09-29T22:38:48.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:38:48.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y= 2023-09-29T22:38:48.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1l53VN1ep6VAcPJeAuZWpDdFN6Y=', 'Content-Length': '0'} 2023-09-29T22:38:48.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000065eb9295021c908-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.467 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.467 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.467 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.467 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.467 DEBUG:boto:Method: GET 2023-09-29T22:38:48.468 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.468 DEBUG:boto:Data: 2023-09-29T22:38:48.468 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.468 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.468 DEBUG:boto:Params: {} 2023-09-29T22:38:48.468 DEBUG:boto:Token: None 2023-09-29T22:38:48.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.474 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cfa019f0f191381f-00651751f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.474 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.475 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.475 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.475 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.475 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.475 DEBUG:boto:Method: GET 2023-09-29T22:38:48.475 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.475 DEBUG:boto:Data: 2023-09-29T22:38:48.475 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.476 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.476 DEBUG:boto:Params: {} 2023-09-29T22:38:48.476 DEBUG:boto:Token: None 2023-09-29T22:38:48.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7f23ba8a0908a51-00651751f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.522 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.523 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.523 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.523 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.523 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:38:48.523 DEBUG:boto:Data: 2023-09-29T22:38:48.523 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.524 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.524 DEBUG:boto:Params: {} 2023-09-29T22:38:48.524 DEBUG:boto:Token: None 2023-09-29T22:38:48.524 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85 2023-09-29T22:38:48.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V9fkavHDKlOheNz2ihn8LUPdTCo= 2023-09-29T22:38:48.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V9fkavHDKlOheNz2ihn8LUPdTCo=', 'Content-Length': '0'} 2023-09-29T22:38:48.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f77b72e3357d7040-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.527 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.528 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.528 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.528 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:38:48.528 DEBUG:boto:Data: 2023-09-29T22:38:48.528 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.528 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.528 DEBUG:boto:Params: {} 2023-09-29T22:38:48.528 DEBUG:boto:Token: None 2023-09-29T22:38:48.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85 2023-09-29T22:38:48.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V9fkavHDKlOheNz2ihn8LUPdTCo= 2023-09-29T22:38:48.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V9fkavHDKlOheNz2ihn8LUPdTCo=', 'Content-Length': '0'} 2023-09-29T22:38:48.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022f6174a890e8231-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:38:48.532 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.532 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.532 DEBUG:boto:Method: GET 2023-09-29T22:38:48.532 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:38:48.532 DEBUG:boto:Data: 2023-09-29T22:38:48.533 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.533 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.533 DEBUG:boto:Params: {} 2023-09-29T22:38:48.533 DEBUG:boto:Token: None 2023-09-29T22:38:48.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85 2023-09-29T22:38:48.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h0r3TBr3wkAeGGJEwAqqlQfTSLo= 2023-09-29T22:38:48.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h0r3TBr3wkAeGGJEwAqqlQfTSLo=', 'Content-Length': '0'} 2023-09-29T22:38:48.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e56b5ad14084f0df-00651751f8-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.537 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:38:48.537 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:38:48.537 DEBUG:boto:Method: GET 2023-09-29T22:38:48.537 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:38:48.538 DEBUG:boto:Data: 2023-09-29T22:38:48.538 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.538 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.538 DEBUG:boto:Params: {} 2023-09-29T22:38:48.538 DEBUG:boto:Token: None 2023-09-29T22:38:48.538 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj85 2023-09-29T22:38:48.539 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h0r3TBr3wkAeGGJEwAqqlQfTSLo= 2023-09-29T22:38:48.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h0r3TBr3wkAeGGJEwAqqlQfTSLo=', 'Content-Length': '0'} 2023-09-29T22:38:48.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004367f7352f8ee00e-00651751f8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.542 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.542 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.542 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.542 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.543 DEBUG:boto:Method: GET 2023-09-29T22:38:48.543 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.543 DEBUG:boto:Data: 2023-09-29T22:38:48.543 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:38:48.543 DEBUG:boto:Port: 8001 2023-09-29T22:38:48.544 DEBUG:boto:Params: {} 2023-09-29T22:38:48.544 DEBUG:boto:Token: None 2023-09-29T22:38:48.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b4d53461df721101-00651751f8-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.549 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.550 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.550 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.550 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.550 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.550 DEBUG:boto:Method: GET 2023-09-29T22:38:48.550 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:38:48.550 DEBUG:boto:Data: 2023-09-29T22:38:48.550 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.551 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.551 DEBUG:boto:Params: {} 2023-09-29T22:38:48.551 DEBUG:boto:Token: None 2023-09-29T22:38:48.551 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.551 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0= 2023-09-29T22:38:48.551 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwRROZcc8X0nZkj3FHumtiCljb0=', 'Content-Length': '0'} 2023-09-29T22:38:48.557 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a1c46a8e45502a2a-00651751f8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.557 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:38:48.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:38:48.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:38:48.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:38:48.558 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:38:48.604 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:48.604 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:48.683 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:38:48.683 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-59 2023-09-29T22:38:48.697 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-09-29T22:38:48.792 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.793 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.793 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.793 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:48.793 DEBUG:boto:Data: 2023-09-29T22:38:48.793 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.793 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.794 DEBUG:boto:Params: {} 2023-09-29T22:38:48.794 DEBUG:boto:Token: None 2023-09-29T22:38:48.794 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.794 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg= 2023-09-29T22:38:48.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg=', 'Content-Length': '0'} 2023-09-29T22:38:48.799 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000005562c7123a6f2c9-00651751f8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.799 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:38:48.800 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:38:48.800 DEBUG:boto:Method: PUT 2023-09-29T22:38:48.800 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:38:48.800 DEBUG:boto:Data: 2023-09-29T22:38:48.800 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:38:48.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.800 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.801 DEBUG:boto:Params: {} 2023-09-29T22:38:48.801 DEBUG:boto:Token: None 2023-09-29T22:38:48.801 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj19 2023-09-29T22:38:48.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p6GDE4KzFUkaoyaJIGpFw3uJ2Sc= 2023-09-29T22:38:48.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p6GDE4KzFUkaoyaJIGpFw3uJ2Sc='} 2023-09-29T22:38:48.846 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009e8203abd97881d9-00651751f8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.847 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.847 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.847 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.847 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:48.847 DEBUG:boto:Data: 2023-09-29T22:38:48.847 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.848 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.848 DEBUG:boto:Params: {} 2023-09-29T22:38:48.848 DEBUG:boto:Token: None 2023-09-29T22:38:48.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg= 2023-09-29T22:38:48.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg=', 'Content-Length': '0'} 2023-09-29T22:38:48.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a8ec80bcda406672-00651751f8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.854 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:38:48.854 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:38:48.854 DEBUG:boto:Method: PUT 2023-09-29T22:38:48.854 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:38:48.855 DEBUG:boto:Data: 2023-09-29T22:38:48.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:38:48.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.855 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.855 DEBUG:boto:Params: {} 2023-09-29T22:38:48.855 DEBUG:boto:Token: None 2023-09-29T22:38:48.855 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj38 2023-09-29T22:38:48.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y7QOQoMzOhwhxELDD1S9WmVEQCA= 2023-09-29T22:38:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y7QOQoMzOhwhxELDD1S9WmVEQCA='} 2023-09-29T22:38:48.863 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008d674a883b017fe5-00651751f8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.863 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.863 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.863 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.863 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:48.864 DEBUG:boto:Data: 2023-09-29T22:38:48.864 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.864 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.864 DEBUG:boto:Params: {} 2023-09-29T22:38:48.864 DEBUG:boto:Token: None 2023-09-29T22:38:48.864 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg= 2023-09-29T22:38:48.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg=', 'Content-Length': '0'} 2023-09-29T22:38:48.869 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bf743d5a5301dd3b-00651751f8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.869 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:38:48.869 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:38:48.870 DEBUG:boto:Method: PUT 2023-09-29T22:38:48.870 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:38:48.870 DEBUG:boto:Data: 2023-09-29T22:38:48.870 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:38:48.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.870 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.870 DEBUG:boto:Params: {} 2023-09-29T22:38:48.870 DEBUG:boto:Token: None 2023-09-29T22:38:48.871 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj57 2023-09-29T22:38:48.871 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vy3mOt5UsyM7UiHyhRCugs+qcrc= 2023-09-29T22:38:48.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vy3mOt5UsyM7UiHyhRCugs+qcrc='} 2023-09-29T22:38:48.914 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000097e8dea2133fa830-00651751f8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.915 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.915 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.915 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.915 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:48.915 DEBUG:boto:Data: 2023-09-29T22:38:48.915 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.916 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.916 DEBUG:boto:Params: {} 2023-09-29T22:38:48.916 DEBUG:boto:Token: None 2023-09-29T22:38:48.916 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.916 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg= 2023-09-29T22:38:48.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg=', 'Content-Length': '0'} 2023-09-29T22:38:48.921 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bd7a4203d521f0b3-00651751f8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.921 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:38:48.922 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:38:48.922 DEBUG:boto:Method: PUT 2023-09-29T22:38:48.922 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:38:48.922 DEBUG:boto:Data: 2023-09-29T22:38:48.922 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:38:48.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.922 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.922 DEBUG:boto:Params: {} 2023-09-29T22:38:48.923 DEBUG:boto:Token: None 2023-09-29T22:38:48.923 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj76 2023-09-29T22:38:48.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4azvuXvvCzMq1LKiKQAHI3O6Hsw= 2023-09-29T22:38:48.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4azvuXvvCzMq1LKiKQAHI3O6Hsw='} 2023-09-29T22:38:48.934 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000de432eeaa0714567-00651751f8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.934 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:38:48.934 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:38:48.934 DEBUG:boto:Method: HEAD 2023-09-29T22:38:48.934 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:38:48.935 DEBUG:boto:Data: 2023-09-29T22:38:48.935 DEBUG:boto:Headers: {} 2023-09-29T22:38:48.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.935 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.935 DEBUG:boto:Params: {} 2023-09-29T22:38:48.935 DEBUG:boto:Token: None 2023-09-29T22:38:48.935 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/ 2023-09-29T22:38:48.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg= 2023-09-29T22:38:48.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmORIjvG5SA0C9uo2xWFkNoJehg=', 'Content-Length': '0'} 2023-09-29T22:38:48.940 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000027d1c0e2d856551c-00651751f8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.940 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:38:48.940 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:38:48.940 DEBUG:boto:Method: PUT 2023-09-29T22:38:48.940 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:38:48.940 DEBUG:boto:Data: 2023-09-29T22:38:48.941 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:38:48.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:38:48.941 DEBUG:boto:Port: 8000 2023-09-29T22:38:48.941 DEBUG:boto:Params: {} 2023-09-29T22:38:48.941 DEBUG:boto:Token: None 2023-09-29T22:38:48.941 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:38:48 GMT /hcnphz-59/obj95 2023-09-29T22:38:48.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AiqVImC0azGBRE+oqpnT/lYhi1g= 2023-09-29T22:38:48.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:38:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AiqVImC0azGBRE+oqpnT/lYhi1g='} 2023-09-29T22:38:48.986 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d60faa006488d1b4-00651751f8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:38:48.987 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:38:48.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:38:48.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:38:48.987 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:38:49.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:38:49.042 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:38:49.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:49.109+0000 7f86787eef40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:38:54.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:54.109+0000 7f86787eef40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:38:59.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:38:59.112+0000 7f86787eef40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T22:39:04.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:04.116+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:04.120 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-29T22:39:04.120 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:09.120+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:09.122 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-29T22:39:09.122 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:14.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:14.120+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:14.124 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-29T22:39:14.124 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:19.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:19.124+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:19.127 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-29T22:39:19.127 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:24.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:24.128+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:24.130 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-29T22:39:24.130 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:29.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:29.128+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:29.132 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , ] 2023-09-29T22:39:29.132 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:34.137 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:34.136+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:34.137 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, , , , , , , , , , , , ] 2023-09-29T22:39:34.137 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:39.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:39.139+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:39.140 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, , , , , , , , , , , , ] 2023-09-29T22:39:39.141 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:44.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:44.139+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:44.143 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:39:44.143 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:49.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:49.143+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:49.146 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:39:49.146 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:54.148 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:54.147+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:54.148 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:39:54.148 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:39:59.151 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:39:59.147+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:39:59.246 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:39:59.246 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:04.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:04.151+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:04.153 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:04.153 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:09.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:09.155+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:09.157 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:09.157 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:14.158 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:14.155+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:14.159 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:14.159 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:19.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:19.158+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:19.162 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:19.162 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:24.164 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:24.162+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:24.165 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:24.165 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:29.168 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:29.162+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:29.168 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:29.168 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:34.170 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:34.166+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:34.170 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:34.171 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:39.173 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:39.170+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:39.173 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:39.173 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:44.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:44.174+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:44.176 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:44.176 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:49.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:49.174+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:49.179 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:49.180 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:54.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:54.178+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:54.182 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:54.182 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:40:59.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:40:59.181+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:40:59.185 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000001.90.6, , , , , , , , , , , ] 2023-09-29T22:40:59.185 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:04.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:04.181+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:04.188 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:04.188 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:09.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:09.189+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:09.191 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:09.191 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:14.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:14.193+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:14.193 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:14.194 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:19.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:19.197+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:19.196 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:19.197 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:24.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:24.197+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:24.200 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:24.200 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:29.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:29.201+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:29.202 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:29.202 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:34.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:34.205+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:34.204 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:34.204 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:39.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:39.204+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:39.207 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:39.207 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:44.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:44.208+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:44.210 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:44.211 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:49.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:49.212+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:49.213 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:49.214 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:54.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:54.216+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:54.216 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:54.216 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:41:59.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:41:59.216+0000 7f86787eef40 1 waiting for incremental sync to catch up: 2023-09-29T22:41:59.219 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000002.92.6, , , , , , , , , , , ] 2023-09-29T22:41:59.220 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:42:04.222 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:04.220+0000 7f86787eef40 1 bucket sync caught up with source: 2023-09-29T22:42:04.223 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:42:04.223 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.33.6, 00000000004.96.6, , , , , , , , , , , ] 2023-09-29T22:42:04.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:04.220+0000 7f86787eef40 0 bucket checkpoint complete 2023-09-29T22:42:04.254 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:42:04.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:42:04.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:42:04.255 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:42:04.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:04.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:04.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:04.372+0000 7fbd638c2f40 1 bucket sync caught up with source: 2023-09-29T22:42:04.373 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000001.580.6, 00000000001.228.6, 00000000001.181.6, 00000000001.222.6, , , 00000000002.270.6, 00000000004.183.6, 00000000003.306.6] 2023-09-29T22:42:04.373 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000001.580.6, 00000000001.228.6, 00000000001.181.6, 00000000001.222.6, , , 00000000002.270.6, 00000000004.183.6, 00000000003.306.6] 2023-09-29T22:42:04.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:04.372+0000 7fbd638c2f40 0 bucket checkpoint complete 2023-09-29T22:42:04.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:42:04.400 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.400 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.400 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.400 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:04.400 DEBUG:boto:Data: 2023-09-29T22:42:04.400 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.401 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.401 DEBUG:boto:Params: {} 2023-09-29T22:42:04.401 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:42:04.401 DEBUG:boto:Token: None 2023-09-29T22:42:04.402 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4qqvjKvTdjwzqH8BiUz9oQcPjnY= 2023-09-29T22:42:04.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4qqvjKvTdjwzqH8BiUz9oQcPjnY=', 'Content-Length': '0'} 2023-09-29T22:42:04.408 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3620b158605f9e7-00651752bc-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.408 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.408 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.408 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.408 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:04.408 DEBUG:boto:Data: 2023-09-29T22:42:04.409 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.409 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.409 DEBUG:boto:Params: {} 2023-09-29T22:42:04.409 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:42:04.409 DEBUG:boto:Token: None 2023-09-29T22:42:04.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4qqvjKvTdjwzqH8BiUz9oQcPjnY= 2023-09-29T22:42:04.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4qqvjKvTdjwzqH8BiUz9oQcPjnY=', 'Content-Length': '0'} 2023-09-29T22:42:04.416 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000483a1c429d438367-00651752bc-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.416 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:42:04.416 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.416 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.416 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.416 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.417 DEBUG:boto:Method: GET 2023-09-29T22:42:04.417 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.417 DEBUG:boto:Data: 2023-09-29T22:42:04.417 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.417 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.417 DEBUG:boto:Params: {} 2023-09-29T22:42:04.417 DEBUG:boto:Token: None 2023-09-29T22:42:04.417 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/?versions 2023-09-29T22:42:04.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q= 2023-09-29T22:42:04.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q=', 'Content-Length': '0'} 2023-09-29T22:42:04.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d9da7d6fc77523d-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.422 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:42:04.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:42:04.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:42:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:42:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:42:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:42:04.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:42:04.425 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:42:04.425 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.425 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.425 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.426 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.426 DEBUG:boto:Method: GET 2023-09-29T22:42:04.426 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.426 DEBUG:boto:Data: 2023-09-29T22:42:04.426 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.426 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.426 DEBUG:boto:Params: {} 2023-09-29T22:42:04.427 DEBUG:boto:Token: None 2023-09-29T22:42:04.427 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/?versions 2023-09-29T22:42:04.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q= 2023-09-29T22:42:04.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q=', 'Content-Length': '0'} 2023-09-29T22:42:04.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068ed1ec9ab4db69d-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.432 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:42:04.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:42:04.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:42:04.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:42:04.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:42:04.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:42:04.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:42:04.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:42:04.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:42:04.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:42:04.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:42:04.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:42:04.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:42:04.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:42:04.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:42:04.436 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.436 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.436 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.436 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.436 DEBUG:boto:Method: GET 2023-09-29T22:42:04.436 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.436 DEBUG:boto:Data: 2023-09-29T22:42:04.436 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.437 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.437 DEBUG:boto:Params: {} 2023-09-29T22:42:04.437 DEBUG:boto:Token: None 2023-09-29T22:42:04.437 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/?versions 2023-09-29T22:42:04.437 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q= 2023-09-29T22:42:04.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q=', 'Content-Length': '0'} 2023-09-29T22:42:04.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000032f3ae096c82b4f-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.442 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.443 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.443 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.443 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.443 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.443 DEBUG:boto:Method: GET 2023-09-29T22:42:04.443 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:42:04.443 DEBUG:boto:Data: 2023-09-29T22:42:04.443 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.444 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.444 DEBUG:boto:Params: {} 2023-09-29T22:42:04.444 DEBUG:boto:Token: None 2023-09-29T22:42:04.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/?versions 2023-09-29T22:42:04.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q= 2023-09-29T22:42:04.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jsNIuooUiJK/YR+2BabLXlJMe8Q=', 'Content-Length': '0'} 2023-09-29T22:42:04.449 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f808580af77ae4bb-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.450 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:42:04.451 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.451 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.451 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.451 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.451 DEBUG:boto:Method: GET 2023-09-29T22:42:04.452 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.452 DEBUG:boto:Data: 2023-09-29T22:42:04.452 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.452 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.452 DEBUG:boto:Params: {} 2023-09-29T22:42:04.452 DEBUG:boto:Token: None 2023-09-29T22:42:04.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g= 2023-09-29T22:42:04.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g=', 'Content-Length': '0'} 2023-09-29T22:42:04.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9d36a27c672a7e5-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.455 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.455 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.455 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.455 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.456 DEBUG:boto:Method: GET 2023-09-29T22:42:04.456 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.456 DEBUG:boto:Data: 2023-09-29T22:42:04.456 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.456 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.456 DEBUG:boto:Params: {} 2023-09-29T22:42:04.456 DEBUG:boto:Token: None 2023-09-29T22:42:04.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g= 2023-09-29T22:42:04.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g=', 'Content-Length': '0'} 2023-09-29T22:42:04.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b33a2a3c9295a25-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.506 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.507 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.507 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.507 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.507 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.507 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.507 DEBUG:boto:Data: 2023-09-29T22:42:04.507 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.508 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.508 DEBUG:boto:Params: {} 2023-09-29T22:42:04.508 DEBUG:boto:Token: None 2023-09-29T22:42:04.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4MJWtHNsufcLKRuoYlwAQwqit5g= 2023-09-29T22:42:04.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4MJWtHNsufcLKRuoYlwAQwqit5g=', 'Content-Length': '0'} 2023-09-29T22:42:04.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000218bbbfc33987a7-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.512 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.512 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.512 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.512 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.512 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.512 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.512 DEBUG:boto:Data: 2023-09-29T22:42:04.512 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.513 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.513 DEBUG:boto:Params: {} 2023-09-29T22:42:04.513 DEBUG:boto:Token: None 2023-09-29T22:42:04.513 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4MJWtHNsufcLKRuoYlwAQwqit5g= 2023-09-29T22:42:04.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4MJWtHNsufcLKRuoYlwAQwqit5g=', 'Content-Length': '0'} 2023-09-29T22:42:04.516 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c59d3b1c14b5c548-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:42:04.516 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.517 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.517 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.517 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.517 DEBUG:boto:Method: GET 2023-09-29T22:42:04.517 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.517 DEBUG:boto:Data: 2023-09-29T22:42:04.517 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.517 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.518 DEBUG:boto:Params: {} 2023-09-29T22:42:04.518 DEBUG:boto:Token: None 2023-09-29T22:42:04.518 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.518 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g= 2023-09-29T22:42:04.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g=', 'Content-Length': '0'} 2023-09-29T22:42:04.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f7d12f0dad99115-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.562 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.563 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.563 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.563 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.563 DEBUG:boto:Method: GET 2023-09-29T22:42:04.563 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.563 DEBUG:boto:Data: 2023-09-29T22:42:04.563 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.564 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.564 DEBUG:boto:Params: {} 2023-09-29T22:42:04.564 DEBUG:boto:Token: None 2023-09-29T22:42:04.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:42:04.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g= 2023-09-29T22:42:04.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yTGh0JgqlyIWyOZnR0PRQ1GEg7g=', 'Content-Length': '0'} 2023-09-29T22:42:04.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e88163fceb951ec2-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.568 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.568 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.568 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.568 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.568 DEBUG:boto:Method: GET 2023-09-29T22:42:04.568 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.568 DEBUG:boto:Data: 2023-09-29T22:42:04.569 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.569 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.569 DEBUG:boto:Params: {} 2023-09-29T22:42:04.569 DEBUG:boto:Token: None 2023-09-29T22:42:04.569 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.573 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000175e7ddd9bb91425-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.574 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.574 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.575 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.575 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.575 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.575 DEBUG:boto:Method: GET 2023-09-29T22:42:04.575 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.575 DEBUG:boto:Data: 2023-09-29T22:42:04.575 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.575 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.575 DEBUG:boto:Params: {} 2023-09-29T22:42:04.576 DEBUG:boto:Token: None 2023-09-29T22:42:04.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd1ceeea18a23085-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.582 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.582 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.582 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.583 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.583 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:42:04.583 DEBUG:boto:Data: 2023-09-29T22:42:04.583 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.583 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.583 DEBUG:boto:Params: {} 2023-09-29T22:42:04.583 DEBUG:boto:Token: None 2023-09-29T22:42:04.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11 2023-09-29T22:42:04.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3MlwXdRnlJw2pQ3yWFpbUiH9IAQ= 2023-09-29T22:42:04.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3MlwXdRnlJw2pQ3yWFpbUiH9IAQ=', 'Content-Length': '0'} 2023-09-29T22:42:04.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b99ba5ab4cc771a6-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.588 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.588 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.588 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.588 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:42:04.588 DEBUG:boto:Data: 2023-09-29T22:42:04.589 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.589 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.589 DEBUG:boto:Params: {} 2023-09-29T22:42:04.589 DEBUG:boto:Token: None 2023-09-29T22:42:04.589 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11 2023-09-29T22:42:04.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3MlwXdRnlJw2pQ3yWFpbUiH9IAQ= 2023-09-29T22:42:04.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3MlwXdRnlJw2pQ3yWFpbUiH9IAQ=', 'Content-Length': '0'} 2023-09-29T22:42:04.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b60e2e69e1e39eb-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:42:04.594 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.595 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.595 DEBUG:boto:Method: GET 2023-09-29T22:42:04.595 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:42:04.595 DEBUG:boto:Data: 2023-09-29T22:42:04.595 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.595 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.595 DEBUG:boto:Params: {} 2023-09-29T22:42:04.595 DEBUG:boto:Token: None 2023-09-29T22:42:04.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11 2023-09-29T22:42:04.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kYxeiWfONmfB/mJsDmM/S7zqluM= 2023-09-29T22:42:04.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kYxeiWfONmfB/mJsDmM/S7zqluM=', 'Content-Length': '0'} 2023-09-29T22:42:04.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006719124fa1a2bf4-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.646 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:42:04.646 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:42:04.647 DEBUG:boto:Method: GET 2023-09-29T22:42:04.647 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:42:04.647 DEBUG:boto:Data: 2023-09-29T22:42:04.647 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.647 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.647 DEBUG:boto:Params: {} 2023-09-29T22:42:04.648 DEBUG:boto:Token: None 2023-09-29T22:42:04.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj11 2023-09-29T22:42:04.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kYxeiWfONmfB/mJsDmM/S7zqluM= 2023-09-29T22:42:04.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kYxeiWfONmfB/mJsDmM/S7zqluM=', 'Content-Length': '0'} 2023-09-29T22:42:04.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054ae90a9ce7e2c99-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.652 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.652 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.652 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.652 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.652 DEBUG:boto:Method: GET 2023-09-29T22:42:04.652 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.652 DEBUG:boto:Data: 2023-09-29T22:42:04.653 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.653 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.653 DEBUG:boto:Params: {} 2023-09-29T22:42:04.653 DEBUG:boto:Token: None 2023-09-29T22:42:04.653 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde8aea596a49009-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.658 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.659 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.659 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.659 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.659 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.659 DEBUG:boto:Method: GET 2023-09-29T22:42:04.659 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:42:04.660 DEBUG:boto:Data: 2023-09-29T22:42:04.660 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.660 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.660 DEBUG:boto:Params: {} 2023-09-29T22:42:04.660 DEBUG:boto:Token: None 2023-09-29T22:42:04.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de61be25cc91dd94-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.666 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:42:04.667 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.667 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.667 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.668 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.668 DEBUG:boto:Method: GET 2023-09-29T22:42:04.668 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.668 DEBUG:boto:Data: 2023-09-29T22:42:04.668 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.669 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.669 DEBUG:boto:Params: {} 2023-09-29T22:42:04.669 DEBUG:boto:Token: None 2023-09-29T22:42:04.669 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ= 2023-09-29T22:42:04.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ=', 'Content-Length': '0'} 2023-09-29T22:42:04.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c514d696a1f1458-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.673 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.673 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.674 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.674 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.674 DEBUG:boto:Method: GET 2023-09-29T22:42:04.674 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.674 DEBUG:boto:Data: 2023-09-29T22:42:04.674 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.675 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.675 DEBUG:boto:Params: {} 2023-09-29T22:42:04.675 DEBUG:boto:Token: None 2023-09-29T22:42:04.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ= 2023-09-29T22:42:04.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ=', 'Content-Length': '0'} 2023-09-29T22:42:04.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d4a3f0535b7d0d9-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.679 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.679 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.679 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.680 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.680 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.680 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.680 DEBUG:boto:Data: 2023-09-29T22:42:04.680 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.681 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.681 DEBUG:boto:Params: {} 2023-09-29T22:42:04.681 DEBUG:boto:Token: None 2023-09-29T22:42:04.681 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.682 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5ZhsXGpQvnIFg4XEf+ELvixa3/g= 2023-09-29T22:42:04.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5ZhsXGpQvnIFg4XEf+ELvixa3/g=', 'Content-Length': '0'} 2023-09-29T22:42:04.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a02dece5cd829c54-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.685 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.686 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.686 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.686 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.687 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.687 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.687 DEBUG:boto:Data: 2023-09-29T22:42:04.687 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.687 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.688 DEBUG:boto:Params: {} 2023-09-29T22:42:04.688 DEBUG:boto:Token: None 2023-09-29T22:42:04.688 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5ZhsXGpQvnIFg4XEf+ELvixa3/g= 2023-09-29T22:42:04.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5ZhsXGpQvnIFg4XEf+ELvixa3/g=', 'Content-Length': '0'} 2023-09-29T22:42:04.691 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb48a98b51c738b6-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:42:04.691 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.692 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.692 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.692 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.692 DEBUG:boto:Method: GET 2023-09-29T22:42:04.692 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.692 DEBUG:boto:Data: 2023-09-29T22:42:04.693 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.693 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.693 DEBUG:boto:Params: {} 2023-09-29T22:42:04.693 DEBUG:boto:Token: None 2023-09-29T22:42:04.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ= 2023-09-29T22:42:04.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ=', 'Content-Length': '0'} 2023-09-29T22:42:04.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003035e1de022898eb-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.696 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.697 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.697 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.697 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.697 DEBUG:boto:Method: GET 2023-09-29T22:42:04.697 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.697 DEBUG:boto:Data: 2023-09-29T22:42:04.697 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.698 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.698 DEBUG:boto:Params: {} 2023-09-29T22:42:04.698 DEBUG:boto:Token: None 2023-09-29T22:42:04.698 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:42:04.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ= 2023-09-29T22:42:04.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b4GQDqeQbSEMpaesJca5HT7fdSQ=', 'Content-Length': '0'} 2023-09-29T22:42:04.701 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c0e3077fee81b2c-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.743 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.743 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.743 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.743 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.743 DEBUG:boto:Method: GET 2023-09-29T22:42:04.744 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.744 DEBUG:boto:Data: 2023-09-29T22:42:04.744 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.744 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.744 DEBUG:boto:Params: {} 2023-09-29T22:42:04.744 DEBUG:boto:Token: None 2023-09-29T22:42:04.745 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006358d0edaa979b4c-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.751 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.752 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.752 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.752 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.752 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.753 DEBUG:boto:Method: GET 2023-09-29T22:42:04.753 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.753 DEBUG:boto:Data: 2023-09-29T22:42:04.753 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.753 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.753 DEBUG:boto:Params: {} 2023-09-29T22:42:04.754 DEBUG:boto:Token: None 2023-09-29T22:42:04.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.754 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.760 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006311c9823d437370-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.760 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.761 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.761 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.761 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.762 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:42:04.762 DEBUG:boto:Data: 2023-09-29T22:42:04.762 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.763 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.763 DEBUG:boto:Params: {} 2023-09-29T22:42:04.763 DEBUG:boto:Token: None 2023-09-29T22:42:04.764 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17 2023-09-29T22:42:04.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jn5Fs7x4SkUZX90vdd2MC7QBHMA= 2023-09-29T22:42:04.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jn5Fs7x4SkUZX90vdd2MC7QBHMA=', 'Content-Length': '0'} 2023-09-29T22:42:04.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5c41fce66faa4d3-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.767 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.767 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.767 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.767 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:42:04.767 DEBUG:boto:Data: 2023-09-29T22:42:04.768 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.768 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.768 DEBUG:boto:Params: {} 2023-09-29T22:42:04.768 DEBUG:boto:Token: None 2023-09-29T22:42:04.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17 2023-09-29T22:42:04.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jn5Fs7x4SkUZX90vdd2MC7QBHMA= 2023-09-29T22:42:04.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jn5Fs7x4SkUZX90vdd2MC7QBHMA=', 'Content-Length': '0'} 2023-09-29T22:42:04.771 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000113d1dd041eca31a-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:42:04.771 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.772 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.772 DEBUG:boto:Method: GET 2023-09-29T22:42:04.772 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:42:04.773 DEBUG:boto:Data: 2023-09-29T22:42:04.773 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.773 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.773 DEBUG:boto:Params: {} 2023-09-29T22:42:04.774 DEBUG:boto:Token: None 2023-09-29T22:42:04.774 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17 2023-09-29T22:42:04.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wvSv4Wvdg/DhvvVh6WCEmidm/FU= 2023-09-29T22:42:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wvSv4Wvdg/DhvvVh6WCEmidm/FU=', 'Content-Length': '0'} 2023-09-29T22:42:04.777 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000af62e36a4cf8c25-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.822 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:42:04.822 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:42:04.823 DEBUG:boto:Method: GET 2023-09-29T22:42:04.823 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:42:04.823 DEBUG:boto:Data: 2023-09-29T22:42:04.823 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.823 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.823 DEBUG:boto:Params: {} 2023-09-29T22:42:04.823 DEBUG:boto:Token: None 2023-09-29T22:42:04.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj17 2023-09-29T22:42:04.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wvSv4Wvdg/DhvvVh6WCEmidm/FU= 2023-09-29T22:42:04.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wvSv4Wvdg/DhvvVh6WCEmidm/FU=', 'Content-Length': '0'} 2023-09-29T22:42:04.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6345513d2ac49ce-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.827 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.827 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.828 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.828 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.828 DEBUG:boto:Method: GET 2023-09-29T22:42:04.828 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.828 DEBUG:boto:Data: 2023-09-29T22:42:04.828 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.828 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.828 DEBUG:boto:Params: {} 2023-09-29T22:42:04.828 DEBUG:boto:Token: None 2023-09-29T22:42:04.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.834 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b70277912a05c7b-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.834 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.834 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.834 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.834 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.835 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.835 DEBUG:boto:Method: GET 2023-09-29T22:42:04.835 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:42:04.835 DEBUG:boto:Data: 2023-09-29T22:42:04.835 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.835 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.835 DEBUG:boto:Params: {} 2023-09-29T22:42:04.835 DEBUG:boto:Token: None 2023-09-29T22:42:04.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.841 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba6a7f21e1d9f23b-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.882 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:42:04.883 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:04.884 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:04.884 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.884 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.884 DEBUG:boto:Method: GET 2023-09-29T22:42:04.884 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.884 DEBUG:boto:Data: 2023-09-29T22:42:04.885 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.885 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.885 DEBUG:boto:Params: {} 2023-09-29T22:42:04.885 DEBUG:boto:Token: None 2023-09-29T22:42:04.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY= 2023-09-29T22:42:04.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY=', 'Content-Length': '0'} 2023-09-29T22:42:04.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf228900ec19ee88-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.889 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:04.889 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:04.889 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.889 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.890 DEBUG:boto:Method: GET 2023-09-29T22:42:04.890 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.890 DEBUG:boto:Data: 2023-09-29T22:42:04.890 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.890 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.890 DEBUG:boto:Params: {} 2023-09-29T22:42:04.891 DEBUG:boto:Token: None 2023-09-29T22:42:04.891 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.891 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY= 2023-09-29T22:42:04.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY=', 'Content-Length': '0'} 2023-09-29T22:42:04.894 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3c5e1cd19c1e16f-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.894 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:04.895 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:04.895 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.895 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.895 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.895 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.895 DEBUG:boto:Data: 2023-09-29T22:42:04.895 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.895 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.896 DEBUG:boto:Params: {} 2023-09-29T22:42:04.896 DEBUG:boto:Token: None 2023-09-29T22:42:04.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a8N0YPXzx9u/1WSWmFqOsC//Npw= 2023-09-29T22:42:04.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a8N0YPXzx9u/1WSWmFqOsC//Npw=', 'Content-Length': '0'} 2023-09-29T22:42:04.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5308dc718998dbf-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.899 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:04.899 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:04.899 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.899 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.899 DEBUG:boto:Method: HEAD 2023-09-29T22:42:04.900 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.900 DEBUG:boto:Data: 2023-09-29T22:42:04.900 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.900 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.900 DEBUG:boto:Params: {} 2023-09-29T22:42:04.900 DEBUG:boto:Token: None 2023-09-29T22:42:04.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a8N0YPXzx9u/1WSWmFqOsC//Npw= 2023-09-29T22:42:04.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a8N0YPXzx9u/1WSWmFqOsC//Npw=', 'Content-Length': '0'} 2023-09-29T22:42:04.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000169d194a6d5ac7e1-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:42:04.904 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:04.904 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:04.904 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.904 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.904 DEBUG:boto:Method: GET 2023-09-29T22:42:04.904 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.904 DEBUG:boto:Data: 2023-09-29T22:42:04.904 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.905 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.905 DEBUG:boto:Params: {} 2023-09-29T22:42:04.905 DEBUG:boto:Token: None 2023-09-29T22:42:04.905 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY= 2023-09-29T22:42:04.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY=', 'Content-Length': '0'} 2023-09-29T22:42:04.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e98261d937c6da5-00651752bc-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.954 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:04.955 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:04.955 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.955 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.955 DEBUG:boto:Method: GET 2023-09-29T22:42:04.955 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.955 DEBUG:boto:Data: 2023-09-29T22:42:04.955 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.956 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.956 DEBUG:boto:Params: {} 2023-09-29T22:42:04.956 DEBUG:boto:Token: None 2023-09-29T22:42:04.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:42:04.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY= 2023-09-29T22:42:04.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xvx+wrXOmm0varyGPJfXeL4ikwY=', 'Content-Length': '0'} 2023-09-29T22:42:04.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060433eb9bc8ab91a-00651752bc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.960 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.960 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.960 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:04.960 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:04.961 DEBUG:boto:Method: GET 2023-09-29T22:42:04.961 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:42:04.961 DEBUG:boto:Data: 2023-09-29T22:42:04.961 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:04.961 DEBUG:boto:Port: 8001 2023-09-29T22:42:04.961 DEBUG:boto:Params: {} 2023-09-29T22:42:04.961 DEBUG:boto:Token: None 2023-09-29T22:42:04.962 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.962 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7093b97622d2d5a-00651752bc-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:04.967 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:04.967 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:04.967 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:04.968 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:04.968 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:04.968 DEBUG:boto:Method: GET 2023-09-29T22:42:04.968 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:42:04.968 DEBUG:boto:Data: 2023-09-29T22:42:04.968 DEBUG:boto:Headers: {} 2023-09-29T22:42:04.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:04.969 DEBUG:boto:Port: 8000 2023-09-29T22:42:04.969 DEBUG:boto:Params: {} 2023-09-29T22:42:04.969 DEBUG:boto:Token: None 2023-09-29T22:42:04.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:04 GMT /hcnphz-59/ 2023-09-29T22:42:04.970 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo= 2023-09-29T22:42:04.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9r85Oa6BBAVn8bUSR9H3h+BoXyo=', 'Content-Length': '0'} 2023-09-29T22:42:04.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ac79d7c034baf9d-00651752bc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.018 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.019 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:05.019 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:05.019 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.019 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:42:05.019 DEBUG:boto:Data: 2023-09-29T22:42:05.019 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.020 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.020 DEBUG:boto:Params: {} 2023-09-29T22:42:05.020 DEBUG:boto:Token: None 2023-09-29T22:42:05.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj19 2023-09-29T22:42:05.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kg7nfgFqEOg2QYmfKeOJEFg4SaA= 2023-09-29T22:42:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kg7nfgFqEOg2QYmfKeOJEFg4SaA=', 'Content-Length': '0'} 2023-09-29T22:42:05.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f9c18f583e681c7-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.023 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:05.023 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:05.023 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.024 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:42:05.024 DEBUG:boto:Data: 2023-09-29T22:42:05.024 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.024 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.024 DEBUG:boto:Params: {} 2023-09-29T22:42:05.024 DEBUG:boto:Token: None 2023-09-29T22:42:05.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj19 2023-09-29T22:42:05.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kg7nfgFqEOg2QYmfKeOJEFg4SaA= 2023-09-29T22:42:05.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kg7nfgFqEOg2QYmfKeOJEFg4SaA=', 'Content-Length': '0'} 2023-09-29T22:42:05.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a27bed2f62345070-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:42:05.028 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:05.028 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:05.028 DEBUG:boto:Method: GET 2023-09-29T22:42:05.028 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:42:05.028 DEBUG:boto:Data: 2023-09-29T22:42:05.028 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.029 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.029 DEBUG:boto:Params: {} 2023-09-29T22:42:05.029 DEBUG:boto:Token: None 2023-09-29T22:42:05.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj19 2023-09-29T22:42:05.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vTrU8ECtozMbz9qy4hJLR1xBmi4= 2023-09-29T22:42:05.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vTrU8ECtozMbz9qy4hJLR1xBmi4=', 'Content-Length': '0'} 2023-09-29T22:42:05.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d3289efcd3c50e-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.034 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:42:05.034 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:42:05.034 DEBUG:boto:Method: GET 2023-09-29T22:42:05.034 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:42:05.034 DEBUG:boto:Data: 2023-09-29T22:42:05.034 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.035 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.035 DEBUG:boto:Params: {} 2023-09-29T22:42:05.035 DEBUG:boto:Token: None 2023-09-29T22:42:05.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj19 2023-09-29T22:42:05.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vTrU8ECtozMbz9qy4hJLR1xBmi4= 2023-09-29T22:42:05.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vTrU8ECtozMbz9qy4hJLR1xBmi4=', 'Content-Length': '0'} 2023-09-29T22:42:05.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008947efed7e943b82-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.039 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.039 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.039 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:05.039 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:05.039 DEBUG:boto:Method: GET 2023-09-29T22:42:05.039 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:42:05.039 DEBUG:boto:Data: 2023-09-29T22:42:05.039 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.039 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.040 DEBUG:boto:Params: {} 2023-09-29T22:42:05.040 DEBUG:boto:Token: None 2023-09-29T22:42:05.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094af08d60c2073e9-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.045 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.046 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.046 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.046 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:05.046 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:42:05.047 DEBUG:boto:Method: GET 2023-09-29T22:42:05.047 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:42:05.047 DEBUG:boto:Data: 2023-09-29T22:42:05.047 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.048 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.048 DEBUG:boto:Params: {} 2023-09-29T22:42:05.048 DEBUG:boto:Token: None 2023-09-29T22:42:05.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d911a7aa47ea25ee-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.094 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:42:05.095 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.095 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.095 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.095 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.095 DEBUG:boto:Method: GET 2023-09-29T22:42:05.096 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.096 DEBUG:boto:Data: 2023-09-29T22:42:05.096 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.096 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.096 DEBUG:boto:Params: {} 2023-09-29T22:42:05.096 DEBUG:boto:Token: None 2023-09-29T22:42:05.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ= 2023-09-29T22:42:05.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ=', 'Content-Length': '0'} 2023-09-29T22:42:05.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c46fb6d7f2f1b66-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.100 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.100 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.100 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.100 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.100 DEBUG:boto:Method: GET 2023-09-29T22:42:05.100 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.100 DEBUG:boto:Data: 2023-09-29T22:42:05.100 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.101 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.101 DEBUG:boto:Params: {} 2023-09-29T22:42:05.101 DEBUG:boto:Token: None 2023-09-29T22:42:05.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.101 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ= 2023-09-29T22:42:05.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ=', 'Content-Length': '0'} 2023-09-29T22:42:05.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fff80c73ec31bb4-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.104 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.105 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.105 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.105 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.105 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.105 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.105 DEBUG:boto:Data: 2023-09-29T22:42:05.105 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.105 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.106 DEBUG:boto:Params: {} 2023-09-29T22:42:05.106 DEBUG:boto:Token: None 2023-09-29T22:42:05.106 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.106 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M9XkMK8IneGNwYTCxTQTy7fE+9E= 2023-09-29T22:42:05.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M9XkMK8IneGNwYTCxTQTy7fE+9E=', 'Content-Length': '0'} 2023-09-29T22:42:05.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001296ebf21a5d5eab-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.110 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.110 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.110 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.110 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.110 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.110 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.111 DEBUG:boto:Data: 2023-09-29T22:42:05.111 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.111 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.111 DEBUG:boto:Params: {} 2023-09-29T22:42:05.111 DEBUG:boto:Token: None 2023-09-29T22:42:05.111 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:M9XkMK8IneGNwYTCxTQTy7fE+9E= 2023-09-29T22:42:05.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M9XkMK8IneGNwYTCxTQTy7fE+9E=', 'Content-Length': '0'} 2023-09-29T22:42:05.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c6e75d4be3c7a92-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:42:05.116 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.116 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.117 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.117 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.117 DEBUG:boto:Method: GET 2023-09-29T22:42:05.117 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.117 DEBUG:boto:Data: 2023-09-29T22:42:05.117 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.118 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.118 DEBUG:boto:Params: {} 2023-09-29T22:42:05.118 DEBUG:boto:Token: None 2023-09-29T22:42:05.118 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.118 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ= 2023-09-29T22:42:05.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ=', 'Content-Length': '0'} 2023-09-29T22:42:05.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ccf70395a663919-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.162 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.163 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.163 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.163 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.163 DEBUG:boto:Method: GET 2023-09-29T22:42:05.163 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.164 DEBUG:boto:Data: 2023-09-29T22:42:05.164 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.164 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.164 DEBUG:boto:Params: {} 2023-09-29T22:42:05.164 DEBUG:boto:Token: None 2023-09-29T22:42:05.165 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:42:05.165 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ= 2023-09-29T22:42:05.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3l4RDrw0LOjdxZsgCdo1ANqpgRQ=', 'Content-Length': '0'} 2023-09-29T22:42:05.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869f07d7cafde06b-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.168 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.169 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.169 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.169 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.169 DEBUG:boto:Method: GET 2023-09-29T22:42:05.169 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.170 DEBUG:boto:Data: 2023-09-29T22:42:05.170 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.170 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.171 DEBUG:boto:Params: {} 2023-09-29T22:42:05.171 DEBUG:boto:Token: None 2023-09-29T22:42:05.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ac8bf9c38153855-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.177 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.177 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.178 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.178 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.178 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.178 DEBUG:boto:Method: GET 2023-09-29T22:42:05.179 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.179 DEBUG:boto:Data: 2023-09-29T22:42:05.179 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.179 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.180 DEBUG:boto:Params: {} 2023-09-29T22:42:05.180 DEBUG:boto:Token: None 2023-09-29T22:42:05.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.187 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f9d3fbdbb12c33fd-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.230 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.231 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.231 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.231 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.232 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:42:05.232 DEBUG:boto:Data: 2023-09-29T22:42:05.232 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.232 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.232 DEBUG:boto:Params: {} 2023-09-29T22:42:05.233 DEBUG:boto:Token: None 2023-09-29T22:42:05.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22 2023-09-29T22:42:05.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CkVNnvesdW7FJdcpQ/9XIZOCo1Q= 2023-09-29T22:42:05.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CkVNnvesdW7FJdcpQ/9XIZOCo1Q=', 'Content-Length': '0'} 2023-09-29T22:42:05.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000978f4a17a25819f1-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.237 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.237 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.238 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.238 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:42:05.238 DEBUG:boto:Data: 2023-09-29T22:42:05.238 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.238 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.239 DEBUG:boto:Params: {} 2023-09-29T22:42:05.239 DEBUG:boto:Token: None 2023-09-29T22:42:05.239 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22 2023-09-29T22:42:05.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CkVNnvesdW7FJdcpQ/9XIZOCo1Q= 2023-09-29T22:42:05.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CkVNnvesdW7FJdcpQ/9XIZOCo1Q=', 'Content-Length': '0'} 2023-09-29T22:42:05.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000384d29b1c2a4e080-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:42:05.243 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.243 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.243 DEBUG:boto:Method: GET 2023-09-29T22:42:05.243 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:42:05.243 DEBUG:boto:Data: 2023-09-29T22:42:05.243 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.244 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.244 DEBUG:boto:Params: {} 2023-09-29T22:42:05.244 DEBUG:boto:Token: None 2023-09-29T22:42:05.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22 2023-09-29T22:42:05.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KLE79WvzsefeJUifuelEQogzTmc= 2023-09-29T22:42:05.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KLE79WvzsefeJUifuelEQogzTmc=', 'Content-Length': '0'} 2023-09-29T22:42:05.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001da665ccb831b452-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.248 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:42:05.248 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:42:05.248 DEBUG:boto:Method: GET 2023-09-29T22:42:05.248 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:42:05.248 DEBUG:boto:Data: 2023-09-29T22:42:05.249 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.249 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.249 DEBUG:boto:Params: {} 2023-09-29T22:42:05.249 DEBUG:boto:Token: None 2023-09-29T22:42:05.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj22 2023-09-29T22:42:05.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KLE79WvzsefeJUifuelEQogzTmc= 2023-09-29T22:42:05.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KLE79WvzsefeJUifuelEQogzTmc=', 'Content-Length': '0'} 2023-09-29T22:42:05.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000580a8ad55818c31e-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.253 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.254 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.254 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.254 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.254 DEBUG:boto:Method: GET 2023-09-29T22:42:05.254 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.254 DEBUG:boto:Data: 2023-09-29T22:42:05.254 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.255 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.255 DEBUG:boto:Params: {} 2023-09-29T22:42:05.256 DEBUG:boto:Token: None 2023-09-29T22:42:05.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.261 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000586e734214e97861-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.302 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.303 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.303 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.303 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.303 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.304 DEBUG:boto:Method: GET 2023-09-29T22:42:05.304 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:42:05.304 DEBUG:boto:Data: 2023-09-29T22:42:05.304 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.304 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.305 DEBUG:boto:Params: {} 2023-09-29T22:42:05.305 DEBUG:boto:Token: None 2023-09-29T22:42:05.305 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.305 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.314 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef18e22e58f2f454-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.314 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:42:05.317 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.318 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.318 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.318 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.318 DEBUG:boto:Method: GET 2023-09-29T22:42:05.319 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.319 DEBUG:boto:Data: 2023-09-29T22:42:05.319 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.319 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.320 DEBUG:boto:Params: {} 2023-09-29T22:42:05.320 DEBUG:boto:Token: None 2023-09-29T22:42:05.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k= 2023-09-29T22:42:05.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k=', 'Content-Length': '0'} 2023-09-29T22:42:05.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc02563b0acfec17-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.323 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.324 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.324 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.324 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.324 DEBUG:boto:Method: GET 2023-09-29T22:42:05.324 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.324 DEBUG:boto:Data: 2023-09-29T22:42:05.325 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.325 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.325 DEBUG:boto:Params: {} 2023-09-29T22:42:05.325 DEBUG:boto:Token: None 2023-09-29T22:42:05.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.326 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k= 2023-09-29T22:42:05.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k=', 'Content-Length': '0'} 2023-09-29T22:42:05.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46db7862ca35495-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.328 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.329 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.329 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.329 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.329 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.329 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.329 DEBUG:boto:Data: 2023-09-29T22:42:05.330 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.330 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.330 DEBUG:boto:Params: {} 2023-09-29T22:42:05.330 DEBUG:boto:Token: None 2023-09-29T22:42:05.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.331 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XUw11b8S7FOz7MjYJ05ttaIddtg= 2023-09-29T22:42:05.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XUw11b8S7FOz7MjYJ05ttaIddtg=', 'Content-Length': '0'} 2023-09-29T22:42:05.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bc20076ac780e0-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.333 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.334 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.334 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.334 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.334 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.334 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.335 DEBUG:boto:Data: 2023-09-29T22:42:05.335 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.335 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.335 DEBUG:boto:Params: {} 2023-09-29T22:42:05.336 DEBUG:boto:Token: None 2023-09-29T22:42:05.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XUw11b8S7FOz7MjYJ05ttaIddtg= 2023-09-29T22:42:05.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XUw11b8S7FOz7MjYJ05ttaIddtg=', 'Content-Length': '0'} 2023-09-29T22:42:05.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6d9fe0b6807212a-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:42:05.339 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.339 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.340 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.340 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.340 DEBUG:boto:Method: GET 2023-09-29T22:42:05.340 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.340 DEBUG:boto:Data: 2023-09-29T22:42:05.340 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.341 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.341 DEBUG:boto:Params: {} 2023-09-29T22:42:05.341 DEBUG:boto:Token: None 2023-09-29T22:42:05.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k= 2023-09-29T22:42:05.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k=', 'Content-Length': '0'} 2023-09-29T22:42:05.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a6c9646944ad6d0-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.346 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.346 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.346 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.346 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.347 DEBUG:boto:Method: GET 2023-09-29T22:42:05.347 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.347 DEBUG:boto:Data: 2023-09-29T22:42:05.347 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.347 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.348 DEBUG:boto:Params: {} 2023-09-29T22:42:05.348 DEBUG:boto:Token: None 2023-09-29T22:42:05.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:42:05.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k= 2023-09-29T22:42:05.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+2qCiPzpEQY9Lxw9qpb9D2Z1Z1k=', 'Content-Length': '0'} 2023-09-29T22:42:05.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f36a10d9234bb58a-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.394 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.395 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.395 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.395 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.395 DEBUG:boto:Method: GET 2023-09-29T22:42:05.395 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.395 DEBUG:boto:Data: 2023-09-29T22:42:05.395 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.396 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.396 DEBUG:boto:Params: {} 2023-09-29T22:42:05.396 DEBUG:boto:Token: None 2023-09-29T22:42:05.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.402 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000536c42988360a5bf-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.402 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.403 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.403 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.403 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.403 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.404 DEBUG:boto:Method: GET 2023-09-29T22:42:05.404 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.404 DEBUG:boto:Data: 2023-09-29T22:42:05.404 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.404 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.405 DEBUG:boto:Params: {} 2023-09-29T22:42:05.405 DEBUG:boto:Token: None 2023-09-29T22:42:05.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.409 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee06a1d232bd02fd-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.410 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.410 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.410 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.411 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.411 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:42:05.411 DEBUG:boto:Data: 2023-09-29T22:42:05.411 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.412 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.412 DEBUG:boto:Params: {} 2023-09-29T22:42:05.413 DEBUG:boto:Token: None 2023-09-29T22:42:05.413 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33 2023-09-29T22:42:05.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u/+/uF9G0DWu5cye6LsHb5KYJCM= 2023-09-29T22:42:05.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u/+/uF9G0DWu5cye6LsHb5KYJCM=', 'Content-Length': '0'} 2023-09-29T22:42:05.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2270652578783cf-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.416 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.416 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.416 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.416 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:42:05.416 DEBUG:boto:Data: 2023-09-29T22:42:05.417 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.417 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.417 DEBUG:boto:Params: {} 2023-09-29T22:42:05.417 DEBUG:boto:Token: None 2023-09-29T22:42:05.418 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33 2023-09-29T22:42:05.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u/+/uF9G0DWu5cye6LsHb5KYJCM= 2023-09-29T22:42:05.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u/+/uF9G0DWu5cye6LsHb5KYJCM=', 'Content-Length': '0'} 2023-09-29T22:42:05.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b62dc75c165e2ea-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:42:05.421 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.421 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.421 DEBUG:boto:Method: GET 2023-09-29T22:42:05.422 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:42:05.422 DEBUG:boto:Data: 2023-09-29T22:42:05.422 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.423 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.423 DEBUG:boto:Params: {} 2023-09-29T22:42:05.423 DEBUG:boto:Token: None 2023-09-29T22:42:05.423 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33 2023-09-29T22:42:05.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jYRJxgpI+UthcHYT8TMDjOnnzAE= 2023-09-29T22:42:05.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jYRJxgpI+UthcHYT8TMDjOnnzAE=', 'Content-Length': '0'} 2023-09-29T22:42:05.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3b5705e4af59c57-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.470 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:42:05.471 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:42:05.471 DEBUG:boto:Method: GET 2023-09-29T22:42:05.471 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:42:05.471 DEBUG:boto:Data: 2023-09-29T22:42:05.471 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.471 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.471 DEBUG:boto:Params: {} 2023-09-29T22:42:05.472 DEBUG:boto:Token: None 2023-09-29T22:42:05.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj33 2023-09-29T22:42:05.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jYRJxgpI+UthcHYT8TMDjOnnzAE= 2023-09-29T22:42:05.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jYRJxgpI+UthcHYT8TMDjOnnzAE=', 'Content-Length': '0'} 2023-09-29T22:42:05.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4cb4c424acddff4-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.475 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.475 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.476 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.476 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.476 DEBUG:boto:Method: GET 2023-09-29T22:42:05.476 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.476 DEBUG:boto:Data: 2023-09-29T22:42:05.476 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.476 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.476 DEBUG:boto:Params: {} 2023-09-29T22:42:05.477 DEBUG:boto:Token: None 2023-09-29T22:42:05.477 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b417d0d43ec1e0d8-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.482 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.482 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.482 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.482 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.483 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.483 DEBUG:boto:Method: GET 2023-09-29T22:42:05.483 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:42:05.483 DEBUG:boto:Data: 2023-09-29T22:42:05.483 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.483 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.483 DEBUG:boto:Params: {} 2023-09-29T22:42:05.483 DEBUG:boto:Token: None 2023-09-29T22:42:05.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000021d426cf572fcad4-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.534 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:42:05.535 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.535 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.535 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.535 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.536 DEBUG:boto:Method: GET 2023-09-29T22:42:05.536 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.536 DEBUG:boto:Data: 2023-09-29T22:42:05.536 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.536 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.536 DEBUG:boto:Params: {} 2023-09-29T22:42:05.536 DEBUG:boto:Token: None 2023-09-29T22:42:05.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4= 2023-09-29T22:42:05.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4=', 'Content-Length': '0'} 2023-09-29T22:42:05.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b536e15bf731ffa9-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.540 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.540 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.540 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.541 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.541 DEBUG:boto:Method: GET 2023-09-29T22:42:05.541 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.541 DEBUG:boto:Data: 2023-09-29T22:42:05.541 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.541 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.541 DEBUG:boto:Params: {} 2023-09-29T22:42:05.541 DEBUG:boto:Token: None 2023-09-29T22:42:05.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4= 2023-09-29T22:42:05.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4=', 'Content-Length': '0'} 2023-09-29T22:42:05.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2e8956d1bb5b364-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.545 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.546 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.546 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.546 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.546 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.546 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.546 DEBUG:boto:Data: 2023-09-29T22:42:05.546 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.546 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.547 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.547 DEBUG:boto:Params: {} 2023-09-29T22:42:05.547 DEBUG:boto:Token: None 2023-09-29T22:42:05.547 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.547 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkQtDyZ8jD09BFUP+mSv2IKGp80= 2023-09-29T22:42:05.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkQtDyZ8jD09BFUP+mSv2IKGp80=', 'Content-Length': '0'} 2023-09-29T22:42:05.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002c62a74b50227ab-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.551 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.551 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.551 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.551 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.552 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.552 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.552 DEBUG:boto:Data: 2023-09-29T22:42:05.552 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.552 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.552 DEBUG:boto:Params: {} 2023-09-29T22:42:05.552 DEBUG:boto:Token: None 2023-09-29T22:42:05.552 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkQtDyZ8jD09BFUP+mSv2IKGp80= 2023-09-29T22:42:05.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkQtDyZ8jD09BFUP+mSv2IKGp80=', 'Content-Length': '0'} 2023-09-29T22:42:05.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eef7d237e43b824-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:42:05.556 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.556 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.556 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.556 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.556 DEBUG:boto:Method: GET 2023-09-29T22:42:05.557 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.557 DEBUG:boto:Data: 2023-09-29T22:42:05.557 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.557 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.557 DEBUG:boto:Params: {} 2023-09-29T22:42:05.557 DEBUG:boto:Token: None 2023-09-29T22:42:05.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4= 2023-09-29T22:42:05.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4=', 'Content-Length': '0'} 2023-09-29T22:42:05.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e66d4af68b07d1-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.602 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.603 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.603 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.603 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.603 DEBUG:boto:Method: GET 2023-09-29T22:42:05.603 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.603 DEBUG:boto:Data: 2023-09-29T22:42:05.603 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.603 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.604 DEBUG:boto:Params: {} 2023-09-29T22:42:05.604 DEBUG:boto:Token: None 2023-09-29T22:42:05.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:42:05.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4= 2023-09-29T22:42:05.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RwjCbiksbvnqH0Eg93OkOvtLYT4=', 'Content-Length': '0'} 2023-09-29T22:42:05.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f4307383617307-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.608 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.608 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.608 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.608 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.608 DEBUG:boto:Method: GET 2023-09-29T22:42:05.609 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.609 DEBUG:boto:Data: 2023-09-29T22:42:05.609 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.609 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.610 DEBUG:boto:Params: {} 2023-09-29T22:42:05.610 DEBUG:boto:Token: None 2023-09-29T22:42:05.610 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.616 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0429c7ca958472f-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.617 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.617 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.617 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.617 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.618 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.618 DEBUG:boto:Method: GET 2023-09-29T22:42:05.618 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.618 DEBUG:boto:Data: 2023-09-29T22:42:05.618 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.618 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.618 DEBUG:boto:Params: {} 2023-09-29T22:42:05.619 DEBUG:boto:Token: None 2023-09-29T22:42:05.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.624 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca110a2b2124512f-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.625 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.625 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.625 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.625 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.626 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:42:05.626 DEBUG:boto:Data: 2023-09-29T22:42:05.626 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.626 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.626 DEBUG:boto:Params: {} 2023-09-29T22:42:05.626 DEBUG:boto:Token: None 2023-09-29T22:42:05.626 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34 2023-09-29T22:42:05.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8v44Ycj7whoTlnvA1fRsnr7S0/8= 2023-09-29T22:42:05.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8v44Ycj7whoTlnvA1fRsnr7S0/8=', 'Content-Length': '0'} 2023-09-29T22:42:05.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f867aa595a26db4e-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.629 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.630 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.630 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.630 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:42:05.630 DEBUG:boto:Data: 2023-09-29T22:42:05.630 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.630 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.631 DEBUG:boto:Params: {} 2023-09-29T22:42:05.631 DEBUG:boto:Token: None 2023-09-29T22:42:05.631 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34 2023-09-29T22:42:05.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8v44Ycj7whoTlnvA1fRsnr7S0/8= 2023-09-29T22:42:05.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8v44Ycj7whoTlnvA1fRsnr7S0/8=', 'Content-Length': '0'} 2023-09-29T22:42:05.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036cc1097c37b4f71-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:42:05.635 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.635 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.635 DEBUG:boto:Method: GET 2023-09-29T22:42:05.636 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:42:05.636 DEBUG:boto:Data: 2023-09-29T22:42:05.636 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.636 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.636 DEBUG:boto:Params: {} 2023-09-29T22:42:05.636 DEBUG:boto:Token: None 2023-09-29T22:42:05.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34 2023-09-29T22:42:05.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UJtyFFBM7P8yL/s7Vvy5e6u2SjI= 2023-09-29T22:42:05.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UJtyFFBM7P8yL/s7Vvy5e6u2SjI=', 'Content-Length': '0'} 2023-09-29T22:42:05.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc1c3853712a3e5-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.639 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:42:05.639 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:42:05.640 DEBUG:boto:Method: GET 2023-09-29T22:42:05.640 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:42:05.640 DEBUG:boto:Data: 2023-09-29T22:42:05.640 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.640 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.640 DEBUG:boto:Params: {} 2023-09-29T22:42:05.640 DEBUG:boto:Token: None 2023-09-29T22:42:05.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj34 2023-09-29T22:42:05.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UJtyFFBM7P8yL/s7Vvy5e6u2SjI= 2023-09-29T22:42:05.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UJtyFFBM7P8yL/s7Vvy5e6u2SjI=', 'Content-Length': '0'} 2023-09-29T22:42:05.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000489ab4eca74ea2ee-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.686 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.687 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.687 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.687 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.687 DEBUG:boto:Method: GET 2023-09-29T22:42:05.687 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.687 DEBUG:boto:Data: 2023-09-29T22:42:05.687 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.687 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.688 DEBUG:boto:Params: {} 2023-09-29T22:42:05.688 DEBUG:boto:Token: None 2023-09-29T22:42:05.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bb2efce12726871-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.693 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.694 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.694 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.694 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.694 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.695 DEBUG:boto:Method: GET 2023-09-29T22:42:05.695 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:42:05.695 DEBUG:boto:Data: 2023-09-29T22:42:05.695 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.695 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.695 DEBUG:boto:Params: {} 2023-09-29T22:42:05.695 DEBUG:boto:Token: None 2023-09-29T22:42:05.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7c2a68ad98060ec-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.702 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:42:05.702 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.703 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.703 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.703 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.703 DEBUG:boto:Method: GET 2023-09-29T22:42:05.703 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.703 DEBUG:boto:Data: 2023-09-29T22:42:05.704 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.704 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.704 DEBUG:boto:Params: {} 2023-09-29T22:42:05.704 DEBUG:boto:Token: None 2023-09-29T22:42:05.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o= 2023-09-29T22:42:05.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o=', 'Content-Length': '0'} 2023-09-29T22:42:05.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b35f12cd824a9b-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.750 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.751 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.751 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.751 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.751 DEBUG:boto:Method: GET 2023-09-29T22:42:05.752 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.752 DEBUG:boto:Data: 2023-09-29T22:42:05.752 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.752 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.752 DEBUG:boto:Params: {} 2023-09-29T22:42:05.752 DEBUG:boto:Token: None 2023-09-29T22:42:05.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o= 2023-09-29T22:42:05.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o=', 'Content-Length': '0'} 2023-09-29T22:42:05.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0fda36893f30c27-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.757 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.757 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.757 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.757 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.757 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.757 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.757 DEBUG:boto:Data: 2023-09-29T22:42:05.758 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.758 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.758 DEBUG:boto:Params: {} 2023-09-29T22:42:05.758 DEBUG:boto:Token: None 2023-09-29T22:42:05.758 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.758 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CnKF9tbFs/D+0oBQAsQt3zQ7NyY= 2023-09-29T22:42:05.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CnKF9tbFs/D+0oBQAsQt3zQ7NyY=', 'Content-Length': '0'} 2023-09-29T22:42:05.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d262458119fc8293-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.762 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.762 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.762 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.762 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.762 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.762 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.763 DEBUG:boto:Data: 2023-09-29T22:42:05.763 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.763 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.763 DEBUG:boto:Params: {} 2023-09-29T22:42:05.763 DEBUG:boto:Token: None 2023-09-29T22:42:05.763 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.763 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CnKF9tbFs/D+0oBQAsQt3zQ7NyY= 2023-09-29T22:42:05.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CnKF9tbFs/D+0oBQAsQt3zQ7NyY=', 'Content-Length': '0'} 2023-09-29T22:42:05.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e54860ac5082a01-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:42:05.767 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.767 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.767 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.768 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.768 DEBUG:boto:Method: GET 2023-09-29T22:42:05.768 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.768 DEBUG:boto:Data: 2023-09-29T22:42:05.768 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.769 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.769 DEBUG:boto:Params: {} 2023-09-29T22:42:05.769 DEBUG:boto:Token: None 2023-09-29T22:42:05.770 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.770 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o= 2023-09-29T22:42:05.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o=', 'Content-Length': '0'} 2023-09-29T22:42:05.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a39529469d438f9-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.773 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.774 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.774 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.774 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.774 DEBUG:boto:Method: GET 2023-09-29T22:42:05.774 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.775 DEBUG:boto:Data: 2023-09-29T22:42:05.775 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.775 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.775 DEBUG:boto:Params: {} 2023-09-29T22:42:05.775 DEBUG:boto:Token: None 2023-09-29T22:42:05.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:42:05.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o= 2023-09-29T22:42:05.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lE7NvWonwrr7abiTZVyAIYxeJ6o=', 'Content-Length': '0'} 2023-09-29T22:42:05.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d00c127a50f5ae7-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.778 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.779 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.779 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.779 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.779 DEBUG:boto:Method: GET 2023-09-29T22:42:05.779 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.779 DEBUG:boto:Data: 2023-09-29T22:42:05.779 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.780 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.780 DEBUG:boto:Params: {} 2023-09-29T22:42:05.780 DEBUG:boto:Token: None 2023-09-29T22:42:05.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dce6e31780f61e5e-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.784 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.785 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.785 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.785 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.785 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.785 DEBUG:boto:Method: GET 2023-09-29T22:42:05.785 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.786 DEBUG:boto:Data: 2023-09-29T22:42:05.786 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.786 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.786 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.786 DEBUG:boto:Params: {} 2023-09-29T22:42:05.786 DEBUG:boto:Token: None 2023-09-29T22:42:05.786 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.787 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.791 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001130fac08879506d-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.838 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.839 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.839 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.839 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.839 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:42:05.839 DEBUG:boto:Data: 2023-09-29T22:42:05.839 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.840 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.840 DEBUG:boto:Params: {} 2023-09-29T22:42:05.840 DEBUG:boto:Token: None 2023-09-29T22:42:05.840 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38 2023-09-29T22:42:05.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+cqme2z6DGBXnykodi6mOorq5jA= 2023-09-29T22:42:05.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+cqme2z6DGBXnykodi6mOorq5jA=', 'Content-Length': '0'} 2023-09-29T22:42:05.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048202143ca507896-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.844 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.844 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.844 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.845 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:42:05.845 DEBUG:boto:Data: 2023-09-29T22:42:05.845 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.845 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.846 DEBUG:boto:Params: {} 2023-09-29T22:42:05.846 DEBUG:boto:Token: None 2023-09-29T22:42:05.846 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38 2023-09-29T22:42:05.846 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+cqme2z6DGBXnykodi6mOorq5jA= 2023-09-29T22:42:05.846 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+cqme2z6DGBXnykodi6mOorq5jA=', 'Content-Length': '0'} 2023-09-29T22:42:05.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c66df1519e53df8-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:42:05.849 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.849 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.850 DEBUG:boto:Method: GET 2023-09-29T22:42:05.850 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:42:05.850 DEBUG:boto:Data: 2023-09-29T22:42:05.850 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.850 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.850 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.851 DEBUG:boto:Params: {} 2023-09-29T22:42:05.851 DEBUG:boto:Token: None 2023-09-29T22:42:05.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38 2023-09-29T22:42:05.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2boRsu6mpvp3D3GHqSjyRz6l2xY= 2023-09-29T22:42:05.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2boRsu6mpvp3D3GHqSjyRz6l2xY=', 'Content-Length': '0'} 2023-09-29T22:42:05.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8965bbf8c6cf281-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.854 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:42:05.854 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:42:05.854 DEBUG:boto:Method: GET 2023-09-29T22:42:05.855 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:42:05.855 DEBUG:boto:Data: 2023-09-29T22:42:05.855 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.855 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.855 DEBUG:boto:Params: {} 2023-09-29T22:42:05.855 DEBUG:boto:Token: None 2023-09-29T22:42:05.855 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj38 2023-09-29T22:42:05.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2boRsu6mpvp3D3GHqSjyRz6l2xY= 2023-09-29T22:42:05.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2boRsu6mpvp3D3GHqSjyRz6l2xY=', 'Content-Length': '0'} 2023-09-29T22:42:05.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43f1643a2a46435-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.859 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.859 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.859 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.859 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.860 DEBUG:boto:Method: GET 2023-09-29T22:42:05.860 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.860 DEBUG:boto:Data: 2023-09-29T22:42:05.860 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.860 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.860 DEBUG:boto:Params: {} 2023-09-29T22:42:05.860 DEBUG:boto:Token: None 2023-09-29T22:42:05.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039a2ea990724d9a4-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.866 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.867 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.867 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.867 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.867 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.867 DEBUG:boto:Method: GET 2023-09-29T22:42:05.867 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:42:05.867 DEBUG:boto:Data: 2023-09-29T22:42:05.867 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.868 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.868 DEBUG:boto:Params: {} 2023-09-29T22:42:05.868 DEBUG:boto:Token: None 2023-09-29T22:42:05.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:05.874 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000608d7e425083f05b-00651752bd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.874 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:05.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:42:05.874 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:05.875 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:05.875 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.875 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.875 DEBUG:boto:Method: GET 2023-09-29T22:42:05.875 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.875 DEBUG:boto:Data: 2023-09-29T22:42:05.875 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.875 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.876 DEBUG:boto:Params: {} 2023-09-29T22:42:05.876 DEBUG:boto:Token: None 2023-09-29T22:42:05.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I= 2023-09-29T22:42:05.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I=', 'Content-Length': '0'} 2023-09-29T22:42:05.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015a417d8b9a34793-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.922 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:05.922 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:05.923 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.923 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.923 DEBUG:boto:Method: GET 2023-09-29T22:42:05.923 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.923 DEBUG:boto:Data: 2023-09-29T22:42:05.923 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.924 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.924 DEBUG:boto:Params: {} 2023-09-29T22:42:05.924 DEBUG:boto:Token: None 2023-09-29T22:42:05.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I= 2023-09-29T22:42:05.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I=', 'Content-Length': '0'} 2023-09-29T22:42:05.927 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158880b7135331a2-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.928 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:05.928 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:05.928 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.928 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.928 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.928 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.928 DEBUG:boto:Data: 2023-09-29T22:42:05.928 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.929 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.929 DEBUG:boto:Params: {} 2023-09-29T22:42:05.929 DEBUG:boto:Token: None 2023-09-29T22:42:05.929 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.929 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h1yM1hMe9HmQHbX34onovPgHGH0= 2023-09-29T22:42:05.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h1yM1hMe9HmQHbX34onovPgHGH0=', 'Content-Length': '0'} 2023-09-29T22:42:05.932 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000642657e9921394f7-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.932 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:05.933 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:05.933 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.933 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.933 DEBUG:boto:Method: HEAD 2023-09-29T22:42:05.933 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.933 DEBUG:boto:Data: 2023-09-29T22:42:05.933 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.934 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.934 DEBUG:boto:Params: {} 2023-09-29T22:42:05.934 DEBUG:boto:Token: None 2023-09-29T22:42:05.934 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h1yM1hMe9HmQHbX34onovPgHGH0= 2023-09-29T22:42:05.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h1yM1hMe9HmQHbX34onovPgHGH0=', 'Content-Length': '0'} 2023-09-29T22:42:05.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f863cd7cdf28d9c-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:42:05.938 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:05.941 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:05.941 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.941 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.942 DEBUG:boto:Method: GET 2023-09-29T22:42:05.942 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.942 DEBUG:boto:Data: 2023-09-29T22:42:05.942 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.942 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.942 DEBUG:boto:Params: {} 2023-09-29T22:42:05.942 DEBUG:boto:Token: None 2023-09-29T22:42:05.943 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I= 2023-09-29T22:42:05.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I=', 'Content-Length': '0'} 2023-09-29T22:42:05.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df70df98d90f62e8-00651752bd-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.946 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:05.947 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:05.947 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.947 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.947 DEBUG:boto:Method: GET 2023-09-29T22:42:05.947 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.947 DEBUG:boto:Data: 2023-09-29T22:42:05.947 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:05.948 DEBUG:boto:Port: 8000 2023-09-29T22:42:05.948 DEBUG:boto:Params: {} 2023-09-29T22:42:05.948 DEBUG:boto:Token: None 2023-09-29T22:42:05.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:42:05.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I= 2023-09-29T22:42:05.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HNC3fePeH1YQrwEjjikmYWCBR3I=', 'Content-Length': '0'} 2023-09-29T22:42:05.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a541077d8ba41979-00651752bd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:05.994 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:05.995 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:05.995 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:05.996 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:05.996 DEBUG:boto:Method: GET 2023-09-29T22:42:05.996 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:42:05.996 DEBUG:boto:Data: 2023-09-29T22:42:05.996 DEBUG:boto:Headers: {} 2023-09-29T22:42:05.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:05.996 DEBUG:boto:Port: 8001 2023-09-29T22:42:05.996 DEBUG:boto:Params: {} 2023-09-29T22:42:05.996 DEBUG:boto:Token: None 2023-09-29T22:42:05.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:05 GMT /hcnphz-59/ 2023-09-29T22:42:05.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E= 2023-09-29T22:42:05.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ooKXPnqJmlEXPOL1EQtOHZ+Rm2E=', 'Content-Length': '0'} 2023-09-29T22:42:06.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d4e80608462ccb1-00651752bd-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.002 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.003 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.003 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.003 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.003 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.003 DEBUG:boto:Method: GET 2023-09-29T22:42:06.004 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.004 DEBUG:boto:Data: 2023-09-29T22:42:06.004 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.004 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.004 DEBUG:boto:Params: {} 2023-09-29T22:42:06.004 DEBUG:boto:Token: None 2023-09-29T22:42:06.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.011 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000972ab3dc8c6439d6-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.011 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.011 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:06.011 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:06.012 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.012 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:42:06.012 DEBUG:boto:Data: 2023-09-29T22:42:06.012 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.012 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.012 DEBUG:boto:Params: {} 2023-09-29T22:42:06.012 DEBUG:boto:Token: None 2023-09-29T22:42:06.013 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj44 2023-09-29T22:42:06.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CcmZTdxpP2XnzoGhH5RpuFD57vw= 2023-09-29T22:42:06.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CcmZTdxpP2XnzoGhH5RpuFD57vw=', 'Content-Length': '0'} 2023-09-29T22:42:06.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dcbbb849dd94964b-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.015 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:06.015 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:06.015 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.016 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:42:06.016 DEBUG:boto:Data: 2023-09-29T22:42:06.016 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.016 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.016 DEBUG:boto:Params: {} 2023-09-29T22:42:06.016 DEBUG:boto:Token: None 2023-09-29T22:42:06.017 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj44 2023-09-29T22:42:06.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CcmZTdxpP2XnzoGhH5RpuFD57vw= 2023-09-29T22:42:06.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CcmZTdxpP2XnzoGhH5RpuFD57vw=', 'Content-Length': '0'} 2023-09-29T22:42:06.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a722b27b115f412e-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:42:06.019 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:06.019 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:06.020 DEBUG:boto:Method: GET 2023-09-29T22:42:06.020 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:42:06.020 DEBUG:boto:Data: 2023-09-29T22:42:06.020 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.020 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.020 DEBUG:boto:Params: {} 2023-09-29T22:42:06.021 DEBUG:boto:Token: None 2023-09-29T22:42:06.021 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj44 2023-09-29T22:42:06.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k/8cep7J6W/lyqYcgHQBXoxdlWQ= 2023-09-29T22:42:06.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k/8cep7J6W/lyqYcgHQBXoxdlWQ=', 'Content-Length': '0'} 2023-09-29T22:42:06.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000973501e0554ad309-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.066 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:42:06.067 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:42:06.067 DEBUG:boto:Method: GET 2023-09-29T22:42:06.067 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:42:06.067 DEBUG:boto:Data: 2023-09-29T22:42:06.067 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.067 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.067 DEBUG:boto:Params: {} 2023-09-29T22:42:06.068 DEBUG:boto:Token: None 2023-09-29T22:42:06.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj44 2023-09-29T22:42:06.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k/8cep7J6W/lyqYcgHQBXoxdlWQ= 2023-09-29T22:42:06.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k/8cep7J6W/lyqYcgHQBXoxdlWQ=', 'Content-Length': '0'} 2023-09-29T22:42:06.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a37400139e39f6-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.071 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.071 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.072 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.072 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.072 DEBUG:boto:Method: GET 2023-09-29T22:42:06.072 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.072 DEBUG:boto:Data: 2023-09-29T22:42:06.072 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.072 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.072 DEBUG:boto:Params: {} 2023-09-29T22:42:06.073 DEBUG:boto:Token: None 2023-09-29T22:42:06.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.073 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.078 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002e2d2d1677223eaa-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.078 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.079 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.079 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.079 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.079 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.079 DEBUG:boto:Method: GET 2023-09-29T22:42:06.079 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:42:06.079 DEBUG:boto:Data: 2023-09-29T22:42:06.080 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.080 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.080 DEBUG:boto:Params: {} 2023-09-29T22:42:06.080 DEBUG:boto:Token: None 2023-09-29T22:42:06.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000790bee0f98468362-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.126 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:42:06.127 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.127 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.127 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.127 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.128 DEBUG:boto:Method: GET 2023-09-29T22:42:06.128 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.128 DEBUG:boto:Data: 2023-09-29T22:42:06.128 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.128 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.128 DEBUG:boto:Params: {} 2023-09-29T22:42:06.128 DEBUG:boto:Token: None 2023-09-29T22:42:06.129 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc= 2023-09-29T22:42:06.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc=', 'Content-Length': '0'} 2023-09-29T22:42:06.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001855a4395bcb3fbb-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.134 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.134 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.134 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.134 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.134 DEBUG:boto:Method: GET 2023-09-29T22:42:06.135 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.135 DEBUG:boto:Data: 2023-09-29T22:42:06.135 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.135 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.135 DEBUG:boto:Params: {} 2023-09-29T22:42:06.135 DEBUG:boto:Token: None 2023-09-29T22:42:06.135 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.135 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc= 2023-09-29T22:42:06.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc=', 'Content-Length': '0'} 2023-09-29T22:42:06.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086342c547dc66320-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.138 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.138 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.139 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.139 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.139 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.139 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.139 DEBUG:boto:Data: 2023-09-29T22:42:06.140 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.140 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.140 DEBUG:boto:Params: {} 2023-09-29T22:42:06.140 DEBUG:boto:Token: None 2023-09-29T22:42:06.141 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DGXzWsrNlu7ZVPm0pWTucJgh/dA= 2023-09-29T22:42:06.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DGXzWsrNlu7ZVPm0pWTucJgh/dA=', 'Content-Length': '0'} 2023-09-29T22:42:06.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027c532c6e7fd2e9d-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.145 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.146 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.146 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.146 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.146 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.146 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.147 DEBUG:boto:Data: 2023-09-29T22:42:06.147 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.147 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.147 DEBUG:boto:Params: {} 2023-09-29T22:42:06.147 DEBUG:boto:Token: None 2023-09-29T22:42:06.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DGXzWsrNlu7ZVPm0pWTucJgh/dA= 2023-09-29T22:42:06.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DGXzWsrNlu7ZVPm0pWTucJgh/dA=', 'Content-Length': '0'} 2023-09-29T22:42:06.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007faf6712aa744fb5-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:42:06.150 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.151 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.151 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.151 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.151 DEBUG:boto:Method: GET 2023-09-29T22:42:06.151 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.151 DEBUG:boto:Data: 2023-09-29T22:42:06.151 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.152 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.152 DEBUG:boto:Params: {} 2023-09-29T22:42:06.152 DEBUG:boto:Token: None 2023-09-29T22:42:06.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc= 2023-09-29T22:42:06.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc=', 'Content-Length': '0'} 2023-09-29T22:42:06.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b930d76406593a0d-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.198 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.199 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.199 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.199 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.199 DEBUG:boto:Method: GET 2023-09-29T22:42:06.199 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.200 DEBUG:boto:Data: 2023-09-29T22:42:06.200 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.200 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.200 DEBUG:boto:Params: {} 2023-09-29T22:42:06.200 DEBUG:boto:Token: None 2023-09-29T22:42:06.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:42:06.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc= 2023-09-29T22:42:06.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:luE4Ub8cWCS4tnDo7nEu0dMReOc=', 'Content-Length': '0'} 2023-09-29T22:42:06.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068470e8780fb3cdf-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.204 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.205 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.205 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.205 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.205 DEBUG:boto:Method: GET 2023-09-29T22:42:06.205 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.205 DEBUG:boto:Data: 2023-09-29T22:42:06.205 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.205 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.206 DEBUG:boto:Params: {} 2023-09-29T22:42:06.206 DEBUG:boto:Token: None 2023-09-29T22:42:06.206 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.206 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.212 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be7eb4e1e744300-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.212 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.213 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.213 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.213 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.213 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.213 DEBUG:boto:Method: GET 2023-09-29T22:42:06.213 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.214 DEBUG:boto:Data: 2023-09-29T22:42:06.214 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.214 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.214 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.214 DEBUG:boto:Params: {} 2023-09-29T22:42:06.215 DEBUG:boto:Token: None 2023-09-29T22:42:06.215 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.215 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.221 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f7da2a1e48c57f7-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.262 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.263 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.263 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.263 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.263 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:42:06.264 DEBUG:boto:Data: 2023-09-29T22:42:06.264 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.264 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.264 DEBUG:boto:Params: {} 2023-09-29T22:42:06.264 DEBUG:boto:Token: None 2023-09-29T22:42:06.264 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51 2023-09-29T22:42:06.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NBqL0tfDpau7J3z6xhdRYW1dUvo= 2023-09-29T22:42:06.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NBqL0tfDpau7J3z6xhdRYW1dUvo=', 'Content-Length': '0'} 2023-09-29T22:42:06.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b35f0709fa14ec11-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.270 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.270 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.270 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.270 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:42:06.270 DEBUG:boto:Data: 2023-09-29T22:42:06.270 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.271 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.271 DEBUG:boto:Params: {} 2023-09-29T22:42:06.271 DEBUG:boto:Token: None 2023-09-29T22:42:06.271 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51 2023-09-29T22:42:06.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NBqL0tfDpau7J3z6xhdRYW1dUvo= 2023-09-29T22:42:06.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NBqL0tfDpau7J3z6xhdRYW1dUvo=', 'Content-Length': '0'} 2023-09-29T22:42:06.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d51a92fc330e4c9-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:42:06.275 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.275 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.276 DEBUG:boto:Method: GET 2023-09-29T22:42:06.276 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:42:06.276 DEBUG:boto:Data: 2023-09-29T22:42:06.276 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.276 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.276 DEBUG:boto:Params: {} 2023-09-29T22:42:06.276 DEBUG:boto:Token: None 2023-09-29T22:42:06.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51 2023-09-29T22:42:06.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w1S8CGjZ2hH1iWfqEZ/zrXODyW4= 2023-09-29T22:42:06.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w1S8CGjZ2hH1iWfqEZ/zrXODyW4=', 'Content-Length': '0'} 2023-09-29T22:42:06.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008091b137b7d1807-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.280 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:42:06.280 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:42:06.280 DEBUG:boto:Method: GET 2023-09-29T22:42:06.280 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:42:06.280 DEBUG:boto:Data: 2023-09-29T22:42:06.280 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.281 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.281 DEBUG:boto:Params: {} 2023-09-29T22:42:06.281 DEBUG:boto:Token: None 2023-09-29T22:42:06.281 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj51 2023-09-29T22:42:06.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w1S8CGjZ2hH1iWfqEZ/zrXODyW4= 2023-09-29T22:42:06.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w1S8CGjZ2hH1iWfqEZ/zrXODyW4=', 'Content-Length': '0'} 2023-09-29T22:42:06.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011541a1b8babf1df-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.284 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.284 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.285 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.285 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.285 DEBUG:boto:Method: GET 2023-09-29T22:42:06.285 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.285 DEBUG:boto:Data: 2023-09-29T22:42:06.285 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.285 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.285 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.285 DEBUG:boto:Params: {} 2023-09-29T22:42:06.286 DEBUG:boto:Token: None 2023-09-29T22:42:06.286 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.286 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.286 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.291 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a13ecc5bbec1fa2-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.338 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.339 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.339 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.339 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.339 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.339 DEBUG:boto:Method: GET 2023-09-29T22:42:06.339 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:42:06.340 DEBUG:boto:Data: 2023-09-29T22:42:06.340 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.340 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.340 DEBUG:boto:Params: {} 2023-09-29T22:42:06.340 DEBUG:boto:Token: None 2023-09-29T22:42:06.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.341 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.347 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000fb908ccbafd8263-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.348 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:42:06.348 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.349 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.349 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.349 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.349 DEBUG:boto:Method: GET 2023-09-29T22:42:06.349 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.349 DEBUG:boto:Data: 2023-09-29T22:42:06.349 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.350 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.350 DEBUG:boto:Params: {} 2023-09-29T22:42:06.350 DEBUG:boto:Token: None 2023-09-29T22:42:06.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.350 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE= 2023-09-29T22:42:06.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE=', 'Content-Length': '0'} 2023-09-29T22:42:06.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f20f88e5eba2cdd-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.354 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.354 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.354 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.354 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.354 DEBUG:boto:Method: GET 2023-09-29T22:42:06.355 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.355 DEBUG:boto:Data: 2023-09-29T22:42:06.355 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.355 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.355 DEBUG:boto:Params: {} 2023-09-29T22:42:06.355 DEBUG:boto:Token: None 2023-09-29T22:42:06.355 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE= 2023-09-29T22:42:06.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE=', 'Content-Length': '0'} 2023-09-29T22:42:06.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bfb5b7236756b8b-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.402 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.403 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.403 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.403 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.403 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.403 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.403 DEBUG:boto:Data: 2023-09-29T22:42:06.403 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.403 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.403 DEBUG:boto:Params: {} 2023-09-29T22:42:06.404 DEBUG:boto:Token: None 2023-09-29T22:42:06.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gTf5B9ZrcyD0VrH6AK8N4yO2yuE= 2023-09-29T22:42:06.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gTf5B9ZrcyD0VrH6AK8N4yO2yuE=', 'Content-Length': '0'} 2023-09-29T22:42:06.407 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e285f5320740bb1-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.407 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.407 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.407 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.407 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.408 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.408 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.408 DEBUG:boto:Data: 2023-09-29T22:42:06.408 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.408 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.408 DEBUG:boto:Params: {} 2023-09-29T22:42:06.408 DEBUG:boto:Token: None 2023-09-29T22:42:06.409 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gTf5B9ZrcyD0VrH6AK8N4yO2yuE= 2023-09-29T22:42:06.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gTf5B9ZrcyD0VrH6AK8N4yO2yuE=', 'Content-Length': '0'} 2023-09-29T22:42:06.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e188b175a45625f0-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:42:06.412 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.412 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.412 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.413 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.413 DEBUG:boto:Method: GET 2023-09-29T22:42:06.413 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.413 DEBUG:boto:Data: 2023-09-29T22:42:06.413 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.414 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.414 DEBUG:boto:Params: {} 2023-09-29T22:42:06.414 DEBUG:boto:Token: None 2023-09-29T22:42:06.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE= 2023-09-29T22:42:06.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE=', 'Content-Length': '0'} 2023-09-29T22:42:06.417 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db2a8d3a209533b5-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.417 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.417 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.418 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.418 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.418 DEBUG:boto:Method: GET 2023-09-29T22:42:06.418 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.418 DEBUG:boto:Data: 2023-09-29T22:42:06.418 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.418 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.418 DEBUG:boto:Params: {} 2023-09-29T22:42:06.419 DEBUG:boto:Token: None 2023-09-29T22:42:06.419 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:42:06.419 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE= 2023-09-29T22:42:06.419 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:obfJDbkz5Rtc3bZZdzjn1M2oAfE=', 'Content-Length': '0'} 2023-09-29T22:42:06.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095f9d484b1f9e2d3-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.422 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.422 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.422 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.423 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.423 DEBUG:boto:Method: GET 2023-09-29T22:42:06.423 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.423 DEBUG:boto:Data: 2023-09-29T22:42:06.423 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.423 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.423 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.423 DEBUG:boto:Params: {} 2023-09-29T22:42:06.423 DEBUG:boto:Token: None 2023-09-29T22:42:06.423 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e2238111895380f-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.429 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.429 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.429 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.429 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.429 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.429 DEBUG:boto:Method: GET 2023-09-29T22:42:06.430 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.430 DEBUG:boto:Data: 2023-09-29T22:42:06.430 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.430 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.430 DEBUG:boto:Params: {} 2023-09-29T22:42:06.430 DEBUG:boto:Token: None 2023-09-29T22:42:06.431 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.431 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.436 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007f74224d45215de-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.436 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.436 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.436 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.437 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.437 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:42:06.437 DEBUG:boto:Data: 2023-09-29T22:42:06.437 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.437 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.437 DEBUG:boto:Params: {} 2023-09-29T22:42:06.437 DEBUG:boto:Token: None 2023-09-29T22:42:06.438 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55 2023-09-29T22:42:06.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GhYe345YNMjvKurw65zKmlPuUh8= 2023-09-29T22:42:06.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GhYe345YNMjvKurw65zKmlPuUh8=', 'Content-Length': '0'} 2023-09-29T22:42:06.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002722c7189befc205-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.440 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.440 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.440 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.440 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:42:06.441 DEBUG:boto:Data: 2023-09-29T22:42:06.441 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.441 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.441 DEBUG:boto:Params: {} 2023-09-29T22:42:06.441 DEBUG:boto:Token: None 2023-09-29T22:42:06.441 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55 2023-09-29T22:42:06.441 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GhYe345YNMjvKurw65zKmlPuUh8= 2023-09-29T22:42:06.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GhYe345YNMjvKurw65zKmlPuUh8=', 'Content-Length': '0'} 2023-09-29T22:42:06.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051f56c1a474429d3-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:42:06.444 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.444 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.444 DEBUG:boto:Method: GET 2023-09-29T22:42:06.444 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:42:06.444 DEBUG:boto:Data: 2023-09-29T22:42:06.444 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.444 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.444 DEBUG:boto:Params: {} 2023-09-29T22:42:06.444 DEBUG:boto:Token: None 2023-09-29T22:42:06.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55 2023-09-29T22:42:06.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oCsw9IPZieEe07CA1N7C2bJ/dCE= 2023-09-29T22:42:06.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oCsw9IPZieEe07CA1N7C2bJ/dCE=', 'Content-Length': '0'} 2023-09-29T22:42:06.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d45825607cb1520-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.490 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:42:06.491 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:42:06.491 DEBUG:boto:Method: GET 2023-09-29T22:42:06.491 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:42:06.491 DEBUG:boto:Data: 2023-09-29T22:42:06.491 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.491 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.491 DEBUG:boto:Params: {} 2023-09-29T22:42:06.491 DEBUG:boto:Token: None 2023-09-29T22:42:06.492 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj55 2023-09-29T22:42:06.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oCsw9IPZieEe07CA1N7C2bJ/dCE= 2023-09-29T22:42:06.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oCsw9IPZieEe07CA1N7C2bJ/dCE=', 'Content-Length': '0'} 2023-09-29T22:42:06.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd7438a5fc98c5a0-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.495 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.495 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.495 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.495 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.495 DEBUG:boto:Method: GET 2023-09-29T22:42:06.496 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.496 DEBUG:boto:Data: 2023-09-29T22:42:06.496 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.496 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.496 DEBUG:boto:Params: {} 2023-09-29T22:42:06.496 DEBUG:boto:Token: None 2023-09-29T22:42:06.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.500 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e7f0b11536595ad-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.501 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.501 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.501 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.501 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.501 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.502 DEBUG:boto:Method: GET 2023-09-29T22:42:06.502 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:42:06.502 DEBUG:boto:Data: 2023-09-29T22:42:06.502 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.502 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.502 DEBUG:boto:Params: {} 2023-09-29T22:42:06.502 DEBUG:boto:Token: None 2023-09-29T22:42:06.503 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005057935d09ba88e3-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.508 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:42:06.509 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.509 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.509 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.509 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.509 DEBUG:boto:Method: GET 2023-09-29T22:42:06.509 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.509 DEBUG:boto:Data: 2023-09-29T22:42:06.509 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.510 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.510 DEBUG:boto:Params: {} 2023-09-29T22:42:06.510 DEBUG:boto:Token: None 2023-09-29T22:42:06.510 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.510 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98= 2023-09-29T22:42:06.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98=', 'Content-Length': '0'} 2023-09-29T22:42:06.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbeb3982f0701aef-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.513 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.513 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.513 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.513 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.513 DEBUG:boto:Method: GET 2023-09-29T22:42:06.514 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.514 DEBUG:boto:Data: 2023-09-29T22:42:06.514 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.514 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.514 DEBUG:boto:Params: {} 2023-09-29T22:42:06.514 DEBUG:boto:Token: None 2023-09-29T22:42:06.514 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98= 2023-09-29T22:42:06.515 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98=', 'Content-Length': '0'} 2023-09-29T22:42:06.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da7fea4d1de2d384-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.517 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.517 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.518 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.518 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.518 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.518 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.518 DEBUG:boto:Data: 2023-09-29T22:42:06.518 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.518 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.518 DEBUG:boto:Params: {} 2023-09-29T22:42:06.519 DEBUG:boto:Token: None 2023-09-29T22:42:06.519 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.519 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iEo98xsKvzJJ1VeAJB8AgV+1jcQ= 2023-09-29T22:42:06.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iEo98xsKvzJJ1VeAJB8AgV+1jcQ=', 'Content-Length': '0'} 2023-09-29T22:42:06.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b181e204c8e980-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.521 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.522 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.522 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.522 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.522 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.522 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.522 DEBUG:boto:Data: 2023-09-29T22:42:06.522 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.523 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.523 DEBUG:boto:Params: {} 2023-09-29T22:42:06.523 DEBUG:boto:Token: None 2023-09-29T22:42:06.523 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iEo98xsKvzJJ1VeAJB8AgV+1jcQ= 2023-09-29T22:42:06.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iEo98xsKvzJJ1VeAJB8AgV+1jcQ=', 'Content-Length': '0'} 2023-09-29T22:42:06.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000e2d8d8d689baa3-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:42:06.526 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.526 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.526 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.526 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.526 DEBUG:boto:Method: GET 2023-09-29T22:42:06.526 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.526 DEBUG:boto:Data: 2023-09-29T22:42:06.526 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.527 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.527 DEBUG:boto:Params: {} 2023-09-29T22:42:06.527 DEBUG:boto:Token: None 2023-09-29T22:42:06.527 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98= 2023-09-29T22:42:06.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98=', 'Content-Length': '0'} 2023-09-29T22:42:06.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eff8683c7b28df45-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.530 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.530 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.530 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.530 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.530 DEBUG:boto:Method: GET 2023-09-29T22:42:06.531 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.531 DEBUG:boto:Data: 2023-09-29T22:42:06.531 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.531 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.531 DEBUG:boto:Params: {} 2023-09-29T22:42:06.531 DEBUG:boto:Token: None 2023-09-29T22:42:06.531 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:42:06.531 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98= 2023-09-29T22:42:06.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qcMWYC+vxGIKUNlXeng7e5uge98=', 'Content-Length': '0'} 2023-09-29T22:42:06.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc3cb12378332d08-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.535 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.535 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.535 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.535 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.535 DEBUG:boto:Method: GET 2023-09-29T22:42:06.535 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.536 DEBUG:boto:Data: 2023-09-29T22:42:06.536 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.536 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.536 DEBUG:boto:Params: {} 2023-09-29T22:42:06.536 DEBUG:boto:Token: None 2023-09-29T22:42:06.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6298c842aedd7f7-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.542 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.543 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.543 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.543 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.543 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.543 DEBUG:boto:Method: GET 2023-09-29T22:42:06.543 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.544 DEBUG:boto:Data: 2023-09-29T22:42:06.544 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.544 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.544 DEBUG:boto:Params: {} 2023-09-29T22:42:06.544 DEBUG:boto:Token: None 2023-09-29T22:42:06.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f97368e02a0b0467-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.598 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.599 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.599 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.599 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.599 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:42:06.599 DEBUG:boto:Data: 2023-09-29T22:42:06.600 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.600 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.600 DEBUG:boto:Params: {} 2023-09-29T22:42:06.600 DEBUG:boto:Token: None 2023-09-29T22:42:06.600 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57 2023-09-29T22:42:06.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h1zinJ+mC6K91I766ynlCL6D2Ds= 2023-09-29T22:42:06.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h1zinJ+mC6K91I766ynlCL6D2Ds=', 'Content-Length': '0'} 2023-09-29T22:42:06.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b03514ca98fe63f7-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.604 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.604 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.604 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.604 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:42:06.604 DEBUG:boto:Data: 2023-09-29T22:42:06.605 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.605 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.605 DEBUG:boto:Params: {} 2023-09-29T22:42:06.606 DEBUG:boto:Token: None 2023-09-29T22:42:06.606 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57 2023-09-29T22:42:06.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h1zinJ+mC6K91I766ynlCL6D2Ds= 2023-09-29T22:42:06.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h1zinJ+mC6K91I766ynlCL6D2Ds=', 'Content-Length': '0'} 2023-09-29T22:42:06.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d803c76fe0296470-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:42:06.610 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.610 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.610 DEBUG:boto:Method: GET 2023-09-29T22:42:06.611 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:42:06.611 DEBUG:boto:Data: 2023-09-29T22:42:06.611 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.611 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.612 DEBUG:boto:Params: {} 2023-09-29T22:42:06.612 DEBUG:boto:Token: None 2023-09-29T22:42:06.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57 2023-09-29T22:42:06.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X5b9MaX1OLIwNmhPySub0SuTiyA= 2023-09-29T22:42:06.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X5b9MaX1OLIwNmhPySub0SuTiyA=', 'Content-Length': '0'} 2023-09-29T22:42:06.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a113d5e53632b59b-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.658 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:42:06.659 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:42:06.659 DEBUG:boto:Method: GET 2023-09-29T22:42:06.659 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:42:06.659 DEBUG:boto:Data: 2023-09-29T22:42:06.659 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.659 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.660 DEBUG:boto:Params: {} 2023-09-29T22:42:06.660 DEBUG:boto:Token: None 2023-09-29T22:42:06.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj57 2023-09-29T22:42:06.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X5b9MaX1OLIwNmhPySub0SuTiyA= 2023-09-29T22:42:06.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X5b9MaX1OLIwNmhPySub0SuTiyA=', 'Content-Length': '0'} 2023-09-29T22:42:06.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000967b91843b3143b3-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.663 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.664 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.664 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.664 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.664 DEBUG:boto:Method: GET 2023-09-29T22:42:06.664 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.664 DEBUG:boto:Data: 2023-09-29T22:42:06.664 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.664 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.665 DEBUG:boto:Params: {} 2023-09-29T22:42:06.665 DEBUG:boto:Token: None 2023-09-29T22:42:06.665 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbd9c5403ce6da9d-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.671 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.671 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.671 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.672 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.672 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.672 DEBUG:boto:Method: GET 2023-09-29T22:42:06.672 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:42:06.672 DEBUG:boto:Data: 2023-09-29T22:42:06.672 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.672 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.672 DEBUG:boto:Params: {} 2023-09-29T22:42:06.673 DEBUG:boto:Token: None 2023-09-29T22:42:06.673 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f54e0c0d58be3228-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.679 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:42:06.679 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.679 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.680 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.680 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.680 DEBUG:boto:Method: GET 2023-09-29T22:42:06.680 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.680 DEBUG:boto:Data: 2023-09-29T22:42:06.680 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.681 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.681 DEBUG:boto:Params: {} 2023-09-29T22:42:06.681 DEBUG:boto:Token: None 2023-09-29T22:42:06.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw= 2023-09-29T22:42:06.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw=', 'Content-Length': '0'} 2023-09-29T22:42:06.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a872aaa55e8f83d-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.684 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.684 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.684 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.684 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.685 DEBUG:boto:Method: GET 2023-09-29T22:42:06.685 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.685 DEBUG:boto:Data: 2023-09-29T22:42:06.685 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.685 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.685 DEBUG:boto:Params: {} 2023-09-29T22:42:06.685 DEBUG:boto:Token: None 2023-09-29T22:42:06.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw= 2023-09-29T22:42:06.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw=', 'Content-Length': '0'} 2023-09-29T22:42:06.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d5fef747544248b-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.730 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.731 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.731 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.731 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.731 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.731 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.731 DEBUG:boto:Data: 2023-09-29T22:42:06.731 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.732 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.732 DEBUG:boto:Params: {} 2023-09-29T22:42:06.732 DEBUG:boto:Token: None 2023-09-29T22:42:06.732 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AQUQsK1Xd2FzjtLJa1mOFBWuD34= 2023-09-29T22:42:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AQUQsK1Xd2FzjtLJa1mOFBWuD34=', 'Content-Length': '0'} 2023-09-29T22:42:06.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047434b88091bbd01-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.737 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.737 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.737 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.737 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.738 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.738 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.738 DEBUG:boto:Data: 2023-09-29T22:42:06.738 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.738 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.738 DEBUG:boto:Params: {} 2023-09-29T22:42:06.738 DEBUG:boto:Token: None 2023-09-29T22:42:06.739 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AQUQsK1Xd2FzjtLJa1mOFBWuD34= 2023-09-29T22:42:06.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AQUQsK1Xd2FzjtLJa1mOFBWuD34=', 'Content-Length': '0'} 2023-09-29T22:42:06.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a1a3093526e9618-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:42:06.742 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.742 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.742 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.742 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.742 DEBUG:boto:Method: GET 2023-09-29T22:42:06.743 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.743 DEBUG:boto:Data: 2023-09-29T22:42:06.743 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.744 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.744 DEBUG:boto:Params: {} 2023-09-29T22:42:06.744 DEBUG:boto:Token: None 2023-09-29T22:42:06.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw= 2023-09-29T22:42:06.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw=', 'Content-Length': '0'} 2023-09-29T22:42:06.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8506bc44e88412-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.790 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.791 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.791 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.791 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.791 DEBUG:boto:Method: GET 2023-09-29T22:42:06.791 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.791 DEBUG:boto:Data: 2023-09-29T22:42:06.791 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.791 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.792 DEBUG:boto:Params: {} 2023-09-29T22:42:06.792 DEBUG:boto:Token: None 2023-09-29T22:42:06.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:42:06.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw= 2023-09-29T22:42:06.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5n8asY8HEhKWufMJh9op37esOkw=', 'Content-Length': '0'} 2023-09-29T22:42:06.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cffa5b433b1a312-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.795 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.795 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.796 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.796 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.796 DEBUG:boto:Method: GET 2023-09-29T22:42:06.796 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.796 DEBUG:boto:Data: 2023-09-29T22:42:06.796 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.796 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.796 DEBUG:boto:Params: {} 2023-09-29T22:42:06.796 DEBUG:boto:Token: None 2023-09-29T22:42:06.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.801 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b38439ca8e42c7-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.801 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.802 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.802 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.802 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.802 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.802 DEBUG:boto:Method: GET 2023-09-29T22:42:06.802 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.803 DEBUG:boto:Data: 2023-09-29T22:42:06.803 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.803 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.803 DEBUG:boto:Params: {} 2023-09-29T22:42:06.803 DEBUG:boto:Token: None 2023-09-29T22:42:06.803 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.803 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ed558aed60ef9ed-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.809 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.809 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.809 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.810 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.810 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:42:06.810 DEBUG:boto:Data: 2023-09-29T22:42:06.810 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.810 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.810 DEBUG:boto:Params: {} 2023-09-29T22:42:06.810 DEBUG:boto:Token: None 2023-09-29T22:42:06.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68 2023-09-29T22:42:06.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bcu1PchpKC7hY0Ohza2XD+yNbks= 2023-09-29T22:42:06.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bcu1PchpKC7hY0Ohza2XD+yNbks=', 'Content-Length': '0'} 2023-09-29T22:42:06.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af618b0f1c5da06b-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.813 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.813 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.813 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.813 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:42:06.813 DEBUG:boto:Data: 2023-09-29T22:42:06.813 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.814 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.814 DEBUG:boto:Params: {} 2023-09-29T22:42:06.814 DEBUG:boto:Token: None 2023-09-29T22:42:06.814 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68 2023-09-29T22:42:06.814 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bcu1PchpKC7hY0Ohza2XD+yNbks= 2023-09-29T22:42:06.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bcu1PchpKC7hY0Ohza2XD+yNbks=', 'Content-Length': '0'} 2023-09-29T22:42:06.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5a66e0b2c6f7fef-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:42:06.817 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.817 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.817 DEBUG:boto:Method: GET 2023-09-29T22:42:06.817 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:42:06.817 DEBUG:boto:Data: 2023-09-29T22:42:06.817 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.817 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.818 DEBUG:boto:Params: {} 2023-09-29T22:42:06.818 DEBUG:boto:Token: None 2023-09-29T22:42:06.818 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68 2023-09-29T22:42:06.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ye+WwY+i3Hj48rcnnZT7gprcnVM= 2023-09-29T22:42:06.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ye+WwY+i3Hj48rcnnZT7gprcnVM=', 'Content-Length': '0'} 2023-09-29T22:42:06.820 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507e3d2dddcd9275-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.821 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:42:06.821 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:42:06.821 DEBUG:boto:Method: GET 2023-09-29T22:42:06.821 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:42:06.821 DEBUG:boto:Data: 2023-09-29T22:42:06.821 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.821 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.821 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.821 DEBUG:boto:Params: {} 2023-09-29T22:42:06.822 DEBUG:boto:Token: None 2023-09-29T22:42:06.822 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj68 2023-09-29T22:42:06.822 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ye+WwY+i3Hj48rcnnZT7gprcnVM= 2023-09-29T22:42:06.822 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ye+WwY+i3Hj48rcnnZT7gprcnVM=', 'Content-Length': '0'} 2023-09-29T22:42:06.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035ca19ba779699e1-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.866 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.867 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.867 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.867 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.867 DEBUG:boto:Method: GET 2023-09-29T22:42:06.867 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.867 DEBUG:boto:Data: 2023-09-29T22:42:06.867 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.867 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.868 DEBUG:boto:Params: {} 2023-09-29T22:42:06.868 DEBUG:boto:Token: None 2023-09-29T22:42:06.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d8d2c008853bc13-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.873 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.874 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.874 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.874 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.874 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.874 DEBUG:boto:Method: GET 2023-09-29T22:42:06.874 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:42:06.874 DEBUG:boto:Data: 2023-09-29T22:42:06.874 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.875 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.875 DEBUG:boto:Params: {} 2023-09-29T22:42:06.875 DEBUG:boto:Token: None 2023-09-29T22:42:06.875 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.875 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a4be4c65001bf71-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.881 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:42:06.881 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:06.881 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:06.881 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.882 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.882 DEBUG:boto:Method: GET 2023-09-29T22:42:06.882 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.882 DEBUG:boto:Data: 2023-09-29T22:42:06.882 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.882 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.882 DEBUG:boto:Params: {} 2023-09-29T22:42:06.882 DEBUG:boto:Token: None 2023-09-29T22:42:06.883 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.883 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw= 2023-09-29T22:42:06.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw=', 'Content-Length': '0'} 2023-09-29T22:42:06.885 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e7a06cb20c2355d-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.930 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:06.931 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:06.931 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.931 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.931 DEBUG:boto:Method: GET 2023-09-29T22:42:06.932 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.932 DEBUG:boto:Data: 2023-09-29T22:42:06.932 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.932 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.933 DEBUG:boto:Params: {} 2023-09-29T22:42:06.933 DEBUG:boto:Token: None 2023-09-29T22:42:06.933 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw= 2023-09-29T22:42:06.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw=', 'Content-Length': '0'} 2023-09-29T22:42:06.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8d7e59368e5a1ed-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.937 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:06.938 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:06.938 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.938 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.938 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.938 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.939 DEBUG:boto:Data: 2023-09-29T22:42:06.939 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.939 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.939 DEBUG:boto:Params: {} 2023-09-29T22:42:06.939 DEBUG:boto:Token: None 2023-09-29T22:42:06.940 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aEH88OlZX7TxG29E2nhKfCf/U9c= 2023-09-29T22:42:06.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aEH88OlZX7TxG29E2nhKfCf/U9c=', 'Content-Length': '0'} 2023-09-29T22:42:06.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c22fce1e9861405-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.943 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:06.944 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:06.944 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.944 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.944 DEBUG:boto:Method: HEAD 2023-09-29T22:42:06.944 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.944 DEBUG:boto:Data: 2023-09-29T22:42:06.944 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.944 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.945 DEBUG:boto:Params: {} 2023-09-29T22:42:06.945 DEBUG:boto:Token: None 2023-09-29T22:42:06.945 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aEH88OlZX7TxG29E2nhKfCf/U9c= 2023-09-29T22:42:06.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aEH88OlZX7TxG29E2nhKfCf/U9c=', 'Content-Length': '0'} 2023-09-29T22:42:06.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f8a1eaa4ba08bf9-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:42:06.949 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:06.949 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:06.949 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.949 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.950 DEBUG:boto:Method: GET 2023-09-29T22:42:06.950 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.950 DEBUG:boto:Data: 2023-09-29T22:42:06.950 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.950 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.950 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.950 DEBUG:boto:Params: {} 2023-09-29T22:42:06.950 DEBUG:boto:Token: None 2023-09-29T22:42:06.951 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.951 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw= 2023-09-29T22:42:06.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw=', 'Content-Length': '0'} 2023-09-29T22:42:06.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f80c588e0509357-00651752be-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.954 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:06.954 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:06.954 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.954 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.954 DEBUG:boto:Method: GET 2023-09-29T22:42:06.954 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.955 DEBUG:boto:Data: 2023-09-29T22:42:06.955 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.955 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.955 DEBUG:boto:Params: {} 2023-09-29T22:42:06.955 DEBUG:boto:Token: None 2023-09-29T22:42:06.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:42:06.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw= 2023-09-29T22:42:06.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgOIN2vPfuZsTKC315QnkDYm2Cw=', 'Content-Length': '0'} 2023-09-29T22:42:06.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c55746a43463b250-00651752be-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.958 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.959 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.959 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:06.959 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:06.959 DEBUG:boto:Method: GET 2023-09-29T22:42:06.959 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:42:06.959 DEBUG:boto:Data: 2023-09-29T22:42:06.959 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:06.959 DEBUG:boto:Port: 8001 2023-09-29T22:42:06.959 DEBUG:boto:Params: {} 2023-09-29T22:42:06.960 DEBUG:boto:Token: None 2023-09-29T22:42:06.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b15deb44d48d1411-00651752be-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:06.967 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:06.967 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:06.967 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:06.967 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:06.967 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:06.968 DEBUG:boto:Method: GET 2023-09-29T22:42:06.968 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:42:06.968 DEBUG:boto:Data: 2023-09-29T22:42:06.968 DEBUG:boto:Headers: {} 2023-09-29T22:42:06.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:06.968 DEBUG:boto:Port: 8000 2023-09-29T22:42:06.968 DEBUG:boto:Params: {} 2023-09-29T22:42:06.968 DEBUG:boto:Token: None 2023-09-29T22:42:06.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:06 GMT /hcnphz-59/ 2023-09-29T22:42:06.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY= 2023-09-29T22:42:06.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:buFE83YEzTPZo6QNGzItOvZvzqY=', 'Content-Length': '0'} 2023-09-29T22:42:06.974 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004be7af09ff8c0b58-00651752be-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.014 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.015 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:07.015 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:07.015 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.015 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:42:07.015 DEBUG:boto:Data: 2023-09-29T22:42:07.015 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.016 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.016 DEBUG:boto:Params: {} 2023-09-29T22:42:07.016 DEBUG:boto:Token: None 2023-09-29T22:42:07.016 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj76 2023-09-29T22:42:07.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NRh7jX7wNKUmA9985FtLnz0Ds74= 2023-09-29T22:42:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NRh7jX7wNKUmA9985FtLnz0Ds74=', 'Content-Length': '0'} 2023-09-29T22:42:07.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f31201867a3ea2-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.019 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:07.020 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:07.020 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.020 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:42:07.020 DEBUG:boto:Data: 2023-09-29T22:42:07.020 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.021 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.021 DEBUG:boto:Params: {} 2023-09-29T22:42:07.021 DEBUG:boto:Token: None 2023-09-29T22:42:07.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj76 2023-09-29T22:42:07.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NRh7jX7wNKUmA9985FtLnz0Ds74= 2023-09-29T22:42:07.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NRh7jX7wNKUmA9985FtLnz0Ds74=', 'Content-Length': '0'} 2023-09-29T22:42:07.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa17aa4fe701fdca-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:42:07.025 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:07.025 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:07.025 DEBUG:boto:Method: GET 2023-09-29T22:42:07.025 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:42:07.025 DEBUG:boto:Data: 2023-09-29T22:42:07.025 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.025 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.026 DEBUG:boto:Params: {} 2023-09-29T22:42:07.026 DEBUG:boto:Token: None 2023-09-29T22:42:07.026 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj76 2023-09-29T22:42:07.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAtPsBfKtBNcnWm6Uk7TEoWeuoY= 2023-09-29T22:42:07.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAtPsBfKtBNcnWm6Uk7TEoWeuoY=', 'Content-Length': '0'} 2023-09-29T22:42:07.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7494ee5de93fe36-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.070 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:42:07.071 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:42:07.071 DEBUG:boto:Method: GET 2023-09-29T22:42:07.071 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:42:07.071 DEBUG:boto:Data: 2023-09-29T22:42:07.071 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.071 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.071 DEBUG:boto:Params: {} 2023-09-29T22:42:07.071 DEBUG:boto:Token: None 2023-09-29T22:42:07.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj76 2023-09-29T22:42:07.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AAtPsBfKtBNcnWm6Uk7TEoWeuoY= 2023-09-29T22:42:07.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AAtPsBfKtBNcnWm6Uk7TEoWeuoY=', 'Content-Length': '0'} 2023-09-29T22:42:07.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bd5e0bc1dd3da9a-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.075 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.075 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.075 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:07.075 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:07.075 DEBUG:boto:Method: GET 2023-09-29T22:42:07.076 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:42:07.076 DEBUG:boto:Data: 2023-09-29T22:42:07.076 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.076 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.076 DEBUG:boto:Params: {} 2023-09-29T22:42:07.076 DEBUG:boto:Token: None 2023-09-29T22:42:07.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000578ae0d05f310e29-00651752bf-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.082 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.083 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.083 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.083 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:07.083 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:42:07.083 DEBUG:boto:Method: GET 2023-09-29T22:42:07.083 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:42:07.083 DEBUG:boto:Data: 2023-09-29T22:42:07.083 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.084 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.084 DEBUG:boto:Params: {} 2023-09-29T22:42:07.084 DEBUG:boto:Token: None 2023-09-29T22:42:07.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.090 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973aad7c3dbf4227-00651752bf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.090 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:42:07.090 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.091 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.091 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.091 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.091 DEBUG:boto:Method: GET 2023-09-29T22:42:07.091 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.091 DEBUG:boto:Data: 2023-09-29T22:42:07.091 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.091 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.092 DEBUG:boto:Params: {} 2023-09-29T22:42:07.092 DEBUG:boto:Token: None 2023-09-29T22:42:07.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g= 2023-09-29T22:42:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g=', 'Content-Length': '0'} 2023-09-29T22:42:07.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb521dbc4af72d62-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.095 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.095 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.095 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.095 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.095 DEBUG:boto:Method: GET 2023-09-29T22:42:07.096 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.096 DEBUG:boto:Data: 2023-09-29T22:42:07.096 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.096 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.096 DEBUG:boto:Params: {} 2023-09-29T22:42:07.096 DEBUG:boto:Token: None 2023-09-29T22:42:07.096 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.096 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g= 2023-09-29T22:42:07.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g=', 'Content-Length': '0'} 2023-09-29T22:42:07.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000889f3f85fb8c466a-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.146 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.147 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.147 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.147 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.147 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.147 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.147 DEBUG:boto:Data: 2023-09-29T22:42:07.147 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.148 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.148 DEBUG:boto:Params: {} 2023-09-29T22:42:07.148 DEBUG:boto:Token: None 2023-09-29T22:42:07.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JNFsvD8gt2LSiTcYMG1xX7aP2eA= 2023-09-29T22:42:07.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JNFsvD8gt2LSiTcYMG1xX7aP2eA=', 'Content-Length': '0'} 2023-09-29T22:42:07.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011c237c704af4863-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.151 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.151 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.151 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.152 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.152 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.152 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.152 DEBUG:boto:Data: 2023-09-29T22:42:07.152 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.152 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.152 DEBUG:boto:Params: {} 2023-09-29T22:42:07.152 DEBUG:boto:Token: None 2023-09-29T22:42:07.153 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JNFsvD8gt2LSiTcYMG1xX7aP2eA= 2023-09-29T22:42:07.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JNFsvD8gt2LSiTcYMG1xX7aP2eA=', 'Content-Length': '0'} 2023-09-29T22:42:07.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000084e5507ec27c53ec-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:42:07.156 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.156 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.156 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.156 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.157 DEBUG:boto:Method: GET 2023-09-29T22:42:07.157 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.157 DEBUG:boto:Data: 2023-09-29T22:42:07.157 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.157 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.157 DEBUG:boto:Params: {} 2023-09-29T22:42:07.157 DEBUG:boto:Token: None 2023-09-29T22:42:07.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g= 2023-09-29T22:42:07.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g=', 'Content-Length': '0'} 2023-09-29T22:42:07.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ea1db52ae054010-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.202 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.203 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.203 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.203 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.203 DEBUG:boto:Method: GET 2023-09-29T22:42:07.203 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.203 DEBUG:boto:Data: 2023-09-29T22:42:07.203 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.204 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.204 DEBUG:boto:Params: {} 2023-09-29T22:42:07.204 DEBUG:boto:Token: None 2023-09-29T22:42:07.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:42:07.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g= 2023-09-29T22:42:07.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jG0p6fiO3BZB9G/qY3kv/EOnb5g=', 'Content-Length': '0'} 2023-09-29T22:42:07.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e37230d4ae02965c-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.208 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.208 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.208 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.209 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.209 DEBUG:boto:Method: GET 2023-09-29T22:42:07.209 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.209 DEBUG:boto:Data: 2023-09-29T22:42:07.209 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.209 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.209 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.209 DEBUG:boto:Params: {} 2023-09-29T22:42:07.210 DEBUG:boto:Token: None 2023-09-29T22:42:07.210 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.210 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8193d9021e8ea7-00651752bf-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.215 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.215 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.216 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.216 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.216 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.216 DEBUG:boto:Method: GET 2023-09-29T22:42:07.216 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.216 DEBUG:boto:Data: 2023-09-29T22:42:07.216 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.217 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.217 DEBUG:boto:Params: {} 2023-09-29T22:42:07.217 DEBUG:boto:Token: None 2023-09-29T22:42:07.217 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.223 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3c80b22b58894a8-00651752bf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.223 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.223 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.223 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.224 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.224 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:42:07.224 DEBUG:boto:Data: 2023-09-29T22:42:07.224 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.224 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.224 DEBUG:boto:Params: {} 2023-09-29T22:42:07.224 DEBUG:boto:Token: None 2023-09-29T22:42:07.225 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85 2023-09-29T22:42:07.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hHc/cjgWSJVZa5jmSBCkodUba7Q= 2023-09-29T22:42:07.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hHc/cjgWSJVZa5jmSBCkodUba7Q=', 'Content-Length': '0'} 2023-09-29T22:42:07.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eebe0dd44407424d-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.228 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.228 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.228 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.228 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:42:07.228 DEBUG:boto:Data: 2023-09-29T22:42:07.228 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.229 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.229 DEBUG:boto:Params: {} 2023-09-29T22:42:07.229 DEBUG:boto:Token: None 2023-09-29T22:42:07.229 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85 2023-09-29T22:42:07.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hHc/cjgWSJVZa5jmSBCkodUba7Q= 2023-09-29T22:42:07.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hHc/cjgWSJVZa5jmSBCkodUba7Q=', 'Content-Length': '0'} 2023-09-29T22:42:07.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c1ca239074ad06e-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:42:07.232 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.232 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.233 DEBUG:boto:Method: GET 2023-09-29T22:42:07.233 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:42:07.234 DEBUG:boto:Data: 2023-09-29T22:42:07.234 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.234 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.234 DEBUG:boto:Params: {} 2023-09-29T22:42:07.234 DEBUG:boto:Token: None 2023-09-29T22:42:07.235 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85 2023-09-29T22:42:07.235 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EvpExP0lIRNv16QkbGQFM8GxzDs= 2023-09-29T22:42:07.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EvpExP0lIRNv16QkbGQFM8GxzDs=', 'Content-Length': '0'} 2023-09-29T22:42:07.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cf38f0be09acc67-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.282 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:42:07.283 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:42:07.283 DEBUG:boto:Method: GET 2023-09-29T22:42:07.283 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:42:07.283 DEBUG:boto:Data: 2023-09-29T22:42:07.283 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.283 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.283 DEBUG:boto:Params: {} 2023-09-29T22:42:07.284 DEBUG:boto:Token: None 2023-09-29T22:42:07.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj85 2023-09-29T22:42:07.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EvpExP0lIRNv16QkbGQFM8GxzDs= 2023-09-29T22:42:07.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EvpExP0lIRNv16QkbGQFM8GxzDs=', 'Content-Length': '0'} 2023-09-29T22:42:07.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034afdd73b92918df-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.288 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.288 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.288 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.288 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.288 DEBUG:boto:Method: GET 2023-09-29T22:42:07.288 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.288 DEBUG:boto:Data: 2023-09-29T22:42:07.289 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.289 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.289 DEBUG:boto:Params: {} 2023-09-29T22:42:07.289 DEBUG:boto:Token: None 2023-09-29T22:42:07.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f6b8733bb4c0fd-00651752bf-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.294 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.295 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.295 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.295 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.295 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.295 DEBUG:boto:Method: GET 2023-09-29T22:42:07.295 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:42:07.295 DEBUG:boto:Data: 2023-09-29T22:42:07.296 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.296 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.296 DEBUG:boto:Params: {} 2023-09-29T22:42:07.296 DEBUG:boto:Token: None 2023-09-29T22:42:07.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000025fcb4834b6dee5-00651752bf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.302 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:42:07.303 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.303 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.303 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.303 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.303 DEBUG:boto:Method: GET 2023-09-29T22:42:07.303 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.303 DEBUG:boto:Data: 2023-09-29T22:42:07.304 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.304 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.304 DEBUG:boto:Params: {} 2023-09-29T22:42:07.304 DEBUG:boto:Token: None 2023-09-29T22:42:07.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.304 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM= 2023-09-29T22:42:07.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM=', 'Content-Length': '0'} 2023-09-29T22:42:07.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46f007ed55995eb-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.308 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.308 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.308 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.308 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.308 DEBUG:boto:Method: GET 2023-09-29T22:42:07.308 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.308 DEBUG:boto:Data: 2023-09-29T22:42:07.308 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.309 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.309 DEBUG:boto:Params: {} 2023-09-29T22:42:07.309 DEBUG:boto:Token: None 2023-09-29T22:42:07.309 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.309 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM= 2023-09-29T22:42:07.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM=', 'Content-Length': '0'} 2023-09-29T22:42:07.312 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008476e23d2024f00c-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.312 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.312 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.312 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.312 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.312 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.312 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.313 DEBUG:boto:Data: 2023-09-29T22:42:07.313 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.313 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.313 DEBUG:boto:Params: {} 2023-09-29T22:42:07.313 DEBUG:boto:Token: None 2023-09-29T22:42:07.313 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nN2p7d8p1mXj/v1AdiKUQ5TknMY= 2023-09-29T22:42:07.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nN2p7d8p1mXj/v1AdiKUQ5TknMY=', 'Content-Length': '0'} 2023-09-29T22:42:07.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e02c2df569abed-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.316 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.316 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.316 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.316 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.316 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.316 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.316 DEBUG:boto:Data: 2023-09-29T22:42:07.317 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.317 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.317 DEBUG:boto:Params: {} 2023-09-29T22:42:07.317 DEBUG:boto:Token: None 2023-09-29T22:42:07.317 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nN2p7d8p1mXj/v1AdiKUQ5TknMY= 2023-09-29T22:42:07.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nN2p7d8p1mXj/v1AdiKUQ5TknMY=', 'Content-Length': '0'} 2023-09-29T22:42:07.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008613eb2af31c157-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:42:07.320 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.320 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.320 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.321 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.321 DEBUG:boto:Method: GET 2023-09-29T22:42:07.321 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.321 DEBUG:boto:Data: 2023-09-29T22:42:07.321 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.321 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.321 DEBUG:boto:Params: {} 2023-09-29T22:42:07.321 DEBUG:boto:Token: None 2023-09-29T22:42:07.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.322 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM= 2023-09-29T22:42:07.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM=', 'Content-Length': '0'} 2023-09-29T22:42:07.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebc2fa8aec955125-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.370 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.371 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.371 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.371 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.371 DEBUG:boto:Method: GET 2023-09-29T22:42:07.371 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.371 DEBUG:boto:Data: 2023-09-29T22:42:07.371 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.372 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.372 DEBUG:boto:Params: {} 2023-09-29T22:42:07.372 DEBUG:boto:Token: None 2023-09-29T22:42:07.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:42:07.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM= 2023-09-29T22:42:07.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:teI0QIN1bP1Ms4xHIm2AUd5SBbM=', 'Content-Length': '0'} 2023-09-29T22:42:07.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5201f492b4e1caa-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.375 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.376 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.376 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.376 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.376 DEBUG:boto:Method: GET 2023-09-29T22:42:07.376 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.376 DEBUG:boto:Data: 2023-09-29T22:42:07.376 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.376 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.377 DEBUG:boto:Params: {} 2023-09-29T22:42:07.377 DEBUG:boto:Token: None 2023-09-29T22:42:07.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dada159edc318b3-00651752bf-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.382 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.382 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.382 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.383 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.383 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.383 DEBUG:boto:Method: GET 2023-09-29T22:42:07.383 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.383 DEBUG:boto:Data: 2023-09-29T22:42:07.383 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.383 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.383 DEBUG:boto:Params: {} 2023-09-29T22:42:07.384 DEBUG:boto:Token: None 2023-09-29T22:42:07.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002c55ec0cc8a29a2-00651752bf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.390 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.390 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.390 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.390 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.390 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:42:07.391 DEBUG:boto:Data: 2023-09-29T22:42:07.391 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.391 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.391 DEBUG:boto:Params: {} 2023-09-29T22:42:07.391 DEBUG:boto:Token: None 2023-09-29T22:42:07.391 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95 2023-09-29T22:42:07.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VIAAW+vs1IblFJty346o8MQOdu0= 2023-09-29T22:42:07.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VIAAW+vs1IblFJty346o8MQOdu0=', 'Content-Length': '0'} 2023-09-29T22:42:07.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d171d31fc9f3ce93-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.394 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.394 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.394 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.395 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:42:07.395 DEBUG:boto:Data: 2023-09-29T22:42:07.395 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.395 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.395 DEBUG:boto:Params: {} 2023-09-29T22:42:07.395 DEBUG:boto:Token: None 2023-09-29T22:42:07.395 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95 2023-09-29T22:42:07.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VIAAW+vs1IblFJty346o8MQOdu0= 2023-09-29T22:42:07.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VIAAW+vs1IblFJty346o8MQOdu0=', 'Content-Length': '0'} 2023-09-29T22:42:07.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006547a994f27ff433-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:42:07.398 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.398 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.398 DEBUG:boto:Method: GET 2023-09-29T22:42:07.398 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:42:07.399 DEBUG:boto:Data: 2023-09-29T22:42:07.399 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.399 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.399 DEBUG:boto:Params: {} 2023-09-29T22:42:07.399 DEBUG:boto:Token: None 2023-09-29T22:42:07.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95 2023-09-29T22:42:07.399 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:49qLYGQlQsfTOgs8JK1oe0b0zfE= 2023-09-29T22:42:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:49qLYGQlQsfTOgs8JK1oe0b0zfE=', 'Content-Length': '0'} 2023-09-29T22:42:07.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087d2e6e12f1e36e6-00651752bf-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.402 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:42:07.402 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:42:07.403 DEBUG:boto:Method: GET 2023-09-29T22:42:07.403 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:42:07.403 DEBUG:boto:Data: 2023-09-29T22:42:07.403 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.403 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.403 DEBUG:boto:Params: {} 2023-09-29T22:42:07.403 DEBUG:boto:Token: None 2023-09-29T22:42:07.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj95 2023-09-29T22:42:07.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:49qLYGQlQsfTOgs8JK1oe0b0zfE= 2023-09-29T22:42:07.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:49qLYGQlQsfTOgs8JK1oe0b0zfE=', 'Content-Length': '0'} 2023-09-29T22:42:07.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000461fb5a7ff20c916-00651752bf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.406 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.407 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.407 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.407 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.407 DEBUG:boto:Method: GET 2023-09-29T22:42:07.407 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.407 DEBUG:boto:Data: 2023-09-29T22:42:07.407 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:42:07.407 DEBUG:boto:Port: 8001 2023-09-29T22:42:07.407 DEBUG:boto:Params: {} 2023-09-29T22:42:07.408 DEBUG:boto:Token: None 2023-09-29T22:42:07.408 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.412 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ea489166db881dc-00651752bf-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.412 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.413 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.413 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.413 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.413 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.413 DEBUG:boto:Method: GET 2023-09-29T22:42:07.413 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:42:07.413 DEBUG:boto:Data: 2023-09-29T22:42:07.414 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.414 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.414 DEBUG:boto:Params: {} 2023-09-29T22:42:07.414 DEBUG:boto:Token: None 2023-09-29T22:42:07.414 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M= 2023-09-29T22:42:07.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdxVcmPzQSUTKBTUr+Du411Fu6M=', 'Content-Length': '0'} 2023-09-29T22:42:07.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d5d04db96885c8-00651752bf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.466 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:42:07.467 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:42:07.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:42:07.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:42:07.467 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:42:07.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:07.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:07.599 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:42:07.599 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-59 2023-09-29T22:42:07.613 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-09-29T22:42:07.706 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.707 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.707 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.707 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:07.707 DEBUG:boto:Data: 2023-09-29T22:42:07.707 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.707 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.707 DEBUG:boto:Params: {} 2023-09-29T22:42:07.708 DEBUG:boto:Token: None 2023-09-29T22:42:07.708 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc= 2023-09-29T22:42:07.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc=', 'Content-Length': '0'} 2023-09-29T22:42:07.713 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000209560615b8023c1-00651752bf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.713 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:42:07.714 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:42:07.714 DEBUG:boto:Method: PUT 2023-09-29T22:42:07.714 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:42:07.714 DEBUG:boto:Data: 2023-09-29T22:42:07.714 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:42:07.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.714 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.714 DEBUG:boto:Params: {} 2023-09-29T22:42:07.715 DEBUG:boto:Token: None 2023-09-29T22:42:07.715 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj23 2023-09-29T22:42:07.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eYB1kcIewecPvKH6/6fJWNpMeI0= 2023-09-29T22:42:07.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eYB1kcIewecPvKH6/6fJWNpMeI0='} 2023-09-29T22:42:07.725 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ff470cc826182371-00651752bf-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.725 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.725 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.725 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.725 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:07.725 DEBUG:boto:Data: 2023-09-29T22:42:07.725 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.726 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.726 DEBUG:boto:Params: {} 2023-09-29T22:42:07.726 DEBUG:boto:Token: None 2023-09-29T22:42:07.726 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc= 2023-09-29T22:42:07.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc=', 'Content-Length': '0'} 2023-09-29T22:42:07.731 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000141eb5b307db7507-00651752bf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.731 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:42:07.731 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:42:07.731 DEBUG:boto:Method: PUT 2023-09-29T22:42:07.731 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:42:07.732 DEBUG:boto:Data: 2023-09-29T22:42:07.732 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:42:07.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.732 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.732 DEBUG:boto:Params: {} 2023-09-29T22:42:07.732 DEBUG:boto:Token: None 2023-09-29T22:42:07.732 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj46 2023-09-29T22:42:07.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2al3hzVZ+yVedz2fg066FIl18to= 2023-09-29T22:42:07.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2al3hzVZ+yVedz2fg066FIl18to='} 2023-09-29T22:42:07.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000124a47824c10c7a2-00651752bf-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.775 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.775 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.775 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.775 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:07.775 DEBUG:boto:Data: 2023-09-29T22:42:07.776 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.776 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.776 DEBUG:boto:Params: {} 2023-09-29T22:42:07.776 DEBUG:boto:Token: None 2023-09-29T22:42:07.776 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc= 2023-09-29T22:42:07.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc=', 'Content-Length': '0'} 2023-09-29T22:42:07.786 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b721ede8cc8fc963-00651752bf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.787 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:42:07.787 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:42:07.787 DEBUG:boto:Method: PUT 2023-09-29T22:42:07.787 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:42:07.787 DEBUG:boto:Data: 2023-09-29T22:42:07.787 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:42:07.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.788 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.788 DEBUG:boto:Params: {} 2023-09-29T22:42:07.788 DEBUG:boto:Token: None 2023-09-29T22:42:07.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj69 2023-09-29T22:42:07.789 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:16ud/rnJwdmEdJyNkxO3m/FO7cw= 2023-09-29T22:42:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:16ud/rnJwdmEdJyNkxO3m/FO7cw='} 2023-09-29T22:42:07.800 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009ced0fbd2b96f1c9-00651752bf-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.801 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.801 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.801 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.801 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:07.801 DEBUG:boto:Data: 2023-09-29T22:42:07.801 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.801 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.801 DEBUG:boto:Params: {} 2023-09-29T22:42:07.802 DEBUG:boto:Token: None 2023-09-29T22:42:07.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc= 2023-09-29T22:42:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc=', 'Content-Length': '0'} 2023-09-29T22:42:07.807 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000263bb6841ba8f8ab-00651752bf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.808 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:42:07.808 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:42:07.808 DEBUG:boto:Method: PUT 2023-09-29T22:42:07.808 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:42:07.808 DEBUG:boto:Data: 2023-09-29T22:42:07.808 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:42:07.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.808 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.809 DEBUG:boto:Params: {} 2023-09-29T22:42:07.809 DEBUG:boto:Token: None 2023-09-29T22:42:07.809 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj92 2023-09-29T22:42:07.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZVW+vKjy/aNt7ucyT7yZKiZIinI= 2023-09-29T22:42:07.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZVW+vKjy/aNt7ucyT7yZKiZIinI='} 2023-09-29T22:42:07.858 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000841a72e93874c565-00651752bf-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.859 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:42:07.859 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:42:07.859 DEBUG:boto:Method: HEAD 2023-09-29T22:42:07.859 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:42:07.859 DEBUG:boto:Data: 2023-09-29T22:42:07.859 DEBUG:boto:Headers: {} 2023-09-29T22:42:07.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.860 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.860 DEBUG:boto:Params: {} 2023-09-29T22:42:07.860 DEBUG:boto:Token: None 2023-09-29T22:42:07.860 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/ 2023-09-29T22:42:07.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc= 2023-09-29T22:42:07.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPTshBLHMj+sGOqjNczx1L1Kkuc=', 'Content-Length': '0'} 2023-09-29T22:42:07.866 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047ea89cbce5047af-00651752bf-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.866 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:42:07.866 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:42:07.867 DEBUG:boto:Method: PUT 2023-09-29T22:42:07.867 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:42:07.867 DEBUG:boto:Data: 2023-09-29T22:42:07.867 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:42:07.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:42:07.867 DEBUG:boto:Port: 8000 2023-09-29T22:42:07.867 DEBUG:boto:Params: {} 2023-09-29T22:42:07.867 DEBUG:boto:Token: None 2023-09-29T22:42:07.868 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:42:07 GMT /hcnphz-59/obj115 2023-09-29T22:42:07.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vuQDBKZrnnYd77+xNHpYVCvXzD4= 2023-09-29T22:42:07.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:42:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vuQDBKZrnnYd77+xNHpYVCvXzD4='} 2023-09-29T22:42:07.878 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2e034feaff0f5db-00651752bf-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:42:07.879 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:42:07.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:42:07.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:42:07.879 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:42:07.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:42:07.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:42:08.000 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:08.000+0000 7f84ad3aaf40 1 waiting to reach latest gen 3, on 2.. 2023-09-29T22:42:10.170 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:42:10.166+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:42:13.001 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:13.000+0000 7f84ad3aaf40 1 waiting to reach latest gen 3, on 2.. 2023-09-29T22:42:18.002 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:17.999+0000 7f84ad3aaf40 1 waiting to reach latest gen 3, on 2.. 2023-09-29T22:42:23.007 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:23.003+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:23.008 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:42:23.008 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:28.011 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:28.007+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:28.011 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:42:28.011 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:33.014 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:33.011+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:33.015 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:42:33.015 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:38.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:38.015+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:38.018 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:42:38.019 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:43.022 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:43.019+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:43.076 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:42:43.076 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:48.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:48.023+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:48.025 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:42:48.025 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:53.029 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:53.026+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:53.029 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:42:53.029 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:42:58.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:42:58.030+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:42:58.033 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:42:58.033 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:03.036 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:03.034+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:03.036 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:43:03.037 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:08.040 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:08.038+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:08.040 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:43:08.040 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:13.043 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:13.038+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:13.044 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:43:13.044 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:18.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:18.042+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:18.047 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:43:18.047 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:23.051 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:23.046+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:23.051 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:43:23.051 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:28.054 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:28.050+0000 7f84ad3aaf40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:28.054 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , 00000000002.68.6, , , , , , , , 00000000001.105.6, , , , ] 2023-09-29T22:43:28.055 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:33.053+0000 7f84ad3aaf40 1 bucket sync caught up with source: 2023-09-29T22:43:33.058 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:33.058 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.59.6, , , 00000000002.76.6, , 00000000002.68.6, , , , , , 00000000001.41.6, , 00000000001.105.6, , , , ] 2023-09-29T22:43:33.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:33.053+0000 7f84ad3aaf40 0 bucket checkpoint complete 2023-09-29T22:43:33.086 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:43:33.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:43:33.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:43:33.087 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:43:33.139 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:33.139 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:33.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:33.201+0000 7f9de9d55f40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:33.204 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000001.580.6, 00000000001.228.6, 00000000002.189.6, 00000000001.222.6, , , 00000000002.270.6, 00000000004.183.6, 00000000004.316.6] 2023-09-29T22:43:33.204 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000002.189.6, 00000000002.234.6, , , 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:43:38.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:38.205+0000 7f9de9d55f40 1 waiting for incremental sync to catch up: 2023-09-29T22:43:38.209 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000001.580.6, 00000000001.228.6, 00000000002.189.6, 00000000001.222.6, , , 00000000002.270.6, 00000000004.183.6, 00000000004.316.6] 2023-09-29T22:43:38.209 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000002.189.6, 00000000002.234.6, , , 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:43:43.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:43.205+0000 7f9de9d55f40 1 bucket sync caught up with source: 2023-09-29T22:43:43.214 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000002.189.6, 00000000002.234.6, , , 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:43:43.214 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000002.189.6, 00000000002.234.6, , , 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:43:43.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:43.205+0000 7f9de9d55f40 0 bucket checkpoint complete 2023-09-29T22:43:43.239 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:43:43.239 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.240 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.240 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.240 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:43.240 DEBUG:boto:Data: 2023-09-29T22:43:43.240 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.240 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.240 DEBUG:boto:Params: {} 2023-09-29T22:43:43.241 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:43:43.241 DEBUG:boto:Token: None 2023-09-29T22:43:43.241 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2i4WHd630QtfH+XsS0Golj1sxao= 2023-09-29T22:43:43.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2i4WHd630QtfH+XsS0Golj1sxao=', 'Content-Length': '0'} 2023-09-29T22:43:43.247 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e8c451143e406cac-006517531f-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.248 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.248 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.248 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.248 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:43.248 DEBUG:boto:Data: 2023-09-29T22:43:43.248 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.249 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.249 DEBUG:boto:Params: {} 2023-09-29T22:43:43.249 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:43:43.249 DEBUG:boto:Token: None 2023-09-29T22:43:43.249 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2i4WHd630QtfH+XsS0Golj1sxao= 2023-09-29T22:43:43.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2i4WHd630QtfH+XsS0Golj1sxao=', 'Content-Length': '0'} 2023-09-29T22:43:43.258 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012fecca5e0606f1e-006517531f-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.259 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:43:43.259 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.259 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.259 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.259 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.259 DEBUG:boto:Method: GET 2023-09-29T22:43:43.259 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.260 DEBUG:boto:Data: 2023-09-29T22:43:43.260 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.260 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.260 DEBUG:boto:Params: {} 2023-09-29T22:43:43.260 DEBUG:boto:Token: None 2023-09-29T22:43:43.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/?versions 2023-09-29T22:43:43.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE= 2023-09-29T22:43:43.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE=', 'Content-Length': '0'} 2023-09-29T22:43:43.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e5145b8ba2ce7c2-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.265 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:43:43.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:43:43.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:43:43.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:43:43.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:43:43.269 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.269 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.269 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.269 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.269 DEBUG:boto:Method: GET 2023-09-29T22:43:43.269 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.270 DEBUG:boto:Data: 2023-09-29T22:43:43.270 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.270 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.270 DEBUG:boto:Params: {} 2023-09-29T22:43:43.270 DEBUG:boto:Token: None 2023-09-29T22:43:43.270 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/?versions 2023-09-29T22:43:43.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE= 2023-09-29T22:43:43.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE=', 'Content-Length': '0'} 2023-09-29T22:43:43.276 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035b54cb2cca87f21-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.276 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:43:43.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:43:43.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:43:43.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:43:43.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:43:43.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:43:43.279 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.280 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.280 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.280 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.280 DEBUG:boto:Method: GET 2023-09-29T22:43:43.280 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.280 DEBUG:boto:Data: 2023-09-29T22:43:43.280 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.280 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.281 DEBUG:boto:Params: {} 2023-09-29T22:43:43.281 DEBUG:boto:Token: None 2023-09-29T22:43:43.281 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/?versions 2023-09-29T22:43:43.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE= 2023-09-29T22:43:43.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE=', 'Content-Length': '0'} 2023-09-29T22:43:43.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c78a20375c90b8d-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.286 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.287 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.287 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.288 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.288 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.288 DEBUG:boto:Method: GET 2023-09-29T22:43:43.288 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:43:43.288 DEBUG:boto:Data: 2023-09-29T22:43:43.288 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.288 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.288 DEBUG:boto:Params: {} 2023-09-29T22:43:43.289 DEBUG:boto:Token: None 2023-09-29T22:43:43.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/?versions 2023-09-29T22:43:43.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE= 2023-09-29T22:43:43.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WV7i8UJBLrG6VikjFGXSapEj7UE=', 'Content-Length': '0'} 2023-09-29T22:43:43.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d185398c3cf8ef84-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.294 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:43:43.296 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.296 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.296 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.296 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.296 DEBUG:boto:Method: GET 2023-09-29T22:43:43.296 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.297 DEBUG:boto:Data: 2023-09-29T22:43:43.297 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.297 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.297 DEBUG:boto:Params: {} 2023-09-29T22:43:43.297 DEBUG:boto:Token: None 2023-09-29T22:43:43.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM= 2023-09-29T22:43:43.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM=', 'Content-Length': '0'} 2023-09-29T22:43:43.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f80e587b0b763b0-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.342 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.343 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.343 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.343 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.343 DEBUG:boto:Method: GET 2023-09-29T22:43:43.343 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.343 DEBUG:boto:Data: 2023-09-29T22:43:43.343 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.343 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.344 DEBUG:boto:Params: {} 2023-09-29T22:43:43.344 DEBUG:boto:Token: None 2023-09-29T22:43:43.344 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM= 2023-09-29T22:43:43.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM=', 'Content-Length': '0'} 2023-09-29T22:43:43.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0b65918645113a0-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.347 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.347 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.347 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.348 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.348 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.348 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.348 DEBUG:boto:Data: 2023-09-29T22:43:43.348 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.348 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.348 DEBUG:boto:Params: {} 2023-09-29T22:43:43.348 DEBUG:boto:Token: None 2023-09-29T22:43:43.349 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1hWcv2V3/RJ17zXfKQCs4IrGVU= 2023-09-29T22:43:43.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1hWcv2V3/RJ17zXfKQCs4IrGVU=', 'Content-Length': '0'} 2023-09-29T22:43:43.351 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009037dfa2c7996d97-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.352 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.352 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.352 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.352 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.352 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.352 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.352 DEBUG:boto:Data: 2023-09-29T22:43:43.352 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.353 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.353 DEBUG:boto:Params: {} 2023-09-29T22:43:43.353 DEBUG:boto:Token: None 2023-09-29T22:43:43.353 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.353 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1hWcv2V3/RJ17zXfKQCs4IrGVU= 2023-09-29T22:43:43.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1hWcv2V3/RJ17zXfKQCs4IrGVU=', 'Content-Length': '0'} 2023-09-29T22:43:43.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065358439a423552d-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:43:43.356 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.356 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.356 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.357 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.357 DEBUG:boto:Method: GET 2023-09-29T22:43:43.357 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.357 DEBUG:boto:Data: 2023-09-29T22:43:43.357 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.357 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.357 DEBUG:boto:Params: {} 2023-09-29T22:43:43.357 DEBUG:boto:Token: None 2023-09-29T22:43:43.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM= 2023-09-29T22:43:43.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM=', 'Content-Length': '0'} 2023-09-29T22:43:43.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccfd47725cc9a15a-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.361 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.361 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.361 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.361 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.361 DEBUG:boto:Method: GET 2023-09-29T22:43:43.361 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.361 DEBUG:boto:Data: 2023-09-29T22:43:43.362 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.362 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.362 DEBUG:boto:Params: {} 2023-09-29T22:43:43.362 DEBUG:boto:Token: None 2023-09-29T22:43:43.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:43:43.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM= 2023-09-29T22:43:43.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NdEOpy/GkW0GM4DiYmUF8U69AAM=', 'Content-Length': '0'} 2023-09-29T22:43:43.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fec49f2c426744a-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.406 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.407 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.407 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.407 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.408 DEBUG:boto:Method: GET 2023-09-29T22:43:43.408 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.408 DEBUG:boto:Data: 2023-09-29T22:43:43.408 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.409 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.409 DEBUG:boto:Params: {} 2023-09-29T22:43:43.409 DEBUG:boto:Token: None 2023-09-29T22:43:43.410 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001704403a5418d9ab-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.418 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.419 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.419 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.419 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.419 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.420 DEBUG:boto:Method: GET 2023-09-29T22:43:43.420 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.420 DEBUG:boto:Data: 2023-09-29T22:43:43.420 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.420 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.420 DEBUG:boto:Params: {} 2023-09-29T22:43:43.420 DEBUG:boto:Token: None 2023-09-29T22:43:43.421 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.421 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.428 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ff3c8795c0e71a2-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.428 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.428 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.429 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.429 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.429 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:43:43.429 DEBUG:boto:Data: 2023-09-29T22:43:43.429 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.429 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.429 DEBUG:boto:Params: {} 2023-09-29T22:43:43.429 DEBUG:boto:Token: None 2023-09-29T22:43:43.430 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11 2023-09-29T22:43:43.430 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r2JPlzxOsiUOaAQ5YTp/WWqUvOM= 2023-09-29T22:43:43.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r2JPlzxOsiUOaAQ5YTp/WWqUvOM=', 'Content-Length': '0'} 2023-09-29T22:43:43.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b504e31ed3a54ad4-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.432 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.433 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.433 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.433 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:43:43.433 DEBUG:boto:Data: 2023-09-29T22:43:43.433 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.433 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.433 DEBUG:boto:Params: {} 2023-09-29T22:43:43.434 DEBUG:boto:Token: None 2023-09-29T22:43:43.434 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11 2023-09-29T22:43:43.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r2JPlzxOsiUOaAQ5YTp/WWqUvOM= 2023-09-29T22:43:43.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r2JPlzxOsiUOaAQ5YTp/WWqUvOM=', 'Content-Length': '0'} 2023-09-29T22:43:43.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081165d37c22ca87e-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:43:43.437 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.437 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.437 DEBUG:boto:Method: GET 2023-09-29T22:43:43.437 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:43:43.437 DEBUG:boto:Data: 2023-09-29T22:43:43.437 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.437 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.437 DEBUG:boto:Params: {} 2023-09-29T22:43:43.437 DEBUG:boto:Token: None 2023-09-29T22:43:43.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11 2023-09-29T22:43:43.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ex+vHJlGYK4WlyqpKkepPr/rw74= 2023-09-29T22:43:43.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ex+vHJlGYK4WlyqpKkepPr/rw74=', 'Content-Length': '0'} 2023-09-29T22:43:43.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f68b7fcaa6c0800-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.486 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:43:43.487 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:43:43.487 DEBUG:boto:Method: GET 2023-09-29T22:43:43.487 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:43:43.487 DEBUG:boto:Data: 2023-09-29T22:43:43.487 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.487 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.487 DEBUG:boto:Params: {} 2023-09-29T22:43:43.488 DEBUG:boto:Token: None 2023-09-29T22:43:43.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj11 2023-09-29T22:43:43.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ex+vHJlGYK4WlyqpKkepPr/rw74= 2023-09-29T22:43:43.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ex+vHJlGYK4WlyqpKkepPr/rw74=', 'Content-Length': '0'} 2023-09-29T22:43:43.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b94b3a438e922d43-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.491 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.492 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.492 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.492 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.492 DEBUG:boto:Method: GET 2023-09-29T22:43:43.492 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.492 DEBUG:boto:Data: 2023-09-29T22:43:43.492 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.492 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.492 DEBUG:boto:Params: {} 2023-09-29T22:43:43.493 DEBUG:boto:Token: None 2023-09-29T22:43:43.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001dcd570730e45ee3-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.498 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.499 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.499 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.499 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.499 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.499 DEBUG:boto:Method: GET 2023-09-29T22:43:43.499 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:43:43.499 DEBUG:boto:Data: 2023-09-29T22:43:43.499 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.500 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.500 DEBUG:boto:Params: {} 2023-09-29T22:43:43.500 DEBUG:boto:Token: None 2023-09-29T22:43:43.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005082d580b913d1d2-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.550 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:43:43.551 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.551 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.551 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.552 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.552 DEBUG:boto:Method: GET 2023-09-29T22:43:43.552 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.552 DEBUG:boto:Data: 2023-09-29T22:43:43.552 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.552 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.553 DEBUG:boto:Params: {} 2023-09-29T22:43:43.553 DEBUG:boto:Token: None 2023-09-29T22:43:43.553 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM= 2023-09-29T22:43:43.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM=', 'Content-Length': '0'} 2023-09-29T22:43:43.556 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb477fd839198b5f-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.556 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.557 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.557 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.557 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.557 DEBUG:boto:Method: GET 2023-09-29T22:43:43.557 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.557 DEBUG:boto:Data: 2023-09-29T22:43:43.557 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.557 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.557 DEBUG:boto:Params: {} 2023-09-29T22:43:43.558 DEBUG:boto:Token: None 2023-09-29T22:43:43.558 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.558 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM= 2023-09-29T22:43:43.558 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM=', 'Content-Length': '0'} 2023-09-29T22:43:43.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8b86cc564fbdff7-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.561 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.561 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.561 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.561 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.561 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.561 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.562 DEBUG:boto:Data: 2023-09-29T22:43:43.562 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.562 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.562 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.562 DEBUG:boto:Params: {} 2023-09-29T22:43:43.562 DEBUG:boto:Token: None 2023-09-29T22:43:43.562 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wekcycVjEqNqQfY4wzyH919x6Nc= 2023-09-29T22:43:43.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wekcycVjEqNqQfY4wzyH919x6Nc=', 'Content-Length': '0'} 2023-09-29T22:43:43.565 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005053f5b1af0032cc-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.565 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.565 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.565 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.566 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.566 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.566 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.566 DEBUG:boto:Data: 2023-09-29T22:43:43.566 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.566 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.566 DEBUG:boto:Params: {} 2023-09-29T22:43:43.566 DEBUG:boto:Token: None 2023-09-29T22:43:43.567 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wekcycVjEqNqQfY4wzyH919x6Nc= 2023-09-29T22:43:43.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wekcycVjEqNqQfY4wzyH919x6Nc=', 'Content-Length': '0'} 2023-09-29T22:43:43.569 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088103db22446428f-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:43:43.570 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.570 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.570 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.570 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.570 DEBUG:boto:Method: GET 2023-09-29T22:43:43.570 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.570 DEBUG:boto:Data: 2023-09-29T22:43:43.571 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.571 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.571 DEBUG:boto:Params: {} 2023-09-29T22:43:43.571 DEBUG:boto:Token: None 2023-09-29T22:43:43.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM= 2023-09-29T22:43:43.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM=', 'Content-Length': '0'} 2023-09-29T22:43:43.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041fdeef784b10454-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.574 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.574 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.574 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.575 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.575 DEBUG:boto:Method: GET 2023-09-29T22:43:43.575 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.575 DEBUG:boto:Data: 2023-09-29T22:43:43.575 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.575 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.575 DEBUG:boto:Params: {} 2023-09-29T22:43:43.575 DEBUG:boto:Token: None 2023-09-29T22:43:43.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:43:43.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM= 2023-09-29T22:43:43.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P6syOgUNPF3koF1upqesq3iAmDM=', 'Content-Length': '0'} 2023-09-29T22:43:43.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034a02a0babd0c319-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.578 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.579 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.579 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.579 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.579 DEBUG:boto:Method: GET 2023-09-29T22:43:43.579 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.579 DEBUG:boto:Data: 2023-09-29T22:43:43.579 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.579 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.580 DEBUG:boto:Params: {} 2023-09-29T22:43:43.580 DEBUG:boto:Token: None 2023-09-29T22:43:43.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590c522d8b9e6b96-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.585 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.585 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.586 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.586 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.586 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.586 DEBUG:boto:Method: GET 2023-09-29T22:43:43.586 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.586 DEBUG:boto:Data: 2023-09-29T22:43:43.586 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.587 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.587 DEBUG:boto:Params: {} 2023-09-29T22:43:43.587 DEBUG:boto:Token: None 2023-09-29T22:43:43.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.593 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2f8afe0bde662ee-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.593 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.594 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.594 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.594 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.594 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:43:43.594 DEBUG:boto:Data: 2023-09-29T22:43:43.595 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.595 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.595 DEBUG:boto:Params: {} 2023-09-29T22:43:43.595 DEBUG:boto:Token: None 2023-09-29T22:43:43.595 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115 2023-09-29T22:43:43.595 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9kKYkpl8L5if5Ozwi2JLQl0Ftro= 2023-09-29T22:43:43.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9kKYkpl8L5if5Ozwi2JLQl0Ftro=', 'Content-Length': '0'} 2023-09-29T22:43:43.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d6aad2b3774c7e7-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.598 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.598 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.598 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.598 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:43:43.598 DEBUG:boto:Data: 2023-09-29T22:43:43.598 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.599 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.599 DEBUG:boto:Params: {} 2023-09-29T22:43:43.599 DEBUG:boto:Token: None 2023-09-29T22:43:43.599 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115 2023-09-29T22:43:43.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9kKYkpl8L5if5Ozwi2JLQl0Ftro= 2023-09-29T22:43:43.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9kKYkpl8L5if5Ozwi2JLQl0Ftro=', 'Content-Length': '0'} 2023-09-29T22:43:43.601 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f43d5e1380e4f8cc-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:43:43.602 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.602 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.602 DEBUG:boto:Method: GET 2023-09-29T22:43:43.602 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:43:43.603 DEBUG:boto:Data: 2023-09-29T22:43:43.603 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.603 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.603 DEBUG:boto:Params: {} 2023-09-29T22:43:43.603 DEBUG:boto:Token: None 2023-09-29T22:43:43.603 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115 2023-09-29T22:43:43.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d4RNtz3/toM4d7+75hgpBcgblYQ= 2023-09-29T22:43:43.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d4RNtz3/toM4d7+75hgpBcgblYQ=', 'Content-Length': '0'} 2023-09-29T22:43:43.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3537e04efa2761c-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.646 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:43:43.647 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:43:43.647 DEBUG:boto:Method: GET 2023-09-29T22:43:43.647 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:43:43.647 DEBUG:boto:Data: 2023-09-29T22:43:43.647 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.647 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.647 DEBUG:boto:Params: {} 2023-09-29T22:43:43.647 DEBUG:boto:Token: None 2023-09-29T22:43:43.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj115 2023-09-29T22:43:43.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d4RNtz3/toM4d7+75hgpBcgblYQ= 2023-09-29T22:43:43.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d4RNtz3/toM4d7+75hgpBcgblYQ=', 'Content-Length': '0'} 2023-09-29T22:43:43.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078fe8331b6b646ce-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.651 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.651 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.651 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.651 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.651 DEBUG:boto:Method: GET 2023-09-29T22:43:43.652 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.652 DEBUG:boto:Data: 2023-09-29T22:43:43.652 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.652 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.652 DEBUG:boto:Params: {} 2023-09-29T22:43:43.652 DEBUG:boto:Token: None 2023-09-29T22:43:43.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077f3ce5b6f143c3b-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.658 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.658 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.659 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.659 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.659 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.659 DEBUG:boto:Method: GET 2023-09-29T22:43:43.659 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:43:43.659 DEBUG:boto:Data: 2023-09-29T22:43:43.659 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.659 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.659 DEBUG:boto:Params: {} 2023-09-29T22:43:43.660 DEBUG:boto:Token: None 2023-09-29T22:43:43.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.667 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2ca9654397f2473-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.667 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:43:43.668 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.668 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.668 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.668 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.668 DEBUG:boto:Method: GET 2023-09-29T22:43:43.668 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.669 DEBUG:boto:Data: 2023-09-29T22:43:43.669 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.669 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.669 DEBUG:boto:Params: {} 2023-09-29T22:43:43.669 DEBUG:boto:Token: None 2023-09-29T22:43:43.669 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU= 2023-09-29T22:43:43.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU=', 'Content-Length': '0'} 2023-09-29T22:43:43.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b10dc044b75f6419-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.672 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.672 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.672 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.673 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.673 DEBUG:boto:Method: GET 2023-09-29T22:43:43.673 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.673 DEBUG:boto:Data: 2023-09-29T22:43:43.673 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.673 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.673 DEBUG:boto:Params: {} 2023-09-29T22:43:43.673 DEBUG:boto:Token: None 2023-09-29T22:43:43.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU= 2023-09-29T22:43:43.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU=', 'Content-Length': '0'} 2023-09-29T22:43:43.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000184a10adebf236dc-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.676 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.676 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.676 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.677 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.677 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.677 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.677 DEBUG:boto:Data: 2023-09-29T22:43:43.677 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.677 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.677 DEBUG:boto:Params: {} 2023-09-29T22:43:43.677 DEBUG:boto:Token: None 2023-09-29T22:43:43.678 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.678 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F3mMXH99dWkBhKBA2RMG05C9H78= 2023-09-29T22:43:43.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F3mMXH99dWkBhKBA2RMG05C9H78=', 'Content-Length': '0'} 2023-09-29T22:43:43.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021604e436de38de5-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.680 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.680 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.680 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.681 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.681 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.681 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.681 DEBUG:boto:Data: 2023-09-29T22:43:43.681 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.681 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.681 DEBUG:boto:Params: {} 2023-09-29T22:43:43.681 DEBUG:boto:Token: None 2023-09-29T22:43:43.682 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.682 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:F3mMXH99dWkBhKBA2RMG05C9H78= 2023-09-29T22:43:43.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:F3mMXH99dWkBhKBA2RMG05C9H78=', 'Content-Length': '0'} 2023-09-29T22:43:43.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000725d3dc3c590e435-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:43:43.685 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.685 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.685 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.685 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.685 DEBUG:boto:Method: GET 2023-09-29T22:43:43.685 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.685 DEBUG:boto:Data: 2023-09-29T22:43:43.685 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.686 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.686 DEBUG:boto:Params: {} 2023-09-29T22:43:43.686 DEBUG:boto:Token: None 2023-09-29T22:43:43.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU= 2023-09-29T22:43:43.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU=', 'Content-Length': '0'} 2023-09-29T22:43:43.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb2099b2461698d3-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.689 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.689 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.689 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.689 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.690 DEBUG:boto:Method: GET 2023-09-29T22:43:43.690 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.690 DEBUG:boto:Data: 2023-09-29T22:43:43.690 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.690 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.690 DEBUG:boto:Params: {} 2023-09-29T22:43:43.690 DEBUG:boto:Token: None 2023-09-29T22:43:43.690 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:43:43.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU= 2023-09-29T22:43:43.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AlW+QUwtpo3SFpXCVz2L7klbtJU=', 'Content-Length': '0'} 2023-09-29T22:43:43.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6d3bf993443b798-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.693 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.693 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.694 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.694 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.694 DEBUG:boto:Method: GET 2023-09-29T22:43:43.694 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.694 DEBUG:boto:Data: 2023-09-29T22:43:43.694 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.694 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.694 DEBUG:boto:Params: {} 2023-09-29T22:43:43.695 DEBUG:boto:Token: None 2023-09-29T22:43:43.695 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.695 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba603f1520205e8-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.700 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.700 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.700 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.700 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.700 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.701 DEBUG:boto:Method: GET 2023-09-29T22:43:43.701 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.701 DEBUG:boto:Data: 2023-09-29T22:43:43.701 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.701 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.701 DEBUG:boto:Params: {} 2023-09-29T22:43:43.701 DEBUG:boto:Token: None 2023-09-29T22:43:43.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c366502e569e1036-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.708 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.708 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.708 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.709 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.709 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:43:43.709 DEBUG:boto:Data: 2023-09-29T22:43:43.709 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.709 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.709 DEBUG:boto:Params: {} 2023-09-29T22:43:43.709 DEBUG:boto:Token: None 2023-09-29T22:43:43.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17 2023-09-29T22:43:43.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WKqigsI8AArOgBpdfIYNkT1+lmM= 2023-09-29T22:43:43.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WKqigsI8AArOgBpdfIYNkT1+lmM=', 'Content-Length': '0'} 2023-09-29T22:43:43.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000191186f63721821e-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.712 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.712 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.713 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.713 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:43:43.713 DEBUG:boto:Data: 2023-09-29T22:43:43.713 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.713 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.713 DEBUG:boto:Params: {} 2023-09-29T22:43:43.713 DEBUG:boto:Token: None 2023-09-29T22:43:43.713 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17 2023-09-29T22:43:43.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WKqigsI8AArOgBpdfIYNkT1+lmM= 2023-09-29T22:43:43.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WKqigsI8AArOgBpdfIYNkT1+lmM=', 'Content-Length': '0'} 2023-09-29T22:43:43.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003512f8a78cba50a7-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:43:43.716 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.716 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.716 DEBUG:boto:Method: GET 2023-09-29T22:43:43.717 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:43:43.717 DEBUG:boto:Data: 2023-09-29T22:43:43.717 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.717 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.717 DEBUG:boto:Params: {} 2023-09-29T22:43:43.717 DEBUG:boto:Token: None 2023-09-29T22:43:43.717 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17 2023-09-29T22:43:43.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZmOku2GygSypmJke2K8xCN6eGnw= 2023-09-29T22:43:43.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZmOku2GygSypmJke2K8xCN6eGnw=', 'Content-Length': '0'} 2023-09-29T22:43:43.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b120847cd7350602-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.762 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:43:43.763 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:43:43.763 DEBUG:boto:Method: GET 2023-09-29T22:43:43.763 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:43:43.763 DEBUG:boto:Data: 2023-09-29T22:43:43.763 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.763 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.763 DEBUG:boto:Params: {} 2023-09-29T22:43:43.764 DEBUG:boto:Token: None 2023-09-29T22:43:43.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj17 2023-09-29T22:43:43.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZmOku2GygSypmJke2K8xCN6eGnw= 2023-09-29T22:43:43.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZmOku2GygSypmJke2K8xCN6eGnw=', 'Content-Length': '0'} 2023-09-29T22:43:43.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d843cb33b497c75a-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.767 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.767 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.768 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.768 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.768 DEBUG:boto:Method: GET 2023-09-29T22:43:43.768 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.768 DEBUG:boto:Data: 2023-09-29T22:43:43.768 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.768 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.768 DEBUG:boto:Params: {} 2023-09-29T22:43:43.769 DEBUG:boto:Token: None 2023-09-29T22:43:43.769 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.769 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a189eaa444f899d-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.774 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.774 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.774 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.774 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.775 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.775 DEBUG:boto:Method: GET 2023-09-29T22:43:43.775 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:43:43.775 DEBUG:boto:Data: 2023-09-29T22:43:43.775 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.775 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.775 DEBUG:boto:Params: {} 2023-09-29T22:43:43.775 DEBUG:boto:Token: None 2023-09-29T22:43:43.775 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.782 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034a17e6d5a614d6f-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.783 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:43:43.783 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.783 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.783 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.783 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.784 DEBUG:boto:Method: GET 2023-09-29T22:43:43.784 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.784 DEBUG:boto:Data: 2023-09-29T22:43:43.784 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.784 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.784 DEBUG:boto:Params: {} 2023-09-29T22:43:43.784 DEBUG:boto:Token: None 2023-09-29T22:43:43.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.785 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ= 2023-09-29T22:43:43.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ=', 'Content-Length': '0'} 2023-09-29T22:43:43.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fd87196d6b16c8d-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.787 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.787 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.787 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.787 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.787 DEBUG:boto:Method: GET 2023-09-29T22:43:43.788 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.788 DEBUG:boto:Data: 2023-09-29T22:43:43.788 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.788 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.788 DEBUG:boto:Params: {} 2023-09-29T22:43:43.788 DEBUG:boto:Token: None 2023-09-29T22:43:43.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ= 2023-09-29T22:43:43.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ=', 'Content-Length': '0'} 2023-09-29T22:43:43.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c526fda171124fe6-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.791 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.791 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.791 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.791 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.791 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.792 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.792 DEBUG:boto:Data: 2023-09-29T22:43:43.792 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.792 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.792 DEBUG:boto:Params: {} 2023-09-29T22:43:43.792 DEBUG:boto:Token: None 2023-09-29T22:43:43.792 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EI6uTdurDdaUi3S4OP4MZNN8C6U= 2023-09-29T22:43:43.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EI6uTdurDdaUi3S4OP4MZNN8C6U=', 'Content-Length': '0'} 2023-09-29T22:43:43.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050da4c6194315236-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.795 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.795 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.795 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.795 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.795 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.795 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.796 DEBUG:boto:Data: 2023-09-29T22:43:43.796 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.796 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.796 DEBUG:boto:Params: {} 2023-09-29T22:43:43.796 DEBUG:boto:Token: None 2023-09-29T22:43:43.796 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EI6uTdurDdaUi3S4OP4MZNN8C6U= 2023-09-29T22:43:43.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EI6uTdurDdaUi3S4OP4MZNN8C6U=', 'Content-Length': '0'} 2023-09-29T22:43:43.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4e34e589acfbc68-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:43:43.799 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.799 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.799 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.800 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.800 DEBUG:boto:Method: GET 2023-09-29T22:43:43.800 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.800 DEBUG:boto:Data: 2023-09-29T22:43:43.800 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.800 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.800 DEBUG:boto:Params: {} 2023-09-29T22:43:43.800 DEBUG:boto:Token: None 2023-09-29T22:43:43.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ= 2023-09-29T22:43:43.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ=', 'Content-Length': '0'} 2023-09-29T22:43:43.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000998557239c25056c-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.850 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.851 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.851 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.851 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.851 DEBUG:boto:Method: GET 2023-09-29T22:43:43.851 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.851 DEBUG:boto:Data: 2023-09-29T22:43:43.851 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.851 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.851 DEBUG:boto:Params: {} 2023-09-29T22:43:43.852 DEBUG:boto:Token: None 2023-09-29T22:43:43.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:43:43.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ= 2023-09-29T22:43:43.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Cx+nwu103wdgonvymbd7S4rLZyQ=', 'Content-Length': '0'} 2023-09-29T22:43:43.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036b48e9d352c8d70-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.856 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.856 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.856 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.856 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.856 DEBUG:boto:Method: GET 2023-09-29T22:43:43.857 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.857 DEBUG:boto:Data: 2023-09-29T22:43:43.857 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.857 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.857 DEBUG:boto:Params: {} 2023-09-29T22:43:43.858 DEBUG:boto:Token: None 2023-09-29T22:43:43.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.858 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007860bf8a4dfd0141-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.864 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.864 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.864 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.865 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.865 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.865 DEBUG:boto:Method: GET 2023-09-29T22:43:43.865 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.865 DEBUG:boto:Data: 2023-09-29T22:43:43.865 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.866 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.866 DEBUG:boto:Params: {} 2023-09-29T22:43:43.866 DEBUG:boto:Token: None 2023-09-29T22:43:43.867 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.867 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.872 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c9d26b3b03bab08-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.873 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.873 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.873 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.874 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.874 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:43:43.874 DEBUG:boto:Data: 2023-09-29T22:43:43.874 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.874 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.875 DEBUG:boto:Params: {} 2023-09-29T22:43:43.875 DEBUG:boto:Token: None 2023-09-29T22:43:43.875 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19 2023-09-29T22:43:43.875 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DgXXuB/E3mCWlevNbkBa4WLipwA= 2023-09-29T22:43:43.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DgXXuB/E3mCWlevNbkBa4WLipwA=', 'Content-Length': '0'} 2023-09-29T22:43:43.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008a3db8858b457d2-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.878 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.878 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.878 DEBUG:boto:Method: HEAD 2023-09-29T22:43:43.879 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:43:43.879 DEBUG:boto:Data: 2023-09-29T22:43:43.879 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.879 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.879 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.879 DEBUG:boto:Params: {} 2023-09-29T22:43:43.880 DEBUG:boto:Token: None 2023-09-29T22:43:43.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19 2023-09-29T22:43:43.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DgXXuB/E3mCWlevNbkBa4WLipwA= 2023-09-29T22:43:43.880 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DgXXuB/E3mCWlevNbkBa4WLipwA=', 'Content-Length': '0'} 2023-09-29T22:43:43.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001decf2b0e6b26ff-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:43:43.883 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.883 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.884 DEBUG:boto:Method: GET 2023-09-29T22:43:43.884 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:43:43.884 DEBUG:boto:Data: 2023-09-29T22:43:43.884 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.884 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.885 DEBUG:boto:Params: {} 2023-09-29T22:43:43.885 DEBUG:boto:Token: None 2023-09-29T22:43:43.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19 2023-09-29T22:43:43.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:afpJAC5LMLRabCfbKuyB1jYWQ/M= 2023-09-29T22:43:43.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:afpJAC5LMLRabCfbKuyB1jYWQ/M=', 'Content-Length': '0'} 2023-09-29T22:43:43.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3882997b2a7e558-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.888 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:43:43.888 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:43:43.889 DEBUG:boto:Method: GET 2023-09-29T22:43:43.889 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:43:43.889 DEBUG:boto:Data: 2023-09-29T22:43:43.889 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.890 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.890 DEBUG:boto:Params: {} 2023-09-29T22:43:43.890 DEBUG:boto:Token: None 2023-09-29T22:43:43.890 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj19 2023-09-29T22:43:43.890 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:afpJAC5LMLRabCfbKuyB1jYWQ/M= 2023-09-29T22:43:43.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:afpJAC5LMLRabCfbKuyB1jYWQ/M=', 'Content-Length': '0'} 2023-09-29T22:43:43.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b3b1826bc7264dc-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.934 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.935 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.935 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.935 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.935 DEBUG:boto:Method: GET 2023-09-29T22:43:43.936 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.936 DEBUG:boto:Data: 2023-09-29T22:43:43.936 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.936 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.936 DEBUG:boto:Params: {} 2023-09-29T22:43:43.937 DEBUG:boto:Token: None 2023-09-29T22:43:43.937 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.937 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.942 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cce1c67041d20f04-006517531f-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.942 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.943 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:43.943 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:43.943 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.943 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.944 DEBUG:boto:Method: GET 2023-09-29T22:43:43.944 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:43:43.944 DEBUG:boto:Data: 2023-09-29T22:43:43.944 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.944 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:43.945 DEBUG:boto:Port: 8000 2023-09-29T22:43:43.945 DEBUG:boto:Params: {} 2023-09-29T22:43:43.945 DEBUG:boto:Token: None 2023-09-29T22:43:43.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/ 2023-09-29T22:43:43.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk= 2023-09-29T22:43:43.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6pfGEkzxjAWRa7UmiPNUv0Mpgjk=', 'Content-Length': '0'} 2023-09-29T22:43:43.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ae888849d138039b-006517531f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.952 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:43.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:43:43.952 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:43.952 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:43.953 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.953 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.953 DEBUG:boto:Method: GET 2023-09-29T22:43:43.953 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.954 DEBUG:boto:Data: 2023-09-29T22:43:43.954 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:43.954 DEBUG:boto:Port: 8001 2023-09-29T22:43:43.954 DEBUG:boto:Params: {} 2023-09-29T22:43:43.955 DEBUG:boto:Token: None 2023-09-29T22:43:43.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:43 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EPOn5y+ECrI3oZ2FjrxVWs4VgfE= 2023-09-29T22:43:43.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EPOn5y+ECrI3oZ2FjrxVWs4VgfE=', 'Content-Length': '0'} 2023-09-29T22:43:43.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b908ea718c8ea44b-006517531f-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:43.998 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:43.999 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:43.999 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.999 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.999 DEBUG:boto:Method: GET 2023-09-29T22:43:43.999 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:43.999 DEBUG:boto:Data: 2023-09-29T22:43:43.999 DEBUG:boto:Headers: {} 2023-09-29T22:43:43.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.000 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.000 DEBUG:boto:Params: {} 2023-09-29T22:43:44.000 DEBUG:boto:Token: None 2023-09-29T22:43:44.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9PKK9sbU/7gBQqAd+7nHUvw0pt8= 2023-09-29T22:43:44.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9PKK9sbU/7gBQqAd+7nHUvw0pt8=', 'Content-Length': '0'} 2023-09-29T22:43:44.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e5384d7e1214bf8-006517531f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.003 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.004 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.004 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.004 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.004 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.004 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.005 DEBUG:boto:Data: 2023-09-29T22:43:44.005 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.005 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.005 DEBUG:boto:Params: {} 2023-09-29T22:43:44.006 DEBUG:boto:Token: None 2023-09-29T22:43:44.006 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.006 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u/f3dCrgdesE1+BhyWA7MuRwY1o= 2023-09-29T22:43:44.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u/f3dCrgdesE1+BhyWA7MuRwY1o=', 'Content-Length': '0'} 2023-09-29T22:43:44.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcc85bd798ede9dd-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.009 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.010 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.010 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.010 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.010 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.010 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.011 DEBUG:boto:Data: 2023-09-29T22:43:44.011 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.011 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.011 DEBUG:boto:Params: {} 2023-09-29T22:43:44.012 DEBUG:boto:Token: None 2023-09-29T22:43:44.012 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u/f3dCrgdesE1+BhyWA7MuRwY1o= 2023-09-29T22:43:44.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u/f3dCrgdesE1+BhyWA7MuRwY1o=', 'Content-Length': '0'} 2023-09-29T22:43:44.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099dac7b1ed7038cf-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:43:44.016 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.016 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.016 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.017 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.017 DEBUG:boto:Method: GET 2023-09-29T22:43:44.017 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.017 DEBUG:boto:Data: 2023-09-29T22:43:44.017 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.018 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.018 DEBUG:boto:Params: {} 2023-09-29T22:43:44.018 DEBUG:boto:Token: None 2023-09-29T22:43:44.018 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9PKK9sbU/7gBQqAd+7nHUvw0pt8= 2023-09-29T22:43:44.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9PKK9sbU/7gBQqAd+7nHUvw0pt8=', 'Content-Length': '0'} 2023-09-29T22:43:44.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029a0401d09220d60-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.022 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.022 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.022 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.022 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.023 DEBUG:boto:Method: GET 2023-09-29T22:43:44.023 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.023 DEBUG:boto:Data: 2023-09-29T22:43:44.023 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.024 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.024 DEBUG:boto:Params: {} 2023-09-29T22:43:44.024 DEBUG:boto:Token: None 2023-09-29T22:43:44.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:43:44.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9PKK9sbU/7gBQqAd+7nHUvw0pt8= 2023-09-29T22:43:44.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9PKK9sbU/7gBQqAd+7nHUvw0pt8=', 'Content-Length': '0'} 2023-09-29T22:43:44.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53961be7f9e27c4-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.070 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.071 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.071 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.071 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.071 DEBUG:boto:Method: GET 2023-09-29T22:43:44.071 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.071 DEBUG:boto:Data: 2023-09-29T22:43:44.071 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.072 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.072 DEBUG:boto:Params: {} 2023-09-29T22:43:44.072 DEBUG:boto:Token: None 2023-09-29T22:43:44.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ab4edfea61647631-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.078 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.078 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.079 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.079 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.079 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.079 DEBUG:boto:Method: GET 2023-09-29T22:43:44.079 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.079 DEBUG:boto:Data: 2023-09-29T22:43:44.079 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.079 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.079 DEBUG:boto:Params: {} 2023-09-29T22:43:44.080 DEBUG:boto:Token: None 2023-09-29T22:43:44.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003ae6be951a3b1c6-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.085 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.085 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.086 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.086 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.086 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:43:44.086 DEBUG:boto:Data: 2023-09-29T22:43:44.086 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.086 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.086 DEBUG:boto:Params: {} 2023-09-29T22:43:44.087 DEBUG:boto:Token: None 2023-09-29T22:43:44.087 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22 2023-09-29T22:43:44.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hteztCvT3rAr30tkUPuYB182hmg= 2023-09-29T22:43:44.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hteztCvT3rAr30tkUPuYB182hmg=', 'Content-Length': '0'} 2023-09-29T22:43:44.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e78533e2161cef1-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.089 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.089 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.089 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.089 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:43:44.090 DEBUG:boto:Data: 2023-09-29T22:43:44.090 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.090 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.090 DEBUG:boto:Params: {} 2023-09-29T22:43:44.090 DEBUG:boto:Token: None 2023-09-29T22:43:44.090 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22 2023-09-29T22:43:44.090 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hteztCvT3rAr30tkUPuYB182hmg= 2023-09-29T22:43:44.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hteztCvT3rAr30tkUPuYB182hmg=', 'Content-Length': '0'} 2023-09-29T22:43:44.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e45707f88a5cec4-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:43:44.093 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.093 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.093 DEBUG:boto:Method: GET 2023-09-29T22:43:44.093 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:43:44.093 DEBUG:boto:Data: 2023-09-29T22:43:44.093 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.094 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.094 DEBUG:boto:Params: {} 2023-09-29T22:43:44.094 DEBUG:boto:Token: None 2023-09-29T22:43:44.094 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22 2023-09-29T22:43:44.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QCALuPSM1EpFe97Exn6tq3UOMVY= 2023-09-29T22:43:44.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QCALuPSM1EpFe97Exn6tq3UOMVY=', 'Content-Length': '0'} 2023-09-29T22:43:44.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000770a1d2595d85faf-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.097 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:43:44.097 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:43:44.097 DEBUG:boto:Method: GET 2023-09-29T22:43:44.097 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:43:44.097 DEBUG:boto:Data: 2023-09-29T22:43:44.097 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.097 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.098 DEBUG:boto:Params: {} 2023-09-29T22:43:44.098 DEBUG:boto:Token: None 2023-09-29T22:43:44.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj22 2023-09-29T22:43:44.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QCALuPSM1EpFe97Exn6tq3UOMVY= 2023-09-29T22:43:44.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QCALuPSM1EpFe97Exn6tq3UOMVY=', 'Content-Length': '0'} 2023-09-29T22:43:44.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4f1b4627d30595b-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.101 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.101 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.101 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.101 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.101 DEBUG:boto:Method: GET 2023-09-29T22:43:44.101 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.101 DEBUG:boto:Data: 2023-09-29T22:43:44.101 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.101 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.102 DEBUG:boto:Params: {} 2023-09-29T22:43:44.102 DEBUG:boto:Token: None 2023-09-29T22:43:44.102 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000845e204f6dc01726-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.106 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.106 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.107 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.107 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.107 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.107 DEBUG:boto:Method: GET 2023-09-29T22:43:44.107 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:43:44.107 DEBUG:boto:Data: 2023-09-29T22:43:44.107 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.107 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.108 DEBUG:boto:Params: {} 2023-09-29T22:43:44.108 DEBUG:boto:Token: None 2023-09-29T22:43:44.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e12aba43f3ae1b1b-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.117 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:43:44.118 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.118 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.118 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.118 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.119 DEBUG:boto:Method: GET 2023-09-29T22:43:44.119 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.119 DEBUG:boto:Data: 2023-09-29T22:43:44.119 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.119 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.120 DEBUG:boto:Params: {} 2023-09-29T22:43:44.120 DEBUG:boto:Token: None 2023-09-29T22:43:44.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0= 2023-09-29T22:43:44.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0=', 'Content-Length': '0'} 2023-09-29T22:43:44.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcc06ef20868b6c0-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.166 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.167 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.167 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.167 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.167 DEBUG:boto:Method: GET 2023-09-29T22:43:44.167 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.168 DEBUG:boto:Data: 2023-09-29T22:43:44.168 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.168 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.168 DEBUG:boto:Params: {} 2023-09-29T22:43:44.169 DEBUG:boto:Token: None 2023-09-29T22:43:44.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0= 2023-09-29T22:43:44.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0=', 'Content-Length': '0'} 2023-09-29T22:43:44.173 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb6b33db0889e254-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.173 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.173 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.174 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.174 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.174 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.174 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.174 DEBUG:boto:Data: 2023-09-29T22:43:44.175 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.175 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.175 DEBUG:boto:Params: {} 2023-09-29T22:43:44.175 DEBUG:boto:Token: None 2023-09-29T22:43:44.176 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4KlWTFmPCiu2L0PBJbZIZQ38bCg= 2023-09-29T22:43:44.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4KlWTFmPCiu2L0PBJbZIZQ38bCg=', 'Content-Length': '0'} 2023-09-29T22:43:44.179 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e464faa431761bd0-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.179 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.179 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.179 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.180 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.180 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.180 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.180 DEBUG:boto:Data: 2023-09-29T22:43:44.180 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.181 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.181 DEBUG:boto:Params: {} 2023-09-29T22:43:44.181 DEBUG:boto:Token: None 2023-09-29T22:43:44.181 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.182 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4KlWTFmPCiu2L0PBJbZIZQ38bCg= 2023-09-29T22:43:44.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4KlWTFmPCiu2L0PBJbZIZQ38bCg=', 'Content-Length': '0'} 2023-09-29T22:43:44.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea61acbbbbd45c7-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:43:44.185 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.185 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.186 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.186 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.186 DEBUG:boto:Method: GET 2023-09-29T22:43:44.186 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.186 DEBUG:boto:Data: 2023-09-29T22:43:44.187 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.187 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.187 DEBUG:boto:Params: {} 2023-09-29T22:43:44.187 DEBUG:boto:Token: None 2023-09-29T22:43:44.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0= 2023-09-29T22:43:44.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0=', 'Content-Length': '0'} 2023-09-29T22:43:44.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97c59291f95827b-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.192 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.192 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.192 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.192 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.192 DEBUG:boto:Method: GET 2023-09-29T22:43:44.192 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.193 DEBUG:boto:Data: 2023-09-29T22:43:44.193 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.193 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.193 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.193 DEBUG:boto:Params: {} 2023-09-29T22:43:44.193 DEBUG:boto:Token: None 2023-09-29T22:43:44.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:43:44.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0= 2023-09-29T22:43:44.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sOl1aY3YVkLgA9l8ZZCj1epmE+0=', 'Content-Length': '0'} 2023-09-29T22:43:44.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a0e4d6d59c1c687-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.197 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.197 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.197 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.197 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.197 DEBUG:boto:Method: GET 2023-09-29T22:43:44.197 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.197 DEBUG:boto:Data: 2023-09-29T22:43:44.197 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.198 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.198 DEBUG:boto:Params: {} 2023-09-29T22:43:44.198 DEBUG:boto:Token: None 2023-09-29T22:43:44.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046404cbfb5dd7c82-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.250 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.251 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.251 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.251 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.251 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.251 DEBUG:boto:Method: GET 2023-09-29T22:43:44.251 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.252 DEBUG:boto:Data: 2023-09-29T22:43:44.252 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.252 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.252 DEBUG:boto:Params: {} 2023-09-29T22:43:44.252 DEBUG:boto:Token: None 2023-09-29T22:43:44.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.253 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e601ac01579b4f5-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.260 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.261 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.261 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.261 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.261 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:43:44.261 DEBUG:boto:Data: 2023-09-29T22:43:44.262 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.262 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.262 DEBUG:boto:Params: {} 2023-09-29T22:43:44.262 DEBUG:boto:Token: None 2023-09-29T22:43:44.262 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23 2023-09-29T22:43:44.263 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qf1WhKG5sqC6mayBeCqBgyQ/aI= 2023-09-29T22:43:44.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qf1WhKG5sqC6mayBeCqBgyQ/aI=', 'Content-Length': '0'} 2023-09-29T22:43:44.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb47faf9f8404e26-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.265 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.266 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.266 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.266 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:43:44.266 DEBUG:boto:Data: 2023-09-29T22:43:44.266 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.267 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.267 DEBUG:boto:Params: {} 2023-09-29T22:43:44.267 DEBUG:boto:Token: None 2023-09-29T22:43:44.267 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23 2023-09-29T22:43:44.267 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qf1WhKG5sqC6mayBeCqBgyQ/aI= 2023-09-29T22:43:44.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qf1WhKG5sqC6mayBeCqBgyQ/aI=', 'Content-Length': '0'} 2023-09-29T22:43:44.269 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fefc7b5c681fd5ec-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:43:44.270 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.270 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.270 DEBUG:boto:Method: GET 2023-09-29T22:43:44.270 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:43:44.270 DEBUG:boto:Data: 2023-09-29T22:43:44.270 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.270 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.271 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.271 DEBUG:boto:Params: {} 2023-09-29T22:43:44.271 DEBUG:boto:Token: None 2023-09-29T22:43:44.271 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23 2023-09-29T22:43:44.271 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5HiiBmqvVXzFOUqGnRE5J3GqXOo= 2023-09-29T22:43:44.271 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5HiiBmqvVXzFOUqGnRE5J3GqXOo=', 'Content-Length': '0'} 2023-09-29T22:43:44.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f9e398cfb420ed-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.274 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:43:44.274 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:43:44.274 DEBUG:boto:Method: GET 2023-09-29T22:43:44.274 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:43:44.274 DEBUG:boto:Data: 2023-09-29T22:43:44.274 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.274 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.275 DEBUG:boto:Params: {} 2023-09-29T22:43:44.275 DEBUG:boto:Token: None 2023-09-29T22:43:44.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj23 2023-09-29T22:43:44.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5HiiBmqvVXzFOUqGnRE5J3GqXOo= 2023-09-29T22:43:44.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5HiiBmqvVXzFOUqGnRE5J3GqXOo=', 'Content-Length': '0'} 2023-09-29T22:43:44.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022c1342269871ec5-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.318 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.319 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.319 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.319 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.319 DEBUG:boto:Method: GET 2023-09-29T22:43:44.319 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.319 DEBUG:boto:Data: 2023-09-29T22:43:44.319 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.320 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.320 DEBUG:boto:Params: {} 2023-09-29T22:43:44.320 DEBUG:boto:Token: None 2023-09-29T22:43:44.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002d05d884817d1547-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.325 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.326 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.326 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.326 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.326 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.326 DEBUG:boto:Method: GET 2023-09-29T22:43:44.326 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:43:44.326 DEBUG:boto:Data: 2023-09-29T22:43:44.326 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.327 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.327 DEBUG:boto:Params: {} 2023-09-29T22:43:44.327 DEBUG:boto:Token: None 2023-09-29T22:43:44.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.334 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a05a0d526ef839f-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.334 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:43:44.335 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.335 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.335 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.335 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.335 DEBUG:boto:Method: GET 2023-09-29T22:43:44.335 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.335 DEBUG:boto:Data: 2023-09-29T22:43:44.336 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.336 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.336 DEBUG:boto:Params: {} 2023-09-29T22:43:44.336 DEBUG:boto:Token: None 2023-09-29T22:43:44.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA= 2023-09-29T22:43:44.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA=', 'Content-Length': '0'} 2023-09-29T22:43:44.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef42687fa4af3ae8-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.339 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.339 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.339 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.339 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.339 DEBUG:boto:Method: GET 2023-09-29T22:43:44.339 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.339 DEBUG:boto:Data: 2023-09-29T22:43:44.339 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.339 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.340 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.340 DEBUG:boto:Params: {} 2023-09-29T22:43:44.340 DEBUG:boto:Token: None 2023-09-29T22:43:44.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA= 2023-09-29T22:43:44.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA=', 'Content-Length': '0'} 2023-09-29T22:43:44.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072863e754cc5ec2e-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.343 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.343 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.343 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.343 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.343 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.343 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.343 DEBUG:boto:Data: 2023-09-29T22:43:44.343 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.343 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.344 DEBUG:boto:Params: {} 2023-09-29T22:43:44.344 DEBUG:boto:Token: None 2023-09-29T22:43:44.344 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hxKiTBRNUEHa/Qf4uuc9jlVA5bQ= 2023-09-29T22:43:44.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hxKiTBRNUEHa/Qf4uuc9jlVA5bQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813c3f522fd393ce-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.346 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.346 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.346 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.347 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.347 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.347 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.347 DEBUG:boto:Data: 2023-09-29T22:43:44.347 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.347 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.347 DEBUG:boto:Params: {} 2023-09-29T22:43:44.347 DEBUG:boto:Token: None 2023-09-29T22:43:44.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hxKiTBRNUEHa/Qf4uuc9jlVA5bQ= 2023-09-29T22:43:44.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hxKiTBRNUEHa/Qf4uuc9jlVA5bQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.350 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000060cf43360b4f798-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:43:44.350 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.350 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.351 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.351 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.351 DEBUG:boto:Method: GET 2023-09-29T22:43:44.351 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.351 DEBUG:boto:Data: 2023-09-29T22:43:44.351 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.351 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.351 DEBUG:boto:Params: {} 2023-09-29T22:43:44.351 DEBUG:boto:Token: None 2023-09-29T22:43:44.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA= 2023-09-29T22:43:44.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA=', 'Content-Length': '0'} 2023-09-29T22:43:44.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ec72a34702cca8c-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.398 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.399 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.399 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.399 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.399 DEBUG:boto:Method: GET 2023-09-29T22:43:44.399 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.399 DEBUG:boto:Data: 2023-09-29T22:43:44.399 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.399 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.400 DEBUG:boto:Params: {} 2023-09-29T22:43:44.400 DEBUG:boto:Token: None 2023-09-29T22:43:44.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:43:44.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA= 2023-09-29T22:43:44.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TgR/l2RTvKqGQCS0uS7bFqcVEcA=', 'Content-Length': '0'} 2023-09-29T22:43:44.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097773f7acdf6b176-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.403 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.404 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.404 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.404 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.404 DEBUG:boto:Method: GET 2023-09-29T22:43:44.404 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.404 DEBUG:boto:Data: 2023-09-29T22:43:44.404 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.404 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.404 DEBUG:boto:Params: {} 2023-09-29T22:43:44.405 DEBUG:boto:Token: None 2023-09-29T22:43:44.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.410 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8867019423c9066-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.410 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.411 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.411 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.411 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.411 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.411 DEBUG:boto:Method: GET 2023-09-29T22:43:44.411 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.411 DEBUG:boto:Data: 2023-09-29T22:43:44.411 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.412 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.412 DEBUG:boto:Params: {} 2023-09-29T22:43:44.412 DEBUG:boto:Token: None 2023-09-29T22:43:44.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035848fc9cde6a895-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.419 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.419 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.419 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.419 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.419 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:43:44.420 DEBUG:boto:Data: 2023-09-29T22:43:44.420 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.420 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.420 DEBUG:boto:Params: {} 2023-09-29T22:43:44.420 DEBUG:boto:Token: None 2023-09-29T22:43:44.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33 2023-09-29T22:43:44.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:biDoeRsKJpf99Kpzx5h0YU69Mpg= 2023-09-29T22:43:44.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:biDoeRsKJpf99Kpzx5h0YU69Mpg=', 'Content-Length': '0'} 2023-09-29T22:43:44.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043313cde1e627db0-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.423 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.423 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.423 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.423 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:43:44.423 DEBUG:boto:Data: 2023-09-29T22:43:44.424 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.424 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.424 DEBUG:boto:Params: {} 2023-09-29T22:43:44.424 DEBUG:boto:Token: None 2023-09-29T22:43:44.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33 2023-09-29T22:43:44.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:biDoeRsKJpf99Kpzx5h0YU69Mpg= 2023-09-29T22:43:44.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:biDoeRsKJpf99Kpzx5h0YU69Mpg=', 'Content-Length': '0'} 2023-09-29T22:43:44.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f9353f5848d8926-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:43:44.427 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.427 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.428 DEBUG:boto:Method: GET 2023-09-29T22:43:44.428 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:43:44.428 DEBUG:boto:Data: 2023-09-29T22:43:44.428 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.428 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.428 DEBUG:boto:Params: {} 2023-09-29T22:43:44.428 DEBUG:boto:Token: None 2023-09-29T22:43:44.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33 2023-09-29T22:43:44.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ODxLYsvIQrTG2h3N5oETb38mlbA= 2023-09-29T22:43:44.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ODxLYsvIQrTG2h3N5oETb38mlbA=', 'Content-Length': '0'} 2023-09-29T22:43:44.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a895dd7f6bd127e-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.474 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:43:44.475 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:43:44.475 DEBUG:boto:Method: GET 2023-09-29T22:43:44.475 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:43:44.475 DEBUG:boto:Data: 2023-09-29T22:43:44.475 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.475 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.475 DEBUG:boto:Params: {} 2023-09-29T22:43:44.476 DEBUG:boto:Token: None 2023-09-29T22:43:44.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj33 2023-09-29T22:43:44.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ODxLYsvIQrTG2h3N5oETb38mlbA= 2023-09-29T22:43:44.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ODxLYsvIQrTG2h3N5oETb38mlbA=', 'Content-Length': '0'} 2023-09-29T22:43:44.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c405df07aaf946c-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.479 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.479 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.479 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.480 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.480 DEBUG:boto:Method: GET 2023-09-29T22:43:44.480 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.480 DEBUG:boto:Data: 2023-09-29T22:43:44.480 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.480 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.480 DEBUG:boto:Params: {} 2023-09-29T22:43:44.480 DEBUG:boto:Token: None 2023-09-29T22:43:44.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a3adfd3158efeba-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.486 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.486 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.486 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.486 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.486 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.487 DEBUG:boto:Method: GET 2023-09-29T22:43:44.487 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:43:44.487 DEBUG:boto:Data: 2023-09-29T22:43:44.487 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.487 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.487 DEBUG:boto:Params: {} 2023-09-29T22:43:44.487 DEBUG:boto:Token: None 2023-09-29T22:43:44.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000080a470aed5342d6f-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.538 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:43:44.539 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.539 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.539 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.539 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.540 DEBUG:boto:Method: GET 2023-09-29T22:43:44.540 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.540 DEBUG:boto:Data: 2023-09-29T22:43:44.540 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.540 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.540 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.540 DEBUG:boto:Params: {} 2023-09-29T22:43:44.540 DEBUG:boto:Token: None 2023-09-29T22:43:44.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw= 2023-09-29T22:43:44.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw=', 'Content-Length': '0'} 2023-09-29T22:43:44.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a7c8c7683e1eca1-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.544 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.544 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.544 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.544 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.544 DEBUG:boto:Method: GET 2023-09-29T22:43:44.544 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.544 DEBUG:boto:Data: 2023-09-29T22:43:44.544 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.545 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.545 DEBUG:boto:Params: {} 2023-09-29T22:43:44.545 DEBUG:boto:Token: None 2023-09-29T22:43:44.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw= 2023-09-29T22:43:44.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw=', 'Content-Length': '0'} 2023-09-29T22:43:44.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464f19a1ccfc5be5-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.548 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.548 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.548 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.549 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.549 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.549 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.549 DEBUG:boto:Data: 2023-09-29T22:43:44.549 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.549 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.549 DEBUG:boto:Params: {} 2023-09-29T22:43:44.549 DEBUG:boto:Token: None 2023-09-29T22:43:44.550 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.550 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pogVDz7bBU8FAVhPQ2fuhCs4wYs= 2023-09-29T22:43:44.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pogVDz7bBU8FAVhPQ2fuhCs4wYs=', 'Content-Length': '0'} 2023-09-29T22:43:44.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000958d449a84f44fb-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.553 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.553 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.553 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.553 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.553 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.553 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.553 DEBUG:boto:Data: 2023-09-29T22:43:44.553 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.554 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.554 DEBUG:boto:Params: {} 2023-09-29T22:43:44.554 DEBUG:boto:Token: None 2023-09-29T22:43:44.554 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.554 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pogVDz7bBU8FAVhPQ2fuhCs4wYs= 2023-09-29T22:43:44.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pogVDz7bBU8FAVhPQ2fuhCs4wYs=', 'Content-Length': '0'} 2023-09-29T22:43:44.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bba55184d6836d1c-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:43:44.557 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.557 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.558 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.558 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.558 DEBUG:boto:Method: GET 2023-09-29T22:43:44.558 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.558 DEBUG:boto:Data: 2023-09-29T22:43:44.558 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.558 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.558 DEBUG:boto:Params: {} 2023-09-29T22:43:44.559 DEBUG:boto:Token: None 2023-09-29T22:43:44.559 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.559 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw= 2023-09-29T22:43:44.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw=', 'Content-Length': '0'} 2023-09-29T22:43:44.562 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038aa8bdf26e08a87-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.562 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.562 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.562 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.562 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.562 DEBUG:boto:Method: GET 2023-09-29T22:43:44.562 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.563 DEBUG:boto:Data: 2023-09-29T22:43:44.563 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.563 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.563 DEBUG:boto:Params: {} 2023-09-29T22:43:44.563 DEBUG:boto:Token: None 2023-09-29T22:43:44.563 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:43:44.563 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw= 2023-09-29T22:43:44.563 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vse7B9+A9Iih26RDce6H/W4IIYw=', 'Content-Length': '0'} 2023-09-29T22:43:44.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004538378546aa690a-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.566 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.567 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.567 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.567 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.567 DEBUG:boto:Method: GET 2023-09-29T22:43:44.567 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.567 DEBUG:boto:Data: 2023-09-29T22:43:44.567 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.567 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.567 DEBUG:boto:Params: {} 2023-09-29T22:43:44.568 DEBUG:boto:Token: None 2023-09-29T22:43:44.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099812b29786f1f7e-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.573 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.573 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.573 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.573 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.573 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.573 DEBUG:boto:Method: GET 2023-09-29T22:43:44.574 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.574 DEBUG:boto:Data: 2023-09-29T22:43:44.574 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.574 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.574 DEBUG:boto:Params: {} 2023-09-29T22:43:44.574 DEBUG:boto:Token: None 2023-09-29T22:43:44.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.575 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001377ffd5089a4830-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.626 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.627 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.627 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.627 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.627 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:43:44.627 DEBUG:boto:Data: 2023-09-29T22:43:44.627 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.628 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.628 DEBUG:boto:Params: {} 2023-09-29T22:43:44.628 DEBUG:boto:Token: None 2023-09-29T22:43:44.628 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34 2023-09-29T22:43:44.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mHGrP1QRX7BlzGXtJ08o9RLrl20= 2023-09-29T22:43:44.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mHGrP1QRX7BlzGXtJ08o9RLrl20=', 'Content-Length': '0'} 2023-09-29T22:43:44.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60d7f8d282f6165-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.631 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.632 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.632 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.632 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:43:44.632 DEBUG:boto:Data: 2023-09-29T22:43:44.632 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.632 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.632 DEBUG:boto:Params: {} 2023-09-29T22:43:44.632 DEBUG:boto:Token: None 2023-09-29T22:43:44.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34 2023-09-29T22:43:44.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mHGrP1QRX7BlzGXtJ08o9RLrl20= 2023-09-29T22:43:44.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mHGrP1QRX7BlzGXtJ08o9RLrl20=', 'Content-Length': '0'} 2023-09-29T22:43:44.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db0b993040c18ab1-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:43:44.635 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.636 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.636 DEBUG:boto:Method: GET 2023-09-29T22:43:44.636 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:43:44.636 DEBUG:boto:Data: 2023-09-29T22:43:44.636 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.636 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.636 DEBUG:boto:Params: {} 2023-09-29T22:43:44.636 DEBUG:boto:Token: None 2023-09-29T22:43:44.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34 2023-09-29T22:43:44.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wRKcpw5IPDzHnlGkLIwT94ZhkGY= 2023-09-29T22:43:44.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wRKcpw5IPDzHnlGkLIwT94ZhkGY=', 'Content-Length': '0'} 2023-09-29T22:43:44.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080cc8755e4622ede-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.682 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:43:44.836 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:43:44.836 DEBUG:boto:Method: GET 2023-09-29T22:43:44.836 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:43:44.836 DEBUG:boto:Data: 2023-09-29T22:43:44.837 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.837 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.837 DEBUG:boto:Params: {} 2023-09-29T22:43:44.837 DEBUG:boto:Token: None 2023-09-29T22:43:44.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj34 2023-09-29T22:43:44.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wRKcpw5IPDzHnlGkLIwT94ZhkGY= 2023-09-29T22:43:44.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wRKcpw5IPDzHnlGkLIwT94ZhkGY=', 'Content-Length': '0'} 2023-09-29T22:43:44.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3b0956aed84097f-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.841 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.842 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.842 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.842 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.842 DEBUG:boto:Method: GET 2023-09-29T22:43:44.842 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.843 DEBUG:boto:Data: 2023-09-29T22:43:44.843 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.843 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.844 DEBUG:boto:Params: {} 2023-09-29T22:43:44.844 DEBUG:boto:Token: None 2023-09-29T22:43:44.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.844 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.850 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000401588778b0fe15f-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.850 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.851 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.851 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.851 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.851 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.851 DEBUG:boto:Method: GET 2023-09-29T22:43:44.852 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:43:44.852 DEBUG:boto:Data: 2023-09-29T22:43:44.852 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.852 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.853 DEBUG:boto:Params: {} 2023-09-29T22:43:44.853 DEBUG:boto:Token: None 2023-09-29T22:43:44.853 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.860 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000646baf3521ca7391-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.860 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:43:44.860 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.861 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.861 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.861 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.861 DEBUG:boto:Method: GET 2023-09-29T22:43:44.861 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.861 DEBUG:boto:Data: 2023-09-29T22:43:44.862 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.862 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.862 DEBUG:boto:Params: {} 2023-09-29T22:43:44.862 DEBUG:boto:Token: None 2023-09-29T22:43:44.862 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.863 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ= 2023-09-29T22:43:44.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af64309d77d9671f-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.866 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.866 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.866 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.866 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.867 DEBUG:boto:Method: GET 2023-09-29T22:43:44.867 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.867 DEBUG:boto:Data: 2023-09-29T22:43:44.867 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.867 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.867 DEBUG:boto:Params: {} 2023-09-29T22:43:44.868 DEBUG:boto:Token: None 2023-09-29T22:43:44.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ= 2023-09-29T22:43:44.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.870 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000034c64ae8b9aaaf9-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.871 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.871 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.871 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.871 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.871 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.872 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.872 DEBUG:boto:Data: 2023-09-29T22:43:44.872 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.872 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.872 DEBUG:boto:Params: {} 2023-09-29T22:43:44.872 DEBUG:boto:Token: None 2023-09-29T22:43:44.873 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.873 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l4sixMMgPfvAR90QKia0nIWta5E= 2023-09-29T22:43:44.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l4sixMMgPfvAR90QKia0nIWta5E=', 'Content-Length': '0'} 2023-09-29T22:43:44.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bcee27247af1f4e-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.875 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.875 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.875 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.876 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.876 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.876 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.876 DEBUG:boto:Data: 2023-09-29T22:43:44.876 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.876 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.876 DEBUG:boto:Params: {} 2023-09-29T22:43:44.877 DEBUG:boto:Token: None 2023-09-29T22:43:44.877 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:l4sixMMgPfvAR90QKia0nIWta5E= 2023-09-29T22:43:44.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:l4sixMMgPfvAR90QKia0nIWta5E=', 'Content-Length': '0'} 2023-09-29T22:43:44.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed4fd1f5f18758aa-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:43:44.880 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.880 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.880 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.880 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.880 DEBUG:boto:Method: GET 2023-09-29T22:43:44.881 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.881 DEBUG:boto:Data: 2023-09-29T22:43:44.881 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.881 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.881 DEBUG:boto:Params: {} 2023-09-29T22:43:44.881 DEBUG:boto:Token: None 2023-09-29T22:43:44.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.882 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ= 2023-09-29T22:43:44.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a72fc65bc66ed54-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.885 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.885 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.885 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.885 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.885 DEBUG:boto:Method: GET 2023-09-29T22:43:44.885 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.886 DEBUG:boto:Data: 2023-09-29T22:43:44.886 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.886 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.886 DEBUG:boto:Params: {} 2023-09-29T22:43:44.886 DEBUG:boto:Token: None 2023-09-29T22:43:44.886 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:43:44.887 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ= 2023-09-29T22:43:44.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lcmdB025cam0LKxE50G3xq+iQUQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7c59d977ec7821b-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.930 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.931 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.931 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:44.931 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:44.931 DEBUG:boto:Method: GET 2023-09-29T22:43:44.931 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:43:44.931 DEBUG:boto:Data: 2023-09-29T22:43:44.931 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.932 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.932 DEBUG:boto:Params: {} 2023-09-29T22:43:44.932 DEBUG:boto:Token: None 2023-09-29T22:43:44.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7122488d3f3c5a3-0065175320-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.938 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.938 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:44.938 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:44.938 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:44.938 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:44.939 DEBUG:boto:Method: GET 2023-09-29T22:43:44.939 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:43:44.939 DEBUG:boto:Data: 2023-09-29T22:43:44.939 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.939 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.939 DEBUG:boto:Params: {} 2023-09-29T22:43:44.939 DEBUG:boto:Token: None 2023-09-29T22:43:44.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/ 2023-09-29T22:43:44.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ= 2023-09-29T22:43:44.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QkVbRYgZMx4UQkSxKcYBwP9t6aQ=', 'Content-Length': '0'} 2023-09-29T22:43:44.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000759ed8aa48cd2b9-0065175320-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.946 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:44.947 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.947 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.947 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.947 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:43:44.947 DEBUG:boto:Data: 2023-09-29T22:43:44.947 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.947 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.947 DEBUG:boto:Params: {} 2023-09-29T22:43:44.948 DEBUG:boto:Token: None 2023-09-29T22:43:44.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38 2023-09-29T22:43:44.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:92Z0JompdMUDBsYY3hcI044oPZo= 2023-09-29T22:43:44.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:92Z0JompdMUDBsYY3hcI044oPZo=', 'Content-Length': '0'} 2023-09-29T22:43:44.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1032eb56bd575c9-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.950 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.950 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.950 DEBUG:boto:Method: HEAD 2023-09-29T22:43:44.950 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:43:44.951 DEBUG:boto:Data: 2023-09-29T22:43:44.951 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:44.951 DEBUG:boto:Port: 8000 2023-09-29T22:43:44.951 DEBUG:boto:Params: {} 2023-09-29T22:43:44.951 DEBUG:boto:Token: None 2023-09-29T22:43:44.951 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38 2023-09-29T22:43:44.951 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:92Z0JompdMUDBsYY3hcI044oPZo= 2023-09-29T22:43:44.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:92Z0JompdMUDBsYY3hcI044oPZo=', 'Content-Length': '0'} 2023-09-29T22:43:44.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2bcb28139e6a276-0065175320-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:44.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:43:44.954 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:44.954 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:44.954 DEBUG:boto:Method: GET 2023-09-29T22:43:44.954 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:43:44.954 DEBUG:boto:Data: 2023-09-29T22:43:44.954 DEBUG:boto:Headers: {} 2023-09-29T22:43:44.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:44.955 DEBUG:boto:Port: 8001 2023-09-29T22:43:44.955 DEBUG:boto:Params: {} 2023-09-29T22:43:44.955 DEBUG:boto:Token: None 2023-09-29T22:43:44.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:44 GMT /hcnphz-59/obj38 2023-09-29T22:43:44.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L2cAZkCNZkkwjcdeJHa4xqQzm9U= 2023-09-29T22:43:44.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L2cAZkCNZkkwjcdeJHa4xqQzm9U=', 'Content-Length': '0'} 2023-09-29T22:43:44.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018d9d081a4a926b8-0065175320-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.002 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:43:45.003 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:43:45.003 DEBUG:boto:Method: GET 2023-09-29T22:43:45.003 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:43:45.003 DEBUG:boto:Data: 2023-09-29T22:43:45.003 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.003 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.003 DEBUG:boto:Params: {} 2023-09-29T22:43:45.004 DEBUG:boto:Token: None 2023-09-29T22:43:45.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj38 2023-09-29T22:43:45.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n96XGnz5/vlk4btuO84vS+TmawA= 2023-09-29T22:43:45.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n96XGnz5/vlk4btuO84vS+TmawA=', 'Content-Length': '0'} 2023-09-29T22:43:45.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049a3ded2a465727a-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.007 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.007 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.007 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:45.008 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:45.008 DEBUG:boto:Method: GET 2023-09-29T22:43:45.008 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:43:45.008 DEBUG:boto:Data: 2023-09-29T22:43:45.008 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.008 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.008 DEBUG:boto:Params: {} 2023-09-29T22:43:45.008 DEBUG:boto:Token: None 2023-09-29T22:43:45.009 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d8722ae77cf84bec-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.014 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.014 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.015 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.015 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:45.015 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:43:45.015 DEBUG:boto:Method: GET 2023-09-29T22:43:45.015 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:43:45.015 DEBUG:boto:Data: 2023-09-29T22:43:45.015 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.015 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.015 DEBUG:boto:Params: {} 2023-09-29T22:43:45.016 DEBUG:boto:Token: None 2023-09-29T22:43:45.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a33c2917786d9beb-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.066 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:43:45.067 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.067 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.067 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.067 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.068 DEBUG:boto:Method: GET 2023-09-29T22:43:45.068 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.068 DEBUG:boto:Data: 2023-09-29T22:43:45.068 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.068 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.068 DEBUG:boto:Params: {} 2023-09-29T22:43:45.068 DEBUG:boto:Token: None 2023-09-29T22:43:45.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8= 2023-09-29T22:43:45.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8=', 'Content-Length': '0'} 2023-09-29T22:43:45.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001c5e624c7f97784-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.072 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.072 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.072 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.072 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.073 DEBUG:boto:Method: GET 2023-09-29T22:43:45.073 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.073 DEBUG:boto:Data: 2023-09-29T22:43:45.073 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.073 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.073 DEBUG:boto:Params: {} 2023-09-29T22:43:45.073 DEBUG:boto:Token: None 2023-09-29T22:43:45.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.074 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8= 2023-09-29T22:43:45.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8=', 'Content-Length': '0'} 2023-09-29T22:43:45.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f231e7058baaf32-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.077 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.077 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.077 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.077 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.077 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.077 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.077 DEBUG:boto:Data: 2023-09-29T22:43:45.077 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.078 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.078 DEBUG:boto:Params: {} 2023-09-29T22:43:45.078 DEBUG:boto:Token: None 2023-09-29T22:43:45.078 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.078 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+rphu+pAI97cs4pWPHbYo3WL5VM= 2023-09-29T22:43:45.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+rphu+pAI97cs4pWPHbYo3WL5VM=', 'Content-Length': '0'} 2023-09-29T22:43:45.081 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b14c08752850909-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.081 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.081 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.081 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.081 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.082 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.082 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.082 DEBUG:boto:Data: 2023-09-29T22:43:45.082 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.082 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.082 DEBUG:boto:Params: {} 2023-09-29T22:43:45.082 DEBUG:boto:Token: None 2023-09-29T22:43:45.082 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.083 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+rphu+pAI97cs4pWPHbYo3WL5VM= 2023-09-29T22:43:45.083 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+rphu+pAI97cs4pWPHbYo3WL5VM=', 'Content-Length': '0'} 2023-09-29T22:43:45.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd6c48382a459fea-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:43:45.086 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.086 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.086 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.086 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.086 DEBUG:boto:Method: GET 2023-09-29T22:43:45.086 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.087 DEBUG:boto:Data: 2023-09-29T22:43:45.087 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.087 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.087 DEBUG:boto:Params: {} 2023-09-29T22:43:45.087 DEBUG:boto:Token: None 2023-09-29T22:43:45.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8= 2023-09-29T22:43:45.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8=', 'Content-Length': '0'} 2023-09-29T22:43:45.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c705d244824b31ef-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.091 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.091 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.091 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.091 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.091 DEBUG:boto:Method: GET 2023-09-29T22:43:45.091 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.091 DEBUG:boto:Data: 2023-09-29T22:43:45.091 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.091 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.092 DEBUG:boto:Params: {} 2023-09-29T22:43:45.092 DEBUG:boto:Token: None 2023-09-29T22:43:45.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:43:45.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8= 2023-09-29T22:43:45.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2lLUkUhlEkBwNSwbaC9Xw19oDC8=', 'Content-Length': '0'} 2023-09-29T22:43:45.095 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079f64aa2c964c176-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.095 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.095 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.095 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.095 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.096 DEBUG:boto:Method: GET 2023-09-29T22:43:45.096 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.096 DEBUG:boto:Data: 2023-09-29T22:43:45.096 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.096 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.096 DEBUG:boto:Params: {} 2023-09-29T22:43:45.097 DEBUG:boto:Token: None 2023-09-29T22:43:45.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.101 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000990282b903a0dcf7-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.142 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.143 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.143 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.143 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.143 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.143 DEBUG:boto:Method: GET 2023-09-29T22:43:45.143 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.144 DEBUG:boto:Data: 2023-09-29T22:43:45.144 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.144 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.144 DEBUG:boto:Params: {} 2023-09-29T22:43:45.144 DEBUG:boto:Token: None 2023-09-29T22:43:45.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.151 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000206b64935382f00a-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.151 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.152 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.152 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.152 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.152 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:43:45.153 DEBUG:boto:Data: 2023-09-29T22:43:45.153 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.153 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.153 DEBUG:boto:Params: {} 2023-09-29T22:43:45.153 DEBUG:boto:Token: None 2023-09-29T22:43:45.154 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44 2023-09-29T22:43:45.154 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j0JS+0vDpcOCW9pODGZqxpyRjQk= 2023-09-29T22:43:45.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j0JS+0vDpcOCW9pODGZqxpyRjQk=', 'Content-Length': '0'} 2023-09-29T22:43:45.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3cf5a13550b6de3-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.157 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.157 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.157 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.157 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:43:45.158 DEBUG:boto:Data: 2023-09-29T22:43:45.158 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.158 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.158 DEBUG:boto:Params: {} 2023-09-29T22:43:45.158 DEBUG:boto:Token: None 2023-09-29T22:43:45.158 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44 2023-09-29T22:43:45.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j0JS+0vDpcOCW9pODGZqxpyRjQk= 2023-09-29T22:43:45.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j0JS+0vDpcOCW9pODGZqxpyRjQk=', 'Content-Length': '0'} 2023-09-29T22:43:45.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbcfb146d1dca3b-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:43:45.161 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.161 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.161 DEBUG:boto:Method: GET 2023-09-29T22:43:45.161 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:43:45.161 DEBUG:boto:Data: 2023-09-29T22:43:45.161 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.162 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.162 DEBUG:boto:Params: {} 2023-09-29T22:43:45.162 DEBUG:boto:Token: None 2023-09-29T22:43:45.162 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44 2023-09-29T22:43:45.162 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I9JH5MabDjSzVyRorwAQIyzsUL8= 2023-09-29T22:43:45.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I9JH5MabDjSzVyRorwAQIyzsUL8=', 'Content-Length': '0'} 2023-09-29T22:43:45.165 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b94998d3387d32e2-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.165 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:43:45.165 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:43:45.165 DEBUG:boto:Method: GET 2023-09-29T22:43:45.165 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:43:45.166 DEBUG:boto:Data: 2023-09-29T22:43:45.166 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.166 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.166 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.166 DEBUG:boto:Params: {} 2023-09-29T22:43:45.166 DEBUG:boto:Token: None 2023-09-29T22:43:45.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj44 2023-09-29T22:43:45.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I9JH5MabDjSzVyRorwAQIyzsUL8= 2023-09-29T22:43:45.167 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I9JH5MabDjSzVyRorwAQIyzsUL8=', 'Content-Length': '0'} 2023-09-29T22:43:45.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c683b7f1c9371c6b-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.169 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.170 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.170 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.170 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.170 DEBUG:boto:Method: GET 2023-09-29T22:43:45.170 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.170 DEBUG:boto:Data: 2023-09-29T22:43:45.170 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.170 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.170 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.170 DEBUG:boto:Params: {} 2023-09-29T22:43:45.171 DEBUG:boto:Token: None 2023-09-29T22:43:45.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.175 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076e095aac983869c-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.175 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.176 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.176 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.176 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.176 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.176 DEBUG:boto:Method: GET 2023-09-29T22:43:45.176 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:43:45.177 DEBUG:boto:Data: 2023-09-29T22:43:45.177 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.177 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.177 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.177 DEBUG:boto:Params: {} 2023-09-29T22:43:45.177 DEBUG:boto:Token: None 2023-09-29T22:43:45.177 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.177 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.178 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f63a1ffaeb44cda-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.184 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:43:45.184 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.184 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.184 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.184 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.184 DEBUG:boto:Method: GET 2023-09-29T22:43:45.185 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.185 DEBUG:boto:Data: 2023-09-29T22:43:45.185 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.185 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.185 DEBUG:boto:Params: {} 2023-09-29T22:43:45.185 DEBUG:boto:Token: None 2023-09-29T22:43:45.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.186 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso= 2023-09-29T22:43:45.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso=', 'Content-Length': '0'} 2023-09-29T22:43:45.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e10e930b83996c8-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.189 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.189 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.189 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.189 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.189 DEBUG:boto:Method: GET 2023-09-29T22:43:45.190 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.190 DEBUG:boto:Data: 2023-09-29T22:43:45.190 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.190 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.190 DEBUG:boto:Params: {} 2023-09-29T22:43:45.190 DEBUG:boto:Token: None 2023-09-29T22:43:45.190 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso= 2023-09-29T22:43:45.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso=', 'Content-Length': '0'} 2023-09-29T22:43:45.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7d6d1c831755948-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.234 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.235 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.235 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.235 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.235 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.235 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.235 DEBUG:boto:Data: 2023-09-29T22:43:45.235 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.235 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.236 DEBUG:boto:Params: {} 2023-09-29T22:43:45.236 DEBUG:boto:Token: None 2023-09-29T22:43:45.236 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wODMFzKM8zD2NQzupO0xQVXNAG0= 2023-09-29T22:43:45.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wODMFzKM8zD2NQzupO0xQVXNAG0=', 'Content-Length': '0'} 2023-09-29T22:43:45.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fe595c193ffa063-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.239 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.240 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.240 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.240 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.240 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.240 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.240 DEBUG:boto:Data: 2023-09-29T22:43:45.240 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.240 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.240 DEBUG:boto:Params: {} 2023-09-29T22:43:45.241 DEBUG:boto:Token: None 2023-09-29T22:43:45.241 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wODMFzKM8zD2NQzupO0xQVXNAG0= 2023-09-29T22:43:45.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wODMFzKM8zD2NQzupO0xQVXNAG0=', 'Content-Length': '0'} 2023-09-29T22:43:45.244 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d16589a3fccc1cae-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:43:45.244 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.244 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.244 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.244 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.245 DEBUG:boto:Method: GET 2023-09-29T22:43:45.245 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.245 DEBUG:boto:Data: 2023-09-29T22:43:45.245 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.245 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.245 DEBUG:boto:Params: {} 2023-09-29T22:43:45.245 DEBUG:boto:Token: None 2023-09-29T22:43:45.246 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso= 2023-09-29T22:43:45.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso=', 'Content-Length': '0'} 2023-09-29T22:43:45.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bdb721998d410ce-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.290 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.291 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.291 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.291 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.291 DEBUG:boto:Method: GET 2023-09-29T22:43:45.291 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.291 DEBUG:boto:Data: 2023-09-29T22:43:45.291 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.291 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.292 DEBUG:boto:Params: {} 2023-09-29T22:43:45.292 DEBUG:boto:Token: None 2023-09-29T22:43:45.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:43:45.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso= 2023-09-29T22:43:45.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DqFJdQ/ASDDSXfvSw36kzr1HLso=', 'Content-Length': '0'} 2023-09-29T22:43:45.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009de3c049af3de0e3-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.295 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.295 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.296 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.296 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.296 DEBUG:boto:Method: GET 2023-09-29T22:43:45.296 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.296 DEBUG:boto:Data: 2023-09-29T22:43:45.296 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.296 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.297 DEBUG:boto:Params: {} 2023-09-29T22:43:45.297 DEBUG:boto:Token: None 2023-09-29T22:43:45.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094b8438f47d85a12-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.302 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.303 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.303 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.303 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.303 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.303 DEBUG:boto:Method: GET 2023-09-29T22:43:45.303 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.303 DEBUG:boto:Data: 2023-09-29T22:43:45.304 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.304 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.304 DEBUG:boto:Params: {} 2023-09-29T22:43:45.304 DEBUG:boto:Token: None 2023-09-29T22:43:45.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.304 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.310 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9e54ad8f1986fc7-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.311 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.311 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.311 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.311 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.312 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:43:45.312 DEBUG:boto:Data: 2023-09-29T22:43:45.312 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.312 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.312 DEBUG:boto:Params: {} 2023-09-29T22:43:45.312 DEBUG:boto:Token: None 2023-09-29T22:43:45.312 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46 2023-09-29T22:43:45.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ludRT3dVHTMYVu4THlffq8TEJPs= 2023-09-29T22:43:45.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ludRT3dVHTMYVu4THlffq8TEJPs=', 'Content-Length': '0'} 2023-09-29T22:43:45.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4358319b9919dad-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.315 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.316 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.316 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.316 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:43:45.316 DEBUG:boto:Data: 2023-09-29T22:43:45.316 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.316 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.316 DEBUG:boto:Params: {} 2023-09-29T22:43:45.316 DEBUG:boto:Token: None 2023-09-29T22:43:45.317 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46 2023-09-29T22:43:45.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ludRT3dVHTMYVu4THlffq8TEJPs= 2023-09-29T22:43:45.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ludRT3dVHTMYVu4THlffq8TEJPs=', 'Content-Length': '0'} 2023-09-29T22:43:45.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf06ab71a1c2472-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:43:45.319 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.319 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.320 DEBUG:boto:Method: GET 2023-09-29T22:43:45.320 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:43:45.320 DEBUG:boto:Data: 2023-09-29T22:43:45.320 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.320 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.320 DEBUG:boto:Params: {} 2023-09-29T22:43:45.320 DEBUG:boto:Token: None 2023-09-29T22:43:45.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46 2023-09-29T22:43:45.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fI2Mb5hxgxatgWW4Ou1boptdTxE= 2023-09-29T22:43:45.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fI2Mb5hxgxatgWW4Ou1boptdTxE=', 'Content-Length': '0'} 2023-09-29T22:43:45.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b4ca29dbb5e49bc-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.370 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:43:45.371 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:43:45.371 DEBUG:boto:Method: GET 2023-09-29T22:43:45.371 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:43:45.371 DEBUG:boto:Data: 2023-09-29T22:43:45.371 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.371 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.371 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.371 DEBUG:boto:Params: {} 2023-09-29T22:43:45.372 DEBUG:boto:Token: None 2023-09-29T22:43:45.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj46 2023-09-29T22:43:45.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fI2Mb5hxgxatgWW4Ou1boptdTxE= 2023-09-29T22:43:45.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fI2Mb5hxgxatgWW4Ou1boptdTxE=', 'Content-Length': '0'} 2023-09-29T22:43:45.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5cf40fd5c2ded94-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.375 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.375 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.375 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.375 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.376 DEBUG:boto:Method: GET 2023-09-29T22:43:45.376 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.376 DEBUG:boto:Data: 2023-09-29T22:43:45.376 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.376 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.376 DEBUG:boto:Params: {} 2023-09-29T22:43:45.376 DEBUG:boto:Token: None 2023-09-29T22:43:45.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028f8e573b55d1b35-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.382 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.382 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.382 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.382 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.382 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.383 DEBUG:boto:Method: GET 2023-09-29T22:43:45.383 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:43:45.383 DEBUG:boto:Data: 2023-09-29T22:43:45.383 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.383 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.383 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.383 DEBUG:boto:Params: {} 2023-09-29T22:43:45.383 DEBUG:boto:Token: None 2023-09-29T22:43:45.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.384 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.390 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d630daa0ea0301bb-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.390 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:43:45.391 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.391 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.391 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.391 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.391 DEBUG:boto:Method: GET 2023-09-29T22:43:45.391 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.391 DEBUG:boto:Data: 2023-09-29T22:43:45.391 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.392 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.392 DEBUG:boto:Params: {} 2023-09-29T22:43:45.392 DEBUG:boto:Token: None 2023-09-29T22:43:45.392 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM= 2023-09-29T22:43:45.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM=', 'Content-Length': '0'} 2023-09-29T22:43:45.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e40dff8717290d-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.395 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.395 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.395 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.395 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.395 DEBUG:boto:Method: GET 2023-09-29T22:43:45.395 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.395 DEBUG:boto:Data: 2023-09-29T22:43:45.395 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.396 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.396 DEBUG:boto:Params: {} 2023-09-29T22:43:45.396 DEBUG:boto:Token: None 2023-09-29T22:43:45.396 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM= 2023-09-29T22:43:45.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM=', 'Content-Length': '0'} 2023-09-29T22:43:45.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f130f47e4547319a-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.398 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.399 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.399 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.399 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.399 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.399 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.399 DEBUG:boto:Data: 2023-09-29T22:43:45.399 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.399 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.399 DEBUG:boto:Params: {} 2023-09-29T22:43:45.400 DEBUG:boto:Token: None 2023-09-29T22:43:45.400 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f75i6cKToc3Evno7yCZ/1V7PEEo= 2023-09-29T22:43:45.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f75i6cKToc3Evno7yCZ/1V7PEEo=', 'Content-Length': '0'} 2023-09-29T22:43:45.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000073dd1e29a45bb395-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.402 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.402 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.403 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.403 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.403 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.403 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.403 DEBUG:boto:Data: 2023-09-29T22:43:45.403 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.403 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.403 DEBUG:boto:Params: {} 2023-09-29T22:43:45.403 DEBUG:boto:Token: None 2023-09-29T22:43:45.404 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f75i6cKToc3Evno7yCZ/1V7PEEo= 2023-09-29T22:43:45.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f75i6cKToc3Evno7yCZ/1V7PEEo=', 'Content-Length': '0'} 2023-09-29T22:43:45.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f453574a45c627c9-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:43:45.407 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.407 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.407 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.407 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.407 DEBUG:boto:Method: GET 2023-09-29T22:43:45.407 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.407 DEBUG:boto:Data: 2023-09-29T22:43:45.407 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.407 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.408 DEBUG:boto:Params: {} 2023-09-29T22:43:45.408 DEBUG:boto:Token: None 2023-09-29T22:43:45.408 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM= 2023-09-29T22:43:45.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM=', 'Content-Length': '0'} 2023-09-29T22:43:45.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d360637d6cd3645-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.458 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.459 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.459 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.459 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.459 DEBUG:boto:Method: GET 2023-09-29T22:43:45.459 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.459 DEBUG:boto:Data: 2023-09-29T22:43:45.459 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.459 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.459 DEBUG:boto:Params: {} 2023-09-29T22:43:45.460 DEBUG:boto:Token: None 2023-09-29T22:43:45.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:43:45.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM= 2023-09-29T22:43:45.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pai3DDRJIbzRiedU1iEU6efNzuM=', 'Content-Length': '0'} 2023-09-29T22:43:45.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ccf95c2ada58993-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.464 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.465 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.465 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.465 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.465 DEBUG:boto:Method: GET 2023-09-29T22:43:45.465 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.465 DEBUG:boto:Data: 2023-09-29T22:43:45.465 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.465 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.465 DEBUG:boto:Params: {} 2023-09-29T22:43:45.466 DEBUG:boto:Token: None 2023-09-29T22:43:45.466 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.466 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001db7d1b9a7752de6-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.471 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.471 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.471 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.471 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.471 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.472 DEBUG:boto:Method: GET 2023-09-29T22:43:45.472 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.472 DEBUG:boto:Data: 2023-09-29T22:43:45.472 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.472 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.472 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.472 DEBUG:boto:Params: {} 2023-09-29T22:43:45.472 DEBUG:boto:Token: None 2023-09-29T22:43:45.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.473 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.479 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000051231eafecdc48b2-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.479 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.479 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.479 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.479 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.479 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:43:45.480 DEBUG:boto:Data: 2023-09-29T22:43:45.480 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.480 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.480 DEBUG:boto:Params: {} 2023-09-29T22:43:45.480 DEBUG:boto:Token: None 2023-09-29T22:43:45.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51 2023-09-29T22:43:45.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YI/pKOhRgjzEHk2kLI4/g86ux1Y= 2023-09-29T22:43:45.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YI/pKOhRgjzEHk2kLI4/g86ux1Y=', 'Content-Length': '0'} 2023-09-29T22:43:45.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a3c3410a40d858a-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.483 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.483 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.483 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.484 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:43:45.484 DEBUG:boto:Data: 2023-09-29T22:43:45.484 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.484 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.484 DEBUG:boto:Params: {} 2023-09-29T22:43:45.484 DEBUG:boto:Token: None 2023-09-29T22:43:45.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51 2023-09-29T22:43:45.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YI/pKOhRgjzEHk2kLI4/g86ux1Y= 2023-09-29T22:43:45.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YI/pKOhRgjzEHk2kLI4/g86ux1Y=', 'Content-Length': '0'} 2023-09-29T22:43:45.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ff7a90339157104-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:43:45.487 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.487 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.487 DEBUG:boto:Method: GET 2023-09-29T22:43:45.488 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:43:45.488 DEBUG:boto:Data: 2023-09-29T22:43:45.488 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.488 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.488 DEBUG:boto:Params: {} 2023-09-29T22:43:45.488 DEBUG:boto:Token: None 2023-09-29T22:43:45.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51 2023-09-29T22:43:45.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dyJymPIKc0FWQfDMTaO3gaWzPEg= 2023-09-29T22:43:45.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dyJymPIKc0FWQfDMTaO3gaWzPEg=', 'Content-Length': '0'} 2023-09-29T22:43:45.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f27782813e68b59-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.538 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:43:45.539 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:43:45.539 DEBUG:boto:Method: GET 2023-09-29T22:43:45.539 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:43:45.539 DEBUG:boto:Data: 2023-09-29T22:43:45.539 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.539 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.539 DEBUG:boto:Params: {} 2023-09-29T22:43:45.539 DEBUG:boto:Token: None 2023-09-29T22:43:45.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj51 2023-09-29T22:43:45.540 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dyJymPIKc0FWQfDMTaO3gaWzPEg= 2023-09-29T22:43:45.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dyJymPIKc0FWQfDMTaO3gaWzPEg=', 'Content-Length': '0'} 2023-09-29T22:43:45.543 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ad1c64f9920ad79-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.543 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.543 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.543 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.544 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.544 DEBUG:boto:Method: GET 2023-09-29T22:43:45.544 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.544 DEBUG:boto:Data: 2023-09-29T22:43:45.544 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.544 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.544 DEBUG:boto:Params: {} 2023-09-29T22:43:45.544 DEBUG:boto:Token: None 2023-09-29T22:43:45.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d5ded328d89101c-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.550 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.551 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.551 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.551 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.551 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.551 DEBUG:boto:Method: GET 2023-09-29T22:43:45.551 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:43:45.551 DEBUG:boto:Data: 2023-09-29T22:43:45.551 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.552 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.552 DEBUG:boto:Params: {} 2023-09-29T22:43:45.552 DEBUG:boto:Token: None 2023-09-29T22:43:45.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.558 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c37fd01aea73102d-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.598 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:43:45.599 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.599 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.599 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.600 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.600 DEBUG:boto:Method: GET 2023-09-29T22:43:45.600 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.600 DEBUG:boto:Data: 2023-09-29T22:43:45.600 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.600 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.600 DEBUG:boto:Params: {} 2023-09-29T22:43:45.601 DEBUG:boto:Token: None 2023-09-29T22:43:45.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I= 2023-09-29T22:43:45.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I=', 'Content-Length': '0'} 2023-09-29T22:43:45.604 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ffc3eddd31628a1-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.604 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.605 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.605 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.605 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.605 DEBUG:boto:Method: GET 2023-09-29T22:43:45.605 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.605 DEBUG:boto:Data: 2023-09-29T22:43:45.605 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.606 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.606 DEBUG:boto:Params: {} 2023-09-29T22:43:45.606 DEBUG:boto:Token: None 2023-09-29T22:43:45.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.606 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I= 2023-09-29T22:43:45.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I=', 'Content-Length': '0'} 2023-09-29T22:43:45.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c918d179d4611861-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.609 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.609 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.610 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.610 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.610 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.610 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.610 DEBUG:boto:Data: 2023-09-29T22:43:45.610 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.610 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.611 DEBUG:boto:Params: {} 2023-09-29T22:43:45.611 DEBUG:boto:Token: None 2023-09-29T22:43:45.611 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5HZbo182xm+EHPGNGUnifFC2Xk0= 2023-09-29T22:43:45.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5HZbo182xm+EHPGNGUnifFC2Xk0=', 'Content-Length': '0'} 2023-09-29T22:43:45.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000693ba14feecd177-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.614 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.615 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.615 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.615 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.615 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.615 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.615 DEBUG:boto:Data: 2023-09-29T22:43:45.615 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.616 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.616 DEBUG:boto:Params: {} 2023-09-29T22:43:45.616 DEBUG:boto:Token: None 2023-09-29T22:43:45.616 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5HZbo182xm+EHPGNGUnifFC2Xk0= 2023-09-29T22:43:45.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5HZbo182xm+EHPGNGUnifFC2Xk0=', 'Content-Length': '0'} 2023-09-29T22:43:45.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009b35b6d95a1c1c0-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:43:45.620 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.620 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.620 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.620 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.620 DEBUG:boto:Method: GET 2023-09-29T22:43:45.621 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.621 DEBUG:boto:Data: 2023-09-29T22:43:45.621 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.621 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.621 DEBUG:boto:Params: {} 2023-09-29T22:43:45.621 DEBUG:boto:Token: None 2023-09-29T22:43:45.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I= 2023-09-29T22:43:45.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I=', 'Content-Length': '0'} 2023-09-29T22:43:45.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3134f31651ea1ce-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.625 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.626 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.626 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.626 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.626 DEBUG:boto:Method: GET 2023-09-29T22:43:45.626 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.626 DEBUG:boto:Data: 2023-09-29T22:43:45.626 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.627 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.627 DEBUG:boto:Params: {} 2023-09-29T22:43:45.627 DEBUG:boto:Token: None 2023-09-29T22:43:45.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:43:45.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I= 2023-09-29T22:43:45.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mvgecR3DcDKeMhQ1woImoDqsG6I=', 'Content-Length': '0'} 2023-09-29T22:43:45.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6248c153a7e380-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.630 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.630 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.630 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.630 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.630 DEBUG:boto:Method: GET 2023-09-29T22:43:45.630 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.630 DEBUG:boto:Data: 2023-09-29T22:43:45.631 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.631 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.631 DEBUG:boto:Params: {} 2023-09-29T22:43:45.631 DEBUG:boto:Token: None 2023-09-29T22:43:45.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.635 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018030e22811e17c0-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.635 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.635 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.635 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.636 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.636 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.636 DEBUG:boto:Method: GET 2023-09-29T22:43:45.636 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.636 DEBUG:boto:Data: 2023-09-29T22:43:45.636 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.636 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.636 DEBUG:boto:Params: {} 2023-09-29T22:43:45.637 DEBUG:boto:Token: None 2023-09-29T22:43:45.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000baa39be72c15540d-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.642 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.643 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.643 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.643 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.643 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:43:45.643 DEBUG:boto:Data: 2023-09-29T22:43:45.643 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.644 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.644 DEBUG:boto:Params: {} 2023-09-29T22:43:45.644 DEBUG:boto:Token: None 2023-09-29T22:43:45.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55 2023-09-29T22:43:45.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8jJBbh9OcPrHWHkvBl19VtGBgfM= 2023-09-29T22:43:45.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8jJBbh9OcPrHWHkvBl19VtGBgfM=', 'Content-Length': '0'} 2023-09-29T22:43:45.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6c6e0ce1a892e37-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.647 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.647 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.647 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.647 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:43:45.647 DEBUG:boto:Data: 2023-09-29T22:43:45.647 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.648 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.648 DEBUG:boto:Params: {} 2023-09-29T22:43:45.648 DEBUG:boto:Token: None 2023-09-29T22:43:45.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55 2023-09-29T22:43:45.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8jJBbh9OcPrHWHkvBl19VtGBgfM= 2023-09-29T22:43:45.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8jJBbh9OcPrHWHkvBl19VtGBgfM=', 'Content-Length': '0'} 2023-09-29T22:43:45.651 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002baccfde0e53e39-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:43:45.651 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.651 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.651 DEBUG:boto:Method: GET 2023-09-29T22:43:45.652 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:43:45.652 DEBUG:boto:Data: 2023-09-29T22:43:45.652 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.652 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.652 DEBUG:boto:Params: {} 2023-09-29T22:43:45.652 DEBUG:boto:Token: None 2023-09-29T22:43:45.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55 2023-09-29T22:43:45.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sc3ZBogdo0mZgw2d2MGWviKqZUg= 2023-09-29T22:43:45.653 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sc3ZBogdo0mZgw2d2MGWviKqZUg=', 'Content-Length': '0'} 2023-09-29T22:43:45.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f5a4c9e654ff52f-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.698 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:43:45.699 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:43:45.699 DEBUG:boto:Method: GET 2023-09-29T22:43:45.699 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:43:45.699 DEBUG:boto:Data: 2023-09-29T22:43:45.699 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.699 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.699 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.699 DEBUG:boto:Params: {} 2023-09-29T22:43:45.700 DEBUG:boto:Token: None 2023-09-29T22:43:45.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj55 2023-09-29T22:43:45.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Sc3ZBogdo0mZgw2d2MGWviKqZUg= 2023-09-29T22:43:45.700 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Sc3ZBogdo0mZgw2d2MGWviKqZUg=', 'Content-Length': '0'} 2023-09-29T22:43:45.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea8beb8b65fe3b8b-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.703 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.703 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.704 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.704 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.704 DEBUG:boto:Method: GET 2023-09-29T22:43:45.704 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.704 DEBUG:boto:Data: 2023-09-29T22:43:45.704 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.704 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.705 DEBUG:boto:Params: {} 2023-09-29T22:43:45.705 DEBUG:boto:Token: None 2023-09-29T22:43:45.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.710 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000063800d1749bb916-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.710 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.711 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.711 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.711 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.711 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.711 DEBUG:boto:Method: GET 2023-09-29T22:43:45.711 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:43:45.712 DEBUG:boto:Data: 2023-09-29T22:43:45.712 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.712 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.712 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.712 DEBUG:boto:Params: {} 2023-09-29T22:43:45.712 DEBUG:boto:Token: None 2023-09-29T22:43:45.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.718 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000febf89b126d77940-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.718 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:43:45.719 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.719 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.719 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.719 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.720 DEBUG:boto:Method: GET 2023-09-29T22:43:45.720 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.720 DEBUG:boto:Data: 2023-09-29T22:43:45.720 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.720 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.720 DEBUG:boto:Params: {} 2023-09-29T22:43:45.720 DEBUG:boto:Token: None 2023-09-29T22:43:45.721 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA= 2023-09-29T22:43:45.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA=', 'Content-Length': '0'} 2023-09-29T22:43:45.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000591cb7ab744b3c13-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.723 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.723 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.723 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.723 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.724 DEBUG:boto:Method: GET 2023-09-29T22:43:45.724 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.724 DEBUG:boto:Data: 2023-09-29T22:43:45.724 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.724 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.724 DEBUG:boto:Params: {} 2023-09-29T22:43:45.724 DEBUG:boto:Token: None 2023-09-29T22:43:45.725 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA= 2023-09-29T22:43:45.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA=', 'Content-Length': '0'} 2023-09-29T22:43:45.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000497f745d7764cd83-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.728 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.728 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.728 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.728 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.728 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.728 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.728 DEBUG:boto:Data: 2023-09-29T22:43:45.728 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.729 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.729 DEBUG:boto:Params: {} 2023-09-29T22:43:45.729 DEBUG:boto:Token: None 2023-09-29T22:43:45.729 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:htc3FO1KYSE7EJc6VkA7lUDhgPk= 2023-09-29T22:43:45.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:htc3FO1KYSE7EJc6VkA7lUDhgPk=', 'Content-Length': '0'} 2023-09-29T22:43:45.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ee56a2793cf839-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.732 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.732 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.732 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.732 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.732 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.732 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.733 DEBUG:boto:Data: 2023-09-29T22:43:45.733 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.733 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.733 DEBUG:boto:Params: {} 2023-09-29T22:43:45.733 DEBUG:boto:Token: None 2023-09-29T22:43:45.733 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.734 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:htc3FO1KYSE7EJc6VkA7lUDhgPk= 2023-09-29T22:43:45.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:htc3FO1KYSE7EJc6VkA7lUDhgPk=', 'Content-Length': '0'} 2023-09-29T22:43:45.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4ce42964fb509a7-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:43:45.736 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.737 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.737 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.737 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.737 DEBUG:boto:Method: GET 2023-09-29T22:43:45.737 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.737 DEBUG:boto:Data: 2023-09-29T22:43:45.737 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.738 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.738 DEBUG:boto:Params: {} 2023-09-29T22:43:45.738 DEBUG:boto:Token: None 2023-09-29T22:43:45.738 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA= 2023-09-29T22:43:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA=', 'Content-Length': '0'} 2023-09-29T22:43:45.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008add6b1d71d36902-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.782 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.783 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.783 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.783 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.783 DEBUG:boto:Method: GET 2023-09-29T22:43:45.783 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.783 DEBUG:boto:Data: 2023-09-29T22:43:45.783 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.784 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.784 DEBUG:boto:Params: {} 2023-09-29T22:43:45.784 DEBUG:boto:Token: None 2023-09-29T22:43:45.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:43:45.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA= 2023-09-29T22:43:45.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gIuuvmahMpMoLJuGzACStt7KUqA=', 'Content-Length': '0'} 2023-09-29T22:43:45.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3c4896b2924d339-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.787 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.787 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.788 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.788 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.788 DEBUG:boto:Method: GET 2023-09-29T22:43:45.788 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.788 DEBUG:boto:Data: 2023-09-29T22:43:45.788 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.788 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.788 DEBUG:boto:Params: {} 2023-09-29T22:43:45.788 DEBUG:boto:Token: None 2023-09-29T22:43:45.789 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.789 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b7a3c10de146a86-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.794 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.794 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.795 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.795 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.795 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.795 DEBUG:boto:Method: GET 2023-09-29T22:43:45.795 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.795 DEBUG:boto:Data: 2023-09-29T22:43:45.795 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.795 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.795 DEBUG:boto:Params: {} 2023-09-29T22:43:45.796 DEBUG:boto:Token: None 2023-09-29T22:43:45.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.802 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000065f0eedcc1e1920b-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.802 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.803 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.803 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.803 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.803 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:43:45.803 DEBUG:boto:Data: 2023-09-29T22:43:45.803 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.804 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.804 DEBUG:boto:Params: {} 2023-09-29T22:43:45.804 DEBUG:boto:Token: None 2023-09-29T22:43:45.804 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57 2023-09-29T22:43:45.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ld0ddBRYeiyJbaS+ldbytg6JzLk= 2023-09-29T22:43:45.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ld0ddBRYeiyJbaS+ldbytg6JzLk=', 'Content-Length': '0'} 2023-09-29T22:43:45.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e3b38f1ab4bb6b-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.806 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.806 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.807 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.807 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:43:45.807 DEBUG:boto:Data: 2023-09-29T22:43:45.807 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.807 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.807 DEBUG:boto:Params: {} 2023-09-29T22:43:45.807 DEBUG:boto:Token: None 2023-09-29T22:43:45.807 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57 2023-09-29T22:43:45.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ld0ddBRYeiyJbaS+ldbytg6JzLk= 2023-09-29T22:43:45.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ld0ddBRYeiyJbaS+ldbytg6JzLk=', 'Content-Length': '0'} 2023-09-29T22:43:45.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed766425b97be90c-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:43:45.810 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.810 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.810 DEBUG:boto:Method: GET 2023-09-29T22:43:45.810 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:43:45.810 DEBUG:boto:Data: 2023-09-29T22:43:45.810 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.811 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.811 DEBUG:boto:Params: {} 2023-09-29T22:43:45.811 DEBUG:boto:Token: None 2023-09-29T22:43:45.811 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57 2023-09-29T22:43:45.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oBiSZFKNfRAA1Z/fgAK3Gn/deIE= 2023-09-29T22:43:45.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oBiSZFKNfRAA1Z/fgAK3Gn/deIE=', 'Content-Length': '0'} 2023-09-29T22:43:45.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002da564ebd474a256-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.814 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:43:45.814 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:43:45.815 DEBUG:boto:Method: GET 2023-09-29T22:43:45.815 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:43:45.815 DEBUG:boto:Data: 2023-09-29T22:43:45.815 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.815 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.815 DEBUG:boto:Params: {} 2023-09-29T22:43:45.815 DEBUG:boto:Token: None 2023-09-29T22:43:45.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj57 2023-09-29T22:43:45.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oBiSZFKNfRAA1Z/fgAK3Gn/deIE= 2023-09-29T22:43:45.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oBiSZFKNfRAA1Z/fgAK3Gn/deIE=', 'Content-Length': '0'} 2023-09-29T22:43:45.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025056b12f95c55c3-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.819 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.819 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.819 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.819 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.819 DEBUG:boto:Method: GET 2023-09-29T22:43:45.819 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.819 DEBUG:boto:Data: 2023-09-29T22:43:45.820 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.820 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.820 DEBUG:boto:Params: {} 2023-09-29T22:43:45.820 DEBUG:boto:Token: None 2023-09-29T22:43:45.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.825 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba2b8d2919d5d6a6-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.825 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.825 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.826 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.826 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.826 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.826 DEBUG:boto:Method: GET 2023-09-29T22:43:45.826 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:43:45.826 DEBUG:boto:Data: 2023-09-29T22:43:45.826 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.826 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.826 DEBUG:boto:Params: {} 2023-09-29T22:43:45.827 DEBUG:boto:Token: None 2023-09-29T22:43:45.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041e9d492b55fb40d-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.874 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:43:45.875 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.875 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.875 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.876 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.876 DEBUG:boto:Method: GET 2023-09-29T22:43:45.876 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.876 DEBUG:boto:Data: 2023-09-29T22:43:45.876 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.876 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.876 DEBUG:boto:Params: {} 2023-09-29T22:43:45.876 DEBUG:boto:Token: None 2023-09-29T22:43:45.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4= 2023-09-29T22:43:45.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4=', 'Content-Length': '0'} 2023-09-29T22:43:45.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000640c02e890f77f92-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.880 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.880 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.880 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.881 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.881 DEBUG:boto:Method: GET 2023-09-29T22:43:45.881 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.881 DEBUG:boto:Data: 2023-09-29T22:43:45.881 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.881 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.881 DEBUG:boto:Params: {} 2023-09-29T22:43:45.881 DEBUG:boto:Token: None 2023-09-29T22:43:45.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.882 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4= 2023-09-29T22:43:45.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4=', 'Content-Length': '0'} 2023-09-29T22:43:45.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7e18d7f129eb41b-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.885 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.885 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.885 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.885 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.885 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.885 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.885 DEBUG:boto:Data: 2023-09-29T22:43:45.885 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.886 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.886 DEBUG:boto:Params: {} 2023-09-29T22:43:45.886 DEBUG:boto:Token: None 2023-09-29T22:43:45.886 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hf4amiyheBKCuj2BkXM2+WeVZUE= 2023-09-29T22:43:45.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hf4amiyheBKCuj2BkXM2+WeVZUE=', 'Content-Length': '0'} 2023-09-29T22:43:45.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bc5c3ff9a76b414-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.889 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.889 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.889 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.889 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.889 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.889 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.890 DEBUG:boto:Data: 2023-09-29T22:43:45.890 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.890 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.890 DEBUG:boto:Params: {} 2023-09-29T22:43:45.890 DEBUG:boto:Token: None 2023-09-29T22:43:45.890 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.890 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hf4amiyheBKCuj2BkXM2+WeVZUE= 2023-09-29T22:43:45.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hf4amiyheBKCuj2BkXM2+WeVZUE=', 'Content-Length': '0'} 2023-09-29T22:43:45.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bcfe9748e90c969-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:43:45.893 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.894 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.894 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.894 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.894 DEBUG:boto:Method: GET 2023-09-29T22:43:45.894 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.894 DEBUG:boto:Data: 2023-09-29T22:43:45.894 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.894 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.894 DEBUG:boto:Params: {} 2023-09-29T22:43:45.895 DEBUG:boto:Token: None 2023-09-29T22:43:45.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4= 2023-09-29T22:43:45.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4=', 'Content-Length': '0'} 2023-09-29T22:43:45.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cea16dde55e60241-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.942 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.943 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.943 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.943 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.943 DEBUG:boto:Method: GET 2023-09-29T22:43:45.943 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.943 DEBUG:boto:Data: 2023-09-29T22:43:45.943 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.943 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.944 DEBUG:boto:Params: {} 2023-09-29T22:43:45.944 DEBUG:boto:Token: None 2023-09-29T22:43:45.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:43:45.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4= 2023-09-29T22:43:45.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s9MbDaPBdetJGO6c4NUw+GAUPq4=', 'Content-Length': '0'} 2023-09-29T22:43:45.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c1bc8495787ed6a-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.948 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.948 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.948 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:45.949 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:45.949 DEBUG:boto:Method: GET 2023-09-29T22:43:45.949 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:43:45.949 DEBUG:boto:Data: 2023-09-29T22:43:45.949 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.949 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.949 DEBUG:boto:Params: {} 2023-09-29T22:43:45.949 DEBUG:boto:Token: None 2023-09-29T22:43:45.950 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.950 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d38206bbad0b4e23-0065175321-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.955 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.956 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:45.956 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:45.956 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:45.956 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:45.956 DEBUG:boto:Method: GET 2023-09-29T22:43:45.956 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:43:45.956 DEBUG:boto:Data: 2023-09-29T22:43:45.957 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.957 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.957 DEBUG:boto:Params: {} 2023-09-29T22:43:45.957 DEBUG:boto:Token: None 2023-09-29T22:43:45.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/ 2023-09-29T22:43:45.957 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM= 2023-09-29T22:43:45.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N/663c+fp6aLL7879btzLBkr/uM=', 'Content-Length': '0'} 2023-09-29T22:43:45.963 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000537880964aa38b38-0065175321-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.963 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:45.964 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.964 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.964 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.964 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:43:45.964 DEBUG:boto:Data: 2023-09-29T22:43:45.964 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.964 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.964 DEBUG:boto:Params: {} 2023-09-29T22:43:45.965 DEBUG:boto:Token: None 2023-09-29T22:43:45.965 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68 2023-09-29T22:43:45.965 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XglWZnjgvECMQTdmzwXFguoutXw= 2023-09-29T22:43:45.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XglWZnjgvECMQTdmzwXFguoutXw=', 'Content-Length': '0'} 2023-09-29T22:43:45.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d1391042adf0157-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.967 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.968 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.968 DEBUG:boto:Method: HEAD 2023-09-29T22:43:45.968 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:43:45.968 DEBUG:boto:Data: 2023-09-29T22:43:45.968 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:45.968 DEBUG:boto:Port: 8000 2023-09-29T22:43:45.968 DEBUG:boto:Params: {} 2023-09-29T22:43:45.968 DEBUG:boto:Token: None 2023-09-29T22:43:45.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68 2023-09-29T22:43:45.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XglWZnjgvECMQTdmzwXFguoutXw= 2023-09-29T22:43:45.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XglWZnjgvECMQTdmzwXFguoutXw=', 'Content-Length': '0'} 2023-09-29T22:43:45.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d58e2cfe7c22a89b-0065175321-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:45.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:43:45.971 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:45.971 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:45.971 DEBUG:boto:Method: GET 2023-09-29T22:43:45.971 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:43:45.972 DEBUG:boto:Data: 2023-09-29T22:43:45.972 DEBUG:boto:Headers: {} 2023-09-29T22:43:45.972 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:45.972 DEBUG:boto:Port: 8001 2023-09-29T22:43:45.972 DEBUG:boto:Params: {} 2023-09-29T22:43:45.972 DEBUG:boto:Token: None 2023-09-29T22:43:45.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:45 GMT /hcnphz-59/obj68 2023-09-29T22:43:45.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fyDojx60driTBcrTidq/OCn4qxE= 2023-09-29T22:43:45.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fyDojx60driTBcrTidq/OCn4qxE=', 'Content-Length': '0'} 2023-09-29T22:43:45.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb749463d7a5e4f8-0065175321-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.018 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:43:46.019 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:43:46.019 DEBUG:boto:Method: GET 2023-09-29T22:43:46.019 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:43:46.019 DEBUG:boto:Data: 2023-09-29T22:43:46.019 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.019 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.019 DEBUG:boto:Params: {} 2023-09-29T22:43:46.020 DEBUG:boto:Token: None 2023-09-29T22:43:46.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj68 2023-09-29T22:43:46.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AacowRwPzuIHZuwyB3hN37XNWBk= 2023-09-29T22:43:46.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AacowRwPzuIHZuwyB3hN37XNWBk=', 'Content-Length': '0'} 2023-09-29T22:43:46.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c7025519bbe54b-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.023 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.023 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.024 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:46.024 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:46.024 DEBUG:boto:Method: GET 2023-09-29T22:43:46.024 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:43:46.024 DEBUG:boto:Data: 2023-09-29T22:43:46.024 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.024 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.024 DEBUG:boto:Params: {} 2023-09-29T22:43:46.024 DEBUG:boto:Token: None 2023-09-29T22:43:46.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.029 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d84fb4f0d0a50f08-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.029 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.030 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.030 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.030 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:46.030 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:43:46.030 DEBUG:boto:Method: GET 2023-09-29T22:43:46.030 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:43:46.030 DEBUG:boto:Data: 2023-09-29T22:43:46.031 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.031 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.031 DEBUG:boto:Params: {} 2023-09-29T22:43:46.031 DEBUG:boto:Token: None 2023-09-29T22:43:46.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6baa8fd36720eef-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.037 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:43:46.037 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.038 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.038 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.038 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.038 DEBUG:boto:Method: GET 2023-09-29T22:43:46.038 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.038 DEBUG:boto:Data: 2023-09-29T22:43:46.038 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.038 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.039 DEBUG:boto:Params: {} 2023-09-29T22:43:46.039 DEBUG:boto:Token: None 2023-09-29T22:43:46.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.039 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k= 2023-09-29T22:43:46.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k=', 'Content-Length': '0'} 2023-09-29T22:43:46.041 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4fd24747ad781b9-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.042 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.042 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.042 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.042 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.042 DEBUG:boto:Method: GET 2023-09-29T22:43:46.042 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.043 DEBUG:boto:Data: 2023-09-29T22:43:46.043 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.043 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.043 DEBUG:boto:Params: {} 2023-09-29T22:43:46.043 DEBUG:boto:Token: None 2023-09-29T22:43:46.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.043 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k= 2023-09-29T22:43:46.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k=', 'Content-Length': '0'} 2023-09-29T22:43:46.046 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009622fc3bcd3202e3-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.046 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.046 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.046 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.047 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.047 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.047 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.047 DEBUG:boto:Data: 2023-09-29T22:43:46.047 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.047 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.047 DEBUG:boto:Params: {} 2023-09-29T22:43:46.047 DEBUG:boto:Token: None 2023-09-29T22:43:46.047 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PROXh6ki2eIQAqO/3nEgoIZL1gk= 2023-09-29T22:43:46.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PROXh6ki2eIQAqO/3nEgoIZL1gk=', 'Content-Length': '0'} 2023-09-29T22:43:46.050 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039d0a6027b4402d2-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.050 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.050 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.050 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.051 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.051 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.051 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.051 DEBUG:boto:Data: 2023-09-29T22:43:46.051 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.051 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.051 DEBUG:boto:Params: {} 2023-09-29T22:43:46.051 DEBUG:boto:Token: None 2023-09-29T22:43:46.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PROXh6ki2eIQAqO/3nEgoIZL1gk= 2023-09-29T22:43:46.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PROXh6ki2eIQAqO/3nEgoIZL1gk=', 'Content-Length': '0'} 2023-09-29T22:43:46.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d78ad8e05b39dfd-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:43:46.054 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.054 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.055 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.055 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.055 DEBUG:boto:Method: GET 2023-09-29T22:43:46.055 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.055 DEBUG:boto:Data: 2023-09-29T22:43:46.055 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.055 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.055 DEBUG:boto:Params: {} 2023-09-29T22:43:46.055 DEBUG:boto:Token: None 2023-09-29T22:43:46.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k= 2023-09-29T22:43:46.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k=', 'Content-Length': '0'} 2023-09-29T22:43:46.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026e7b31bcfa34f3-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.058 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.059 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.059 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.059 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.059 DEBUG:boto:Method: GET 2023-09-29T22:43:46.059 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.059 DEBUG:boto:Data: 2023-09-29T22:43:46.059 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.059 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.060 DEBUG:boto:Params: {} 2023-09-29T22:43:46.060 DEBUG:boto:Token: None 2023-09-29T22:43:46.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:43:46.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k= 2023-09-29T22:43:46.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EKQ0uzRCmzSFC03jbD6qnMIBw2k=', 'Content-Length': '0'} 2023-09-29T22:43:46.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe61862aefdca07e-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.063 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.063 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.063 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.063 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.064 DEBUG:boto:Method: GET 2023-09-29T22:43:46.064 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.064 DEBUG:boto:Data: 2023-09-29T22:43:46.064 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.064 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.064 DEBUG:boto:Params: {} 2023-09-29T22:43:46.064 DEBUG:boto:Token: None 2023-09-29T22:43:46.064 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ce8d7c541d5e9f-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.114 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.115 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.116 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.116 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.116 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.116 DEBUG:boto:Method: GET 2023-09-29T22:43:46.116 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.116 DEBUG:boto:Data: 2023-09-29T22:43:46.116 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.116 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.117 DEBUG:boto:Params: {} 2023-09-29T22:43:46.117 DEBUG:boto:Token: None 2023-09-29T22:43:46.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002600f406f926c94d-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.127 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.127 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.128 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.128 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.128 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:43:46.128 DEBUG:boto:Data: 2023-09-29T22:43:46.129 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.129 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.129 DEBUG:boto:Params: {} 2023-09-29T22:43:46.129 DEBUG:boto:Token: None 2023-09-29T22:43:46.130 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69 2023-09-29T22:43:46.130 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KyCnbDXlhPt54u2BkqmAAlOv2Vk= 2023-09-29T22:43:46.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KyCnbDXlhPt54u2BkqmAAlOv2Vk=', 'Content-Length': '0'} 2023-09-29T22:43:46.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c2d43802ea157bb-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.133 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.133 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.133 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.134 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:43:46.134 DEBUG:boto:Data: 2023-09-29T22:43:46.134 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.134 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.135 DEBUG:boto:Params: {} 2023-09-29T22:43:46.135 DEBUG:boto:Token: None 2023-09-29T22:43:46.135 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69 2023-09-29T22:43:46.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KyCnbDXlhPt54u2BkqmAAlOv2Vk= 2023-09-29T22:43:46.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KyCnbDXlhPt54u2BkqmAAlOv2Vk=', 'Content-Length': '0'} 2023-09-29T22:43:46.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7b865c75764f9bf-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:43:46.138 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.139 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.139 DEBUG:boto:Method: GET 2023-09-29T22:43:46.139 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:43:46.140 DEBUG:boto:Data: 2023-09-29T22:43:46.140 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.140 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.140 DEBUG:boto:Params: {} 2023-09-29T22:43:46.141 DEBUG:boto:Token: None 2023-09-29T22:43:46.141 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69 2023-09-29T22:43:46.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:outvGgju+iMP/UaWcfiE4JfsAOA= 2023-09-29T22:43:46.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:outvGgju+iMP/UaWcfiE4JfsAOA=', 'Content-Length': '0'} 2023-09-29T22:43:46.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098dac2bd20e5ab9c-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.144 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:43:46.144 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:43:46.144 DEBUG:boto:Method: GET 2023-09-29T22:43:46.144 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:43:46.145 DEBUG:boto:Data: 2023-09-29T22:43:46.145 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.145 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.145 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.145 DEBUG:boto:Params: {} 2023-09-29T22:43:46.146 DEBUG:boto:Token: None 2023-09-29T22:43:46.146 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj69 2023-09-29T22:43:46.146 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:outvGgju+iMP/UaWcfiE4JfsAOA= 2023-09-29T22:43:46.147 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:outvGgju+iMP/UaWcfiE4JfsAOA=', 'Content-Length': '0'} 2023-09-29T22:43:46.149 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011bb3b7cef907cf8-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.194 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.195 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.195 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.195 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.195 DEBUG:boto:Method: GET 2023-09-29T22:43:46.195 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.195 DEBUG:boto:Data: 2023-09-29T22:43:46.195 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.195 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.196 DEBUG:boto:Params: {} 2023-09-29T22:43:46.196 DEBUG:boto:Token: None 2023-09-29T22:43:46.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.201 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc95d8e371508f03-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.201 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.202 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.202 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.202 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.202 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.202 DEBUG:boto:Method: GET 2023-09-29T22:43:46.202 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:43:46.202 DEBUG:boto:Data: 2023-09-29T22:43:46.202 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.203 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.203 DEBUG:boto:Params: {} 2023-09-29T22:43:46.203 DEBUG:boto:Token: None 2023-09-29T22:43:46.203 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.203 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008c0c8e0ce85411f4-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.212 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:43:46.212 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.212 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.213 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.213 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.213 DEBUG:boto:Method: GET 2023-09-29T22:43:46.213 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.213 DEBUG:boto:Data: 2023-09-29T22:43:46.213 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.213 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.213 DEBUG:boto:Params: {} 2023-09-29T22:43:46.214 DEBUG:boto:Token: None 2023-09-29T22:43:46.214 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY= 2023-09-29T22:43:46.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY=', 'Content-Length': '0'} 2023-09-29T22:43:46.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052dac8cde800d121-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.216 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.217 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.217 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.217 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.217 DEBUG:boto:Method: GET 2023-09-29T22:43:46.217 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.217 DEBUG:boto:Data: 2023-09-29T22:43:46.217 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.217 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.218 DEBUG:boto:Params: {} 2023-09-29T22:43:46.218 DEBUG:boto:Token: None 2023-09-29T22:43:46.218 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.218 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY= 2023-09-29T22:43:46.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY=', 'Content-Length': '0'} 2023-09-29T22:43:46.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9a8259535da2fef-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.221 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.221 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.221 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.221 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.221 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.221 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.221 DEBUG:boto:Data: 2023-09-29T22:43:46.221 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.222 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.222 DEBUG:boto:Params: {} 2023-09-29T22:43:46.222 DEBUG:boto:Token: None 2023-09-29T22:43:46.222 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.222 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uQcZi+46Bj2lSbNSrK5cHBYI2yA= 2023-09-29T22:43:46.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uQcZi+46Bj2lSbNSrK5cHBYI2yA=', 'Content-Length': '0'} 2023-09-29T22:43:46.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009045fdb385108fea-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.225 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.225 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.225 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.225 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.225 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.225 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.225 DEBUG:boto:Data: 2023-09-29T22:43:46.225 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.226 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.226 DEBUG:boto:Params: {} 2023-09-29T22:43:46.226 DEBUG:boto:Token: None 2023-09-29T22:43:46.226 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.226 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uQcZi+46Bj2lSbNSrK5cHBYI2yA= 2023-09-29T22:43:46.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uQcZi+46Bj2lSbNSrK5cHBYI2yA=', 'Content-Length': '0'} 2023-09-29T22:43:46.229 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013741a6a891477e3-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:43:46.230 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.230 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.230 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.230 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.231 DEBUG:boto:Method: GET 2023-09-29T22:43:46.231 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.231 DEBUG:boto:Data: 2023-09-29T22:43:46.231 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.231 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.232 DEBUG:boto:Params: {} 2023-09-29T22:43:46.232 DEBUG:boto:Token: None 2023-09-29T22:43:46.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY= 2023-09-29T22:43:46.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY=', 'Content-Length': '0'} 2023-09-29T22:43:46.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e91bf83d3220b9e0-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.235 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.236 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.236 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.236 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.236 DEBUG:boto:Method: GET 2023-09-29T22:43:46.236 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.237 DEBUG:boto:Data: 2023-09-29T22:43:46.237 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.237 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.237 DEBUG:boto:Params: {} 2023-09-29T22:43:46.237 DEBUG:boto:Token: None 2023-09-29T22:43:46.238 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:43:46.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY= 2023-09-29T22:43:46.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:27P0nWegSXyS2DGiDoQQlzEi3PY=', 'Content-Length': '0'} 2023-09-29T22:43:46.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c93c3dd67f120268-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.241 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.241 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.241 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.241 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.242 DEBUG:boto:Method: GET 2023-09-29T22:43:46.242 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.242 DEBUG:boto:Data: 2023-09-29T22:43:46.242 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.242 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.242 DEBUG:boto:Params: {} 2023-09-29T22:43:46.242 DEBUG:boto:Token: None 2023-09-29T22:43:46.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.247 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000688927fd671dfb3f-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.290 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.291 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.291 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.291 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.291 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.291 DEBUG:boto:Method: GET 2023-09-29T22:43:46.292 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.292 DEBUG:boto:Data: 2023-09-29T22:43:46.292 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.292 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.292 DEBUG:boto:Params: {} 2023-09-29T22:43:46.293 DEBUG:boto:Token: None 2023-09-29T22:43:46.293 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.300 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c034d18d9683510b-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.300 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.301 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.301 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.301 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.301 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:43:46.301 DEBUG:boto:Data: 2023-09-29T22:43:46.301 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.302 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.302 DEBUG:boto:Params: {} 2023-09-29T22:43:46.302 DEBUG:boto:Token: None 2023-09-29T22:43:46.302 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76 2023-09-29T22:43:46.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2YukxSkvNL26vtB5uu9lLOFH3VE= 2023-09-29T22:43:46.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2YukxSkvNL26vtB5uu9lLOFH3VE=', 'Content-Length': '0'} 2023-09-29T22:43:46.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000178ab0f2c6a627e-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.305 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.305 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.305 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.305 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:43:46.306 DEBUG:boto:Data: 2023-09-29T22:43:46.306 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.306 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.306 DEBUG:boto:Params: {} 2023-09-29T22:43:46.306 DEBUG:boto:Token: None 2023-09-29T22:43:46.306 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76 2023-09-29T22:43:46.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2YukxSkvNL26vtB5uu9lLOFH3VE= 2023-09-29T22:43:46.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2YukxSkvNL26vtB5uu9lLOFH3VE=', 'Content-Length': '0'} 2023-09-29T22:43:46.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f6680fe87a635c2-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:43:46.309 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.309 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.309 DEBUG:boto:Method: GET 2023-09-29T22:43:46.310 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:43:46.310 DEBUG:boto:Data: 2023-09-29T22:43:46.310 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.310 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.310 DEBUG:boto:Params: {} 2023-09-29T22:43:46.310 DEBUG:boto:Token: None 2023-09-29T22:43:46.310 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76 2023-09-29T22:43:46.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3kESihe+UYhQ26uiM8dPrLl8E0= 2023-09-29T22:43:46.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3kESihe+UYhQ26uiM8dPrLl8E0=', 'Content-Length': '0'} 2023-09-29T22:43:46.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae6a709d6267e61f-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.313 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:43:46.313 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:43:46.314 DEBUG:boto:Method: GET 2023-09-29T22:43:46.314 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:43:46.314 DEBUG:boto:Data: 2023-09-29T22:43:46.314 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.314 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.314 DEBUG:boto:Params: {} 2023-09-29T22:43:46.314 DEBUG:boto:Token: None 2023-09-29T22:43:46.315 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj76 2023-09-29T22:43:46.315 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I3kESihe+UYhQ26uiM8dPrLl8E0= 2023-09-29T22:43:46.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I3kESihe+UYhQ26uiM8dPrLl8E0=', 'Content-Length': '0'} 2023-09-29T22:43:46.317 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077d07eb9d31f92e8-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.358 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.359 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.359 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.359 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.359 DEBUG:boto:Method: GET 2023-09-29T22:43:46.359 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.359 DEBUG:boto:Data: 2023-09-29T22:43:46.359 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.360 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.360 DEBUG:boto:Params: {} 2023-09-29T22:43:46.360 DEBUG:boto:Token: None 2023-09-29T22:43:46.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.365 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004fcd7274b2f60ba0-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.365 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.366 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.366 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.366 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.366 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.366 DEBUG:boto:Method: GET 2023-09-29T22:43:46.367 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:43:46.367 DEBUG:boto:Data: 2023-09-29T22:43:46.367 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.367 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.367 DEBUG:boto:Params: {} 2023-09-29T22:43:46.367 DEBUG:boto:Token: None 2023-09-29T22:43:46.367 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b82cd5122e1d4fe-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.374 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:43:46.375 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.375 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.376 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.376 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.376 DEBUG:boto:Method: GET 2023-09-29T22:43:46.376 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.376 DEBUG:boto:Data: 2023-09-29T22:43:46.376 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.376 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.376 DEBUG:boto:Params: {} 2023-09-29T22:43:46.377 DEBUG:boto:Token: None 2023-09-29T22:43:46.377 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo= 2023-09-29T22:43:46.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo=', 'Content-Length': '0'} 2023-09-29T22:43:46.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050f69eae1a7e41f0-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.379 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.380 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.380 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.380 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.380 DEBUG:boto:Method: GET 2023-09-29T22:43:46.380 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.380 DEBUG:boto:Data: 2023-09-29T22:43:46.380 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.380 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.381 DEBUG:boto:Params: {} 2023-09-29T22:43:46.381 DEBUG:boto:Token: None 2023-09-29T22:43:46.381 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo= 2023-09-29T22:43:46.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo=', 'Content-Length': '0'} 2023-09-29T22:43:46.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000916a5d0b765eabd-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.384 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.384 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.384 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.384 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.384 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.384 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.384 DEBUG:boto:Data: 2023-09-29T22:43:46.384 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.385 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.385 DEBUG:boto:Params: {} 2023-09-29T22:43:46.385 DEBUG:boto:Token: None 2023-09-29T22:43:46.385 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Jc9es/ytiYh4Lbfge7OypUtL6/s= 2023-09-29T22:43:46.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Jc9es/ytiYh4Lbfge7OypUtL6/s=', 'Content-Length': '0'} 2023-09-29T22:43:46.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5b6ae29c1f141d8-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.388 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.388 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.388 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.388 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.388 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.388 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.389 DEBUG:boto:Data: 2023-09-29T22:43:46.389 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.389 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.389 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.389 DEBUG:boto:Params: {} 2023-09-29T22:43:46.389 DEBUG:boto:Token: None 2023-09-29T22:43:46.389 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.390 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Jc9es/ytiYh4Lbfge7OypUtL6/s= 2023-09-29T22:43:46.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Jc9es/ytiYh4Lbfge7OypUtL6/s=', 'Content-Length': '0'} 2023-09-29T22:43:46.392 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d96dd6a40ebed79-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:43:46.392 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.393 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.393 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.393 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.393 DEBUG:boto:Method: GET 2023-09-29T22:43:46.393 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.393 DEBUG:boto:Data: 2023-09-29T22:43:46.393 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.393 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.393 DEBUG:boto:Params: {} 2023-09-29T22:43:46.394 DEBUG:boto:Token: None 2023-09-29T22:43:46.394 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo= 2023-09-29T22:43:46.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo=', 'Content-Length': '0'} 2023-09-29T22:43:46.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000433a0de2a3e43686-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.438 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.439 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.439 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.439 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.439 DEBUG:boto:Method: GET 2023-09-29T22:43:46.439 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.439 DEBUG:boto:Data: 2023-09-29T22:43:46.439 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.440 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.440 DEBUG:boto:Params: {} 2023-09-29T22:43:46.440 DEBUG:boto:Token: None 2023-09-29T22:43:46.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:43:46.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo= 2023-09-29T22:43:46.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PiFRgBFuL+WLMNAC4ETs5XVaHvo=', 'Content-Length': '0'} 2023-09-29T22:43:46.444 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000400fadd30a7c804c-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.444 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.444 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.444 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.444 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.444 DEBUG:boto:Method: GET 2023-09-29T22:43:46.444 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.445 DEBUG:boto:Data: 2023-09-29T22:43:46.445 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.445 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.445 DEBUG:boto:Params: {} 2023-09-29T22:43:46.445 DEBUG:boto:Token: None 2023-09-29T22:43:46.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.450 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f5d9b6297a5be3-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.450 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.451 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.451 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.451 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.451 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.451 DEBUG:boto:Method: GET 2023-09-29T22:43:46.451 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.451 DEBUG:boto:Data: 2023-09-29T22:43:46.451 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.452 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.452 DEBUG:boto:Params: {} 2023-09-29T22:43:46.452 DEBUG:boto:Token: None 2023-09-29T22:43:46.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.459 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a85eefbdc021ff2b-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.459 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.459 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.459 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.460 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.460 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:43:46.460 DEBUG:boto:Data: 2023-09-29T22:43:46.460 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.460 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.460 DEBUG:boto:Params: {} 2023-09-29T22:43:46.460 DEBUG:boto:Token: None 2023-09-29T22:43:46.461 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85 2023-09-29T22:43:46.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O4e7g1FD1NdPTjDSmdDCvfpMXh0= 2023-09-29T22:43:46.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O4e7g1FD1NdPTjDSmdDCvfpMXh0=', 'Content-Length': '0'} 2023-09-29T22:43:46.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072ac47efce9e593e-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.463 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.463 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.464 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.464 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:43:46.464 DEBUG:boto:Data: 2023-09-29T22:43:46.464 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.464 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.464 DEBUG:boto:Params: {} 2023-09-29T22:43:46.464 DEBUG:boto:Token: None 2023-09-29T22:43:46.465 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85 2023-09-29T22:43:46.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O4e7g1FD1NdPTjDSmdDCvfpMXh0= 2023-09-29T22:43:46.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O4e7g1FD1NdPTjDSmdDCvfpMXh0=', 'Content-Length': '0'} 2023-09-29T22:43:46.467 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082b986bf399701d7-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:43:46.467 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.467 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.468 DEBUG:boto:Method: GET 2023-09-29T22:43:46.468 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:43:46.468 DEBUG:boto:Data: 2023-09-29T22:43:46.468 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.468 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.468 DEBUG:boto:Params: {} 2023-09-29T22:43:46.468 DEBUG:boto:Token: None 2023-09-29T22:43:46.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85 2023-09-29T22:43:46.469 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GB5mVU3OXF3I35/6IRYiXbATb5M= 2023-09-29T22:43:46.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GB5mVU3OXF3I35/6IRYiXbATb5M=', 'Content-Length': '0'} 2023-09-29T22:43:46.471 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000033aaa91743e0785-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.518 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:43:46.519 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:43:46.519 DEBUG:boto:Method: GET 2023-09-29T22:43:46.519 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:43:46.519 DEBUG:boto:Data: 2023-09-29T22:43:46.519 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.520 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.520 DEBUG:boto:Params: {} 2023-09-29T22:43:46.520 DEBUG:boto:Token: None 2023-09-29T22:43:46.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj85 2023-09-29T22:43:46.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GB5mVU3OXF3I35/6IRYiXbATb5M= 2023-09-29T22:43:46.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GB5mVU3OXF3I35/6IRYiXbATb5M=', 'Content-Length': '0'} 2023-09-29T22:43:46.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b30c9455318d246a-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.524 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.524 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.524 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.524 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.525 DEBUG:boto:Method: GET 2023-09-29T22:43:46.525 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.525 DEBUG:boto:Data: 2023-09-29T22:43:46.525 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.525 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.525 DEBUG:boto:Params: {} 2023-09-29T22:43:46.525 DEBUG:boto:Token: None 2023-09-29T22:43:46.526 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.526 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.531 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a3f0ac398f450b-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.531 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.532 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.532 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.532 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.532 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.532 DEBUG:boto:Method: GET 2023-09-29T22:43:46.532 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:43:46.532 DEBUG:boto:Data: 2023-09-29T22:43:46.532 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.533 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.533 DEBUG:boto:Params: {} 2023-09-29T22:43:46.533 DEBUG:boto:Token: None 2023-09-29T22:43:46.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.539 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7075dae8d141875-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.586 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:43:46.587 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.587 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.587 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.587 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.588 DEBUG:boto:Method: GET 2023-09-29T22:43:46.588 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.588 DEBUG:boto:Data: 2023-09-29T22:43:46.588 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.588 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.588 DEBUG:boto:Params: {} 2023-09-29T22:43:46.588 DEBUG:boto:Token: None 2023-09-29T22:43:46.589 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc= 2023-09-29T22:43:46.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc=', 'Content-Length': '0'} 2023-09-29T22:43:46.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ae336ec6aca3946-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.592 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.592 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.592 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.592 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.592 DEBUG:boto:Method: GET 2023-09-29T22:43:46.593 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.593 DEBUG:boto:Data: 2023-09-29T22:43:46.593 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.593 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.593 DEBUG:boto:Params: {} 2023-09-29T22:43:46.593 DEBUG:boto:Token: None 2023-09-29T22:43:46.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc= 2023-09-29T22:43:46.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc=', 'Content-Length': '0'} 2023-09-29T22:43:46.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009ddff0c73af6ae2-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.596 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.597 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.597 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.597 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.597 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.597 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.597 DEBUG:boto:Data: 2023-09-29T22:43:46.597 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.597 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.597 DEBUG:boto:Params: {} 2023-09-29T22:43:46.598 DEBUG:boto:Token: None 2023-09-29T22:43:46.598 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.598 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d11O7FwRn1I73EY/WNWAvkz2R5Y= 2023-09-29T22:43:46.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d11O7FwRn1I73EY/WNWAvkz2R5Y=', 'Content-Length': '0'} 2023-09-29T22:43:46.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b3449b9bed7ebbb-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.601 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.601 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.601 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.601 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.601 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.601 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.602 DEBUG:boto:Data: 2023-09-29T22:43:46.602 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.602 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.602 DEBUG:boto:Params: {} 2023-09-29T22:43:46.602 DEBUG:boto:Token: None 2023-09-29T22:43:46.602 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.602 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d11O7FwRn1I73EY/WNWAvkz2R5Y= 2023-09-29T22:43:46.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d11O7FwRn1I73EY/WNWAvkz2R5Y=', 'Content-Length': '0'} 2023-09-29T22:43:46.605 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a4f417a496b09f5-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:43:46.605 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.605 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.605 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.605 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.606 DEBUG:boto:Method: GET 2023-09-29T22:43:46.606 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.606 DEBUG:boto:Data: 2023-09-29T22:43:46.606 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.606 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.606 DEBUG:boto:Params: {} 2023-09-29T22:43:46.606 DEBUG:boto:Token: None 2023-09-29T22:43:46.606 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc= 2023-09-29T22:43:46.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc=', 'Content-Length': '0'} 2023-09-29T22:43:46.609 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065fb1c3ffa0c9b59-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.609 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.609 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.610 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.610 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.610 DEBUG:boto:Method: GET 2023-09-29T22:43:46.610 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.610 DEBUG:boto:Data: 2023-09-29T22:43:46.610 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.610 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.610 DEBUG:boto:Params: {} 2023-09-29T22:43:46.611 DEBUG:boto:Token: None 2023-09-29T22:43:46.611 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:43:46.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc= 2023-09-29T22:43:46.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E1Zvi74TKHwPkhEl9O0hVDV55tc=', 'Content-Length': '0'} 2023-09-29T22:43:46.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000054b597cf692e6df7-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.614 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.614 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.614 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.614 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.614 DEBUG:boto:Method: GET 2023-09-29T22:43:46.614 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.614 DEBUG:boto:Data: 2023-09-29T22:43:46.614 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.615 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.615 DEBUG:boto:Params: {} 2023-09-29T22:43:46.615 DEBUG:boto:Token: None 2023-09-29T22:43:46.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050e1b8b2e2c4564d-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.621 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.621 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.621 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.621 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.622 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.622 DEBUG:boto:Method: GET 2023-09-29T22:43:46.622 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.622 DEBUG:boto:Data: 2023-09-29T22:43:46.622 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.623 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.623 DEBUG:boto:Params: {} 2023-09-29T22:43:46.623 DEBUG:boto:Token: None 2023-09-29T22:43:46.623 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.624 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ae2dac11b8606ba-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.630 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.631 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.631 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.631 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.631 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:43:46.632 DEBUG:boto:Data: 2023-09-29T22:43:46.632 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.632 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.632 DEBUG:boto:Params: {} 2023-09-29T22:43:46.633 DEBUG:boto:Token: None 2023-09-29T22:43:46.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92 2023-09-29T22:43:46.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cJ6XvrODZJEFJmAtQvZB1q+uX7w= 2023-09-29T22:43:46.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cJ6XvrODZJEFJmAtQvZB1q+uX7w=', 'Content-Length': '0'} 2023-09-29T22:43:46.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099fc0f1ffe9c28bd-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.636 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.636 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.636 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.636 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:43:46.637 DEBUG:boto:Data: 2023-09-29T22:43:46.637 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.637 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.637 DEBUG:boto:Params: {} 2023-09-29T22:43:46.638 DEBUG:boto:Token: None 2023-09-29T22:43:46.638 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92 2023-09-29T22:43:46.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cJ6XvrODZJEFJmAtQvZB1q+uX7w= 2023-09-29T22:43:46.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cJ6XvrODZJEFJmAtQvZB1q+uX7w=', 'Content-Length': '0'} 2023-09-29T22:43:46.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdf161a80cc2964b-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:43:46.641 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.641 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.641 DEBUG:boto:Method: GET 2023-09-29T22:43:46.641 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:43:46.641 DEBUG:boto:Data: 2023-09-29T22:43:46.641 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.641 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.641 DEBUG:boto:Params: {} 2023-09-29T22:43:46.642 DEBUG:boto:Token: None 2023-09-29T22:43:46.642 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92 2023-09-29T22:43:46.642 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mU8bARwxW23aAltdmHr6g/+eCxk= 2023-09-29T22:43:46.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mU8bARwxW23aAltdmHr6g/+eCxk=', 'Content-Length': '0'} 2023-09-29T22:43:46.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db8542c6a5d87af-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.690 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:43:46.691 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:43:46.691 DEBUG:boto:Method: GET 2023-09-29T22:43:46.691 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:43:46.691 DEBUG:boto:Data: 2023-09-29T22:43:46.691 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.692 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.692 DEBUG:boto:Params: {} 2023-09-29T22:43:46.692 DEBUG:boto:Token: None 2023-09-29T22:43:46.693 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj92 2023-09-29T22:43:46.693 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mU8bARwxW23aAltdmHr6g/+eCxk= 2023-09-29T22:43:46.693 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mU8bARwxW23aAltdmHr6g/+eCxk=', 'Content-Length': '0'} 2023-09-29T22:43:46.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46bf56a45cce47b-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.696 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.696 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.696 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.697 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.697 DEBUG:boto:Method: GET 2023-09-29T22:43:46.697 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.697 DEBUG:boto:Data: 2023-09-29T22:43:46.697 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.697 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.697 DEBUG:boto:Params: {} 2023-09-29T22:43:46.697 DEBUG:boto:Token: None 2023-09-29T22:43:46.698 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007cb15c2acd71b5d-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.703 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.703 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.703 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.703 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.704 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.704 DEBUG:boto:Method: GET 2023-09-29T22:43:46.704 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:43:46.704 DEBUG:boto:Data: 2023-09-29T22:43:46.704 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.704 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.704 DEBUG:boto:Params: {} 2023-09-29T22:43:46.704 DEBUG:boto:Token: None 2023-09-29T22:43:46.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3b9dc4b6f76c3ce-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.712 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:43:46.712 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.712 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.713 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.713 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.713 DEBUG:boto:Method: GET 2023-09-29T22:43:46.713 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.713 DEBUG:boto:Data: 2023-09-29T22:43:46.713 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.713 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.713 DEBUG:boto:Params: {} 2023-09-29T22:43:46.714 DEBUG:boto:Token: None 2023-09-29T22:43:46.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4= 2023-09-29T22:43:46.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4=', 'Content-Length': '0'} 2023-09-29T22:43:46.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc1dcffaa20f644c-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.717 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.717 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.717 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.717 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.717 DEBUG:boto:Method: GET 2023-09-29T22:43:46.717 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.717 DEBUG:boto:Data: 2023-09-29T22:43:46.718 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.718 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.718 DEBUG:boto:Params: {} 2023-09-29T22:43:46.718 DEBUG:boto:Token: None 2023-09-29T22:43:46.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4= 2023-09-29T22:43:46.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4=', 'Content-Length': '0'} 2023-09-29T22:43:46.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040877eb3fd01c32b-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.721 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.721 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.721 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.721 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.721 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.724 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.724 DEBUG:boto:Data: 2023-09-29T22:43:46.725 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.725 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.725 DEBUG:boto:Params: {} 2023-09-29T22:43:46.725 DEBUG:boto:Token: None 2023-09-29T22:43:46.725 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FZ1nI/XMf7rdjpoj61kTSJV7M1U= 2023-09-29T22:43:46.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FZ1nI/XMf7rdjpoj61kTSJV7M1U=', 'Content-Length': '0'} 2023-09-29T22:43:46.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e07af301cdfdc91-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.728 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.728 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.729 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.729 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.729 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.729 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.729 DEBUG:boto:Data: 2023-09-29T22:43:46.729 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.729 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.729 DEBUG:boto:Params: {} 2023-09-29T22:43:46.730 DEBUG:boto:Token: None 2023-09-29T22:43:46.730 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FZ1nI/XMf7rdjpoj61kTSJV7M1U= 2023-09-29T22:43:46.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FZ1nI/XMf7rdjpoj61kTSJV7M1U=', 'Content-Length': '0'} 2023-09-29T22:43:46.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066d276306b52ad3b-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:43:46.733 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.733 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.733 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.733 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.733 DEBUG:boto:Method: GET 2023-09-29T22:43:46.733 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.734 DEBUG:boto:Data: 2023-09-29T22:43:46.734 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.734 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.734 DEBUG:boto:Params: {} 2023-09-29T22:43:46.734 DEBUG:boto:Token: None 2023-09-29T22:43:46.734 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4= 2023-09-29T22:43:46.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4=', 'Content-Length': '0'} 2023-09-29T22:43:46.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2743b054fd04873-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.778 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.779 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.779 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.779 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.779 DEBUG:boto:Method: GET 2023-09-29T22:43:46.779 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.779 DEBUG:boto:Data: 2023-09-29T22:43:46.779 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.779 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.779 DEBUG:boto:Params: {} 2023-09-29T22:43:46.780 DEBUG:boto:Token: None 2023-09-29T22:43:46.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:43:46.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4= 2023-09-29T22:43:46.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbw3mXo2fDjNj2lisAysdTi/2j4=', 'Content-Length': '0'} 2023-09-29T22:43:46.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ec9b94a20231d74-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.783 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.783 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.784 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.784 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.784 DEBUG:boto:Method: GET 2023-09-29T22:43:46.784 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.784 DEBUG:boto:Data: 2023-09-29T22:43:46.784 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.784 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.784 DEBUG:boto:Params: {} 2023-09-29T22:43:46.784 DEBUG:boto:Token: None 2023-09-29T22:43:46.785 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.785 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.790 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074f4ef2fa5343270-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.790 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.790 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.790 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.790 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.791 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.791 DEBUG:boto:Method: GET 2023-09-29T22:43:46.791 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.791 DEBUG:boto:Data: 2023-09-29T22:43:46.791 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.791 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.791 DEBUG:boto:Params: {} 2023-09-29T22:43:46.791 DEBUG:boto:Token: None 2023-09-29T22:43:46.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa194d811dfc25f4-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.838 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.839 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.839 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.839 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.839 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:43:46.839 DEBUG:boto:Data: 2023-09-29T22:43:46.839 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.840 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.840 DEBUG:boto:Params: {} 2023-09-29T22:43:46.840 DEBUG:boto:Token: None 2023-09-29T22:43:46.840 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95 2023-09-29T22:43:46.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+8qdZWEf06zO5qwtuGXosq1M78Y= 2023-09-29T22:43:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+8qdZWEf06zO5qwtuGXosq1M78Y=', 'Content-Length': '0'} 2023-09-29T22:43:46.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec14d39e66530b9c-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.844 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.844 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.844 DEBUG:boto:Method: HEAD 2023-09-29T22:43:46.844 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:43:46.844 DEBUG:boto:Data: 2023-09-29T22:43:46.844 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.844 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.845 DEBUG:boto:Params: {} 2023-09-29T22:43:46.845 DEBUG:boto:Token: None 2023-09-29T22:43:46.845 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95 2023-09-29T22:43:46.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+8qdZWEf06zO5qwtuGXosq1M78Y= 2023-09-29T22:43:46.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+8qdZWEf06zO5qwtuGXosq1M78Y=', 'Content-Length': '0'} 2023-09-29T22:43:46.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ea3ef457ed47e23-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:43:46.848 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.848 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.848 DEBUG:boto:Method: GET 2023-09-29T22:43:46.848 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:43:46.848 DEBUG:boto:Data: 2023-09-29T22:43:46.849 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.849 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.849 DEBUG:boto:Params: {} 2023-09-29T22:43:46.849 DEBUG:boto:Token: None 2023-09-29T22:43:46.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95 2023-09-29T22:43:46.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WUUG/Xsui1C0bNLgwb92hnm/Pf0= 2023-09-29T22:43:46.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WUUG/Xsui1C0bNLgwb92hnm/Pf0=', 'Content-Length': '0'} 2023-09-29T22:43:46.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c688566f27a72537-0065175322-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.852 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:43:46.853 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:43:46.853 DEBUG:boto:Method: GET 2023-09-29T22:43:46.853 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:43:46.853 DEBUG:boto:Data: 2023-09-29T22:43:46.853 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.853 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.853 DEBUG:boto:Params: {} 2023-09-29T22:43:46.853 DEBUG:boto:Token: None 2023-09-29T22:43:46.854 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/obj95 2023-09-29T22:43:46.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WUUG/Xsui1C0bNLgwb92hnm/Pf0= 2023-09-29T22:43:46.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WUUG/Xsui1C0bNLgwb92hnm/Pf0=', 'Content-Length': '0'} 2023-09-29T22:43:46.857 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037c442eb8cfa9712-0065175322-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.857 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.857 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.857 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.857 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.857 DEBUG:boto:Method: GET 2023-09-29T22:43:46.857 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.858 DEBUG:boto:Data: 2023-09-29T22:43:46.858 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:43:46.858 DEBUG:boto:Port: 8001 2023-09-29T22:43:46.858 DEBUG:boto:Params: {} 2023-09-29T22:43:46.858 DEBUG:boto:Token: None 2023-09-29T22:43:46.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.858 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005482c3b0d7f18172-0065175322-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.906 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.907 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:46.907 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:46.907 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.907 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.907 DEBUG:boto:Method: GET 2023-09-29T22:43:46.907 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:43:46.909 DEBUG:boto:Data: 2023-09-29T22:43:46.909 DEBUG:boto:Headers: {} 2023-09-29T22:43:46.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:46.909 DEBUG:boto:Port: 8000 2023-09-29T22:43:46.909 DEBUG:boto:Params: {} 2023-09-29T22:43:46.910 DEBUG:boto:Token: None 2023-09-29T22:43:46.910 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:43:46 GMT /hcnphz-59/ 2023-09-29T22:43:46.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo= 2023-09-29T22:43:46.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DxP4yOuESCTtejE0Z5cInOKZoKo=', 'Content-Length': '0'} 2023-09-29T22:43:46.917 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe4a58a10cdc1a28-0065175322-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:43:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:46.917 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:43:46.918 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:43:46.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:43:46.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:43:46.919 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:43:46.964 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:46.978 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:47.044 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:43:47.045 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-59 2023-09-29T22:43:47.061 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-09-29T22:43:47.159 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:47.160 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:47.160 DEBUG:boto:Method: HEAD 2023-09-29T22:43:47.160 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:47.160 DEBUG:boto:Data: 2023-09-29T22:43:47.160 DEBUG:boto:Headers: {} 2023-09-29T22:43:47.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.160 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.160 DEBUG:boto:Params: {} 2023-09-29T22:43:47.161 DEBUG:boto:Token: None 2023-09-29T22:43:47.161 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/ 2023-09-29T22:43:47.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI= 2023-09-29T22:43:47.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI=', 'Content-Length': '0'} 2023-09-29T22:43:47.166 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000a0d689ec469a546-0065175323-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.167 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:43:47.167 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:43:47.167 DEBUG:boto:Method: PUT 2023-09-29T22:43:47.167 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:43:47.167 DEBUG:boto:Data: 2023-09-29T22:43:47.167 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:43:47.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.167 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.168 DEBUG:boto:Params: {} 2023-09-29T22:43:47.168 DEBUG:boto:Token: None 2023-09-29T22:43:47.168 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/obj29 2023-09-29T22:43:47.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HsYSiBSHA+A1B/QKKynCgQI0Qx8= 2023-09-29T22:43:47.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HsYSiBSHA+A1B/QKKynCgQI0Qx8='} 2023-09-29T22:43:47.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b6d6e0ed4f3955e3-0065175323-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.211 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:47.211 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:47.211 DEBUG:boto:Method: HEAD 2023-09-29T22:43:47.211 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:47.211 DEBUG:boto:Data: 2023-09-29T22:43:47.211 DEBUG:boto:Headers: {} 2023-09-29T22:43:47.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.212 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.212 DEBUG:boto:Params: {} 2023-09-29T22:43:47.212 DEBUG:boto:Token: None 2023-09-29T22:43:47.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/ 2023-09-29T22:43:47.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI= 2023-09-29T22:43:47.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI=', 'Content-Length': '0'} 2023-09-29T22:43:47.218 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c6e6c1a313cdbea0-0065175323-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.218 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:43:47.219 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:43:47.219 DEBUG:boto:Method: PUT 2023-09-29T22:43:47.219 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:43:47.219 DEBUG:boto:Data: 2023-09-29T22:43:47.219 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:43:47.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.219 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.219 DEBUG:boto:Params: {} 2023-09-29T22:43:47.219 DEBUG:boto:Token: None 2023-09-29T22:43:47.220 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/obj58 2023-09-29T22:43:47.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6wi2/cF7BtrOJwLwPxxn2ZPL0Uk= 2023-09-29T22:43:47.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6wi2/cF7BtrOJwLwPxxn2ZPL0Uk='} 2023-09-29T22:43:47.229 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3b500ac31ad8ef3-0065175323-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.230 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:47.230 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:47.230 DEBUG:boto:Method: HEAD 2023-09-29T22:43:47.230 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:47.230 DEBUG:boto:Data: 2023-09-29T22:43:47.230 DEBUG:boto:Headers: {} 2023-09-29T22:43:47.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.231 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.231 DEBUG:boto:Params: {} 2023-09-29T22:43:47.231 DEBUG:boto:Token: None 2023-09-29T22:43:47.231 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/ 2023-09-29T22:43:47.231 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI= 2023-09-29T22:43:47.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI=', 'Content-Length': '0'} 2023-09-29T22:43:47.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000e70b52cd94ab678-0065175323-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.237 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:43:47.237 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:43:47.237 DEBUG:boto:Method: PUT 2023-09-29T22:43:47.237 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:43:47.237 DEBUG:boto:Data: 2023-09-29T22:43:47.237 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:43:47.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.238 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.238 DEBUG:boto:Params: {} 2023-09-29T22:43:47.238 DEBUG:boto:Token: None 2023-09-29T22:43:47.238 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/obj87 2023-09-29T22:43:47.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rzsS5h4pEHjWMC0rauxyXh/QTA4= 2023-09-29T22:43:47.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rzsS5h4pEHjWMC0rauxyXh/QTA4='} 2023-09-29T22:43:47.282 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f5fbd4e6f8d3c34-0065175323-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.283 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:47.283 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:47.283 DEBUG:boto:Method: HEAD 2023-09-29T22:43:47.283 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:47.283 DEBUG:boto:Data: 2023-09-29T22:43:47.283 DEBUG:boto:Headers: {} 2023-09-29T22:43:47.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.284 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.284 DEBUG:boto:Params: {} 2023-09-29T22:43:47.284 DEBUG:boto:Token: None 2023-09-29T22:43:47.284 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/ 2023-09-29T22:43:47.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI= 2023-09-29T22:43:47.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI=', 'Content-Length': '0'} 2023-09-29T22:43:47.291 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000094b6a0f4f12a49e7-0065175323-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.292 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:43:47.292 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:43:47.292 DEBUG:boto:Method: PUT 2023-09-29T22:43:47.292 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:43:47.292 DEBUG:boto:Data: 2023-09-29T22:43:47.292 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:43:47.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.292 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.293 DEBUG:boto:Params: {} 2023-09-29T22:43:47.293 DEBUG:boto:Token: None 2023-09-29T22:43:47.293 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/obj116 2023-09-29T22:43:47.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NEdXJBdGYb/AS4O8ceeKqG8xGhc= 2023-09-29T22:43:47.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NEdXJBdGYb/AS4O8ceeKqG8xGhc='} 2023-09-29T22:43:47.302 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000030ae6ccc5418cdd8-0065175323-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.302 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:43:47.302 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:43:47.302 DEBUG:boto:Method: HEAD 2023-09-29T22:43:47.302 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:43:47.302 DEBUG:boto:Data: 2023-09-29T22:43:47.303 DEBUG:boto:Headers: {} 2023-09-29T22:43:47.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.303 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.303 DEBUG:boto:Params: {} 2023-09-29T22:43:47.303 DEBUG:boto:Token: None 2023-09-29T22:43:47.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/ 2023-09-29T22:43:47.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI= 2023-09-29T22:43:47.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZKt5CvmeZVaqwYFM8905X2BjLaI=', 'Content-Length': '0'} 2023-09-29T22:43:47.308 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b23d9e7a0deb0148-0065175323-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.308 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:43:47.309 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:43:47.309 DEBUG:boto:Method: PUT 2023-09-29T22:43:47.309 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:43:47.309 DEBUG:boto:Data: 2023-09-29T22:43:47.309 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:43:47.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:43:47.309 DEBUG:boto:Port: 8000 2023-09-29T22:43:47.309 DEBUG:boto:Params: {} 2023-09-29T22:43:47.310 DEBUG:boto:Token: None 2023-09-29T22:43:47.310 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:43:47 GMT /hcnphz-59/obj145 2023-09-29T22:43:47.310 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MpJ1ayJuoZ73VzuqtTcrehgLTzU= 2023-09-29T22:43:47.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:43:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MpJ1ayJuoZ73VzuqtTcrehgLTzU='} 2023-09-29T22:43:47.359 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e36a5d89aaa021a6-0065175323-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:43:47.359 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:43:47.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:43:47.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:43:47.359 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:43:47.411 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:43:47.412 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:43:47.480 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:47.481+0000 7f5f36cbff40 1 waiting to reach latest gen 4, on 3.. 2023-09-29T22:43:52.481 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:52.481+0000 7f5f36cbff40 1 waiting to reach latest gen 4, on 3.. 2023-09-29T22:43:57.482 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:43:57.481+0000 7f5f36cbff40 1 waiting to reach latest gen 4, on 3.. 2023-09-29T22:44:02.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:02.489+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:02.489 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:02.489 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:07.493 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:07.493+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:07.493 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:07.493 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:12.497 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:12.496+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:12.497 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:12.498 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:17.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:17.500+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:17.502 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:17.503 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:22.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:22.504+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:22.507 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:22.507 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:27.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:27.512+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:27.512 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:27.512 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:32.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:32.516+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:32.517 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:32.517 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:37.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:37.520+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:37.522 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:37.522 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:42.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:42.524+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:42.526 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:42.527 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:47.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:47.528+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:47.531 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:47.531 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:52.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:52.531+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:52.535 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:52.535 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:44:57.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:44:57.535+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:44:57.539 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:44:57.540 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:02.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:02.543+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:02.544 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:45:02.544 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:07.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:07.547+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:07.549 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:45:07.549 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:12.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:12.551+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:12.553 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:12.553 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:17.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:17.555+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:17.557 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:17.558 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:22.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:22.559+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:22.563 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:22.563 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:27.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:27.563+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:27.567 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:27.567 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:32.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:32.570+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:32.573 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:32.573 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:37.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:37.574+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:37.578 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:37.578 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:42.582 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:42.578+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:42.583 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:42.583 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:47.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:47.582+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:47.588 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:47.588 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:52.592 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:52.590+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:52.592 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:52.592 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:45:57.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:45:57.594+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:45:57.598 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:45:57.598 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:02.602 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:02.598+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:02.603 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:46:02.603 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:07.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:07.602+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:07.608 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:46:07.608 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:12.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:12.605+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:12.608 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:46:12.608 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:17.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:17.609+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:17.613 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:46:17.613 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:22.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:22.613+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:22.618 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:46:22.618 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:27.622 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:27.621+0000 7f5f36cbff40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:27.622 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, , , , , , , , , , , ] 2023-09-29T22:46:27.622 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:32.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:32.625+0000 7f5f36cbff40 1 bucket sync caught up with source: 2023-09-29T22:46:32.627 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:32.627 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [00000000002.86.6, 00000000001.75.6, , , , , 00000000001.91.6, , , , , , , , , , , 00000000002.88.6, 00000000001.73.6, , , , , , , , , , ] 2023-09-29T22:46:32.627 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:32.625+0000 7f5f36cbff40 0 bucket checkpoint complete 2023-09-29T22:46:32.655 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:46:32.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:46:32.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:46:32.656 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:46:32.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:32.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:32.772 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:32.773+0000 7fd0753a2f40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:32.773 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000003.197.6, 00000000004.242.6, , 00000000001.280.6, 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:46:32.773 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000003.197.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:46:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:37.773+0000 7fd0753a2f40 1 waiting for incremental sync to catch up: 2023-09-29T22:46:37.775 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000003.197.6, 00000000004.242.6, , 00000000001.280.6, 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:46:37.776 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000003.197.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:46:42.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:42.777+0000 7fd0753a2f40 1 bucket sync caught up with source: 2023-09-29T22:46:42.778 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000003.197.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:46:42.779 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000002.261.6, 00000000002.586.6, 00000000001.228.6, 00000000003.197.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000002.270.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:46:42.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:42.777+0000 7fd0753a2f40 0 bucket checkpoint complete 2023-09-29T22:46:42.809 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:46:42.809 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.810 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.810 DEBUG:boto:Method: HEAD 2023-09-29T22:46:42.810 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:42.810 DEBUG:boto:Data: 2023-09-29T22:46:42.810 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.810 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.810 DEBUG:boto:Params: {} 2023-09-29T22:46:42.811 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:46:42.811 DEBUG:boto:Token: None 2023-09-29T22:46:42.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/ 2023-09-29T22:46:42.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OsOXl3KHHF9p/QQ48gylQ+o+cRM= 2023-09-29T22:46:42.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OsOXl3KHHF9p/QQ48gylQ+o+cRM=', 'Content-Length': '0'} 2023-09-29T22:46:42.816 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a25ec9c7f0aa1de7-00651753d2-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.817 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.817 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.817 DEBUG:boto:Method: HEAD 2023-09-29T22:46:42.817 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:42.817 DEBUG:boto:Data: 2023-09-29T22:46:42.817 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.817 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.817 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.818 DEBUG:boto:Params: {} 2023-09-29T22:46:42.818 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:46:42.818 DEBUG:boto:Token: None 2023-09-29T22:46:42.818 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/ 2023-09-29T22:46:42.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OsOXl3KHHF9p/QQ48gylQ+o+cRM= 2023-09-29T22:46:42.818 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OsOXl3KHHF9p/QQ48gylQ+o+cRM=', 'Content-Length': '0'} 2023-09-29T22:46:42.825 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eb84c03082bc233c-00651753d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.825 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:46:42.825 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.826 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.826 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.826 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.826 DEBUG:boto:Method: GET 2023-09-29T22:46:42.826 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.826 DEBUG:boto:Data: 2023-09-29T22:46:42.826 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.826 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.827 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.827 DEBUG:boto:Params: {} 2023-09-29T22:46:42.827 DEBUG:boto:Token: None 2023-09-29T22:46:42.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/?versions 2023-09-29T22:46:42.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak= 2023-09-29T22:46:42.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak=', 'Content-Length': '0'} 2023-09-29T22:46:42.831 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c295c7b4e4a59ec-00651753d2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.832 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:42.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:46:42.834 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:46:42.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:46:42.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:46:42.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:46:42.835 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:46:42.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:46:42.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:46:42.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:46:42.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:46:42.836 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:46:42.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:46:42.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:46:42.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:46:42.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:46:42.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:46:42.837 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:46:42.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:46:42.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:46:42.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:46:42.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:46:42.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:46:42.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:46:42.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:46:42.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:46:42.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:46:42.839 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.840 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.840 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.840 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.840 DEBUG:boto:Method: GET 2023-09-29T22:46:42.840 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.841 DEBUG:boto:Data: 2023-09-29T22:46:42.841 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.841 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.841 DEBUG:boto:Params: {} 2023-09-29T22:46:42.842 DEBUG:boto:Token: None 2023-09-29T22:46:42.842 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/?versions 2023-09-29T22:46:42.842 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak= 2023-09-29T22:46:42.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak=', 'Content-Length': '0'} 2023-09-29T22:46:42.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0abef97ad511e69-00651753d2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.848 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:42.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:46:42.850 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:46:42.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:46:42.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:46:42.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:46:42.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:46:42.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:46:42.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:46:42.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:46:42.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:46:42.853 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.853 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.854 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.854 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.854 DEBUG:boto:Method: GET 2023-09-29T22:46:42.854 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.854 DEBUG:boto:Data: 2023-09-29T22:46:42.854 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.854 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.854 DEBUG:boto:Params: {} 2023-09-29T22:46:42.855 DEBUG:boto:Token: None 2023-09-29T22:46:42.855 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/?versions 2023-09-29T22:46:42.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak= 2023-09-29T22:46:42.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak=', 'Content-Length': '0'} 2023-09-29T22:46:42.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3326110743f3b73-00651753d2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.859 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:42.861 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.861 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.861 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.861 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.861 DEBUG:boto:Method: GET 2023-09-29T22:46:42.861 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:46:42.861 DEBUG:boto:Data: 2023-09-29T22:46:42.861 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.862 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.862 DEBUG:boto:Params: {} 2023-09-29T22:46:42.862 DEBUG:boto:Token: None 2023-09-29T22:46:42.862 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/?versions 2023-09-29T22:46:42.862 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak= 2023-09-29T22:46:42.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D9pb/9+JNmS0B5wFrVRyB4IQRak=', 'Content-Length': '0'} 2023-09-29T22:46:42.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074df55ca1176ea83-00651753d2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.869 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:42.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:46:42.870 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.870 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.871 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.871 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.871 DEBUG:boto:Method: GET 2023-09-29T22:46:42.871 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.871 DEBUG:boto:Data: 2023-09-29T22:46:42.871 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.871 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.871 DEBUG:boto:Params: {} 2023-09-29T22:46:42.872 DEBUG:boto:Token: None 2023-09-29T22:46:42.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk= 2023-09-29T22:46:42.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk=', 'Content-Length': '0'} 2023-09-29T22:46:42.874 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076b22f97fdde4bee-00651753d2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.914 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.915 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.915 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.915 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.915 DEBUG:boto:Method: GET 2023-09-29T22:46:42.915 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.916 DEBUG:boto:Data: 2023-09-29T22:46:42.916 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.916 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.917 DEBUG:boto:Params: {} 2023-09-29T22:46:42.917 DEBUG:boto:Token: None 2023-09-29T22:46:42.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk= 2023-09-29T22:46:42.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk=', 'Content-Length': '0'} 2023-09-29T22:46:42.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bbae49fa7055af0-00651753d2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.921 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.921 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.921 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.921 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.921 DEBUG:boto:Method: HEAD 2023-09-29T22:46:42.922 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.922 DEBUG:boto:Data: 2023-09-29T22:46:42.922 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.922 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.922 DEBUG:boto:Params: {} 2023-09-29T22:46:42.922 DEBUG:boto:Token: None 2023-09-29T22:46:42.922 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zkVU223haYRMxkzJGV+hpXKDx7M= 2023-09-29T22:46:42.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zkVU223haYRMxkzJGV+hpXKDx7M=', 'Content-Length': '0'} 2023-09-29T22:46:42.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ec8b987dcf6fd02-00651753d2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.925 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.926 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.926 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.926 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.926 DEBUG:boto:Method: HEAD 2023-09-29T22:46:42.926 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.926 DEBUG:boto:Data: 2023-09-29T22:46:42.926 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.927 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.927 DEBUG:boto:Params: {} 2023-09-29T22:46:42.927 DEBUG:boto:Token: None 2023-09-29T22:46:42.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zkVU223haYRMxkzJGV+hpXKDx7M= 2023-09-29T22:46:42.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zkVU223haYRMxkzJGV+hpXKDx7M=', 'Content-Length': '0'} 2023-09-29T22:46:42.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000026ac12353b1013e-00651753d2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:46:42.930 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.930 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.931 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.931 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.931 DEBUG:boto:Method: GET 2023-09-29T22:46:42.931 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.931 DEBUG:boto:Data: 2023-09-29T22:46:42.931 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.931 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.931 DEBUG:boto:Params: {} 2023-09-29T22:46:42.932 DEBUG:boto:Token: None 2023-09-29T22:46:42.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk= 2023-09-29T22:46:42.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk=', 'Content-Length': '0'} 2023-09-29T22:46:42.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c1a04fe905165e-00651753d2-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.935 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.935 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.935 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.935 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.935 DEBUG:boto:Method: GET 2023-09-29T22:46:42.936 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.936 DEBUG:boto:Data: 2023-09-29T22:46:42.936 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.936 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.936 DEBUG:boto:Params: {} 2023-09-29T22:46:42.936 DEBUG:boto:Token: None 2023-09-29T22:46:42.937 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:46:42.937 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk= 2023-09-29T22:46:42.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YsalFKTVToMiqpR4aarOt6SsFgk=', 'Content-Length': '0'} 2023-09-29T22:46:42.939 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4aafa03175f8a8e-00651753d2-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.940 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.940 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.940 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:42.940 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:42.940 DEBUG:boto:Method: GET 2023-09-29T22:46:42.941 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:46:42.941 DEBUG:boto:Data: 2023-09-29T22:46:42.941 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:42.941 DEBUG:boto:Port: 8001 2023-09-29T22:46:42.941 DEBUG:boto:Params: {} 2023-09-29T22:46:42.941 DEBUG:boto:Token: None 2023-09-29T22:46:42.941 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/ 2023-09-29T22:46:42.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s7+DJwpM4pYp0pHC59hQjQCbOeY= 2023-09-29T22:46:42.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s7+DJwpM4pYp0pHC59hQjQCbOeY=', 'Content-Length': '0'} 2023-09-29T22:46:42.946 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d228c4199ccf28-00651753d2-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.946 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:42.947 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:42.947 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:42.948 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:42.948 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:42.948 DEBUG:boto:Method: GET 2023-09-29T22:46:42.948 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:46:42.948 DEBUG:boto:Data: 2023-09-29T22:46:42.948 DEBUG:boto:Headers: {} 2023-09-29T22:46:42.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:42.949 DEBUG:boto:Port: 8000 2023-09-29T22:46:42.949 DEBUG:boto:Params: {} 2023-09-29T22:46:42.949 DEBUG:boto:Token: None 2023-09-29T22:46:42.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:42 GMT /hcnphz-59/ 2023-09-29T22:46:42.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s7+DJwpM4pYp0pHC59hQjQCbOeY= 2023-09-29T22:46:42.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s7+DJwpM4pYp0pHC59hQjQCbOeY=', 'Content-Length': '0'} 2023-09-29T22:46:42.955 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010d7337e7d8ef0fd-00651753d2-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:42.998 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:42.999 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:42.999 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:42.999 DEBUG:boto:Method: HEAD 2023-09-29T22:46:42.999 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:46:42.999 DEBUG:boto:Data: 2023-09-29T22:46:43.000 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.000 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.000 DEBUG:boto:Params: {} 2023-09-29T22:46:43.000 DEBUG:boto:Token: None 2023-09-29T22:46:43.000 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj11 2023-09-29T22:46:43.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rkx/dfybZ53D5hkVqCCG4ZDG4Eg= 2023-09-29T22:46:43.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rkx/dfybZ53D5hkVqCCG4ZDG4Eg=', 'Content-Length': '0'} 2023-09-29T22:46:43.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f530f6adee6625ef-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.004 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:43.004 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:43.004 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.004 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:46:43.004 DEBUG:boto:Data: 2023-09-29T22:46:43.004 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.004 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.004 DEBUG:boto:Params: {} 2023-09-29T22:46:43.005 DEBUG:boto:Token: None 2023-09-29T22:46:43.005 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj11 2023-09-29T22:46:43.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rkx/dfybZ53D5hkVqCCG4ZDG4Eg= 2023-09-29T22:46:43.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rkx/dfybZ53D5hkVqCCG4ZDG4Eg=', 'Content-Length': '0'} 2023-09-29T22:46:43.009 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb832ac872d258b2-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:46:43.009 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:43.009 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:43.010 DEBUG:boto:Method: GET 2023-09-29T22:46:43.010 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:46:43.010 DEBUG:boto:Data: 2023-09-29T22:46:43.010 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.010 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.010 DEBUG:boto:Params: {} 2023-09-29T22:46:43.011 DEBUG:boto:Token: None 2023-09-29T22:46:43.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj11 2023-09-29T22:46:43.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yInGUawxBN3I56KMpCMS9uQOKtQ= 2023-09-29T22:46:43.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yInGUawxBN3I56KMpCMS9uQOKtQ=', 'Content-Length': '0'} 2023-09-29T22:46:43.014 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bff7d1ba4df7e182-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.054 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:46:43.055 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:46:43.055 DEBUG:boto:Method: GET 2023-09-29T22:46:43.055 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:46:43.055 DEBUG:boto:Data: 2023-09-29T22:46:43.055 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.055 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.055 DEBUG:boto:Params: {} 2023-09-29T22:46:43.056 DEBUG:boto:Token: None 2023-09-29T22:46:43.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj11 2023-09-29T22:46:43.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yInGUawxBN3I56KMpCMS9uQOKtQ= 2023-09-29T22:46:43.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yInGUawxBN3I56KMpCMS9uQOKtQ=', 'Content-Length': '0'} 2023-09-29T22:46:43.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002b482a14f06598c3-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.059 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.059 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.059 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:43.059 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:43.059 DEBUG:boto:Method: GET 2023-09-29T22:46:43.060 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:46:43.060 DEBUG:boto:Data: 2023-09-29T22:46:43.060 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.060 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.060 DEBUG:boto:Params: {} 2023-09-29T22:46:43.060 DEBUG:boto:Token: None 2023-09-29T22:46:43.060 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.065 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d2efe03db93d584-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.065 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.066 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.066 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.066 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:43.066 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:46:43.066 DEBUG:boto:Method: GET 2023-09-29T22:46:43.067 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:46:43.067 DEBUG:boto:Data: 2023-09-29T22:46:43.067 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.067 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.067 DEBUG:boto:Params: {} 2023-09-29T22:46:43.067 DEBUG:boto:Token: None 2023-09-29T22:46:43.067 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.067 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.073 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000994ab0fe882bcdd8-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.074 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:46:43.074 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.074 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.074 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.075 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.075 DEBUG:boto:Method: GET 2023-09-29T22:46:43.075 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.075 DEBUG:boto:Data: 2023-09-29T22:46:43.075 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.075 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.075 DEBUG:boto:Params: {} 2023-09-29T22:46:43.075 DEBUG:boto:Token: None 2023-09-29T22:46:43.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8= 2023-09-29T22:46:43.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8=', 'Content-Length': '0'} 2023-09-29T22:46:43.078 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005343829b9d5bf7b-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.078 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.078 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.078 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.079 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.079 DEBUG:boto:Method: GET 2023-09-29T22:46:43.079 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.079 DEBUG:boto:Data: 2023-09-29T22:46:43.079 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.079 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.079 DEBUG:boto:Params: {} 2023-09-29T22:46:43.079 DEBUG:boto:Token: None 2023-09-29T22:46:43.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8= 2023-09-29T22:46:43.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8=', 'Content-Length': '0'} 2023-09-29T22:46:43.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb8ffb2c4529e27a-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.122 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.123 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.123 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.123 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.123 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.123 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.123 DEBUG:boto:Data: 2023-09-29T22:46:43.123 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.124 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.124 DEBUG:boto:Params: {} 2023-09-29T22:46:43.124 DEBUG:boto:Token: None 2023-09-29T22:46:43.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OS1gnDhzYiCKOfR54JiQSEGTXBw= 2023-09-29T22:46:43.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OS1gnDhzYiCKOfR54JiQSEGTXBw=', 'Content-Length': '0'} 2023-09-29T22:46:43.127 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75ced116e303b03-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.127 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.127 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.128 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.128 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.128 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.128 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.128 DEBUG:boto:Data: 2023-09-29T22:46:43.128 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.128 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.128 DEBUG:boto:Params: {} 2023-09-29T22:46:43.129 DEBUG:boto:Token: None 2023-09-29T22:46:43.129 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OS1gnDhzYiCKOfR54JiQSEGTXBw= 2023-09-29T22:46:43.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OS1gnDhzYiCKOfR54JiQSEGTXBw=', 'Content-Length': '0'} 2023-09-29T22:46:43.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd626a8fbafd9f23-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:46:43.132 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.132 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.132 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.132 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.132 DEBUG:boto:Method: GET 2023-09-29T22:46:43.132 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.133 DEBUG:boto:Data: 2023-09-29T22:46:43.133 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.133 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.133 DEBUG:boto:Params: {} 2023-09-29T22:46:43.133 DEBUG:boto:Token: None 2023-09-29T22:46:43.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8= 2023-09-29T22:46:43.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8=', 'Content-Length': '0'} 2023-09-29T22:46:43.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094f4b9f7c9ea0e05-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.178 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.179 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.179 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.179 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.179 DEBUG:boto:Method: GET 2023-09-29T22:46:43.179 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.179 DEBUG:boto:Data: 2023-09-29T22:46:43.179 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.179 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.179 DEBUG:boto:Params: {} 2023-09-29T22:46:43.180 DEBUG:boto:Token: None 2023-09-29T22:46:43.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:46:43.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8= 2023-09-29T22:46:43.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:61eArGAPLPK2U5PGkgIE8m/ycw8=', 'Content-Length': '0'} 2023-09-29T22:46:43.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068f30af134dc308e-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.183 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.183 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.183 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.184 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.184 DEBUG:boto:Method: GET 2023-09-29T22:46:43.184 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.184 DEBUG:boto:Data: 2023-09-29T22:46:43.184 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.184 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.184 DEBUG:boto:Params: {} 2023-09-29T22:46:43.184 DEBUG:boto:Token: None 2023-09-29T22:46:43.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.185 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003681d21d300c903c-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.189 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.190 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.190 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.190 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.190 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.190 DEBUG:boto:Method: GET 2023-09-29T22:46:43.190 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.191 DEBUG:boto:Data: 2023-09-29T22:46:43.191 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.191 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.191 DEBUG:boto:Params: {} 2023-09-29T22:46:43.191 DEBUG:boto:Token: None 2023-09-29T22:46:43.191 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f5ff317eb2357d5-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.198 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.199 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.199 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.199 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.199 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:46:43.199 DEBUG:boto:Data: 2023-09-29T22:46:43.199 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.200 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.200 DEBUG:boto:Params: {} 2023-09-29T22:46:43.200 DEBUG:boto:Token: None 2023-09-29T22:46:43.200 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115 2023-09-29T22:46:43.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I1TDtFp4L7vAlBpqN56N2+P2XqM= 2023-09-29T22:46:43.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I1TDtFp4L7vAlBpqN56N2+P2XqM=', 'Content-Length': '0'} 2023-09-29T22:46:43.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e953f5f36c3cda-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.203 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.203 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.203 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.203 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:46:43.203 DEBUG:boto:Data: 2023-09-29T22:46:43.203 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.203 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.203 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.203 DEBUG:boto:Params: {} 2023-09-29T22:46:43.204 DEBUG:boto:Token: None 2023-09-29T22:46:43.204 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115 2023-09-29T22:46:43.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I1TDtFp4L7vAlBpqN56N2+P2XqM= 2023-09-29T22:46:43.204 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I1TDtFp4L7vAlBpqN56N2+P2XqM=', 'Content-Length': '0'} 2023-09-29T22:46:43.206 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007995c75e4a52c1c6-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:46:43.207 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.207 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.207 DEBUG:boto:Method: GET 2023-09-29T22:46:43.207 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:46:43.207 DEBUG:boto:Data: 2023-09-29T22:46:43.207 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.207 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.208 DEBUG:boto:Params: {} 2023-09-29T22:46:43.208 DEBUG:boto:Token: None 2023-09-29T22:46:43.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115 2023-09-29T22:46:43.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ah67wvxCz0plvbEiEfBkiVeRkc= 2023-09-29T22:46:43.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ah67wvxCz0plvbEiEfBkiVeRkc=', 'Content-Length': '0'} 2023-09-29T22:46:43.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0916bd47439ab3a-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.254 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:46:43.255 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:46:43.255 DEBUG:boto:Method: GET 2023-09-29T22:46:43.255 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:46:43.255 DEBUG:boto:Data: 2023-09-29T22:46:43.255 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.255 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.255 DEBUG:boto:Params: {} 2023-09-29T22:46:43.255 DEBUG:boto:Token: None 2023-09-29T22:46:43.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj115 2023-09-29T22:46:43.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2ah67wvxCz0plvbEiEfBkiVeRkc= 2023-09-29T22:46:43.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ah67wvxCz0plvbEiEfBkiVeRkc=', 'Content-Length': '0'} 2023-09-29T22:46:43.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4c9717e9751f4d1-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.259 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.259 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.260 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.260 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.260 DEBUG:boto:Method: GET 2023-09-29T22:46:43.260 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.260 DEBUG:boto:Data: 2023-09-29T22:46:43.260 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.260 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.261 DEBUG:boto:Params: {} 2023-09-29T22:46:43.261 DEBUG:boto:Token: None 2023-09-29T22:46:43.261 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.261 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.265 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d3c3b80b5dc47f87-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.266 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.266 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.266 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.266 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.266 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.266 DEBUG:boto:Method: GET 2023-09-29T22:46:43.267 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:46:43.267 DEBUG:boto:Data: 2023-09-29T22:46:43.267 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.267 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.267 DEBUG:boto:Params: {} 2023-09-29T22:46:43.267 DEBUG:boto:Token: None 2023-09-29T22:46:43.267 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000168a5fabd2f5047d-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.274 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:46:43.275 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.275 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.275 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.275 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.275 DEBUG:boto:Method: GET 2023-09-29T22:46:43.275 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.275 DEBUG:boto:Data: 2023-09-29T22:46:43.275 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.276 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.276 DEBUG:boto:Params: {} 2023-09-29T22:46:43.276 DEBUG:boto:Token: None 2023-09-29T22:46:43.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4= 2023-09-29T22:46:43.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4=', 'Content-Length': '0'} 2023-09-29T22:46:43.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7f79368fe02b4f4-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.279 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.279 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.279 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.279 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.279 DEBUG:boto:Method: GET 2023-09-29T22:46:43.279 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.279 DEBUG:boto:Data: 2023-09-29T22:46:43.279 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.280 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.280 DEBUG:boto:Params: {} 2023-09-29T22:46:43.280 DEBUG:boto:Token: None 2023-09-29T22:46:43.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4= 2023-09-29T22:46:43.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4=', 'Content-Length': '0'} 2023-09-29T22:46:43.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de58d5c2bb1407ed-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.283 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.283 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.283 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.283 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.283 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.283 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.283 DEBUG:boto:Data: 2023-09-29T22:46:43.283 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.284 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.284 DEBUG:boto:Params: {} 2023-09-29T22:46:43.284 DEBUG:boto:Token: None 2023-09-29T22:46:43.284 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:migztPR193DuMGkBz8Vm8jLmcdk= 2023-09-29T22:46:43.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:migztPR193DuMGkBz8Vm8jLmcdk=', 'Content-Length': '0'} 2023-09-29T22:46:43.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9754f5d238894f3-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.287 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.287 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.287 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.287 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.287 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.287 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.287 DEBUG:boto:Data: 2023-09-29T22:46:43.287 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.287 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.288 DEBUG:boto:Params: {} 2023-09-29T22:46:43.288 DEBUG:boto:Token: None 2023-09-29T22:46:43.288 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:migztPR193DuMGkBz8Vm8jLmcdk= 2023-09-29T22:46:43.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:migztPR193DuMGkBz8Vm8jLmcdk=', 'Content-Length': '0'} 2023-09-29T22:46:43.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc3de05c33b55094-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:46:43.290 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.291 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.291 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.291 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.291 DEBUG:boto:Method: GET 2023-09-29T22:46:43.291 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.291 DEBUG:boto:Data: 2023-09-29T22:46:43.291 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.291 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.292 DEBUG:boto:Params: {} 2023-09-29T22:46:43.292 DEBUG:boto:Token: None 2023-09-29T22:46:43.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4= 2023-09-29T22:46:43.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4=', 'Content-Length': '0'} 2023-09-29T22:46:43.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3a6eb4db8ac688c-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.334 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.335 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.335 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.335 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.335 DEBUG:boto:Method: GET 2023-09-29T22:46:43.335 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.335 DEBUG:boto:Data: 2023-09-29T22:46:43.335 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.336 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.336 DEBUG:boto:Params: {} 2023-09-29T22:46:43.336 DEBUG:boto:Token: None 2023-09-29T22:46:43.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:46:43.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4= 2023-09-29T22:46:43.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wglNqXBtt2xT7bG0KHoL+6cRPX4=', 'Content-Length': '0'} 2023-09-29T22:46:43.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da83e8742a183fce-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.340 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.340 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.340 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.340 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.340 DEBUG:boto:Method: GET 2023-09-29T22:46:43.340 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.341 DEBUG:boto:Data: 2023-09-29T22:46:43.341 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.341 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.341 DEBUG:boto:Params: {} 2023-09-29T22:46:43.341 DEBUG:boto:Token: None 2023-09-29T22:46:43.341 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.341 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.346 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000013fc29afd03d4bc-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.346 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.347 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.347 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.347 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.347 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.347 DEBUG:boto:Method: GET 2023-09-29T22:46:43.347 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.347 DEBUG:boto:Data: 2023-09-29T22:46:43.348 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.348 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.348 DEBUG:boto:Params: {} 2023-09-29T22:46:43.348 DEBUG:boto:Token: None 2023-09-29T22:46:43.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a33cc49b4835854-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.356 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.356 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.356 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.356 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.356 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:46:43.357 DEBUG:boto:Data: 2023-09-29T22:46:43.357 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.357 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.357 DEBUG:boto:Params: {} 2023-09-29T22:46:43.357 DEBUG:boto:Token: None 2023-09-29T22:46:43.357 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116 2023-09-29T22:46:43.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/w5FSpOS6GqjxEo2JPsWuUMBK0g= 2023-09-29T22:46:43.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/w5FSpOS6GqjxEo2JPsWuUMBK0g=', 'Content-Length': '0'} 2023-09-29T22:46:43.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b06249f54848190-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.360 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.360 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.361 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.361 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:46:43.361 DEBUG:boto:Data: 2023-09-29T22:46:43.361 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.361 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.362 DEBUG:boto:Params: {} 2023-09-29T22:46:43.362 DEBUG:boto:Token: None 2023-09-29T22:46:43.362 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116 2023-09-29T22:46:43.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/w5FSpOS6GqjxEo2JPsWuUMBK0g= 2023-09-29T22:46:43.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/w5FSpOS6GqjxEo2JPsWuUMBK0g=', 'Content-Length': '0'} 2023-09-29T22:46:43.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d630a5e0cf06f25-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:46:43.365 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.366 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.366 DEBUG:boto:Method: GET 2023-09-29T22:46:43.366 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:46:43.366 DEBUG:boto:Data: 2023-09-29T22:46:43.367 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.367 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.367 DEBUG:boto:Params: {} 2023-09-29T22:46:43.367 DEBUG:boto:Token: None 2023-09-29T22:46:43.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116 2023-09-29T22:46:43.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NrRt11xK7nxp9PIyO6lHXV1+gBs= 2023-09-29T22:46:43.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NrRt11xK7nxp9PIyO6lHXV1+gBs=', 'Content-Length': '0'} 2023-09-29T22:46:43.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e9dfb9eb16f8734-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.371 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:46:43.371 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:46:43.371 DEBUG:boto:Method: GET 2023-09-29T22:46:43.372 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:46:43.372 DEBUG:boto:Data: 2023-09-29T22:46:43.372 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.372 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.373 DEBUG:boto:Params: {} 2023-09-29T22:46:43.373 DEBUG:boto:Token: None 2023-09-29T22:46:43.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj116 2023-09-29T22:46:43.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NrRt11xK7nxp9PIyO6lHXV1+gBs= 2023-09-29T22:46:43.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NrRt11xK7nxp9PIyO6lHXV1+gBs=', 'Content-Length': '0'} 2023-09-29T22:46:43.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a81b3192823d0578-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.377 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.377 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.377 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.377 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.378 DEBUG:boto:Method: GET 2023-09-29T22:46:43.378 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.378 DEBUG:boto:Data: 2023-09-29T22:46:43.378 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.379 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.379 DEBUG:boto:Params: {} 2023-09-29T22:46:43.379 DEBUG:boto:Token: None 2023-09-29T22:46:43.379 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.385 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f596864d0464b30-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.426 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.427 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.427 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.427 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.427 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.427 DEBUG:boto:Method: GET 2023-09-29T22:46:43.427 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:46:43.428 DEBUG:boto:Data: 2023-09-29T22:46:43.428 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.428 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.428 DEBUG:boto:Params: {} 2023-09-29T22:46:43.428 DEBUG:boto:Token: None 2023-09-29T22:46:43.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d6b7a313098b4939-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.437 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:46:43.438 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.438 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.438 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.438 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.438 DEBUG:boto:Method: GET 2023-09-29T22:46:43.439 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.439 DEBUG:boto:Data: 2023-09-29T22:46:43.439 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.439 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.439 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.439 DEBUG:boto:Params: {} 2023-09-29T22:46:43.440 DEBUG:boto:Token: None 2023-09-29T22:46:43.440 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.440 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU= 2023-09-29T22:46:43.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU=', 'Content-Length': '0'} 2023-09-29T22:46:43.443 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c99f1626b05579ed-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.443 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.444 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.444 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.444 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.444 DEBUG:boto:Method: GET 2023-09-29T22:46:43.444 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.444 DEBUG:boto:Data: 2023-09-29T22:46:43.444 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.445 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.445 DEBUG:boto:Params: {} 2023-09-29T22:46:43.445 DEBUG:boto:Token: None 2023-09-29T22:46:43.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU= 2023-09-29T22:46:43.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU=', 'Content-Length': '0'} 2023-09-29T22:46:43.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016661ecb3b4761f3-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.490 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.491 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.491 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.491 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.491 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.491 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.491 DEBUG:boto:Data: 2023-09-29T22:46:43.491 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.491 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.491 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.492 DEBUG:boto:Params: {} 2023-09-29T22:46:43.492 DEBUG:boto:Token: None 2023-09-29T22:46:43.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yigwWXO2hccsgLwiB1vmx2Rtjrs= 2023-09-29T22:46:43.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yigwWXO2hccsgLwiB1vmx2Rtjrs=', 'Content-Length': '0'} 2023-09-29T22:46:43.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2b6fc38abf885ec-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.495 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.495 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.496 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.496 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.496 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.496 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.496 DEBUG:boto:Data: 2023-09-29T22:46:43.496 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.496 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.496 DEBUG:boto:Params: {} 2023-09-29T22:46:43.497 DEBUG:boto:Token: None 2023-09-29T22:46:43.497 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yigwWXO2hccsgLwiB1vmx2Rtjrs= 2023-09-29T22:46:43.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yigwWXO2hccsgLwiB1vmx2Rtjrs=', 'Content-Length': '0'} 2023-09-29T22:46:43.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002df7c02b10f43fe0-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:46:43.500 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.500 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.500 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.500 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.500 DEBUG:boto:Method: GET 2023-09-29T22:46:43.500 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.501 DEBUG:boto:Data: 2023-09-29T22:46:43.501 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.501 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.501 DEBUG:boto:Params: {} 2023-09-29T22:46:43.501 DEBUG:boto:Token: None 2023-09-29T22:46:43.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU= 2023-09-29T22:46:43.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU=', 'Content-Length': '0'} 2023-09-29T22:46:43.504 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c41b10ac8962cd0-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.504 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.505 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.505 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.505 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.505 DEBUG:boto:Method: GET 2023-09-29T22:46:43.505 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.505 DEBUG:boto:Data: 2023-09-29T22:46:43.505 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.505 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.506 DEBUG:boto:Params: {} 2023-09-29T22:46:43.506 DEBUG:boto:Token: None 2023-09-29T22:46:43.506 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:46:43.506 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU= 2023-09-29T22:46:43.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FmYQeJyL/boqYGMU+AJSNwDeXhU=', 'Content-Length': '0'} 2023-09-29T22:46:43.509 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080f1adf371e1f10b-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.509 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.509 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.509 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.509 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.509 DEBUG:boto:Method: GET 2023-09-29T22:46:43.509 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.510 DEBUG:boto:Data: 2023-09-29T22:46:43.510 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.510 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.510 DEBUG:boto:Params: {} 2023-09-29T22:46:43.510 DEBUG:boto:Token: None 2023-09-29T22:46:43.510 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.510 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.515 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c505f94a024e47f9-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.558 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.559 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.559 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.559 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.559 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.559 DEBUG:boto:Method: GET 2023-09-29T22:46:43.560 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.560 DEBUG:boto:Data: 2023-09-29T22:46:43.560 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.560 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.560 DEBUG:boto:Params: {} 2023-09-29T22:46:43.560 DEBUG:boto:Token: None 2023-09-29T22:46:43.561 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097ddc0a8dfb2932b-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.569 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.569 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.569 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.569 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.570 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:46:43.570 DEBUG:boto:Data: 2023-09-29T22:46:43.570 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.570 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.570 DEBUG:boto:Params: {} 2023-09-29T22:46:43.570 DEBUG:boto:Token: None 2023-09-29T22:46:43.570 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145 2023-09-29T22:46:43.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ex1Rxn+/1DhqAnPEduhcfIsPLfw= 2023-09-29T22:46:43.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ex1Rxn+/1DhqAnPEduhcfIsPLfw=', 'Content-Length': '0'} 2023-09-29T22:46:43.573 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bbb8a7b7c13984b-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.573 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.573 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.574 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.574 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:46:43.574 DEBUG:boto:Data: 2023-09-29T22:46:43.574 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.574 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.574 DEBUG:boto:Params: {} 2023-09-29T22:46:43.574 DEBUG:boto:Token: None 2023-09-29T22:46:43.574 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145 2023-09-29T22:46:43.575 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ex1Rxn+/1DhqAnPEduhcfIsPLfw= 2023-09-29T22:46:43.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ex1Rxn+/1DhqAnPEduhcfIsPLfw=', 'Content-Length': '0'} 2023-09-29T22:46:43.577 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffb06ed831bcac7d-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:46:43.577 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.577 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.578 DEBUG:boto:Method: GET 2023-09-29T22:46:43.578 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:46:43.578 DEBUG:boto:Data: 2023-09-29T22:46:43.578 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.578 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.578 DEBUG:boto:Params: {} 2023-09-29T22:46:43.578 DEBUG:boto:Token: None 2023-09-29T22:46:43.578 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145 2023-09-29T22:46:43.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9o2qpjmdnF6fj/HA2EGmO1LAby8= 2023-09-29T22:46:43.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9o2qpjmdnF6fj/HA2EGmO1LAby8=', 'Content-Length': '0'} 2023-09-29T22:46:43.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729537577a758391-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.581 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:46:43.581 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:46:43.581 DEBUG:boto:Method: GET 2023-09-29T22:46:43.581 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:46:43.581 DEBUG:boto:Data: 2023-09-29T22:46:43.581 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.582 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.582 DEBUG:boto:Params: {} 2023-09-29T22:46:43.582 DEBUG:boto:Token: None 2023-09-29T22:46:43.582 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj145 2023-09-29T22:46:43.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9o2qpjmdnF6fj/HA2EGmO1LAby8= 2023-09-29T22:46:43.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9o2qpjmdnF6fj/HA2EGmO1LAby8=', 'Content-Length': '0'} 2023-09-29T22:46:43.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000761b7f21b602e738-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.626 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.626 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.627 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.627 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.627 DEBUG:boto:Method: GET 2023-09-29T22:46:43.627 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.627 DEBUG:boto:Data: 2023-09-29T22:46:43.627 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.627 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.627 DEBUG:boto:Params: {} 2023-09-29T22:46:43.628 DEBUG:boto:Token: None 2023-09-29T22:46:43.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004a61bd0183f9b9d3-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.633 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.634 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.634 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.634 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.634 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.634 DEBUG:boto:Method: GET 2023-09-29T22:46:43.634 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:46:43.634 DEBUG:boto:Data: 2023-09-29T22:46:43.635 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.635 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.635 DEBUG:boto:Params: {} 2023-09-29T22:46:43.635 DEBUG:boto:Token: None 2023-09-29T22:46:43.635 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.635 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.635 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.641 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e45269e92c0894af-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.642 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:46:43.643 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.643 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.643 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.643 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.643 DEBUG:boto:Method: GET 2023-09-29T22:46:43.644 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.644 DEBUG:boto:Data: 2023-09-29T22:46:43.644 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.644 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.644 DEBUG:boto:Params: {} 2023-09-29T22:46:43.645 DEBUG:boto:Token: None 2023-09-29T22:46:43.645 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM= 2023-09-29T22:46:43.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM=', 'Content-Length': '0'} 2023-09-29T22:46:43.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350d8ce6450567d8-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.648 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.648 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.648 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.649 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.649 DEBUG:boto:Method: GET 2023-09-29T22:46:43.649 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.649 DEBUG:boto:Data: 2023-09-29T22:46:43.649 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.650 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.650 DEBUG:boto:Params: {} 2023-09-29T22:46:43.650 DEBUG:boto:Token: None 2023-09-29T22:46:43.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM= 2023-09-29T22:46:43.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM=', 'Content-Length': '0'} 2023-09-29T22:46:43.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007888c79ad5afc7af-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.654 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.654 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.654 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.654 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.654 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.655 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.655 DEBUG:boto:Data: 2023-09-29T22:46:43.655 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.655 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.655 DEBUG:boto:Params: {} 2023-09-29T22:46:43.655 DEBUG:boto:Token: None 2023-09-29T22:46:43.655 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jl1qZ3QSgrqhEiBt56bppgMOcBw= 2023-09-29T22:46:43.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jl1qZ3QSgrqhEiBt56bppgMOcBw=', 'Content-Length': '0'} 2023-09-29T22:46:43.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067bd1148bc80cb1b-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.658 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.658 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.658 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.658 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.659 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.659 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.659 DEBUG:boto:Data: 2023-09-29T22:46:43.659 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.659 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.659 DEBUG:boto:Params: {} 2023-09-29T22:46:43.659 DEBUG:boto:Token: None 2023-09-29T22:46:43.659 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jl1qZ3QSgrqhEiBt56bppgMOcBw= 2023-09-29T22:46:43.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jl1qZ3QSgrqhEiBt56bppgMOcBw=', 'Content-Length': '0'} 2023-09-29T22:46:43.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001284f43e636aea36-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:46:43.663 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.663 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.663 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.663 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.663 DEBUG:boto:Method: GET 2023-09-29T22:46:43.663 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.663 DEBUG:boto:Data: 2023-09-29T22:46:43.663 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.663 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.664 DEBUG:boto:Params: {} 2023-09-29T22:46:43.664 DEBUG:boto:Token: None 2023-09-29T22:46:43.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM= 2023-09-29T22:46:43.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM=', 'Content-Length': '0'} 2023-09-29T22:46:43.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2993e6eeb414b69-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.710 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.711 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.711 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.711 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.711 DEBUG:boto:Method: GET 2023-09-29T22:46:43.711 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.711 DEBUG:boto:Data: 2023-09-29T22:46:43.711 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.711 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.712 DEBUG:boto:Params: {} 2023-09-29T22:46:43.712 DEBUG:boto:Token: None 2023-09-29T22:46:43.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:46:43.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM= 2023-09-29T22:46:43.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oY2Kc3YW7z24iLAVZBDM4+T6NXM=', 'Content-Length': '0'} 2023-09-29T22:46:43.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cfadbff9f18abe9-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.716 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.716 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.716 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.716 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.716 DEBUG:boto:Method: GET 2023-09-29T22:46:43.716 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.717 DEBUG:boto:Data: 2023-09-29T22:46:43.717 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.717 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.717 DEBUG:boto:Params: {} 2023-09-29T22:46:43.717 DEBUG:boto:Token: None 2023-09-29T22:46:43.717 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.717 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eff8ecda3f2bcd4c-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.722 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.723 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.723 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.723 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.723 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.723 DEBUG:boto:Method: GET 2023-09-29T22:46:43.724 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.724 DEBUG:boto:Data: 2023-09-29T22:46:43.724 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.724 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.724 DEBUG:boto:Params: {} 2023-09-29T22:46:43.724 DEBUG:boto:Token: None 2023-09-29T22:46:43.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c845dcb9a4281809-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.774 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.775 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.775 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.775 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.775 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:46:43.775 DEBUG:boto:Data: 2023-09-29T22:46:43.775 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.775 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.776 DEBUG:boto:Params: {} 2023-09-29T22:46:43.776 DEBUG:boto:Token: None 2023-09-29T22:46:43.776 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17 2023-09-29T22:46:43.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qA+Fp3OS/sX38oRM3YY8i7oF/cg= 2023-09-29T22:46:43.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qA+Fp3OS/sX38oRM3YY8i7oF/cg=', 'Content-Length': '0'} 2023-09-29T22:46:43.778 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e56e809e5dc5c524-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.779 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.779 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.779 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.779 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:46:43.779 DEBUG:boto:Data: 2023-09-29T22:46:43.779 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.779 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.780 DEBUG:boto:Params: {} 2023-09-29T22:46:43.780 DEBUG:boto:Token: None 2023-09-29T22:46:43.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17 2023-09-29T22:46:43.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qA+Fp3OS/sX38oRM3YY8i7oF/cg= 2023-09-29T22:46:43.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qA+Fp3OS/sX38oRM3YY8i7oF/cg=', 'Content-Length': '0'} 2023-09-29T22:46:43.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f2498ef3b12e56-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:46:43.783 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.783 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.783 DEBUG:boto:Method: GET 2023-09-29T22:46:43.784 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:46:43.784 DEBUG:boto:Data: 2023-09-29T22:46:43.784 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.784 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.784 DEBUG:boto:Params: {} 2023-09-29T22:46:43.784 DEBUG:boto:Token: None 2023-09-29T22:46:43.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17 2023-09-29T22:46:43.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zqGqc6qEj3By4LfQx6KymIeo4fo= 2023-09-29T22:46:43.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zqGqc6qEj3By4LfQx6KymIeo4fo=', 'Content-Length': '0'} 2023-09-29T22:46:43.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b162a2e8bfe3157e-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.787 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:46:43.787 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:46:43.787 DEBUG:boto:Method: GET 2023-09-29T22:46:43.788 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:46:43.788 DEBUG:boto:Data: 2023-09-29T22:46:43.788 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.788 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.788 DEBUG:boto:Params: {} 2023-09-29T22:46:43.788 DEBUG:boto:Token: None 2023-09-29T22:46:43.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj17 2023-09-29T22:46:43.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zqGqc6qEj3By4LfQx6KymIeo4fo= 2023-09-29T22:46:43.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zqGqc6qEj3By4LfQx6KymIeo4fo=', 'Content-Length': '0'} 2023-09-29T22:46:43.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fb8406b543af9f1-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.792 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.792 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.792 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.792 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.792 DEBUG:boto:Method: GET 2023-09-29T22:46:43.792 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.792 DEBUG:boto:Data: 2023-09-29T22:46:43.792 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.793 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.793 DEBUG:boto:Params: {} 2023-09-29T22:46:43.793 DEBUG:boto:Token: None 2023-09-29T22:46:43.793 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012feba0456d267ef-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.838 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.839 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.839 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.839 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.839 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.839 DEBUG:boto:Method: GET 2023-09-29T22:46:43.839 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:46:43.839 DEBUG:boto:Data: 2023-09-29T22:46:43.839 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.840 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.840 DEBUG:boto:Params: {} 2023-09-29T22:46:43.840 DEBUG:boto:Token: None 2023-09-29T22:46:43.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3cffa0ac7465d3-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.848 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:46:43.849 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.849 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.849 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.849 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.849 DEBUG:boto:Method: GET 2023-09-29T22:46:43.849 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.849 DEBUG:boto:Data: 2023-09-29T22:46:43.849 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.849 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.850 DEBUG:boto:Params: {} 2023-09-29T22:46:43.850 DEBUG:boto:Token: None 2023-09-29T22:46:43.850 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8= 2023-09-29T22:46:43.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8=', 'Content-Length': '0'} 2023-09-29T22:46:43.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e95ec2d2f31a0b81-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.853 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.853 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.853 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.853 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.853 DEBUG:boto:Method: GET 2023-09-29T22:46:43.853 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.853 DEBUG:boto:Data: 2023-09-29T22:46:43.854 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.854 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.854 DEBUG:boto:Params: {} 2023-09-29T22:46:43.854 DEBUG:boto:Token: None 2023-09-29T22:46:43.854 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8= 2023-09-29T22:46:43.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8=', 'Content-Length': '0'} 2023-09-29T22:46:43.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000105cf6194cce824e-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.902 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.902 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.903 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.903 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.903 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.903 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.903 DEBUG:boto:Data: 2023-09-29T22:46:43.903 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.903 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.903 DEBUG:boto:Params: {} 2023-09-29T22:46:43.903 DEBUG:boto:Token: None 2023-09-29T22:46:43.904 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hDsnSp1hUK3WUo1vWoXQR/+A6AA= 2023-09-29T22:46:43.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hDsnSp1hUK3WUo1vWoXQR/+A6AA=', 'Content-Length': '0'} 2023-09-29T22:46:43.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0affae8a0464321-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.907 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.907 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.907 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.907 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.907 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.907 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.907 DEBUG:boto:Data: 2023-09-29T22:46:43.907 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.908 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.908 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.908 DEBUG:boto:Params: {} 2023-09-29T22:46:43.908 DEBUG:boto:Token: None 2023-09-29T22:46:43.908 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hDsnSp1hUK3WUo1vWoXQR/+A6AA= 2023-09-29T22:46:43.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hDsnSp1hUK3WUo1vWoXQR/+A6AA=', 'Content-Length': '0'} 2023-09-29T22:46:43.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6e94ecee2e0af9-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:46:43.911 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.911 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.911 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.911 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.911 DEBUG:boto:Method: GET 2023-09-29T22:46:43.911 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.912 DEBUG:boto:Data: 2023-09-29T22:46:43.912 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.912 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.912 DEBUG:boto:Params: {} 2023-09-29T22:46:43.912 DEBUG:boto:Token: None 2023-09-29T22:46:43.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8= 2023-09-29T22:46:43.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8=', 'Content-Length': '0'} 2023-09-29T22:46:43.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2ce68418f071027-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.915 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.917 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.917 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.917 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.917 DEBUG:boto:Method: GET 2023-09-29T22:46:43.917 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.918 DEBUG:boto:Data: 2023-09-29T22:46:43.918 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.918 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.918 DEBUG:boto:Params: {} 2023-09-29T22:46:43.918 DEBUG:boto:Token: None 2023-09-29T22:46:43.918 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:46:43.918 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8= 2023-09-29T22:46:43.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pF49yRAYV04yFgjWWO30SO/UFQ8=', 'Content-Length': '0'} 2023-09-29T22:46:43.921 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b16d2980a5cd83d3-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.921 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.921 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.922 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:43.922 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:43.922 DEBUG:boto:Method: GET 2023-09-29T22:46:43.922 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:46:43.922 DEBUG:boto:Data: 2023-09-29T22:46:43.922 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.922 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.922 DEBUG:boto:Params: {} 2023-09-29T22:46:43.923 DEBUG:boto:Token: None 2023-09-29T22:46:43.923 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.927 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ad2d66b4cab0cc7f-00651753d3-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.970 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.971 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:43.971 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:43.971 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:43.971 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:43.971 DEBUG:boto:Method: GET 2023-09-29T22:46:43.971 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:46:43.971 DEBUG:boto:Data: 2023-09-29T22:46:43.971 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.972 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.972 DEBUG:boto:Params: {} 2023-09-29T22:46:43.972 DEBUG:boto:Token: None 2023-09-29T22:46:43.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/ 2023-09-29T22:46:43.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY= 2023-09-29T22:46:43.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOB8p9qi85xvATI9+oe9R1Gg4VY=', 'Content-Length': '0'} 2023-09-29T22:46:43.979 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002546942de43ac72e-00651753d3-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.980 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:43.980 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.980 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.980 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.981 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:46:43.981 DEBUG:boto:Data: 2023-09-29T22:46:43.981 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.981 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.981 DEBUG:boto:Params: {} 2023-09-29T22:46:43.981 DEBUG:boto:Token: None 2023-09-29T22:46:43.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19 2023-09-29T22:46:43.982 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7qkeNqVMPfr7/+RpV7lFLYd0wjo= 2023-09-29T22:46:43.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7qkeNqVMPfr7/+RpV7lFLYd0wjo=', 'Content-Length': '0'} 2023-09-29T22:46:43.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079601726c1a16b04-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.984 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.984 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.984 DEBUG:boto:Method: HEAD 2023-09-29T22:46:43.984 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:46:43.984 DEBUG:boto:Data: 2023-09-29T22:46:43.984 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.985 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.985 DEBUG:boto:Params: {} 2023-09-29T22:46:43.985 DEBUG:boto:Token: None 2023-09-29T22:46:43.985 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19 2023-09-29T22:46:43.985 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7qkeNqVMPfr7/+RpV7lFLYd0wjo= 2023-09-29T22:46:43.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7qkeNqVMPfr7/+RpV7lFLYd0wjo=', 'Content-Length': '0'} 2023-09-29T22:46:43.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f01877416c1cc5d8-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:46:43.988 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.988 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.988 DEBUG:boto:Method: GET 2023-09-29T22:46:43.988 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:46:43.988 DEBUG:boto:Data: 2023-09-29T22:46:43.988 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:43.988 DEBUG:boto:Port: 8001 2023-09-29T22:46:43.988 DEBUG:boto:Params: {} 2023-09-29T22:46:43.989 DEBUG:boto:Token: None 2023-09-29T22:46:43.989 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19 2023-09-29T22:46:43.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7B3Pq/T3dOd2OI99jz0j10gm2Sw= 2023-09-29T22:46:43.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7B3Pq/T3dOd2OI99jz0j10gm2Sw=', 'Content-Length': '0'} 2023-09-29T22:46:43.991 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000688d8b05270f71ee-00651753d3-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:43.991 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:46:43.991 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:46:43.992 DEBUG:boto:Method: GET 2023-09-29T22:46:43.992 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:46:43.992 DEBUG:boto:Data: 2023-09-29T22:46:43.992 DEBUG:boto:Headers: {} 2023-09-29T22:46:43.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:43.992 DEBUG:boto:Port: 8000 2023-09-29T22:46:43.992 DEBUG:boto:Params: {} 2023-09-29T22:46:43.992 DEBUG:boto:Token: None 2023-09-29T22:46:43.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:43 GMT /hcnphz-59/obj19 2023-09-29T22:46:43.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7B3Pq/T3dOd2OI99jz0j10gm2Sw= 2023-09-29T22:46:43.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:43 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7B3Pq/T3dOd2OI99jz0j10gm2Sw=', 'Content-Length': '0'} 2023-09-29T22:46:43.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b439af229d4e0d86-00651753d3-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:43 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.038 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.038 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.039 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:44.039 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:44.039 DEBUG:boto:Method: GET 2023-09-29T22:46:44.039 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:46:44.039 DEBUG:boto:Data: 2023-09-29T22:46:44.039 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.039 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.039 DEBUG:boto:Params: {} 2023-09-29T22:46:44.040 DEBUG:boto:Token: None 2023-09-29T22:46:44.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c2e58ae3cc80108-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.045 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.045 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.046 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.046 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:44.046 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:46:44.046 DEBUG:boto:Method: GET 2023-09-29T22:46:44.046 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:46:44.046 DEBUG:boto:Data: 2023-09-29T22:46:44.046 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.046 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.046 DEBUG:boto:Params: {} 2023-09-29T22:46:44.047 DEBUG:boto:Token: None 2023-09-29T22:46:44.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb6e292c8cec79ce-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.054 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:46:44.054 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.054 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.054 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.055 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.055 DEBUG:boto:Method: GET 2023-09-29T22:46:44.055 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.055 DEBUG:boto:Data: 2023-09-29T22:46:44.055 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.055 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.055 DEBUG:boto:Params: {} 2023-09-29T22:46:44.055 DEBUG:boto:Token: None 2023-09-29T22:46:44.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw= 2023-09-29T22:46:44.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw=', 'Content-Length': '0'} 2023-09-29T22:46:44.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d165d9d5a600e1b-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.098 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.098 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.099 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.099 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.099 DEBUG:boto:Method: GET 2023-09-29T22:46:44.099 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.099 DEBUG:boto:Data: 2023-09-29T22:46:44.099 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.099 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.099 DEBUG:boto:Params: {} 2023-09-29T22:46:44.099 DEBUG:boto:Token: None 2023-09-29T22:46:44.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw= 2023-09-29T22:46:44.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw=', 'Content-Length': '0'} 2023-09-29T22:46:44.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000109ea89bfe24e3ca-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.103 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.103 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.103 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.103 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.103 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.104 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.104 DEBUG:boto:Data: 2023-09-29T22:46:44.104 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.104 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.104 DEBUG:boto:Params: {} 2023-09-29T22:46:44.104 DEBUG:boto:Token: None 2023-09-29T22:46:44.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BnnRI7JTNwx+vaQPUBDPR6MMdwg= 2023-09-29T22:46:44.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BnnRI7JTNwx+vaQPUBDPR6MMdwg=', 'Content-Length': '0'} 2023-09-29T22:46:44.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bc949c708f086ac-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.107 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.107 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.107 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.108 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.108 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.108 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.108 DEBUG:boto:Data: 2023-09-29T22:46:44.108 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.108 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.108 DEBUG:boto:Params: {} 2023-09-29T22:46:44.108 DEBUG:boto:Token: None 2023-09-29T22:46:44.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BnnRI7JTNwx+vaQPUBDPR6MMdwg= 2023-09-29T22:46:44.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BnnRI7JTNwx+vaQPUBDPR6MMdwg=', 'Content-Length': '0'} 2023-09-29T22:46:44.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d538144cd49cb074-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:46:44.112 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.112 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.112 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.112 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.112 DEBUG:boto:Method: GET 2023-09-29T22:46:44.112 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.112 DEBUG:boto:Data: 2023-09-29T22:46:44.113 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.113 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.113 DEBUG:boto:Params: {} 2023-09-29T22:46:44.113 DEBUG:boto:Token: None 2023-09-29T22:46:44.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw= 2023-09-29T22:46:44.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw=', 'Content-Length': '0'} 2023-09-29T22:46:44.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08c786e0a3e6fca-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.116 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.117 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.117 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.117 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.117 DEBUG:boto:Method: GET 2023-09-29T22:46:44.117 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.117 DEBUG:boto:Data: 2023-09-29T22:46:44.117 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.117 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.117 DEBUG:boto:Params: {} 2023-09-29T22:46:44.118 DEBUG:boto:Token: None 2023-09-29T22:46:44.118 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:46:44.118 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw= 2023-09-29T22:46:44.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9HifLUi4Lt2ReYgcQAWTD1loOTw=', 'Content-Length': '0'} 2023-09-29T22:46:44.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065a9c0d540c06d17-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.162 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.162 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.163 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.163 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.163 DEBUG:boto:Method: GET 2023-09-29T22:46:44.163 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.163 DEBUG:boto:Data: 2023-09-29T22:46:44.163 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.163 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.163 DEBUG:boto:Params: {} 2023-09-29T22:46:44.163 DEBUG:boto:Token: None 2023-09-29T22:46:44.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.167 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dae990d3cca3ca51-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.167 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.167 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.168 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.168 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.168 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.168 DEBUG:boto:Method: GET 2023-09-29T22:46:44.168 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.168 DEBUG:boto:Data: 2023-09-29T22:46:44.168 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.168 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.169 DEBUG:boto:Params: {} 2023-09-29T22:46:44.169 DEBUG:boto:Token: None 2023-09-29T22:46:44.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079f1d36dff1a2d1f-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.174 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.175 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.175 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.175 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.175 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:46:44.175 DEBUG:boto:Data: 2023-09-29T22:46:44.175 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.176 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.176 DEBUG:boto:Params: {} 2023-09-29T22:46:44.176 DEBUG:boto:Token: None 2023-09-29T22:46:44.176 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22 2023-09-29T22:46:44.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4BelHn+4+mDccgnbBwD3Wd5mw4U= 2023-09-29T22:46:44.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4BelHn+4+mDccgnbBwD3Wd5mw4U=', 'Content-Length': '0'} 2023-09-29T22:46:44.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e648a87fbf56d33-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.179 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.179 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.179 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.179 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:46:44.179 DEBUG:boto:Data: 2023-09-29T22:46:44.179 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.179 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.180 DEBUG:boto:Params: {} 2023-09-29T22:46:44.180 DEBUG:boto:Token: None 2023-09-29T22:46:44.180 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22 2023-09-29T22:46:44.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4BelHn+4+mDccgnbBwD3Wd5mw4U= 2023-09-29T22:46:44.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4BelHn+4+mDccgnbBwD3Wd5mw4U=', 'Content-Length': '0'} 2023-09-29T22:46:44.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061ce99ecfe7e7213-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:46:44.183 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.183 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.183 DEBUG:boto:Method: GET 2023-09-29T22:46:44.183 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:46:44.183 DEBUG:boto:Data: 2023-09-29T22:46:44.183 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.183 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.184 DEBUG:boto:Params: {} 2023-09-29T22:46:44.184 DEBUG:boto:Token: None 2023-09-29T22:46:44.184 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22 2023-09-29T22:46:44.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ojcGHw14KOm9i74/IwOrDyCT8vU= 2023-09-29T22:46:44.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ojcGHw14KOm9i74/IwOrDyCT8vU=', 'Content-Length': '0'} 2023-09-29T22:46:44.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3ef118bf9190fef-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.226 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:46:44.226 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:46:44.227 DEBUG:boto:Method: GET 2023-09-29T22:46:44.227 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:46:44.227 DEBUG:boto:Data: 2023-09-29T22:46:44.227 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.227 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.227 DEBUG:boto:Params: {} 2023-09-29T22:46:44.227 DEBUG:boto:Token: None 2023-09-29T22:46:44.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj22 2023-09-29T22:46:44.227 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ojcGHw14KOm9i74/IwOrDyCT8vU= 2023-09-29T22:46:44.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ojcGHw14KOm9i74/IwOrDyCT8vU=', 'Content-Length': '0'} 2023-09-29T22:46:44.230 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f8837fcd21f64b-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.230 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.231 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.231 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.231 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.231 DEBUG:boto:Method: GET 2023-09-29T22:46:44.231 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.231 DEBUG:boto:Data: 2023-09-29T22:46:44.231 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.232 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.232 DEBUG:boto:Params: {} 2023-09-29T22:46:44.232 DEBUG:boto:Token: None 2023-09-29T22:46:44.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.237 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6e36634bb0b731b-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.237 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.238 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.238 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.238 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.238 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.238 DEBUG:boto:Method: GET 2023-09-29T22:46:44.238 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:46:44.238 DEBUG:boto:Data: 2023-09-29T22:46:44.238 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.239 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.239 DEBUG:boto:Params: {} 2023-09-29T22:46:44.239 DEBUG:boto:Token: None 2023-09-29T22:46:44.239 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dcbb87ebc4c5fdd-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.290 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:46:44.291 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.291 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.291 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.291 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.292 DEBUG:boto:Method: GET 2023-09-29T22:46:44.292 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.292 DEBUG:boto:Data: 2023-09-29T22:46:44.292 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.292 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.292 DEBUG:boto:Params: {} 2023-09-29T22:46:44.293 DEBUG:boto:Token: None 2023-09-29T22:46:44.293 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo= 2023-09-29T22:46:44.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo=', 'Content-Length': '0'} 2023-09-29T22:46:44.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021ae932191c15207-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.296 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.296 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.296 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.297 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.297 DEBUG:boto:Method: GET 2023-09-29T22:46:44.297 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.297 DEBUG:boto:Data: 2023-09-29T22:46:44.298 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.298 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.298 DEBUG:boto:Params: {} 2023-09-29T22:46:44.298 DEBUG:boto:Token: None 2023-09-29T22:46:44.298 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo= 2023-09-29T22:46:44.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo=', 'Content-Length': '0'} 2023-09-29T22:46:44.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000820df05cba09212e-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.302 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.302 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.302 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.302 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.302 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.302 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.302 DEBUG:boto:Data: 2023-09-29T22:46:44.303 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.303 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.303 DEBUG:boto:Params: {} 2023-09-29T22:46:44.303 DEBUG:boto:Token: None 2023-09-29T22:46:44.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.304 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u8k43GJuBpVvDGABcTTJLTjC6nw= 2023-09-29T22:46:44.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u8k43GJuBpVvDGABcTTJLTjC6nw=', 'Content-Length': '0'} 2023-09-29T22:46:44.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000371b99500d6efcf2-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.306 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.307 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.307 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.307 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.307 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.307 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.307 DEBUG:boto:Data: 2023-09-29T22:46:44.307 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.307 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.307 DEBUG:boto:Params: {} 2023-09-29T22:46:44.308 DEBUG:boto:Token: None 2023-09-29T22:46:44.308 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.308 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u8k43GJuBpVvDGABcTTJLTjC6nw= 2023-09-29T22:46:44.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u8k43GJuBpVvDGABcTTJLTjC6nw=', 'Content-Length': '0'} 2023-09-29T22:46:44.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000584d06e5f0cd11a7-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:46:44.311 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.312 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.312 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.312 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.312 DEBUG:boto:Method: GET 2023-09-29T22:46:44.312 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.312 DEBUG:boto:Data: 2023-09-29T22:46:44.312 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.313 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.313 DEBUG:boto:Params: {} 2023-09-29T22:46:44.313 DEBUG:boto:Token: None 2023-09-29T22:46:44.313 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo= 2023-09-29T22:46:44.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo=', 'Content-Length': '0'} 2023-09-29T22:46:44.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059fae1b5e406c380-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.358 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.359 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.359 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.359 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.359 DEBUG:boto:Method: GET 2023-09-29T22:46:44.359 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.359 DEBUG:boto:Data: 2023-09-29T22:46:44.359 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.359 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.360 DEBUG:boto:Params: {} 2023-09-29T22:46:44.360 DEBUG:boto:Token: None 2023-09-29T22:46:44.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:46:44.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo= 2023-09-29T22:46:44.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/pp6EzqLR07PYNk1yoLSZ5QsfIo=', 'Content-Length': '0'} 2023-09-29T22:46:44.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c31b14a541826ab-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.364 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.364 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.364 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.364 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.364 DEBUG:boto:Method: GET 2023-09-29T22:46:44.364 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.364 DEBUG:boto:Data: 2023-09-29T22:46:44.365 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.365 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.365 DEBUG:boto:Params: {} 2023-09-29T22:46:44.365 DEBUG:boto:Token: None 2023-09-29T22:46:44.365 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009740d961e7fe6304-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.371 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.372 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.372 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.372 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.373 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.373 DEBUG:boto:Method: GET 2023-09-29T22:46:44.373 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.373 DEBUG:boto:Data: 2023-09-29T22:46:44.373 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.373 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.374 DEBUG:boto:Params: {} 2023-09-29T22:46:44.374 DEBUG:boto:Token: None 2023-09-29T22:46:44.374 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.374 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000756834d04327b671-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.381 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.382 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.382 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.382 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.382 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:46:44.382 DEBUG:boto:Data: 2023-09-29T22:46:44.382 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.383 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.383 DEBUG:boto:Params: {} 2023-09-29T22:46:44.383 DEBUG:boto:Token: None 2023-09-29T22:46:44.383 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23 2023-09-29T22:46:44.383 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9/PIt/pHFB5lb55UQVnxg4BzmtU= 2023-09-29T22:46:44.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9/PIt/pHFB5lb55UQVnxg4BzmtU=', 'Content-Length': '0'} 2023-09-29T22:46:44.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002941f418acfd1ba-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.386 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.386 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.386 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.386 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:46:44.386 DEBUG:boto:Data: 2023-09-29T22:46:44.386 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.387 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.387 DEBUG:boto:Params: {} 2023-09-29T22:46:44.387 DEBUG:boto:Token: None 2023-09-29T22:46:44.387 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23 2023-09-29T22:46:44.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9/PIt/pHFB5lb55UQVnxg4BzmtU= 2023-09-29T22:46:44.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9/PIt/pHFB5lb55UQVnxg4BzmtU=', 'Content-Length': '0'} 2023-09-29T22:46:44.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000922913bac4fac685-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:46:44.390 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.390 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.390 DEBUG:boto:Method: GET 2023-09-29T22:46:44.390 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:46:44.390 DEBUG:boto:Data: 2023-09-29T22:46:44.390 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.391 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.391 DEBUG:boto:Params: {} 2023-09-29T22:46:44.391 DEBUG:boto:Token: None 2023-09-29T22:46:44.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23 2023-09-29T22:46:44.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k0v41e2w4EGbcgktL9GJX8tK5bQ= 2023-09-29T22:46:44.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k0v41e2w4EGbcgktL9GJX8tK5bQ=', 'Content-Length': '0'} 2023-09-29T22:46:44.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfca6394c1797e1c-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.394 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:46:44.394 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:46:44.394 DEBUG:boto:Method: GET 2023-09-29T22:46:44.394 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:46:44.394 DEBUG:boto:Data: 2023-09-29T22:46:44.394 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.395 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.395 DEBUG:boto:Params: {} 2023-09-29T22:46:44.395 DEBUG:boto:Token: None 2023-09-29T22:46:44.395 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj23 2023-09-29T22:46:44.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k0v41e2w4EGbcgktL9GJX8tK5bQ= 2023-09-29T22:46:44.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k0v41e2w4EGbcgktL9GJX8tK5bQ=', 'Content-Length': '0'} 2023-09-29T22:46:44.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b25a0cdda2aa583-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.438 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.439 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.439 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.439 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.439 DEBUG:boto:Method: GET 2023-09-29T22:46:44.441 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.441 DEBUG:boto:Data: 2023-09-29T22:46:44.441 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.442 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.442 DEBUG:boto:Params: {} 2023-09-29T22:46:44.442 DEBUG:boto:Token: None 2023-09-29T22:46:44.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003984df4f9eb2d28-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.448 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.448 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.448 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.448 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.448 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.449 DEBUG:boto:Method: GET 2023-09-29T22:46:44.449 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:46:44.449 DEBUG:boto:Data: 2023-09-29T22:46:44.449 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.449 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.449 DEBUG:boto:Params: {} 2023-09-29T22:46:44.449 DEBUG:boto:Token: None 2023-09-29T22:46:44.449 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.450 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.456 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000402a02330b460427-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.456 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:46:44.457 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.457 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.457 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.457 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.457 DEBUG:boto:Method: GET 2023-09-29T22:46:44.457 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.457 DEBUG:boto:Data: 2023-09-29T22:46:44.458 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.458 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.458 DEBUG:boto:Params: {} 2023-09-29T22:46:44.458 DEBUG:boto:Token: None 2023-09-29T22:46:44.458 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.458 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8= 2023-09-29T22:46:44.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8=', 'Content-Length': '0'} 2023-09-29T22:46:44.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000012a1bb0625753b7-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.502 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.502 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.503 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.503 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.503 DEBUG:boto:Method: GET 2023-09-29T22:46:44.503 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.503 DEBUG:boto:Data: 2023-09-29T22:46:44.503 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.503 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.503 DEBUG:boto:Params: {} 2023-09-29T22:46:44.503 DEBUG:boto:Token: None 2023-09-29T22:46:44.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8= 2023-09-29T22:46:44.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8=', 'Content-Length': '0'} 2023-09-29T22:46:44.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ff8a8e5fc6b9f8-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.507 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.507 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.507 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.507 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.507 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.507 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.507 DEBUG:boto:Data: 2023-09-29T22:46:44.507 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.508 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.508 DEBUG:boto:Params: {} 2023-09-29T22:46:44.508 DEBUG:boto:Token: None 2023-09-29T22:46:44.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i3bkZ09i89GEaZeFV00H9zvUmRk= 2023-09-29T22:46:44.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i3bkZ09i89GEaZeFV00H9zvUmRk=', 'Content-Length': '0'} 2023-09-29T22:46:44.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb0c27126985239c-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.511 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.511 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.511 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.511 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.512 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.512 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.512 DEBUG:boto:Data: 2023-09-29T22:46:44.512 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.512 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.512 DEBUG:boto:Params: {} 2023-09-29T22:46:44.512 DEBUG:boto:Token: None 2023-09-29T22:46:44.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i3bkZ09i89GEaZeFV00H9zvUmRk= 2023-09-29T22:46:44.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i3bkZ09i89GEaZeFV00H9zvUmRk=', 'Content-Length': '0'} 2023-09-29T22:46:44.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022e9461ff6514c5b-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:46:44.516 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.516 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.516 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.516 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.516 DEBUG:boto:Method: GET 2023-09-29T22:46:44.516 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.516 DEBUG:boto:Data: 2023-09-29T22:46:44.516 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.516 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.517 DEBUG:boto:Params: {} 2023-09-29T22:46:44.517 DEBUG:boto:Token: None 2023-09-29T22:46:44.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8= 2023-09-29T22:46:44.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8=', 'Content-Length': '0'} 2023-09-29T22:46:44.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009eb94c84b3ba9bcc-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.520 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.520 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.520 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.521 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.521 DEBUG:boto:Method: GET 2023-09-29T22:46:44.521 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.521 DEBUG:boto:Data: 2023-09-29T22:46:44.521 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.521 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.521 DEBUG:boto:Params: {} 2023-09-29T22:46:44.521 DEBUG:boto:Token: None 2023-09-29T22:46:44.522 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:46:44.522 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8= 2023-09-29T22:46:44.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RUu4/NTkKNn0VLT2bUxQwH7vrh8=', 'Content-Length': '0'} 2023-09-29T22:46:44.524 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac08e5c4c828b5f3-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.566 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.567 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.567 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.567 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.567 DEBUG:boto:Method: GET 2023-09-29T22:46:44.567 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.567 DEBUG:boto:Data: 2023-09-29T22:46:44.567 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.567 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.567 DEBUG:boto:Params: {} 2023-09-29T22:46:44.568 DEBUG:boto:Token: None 2023-09-29T22:46:44.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.572 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f2e33e087e83835-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.573 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.573 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.573 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.573 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.573 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.573 DEBUG:boto:Method: GET 2023-09-29T22:46:44.574 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.574 DEBUG:boto:Data: 2023-09-29T22:46:44.574 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.574 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.574 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.574 DEBUG:boto:Params: {} 2023-09-29T22:46:44.574 DEBUG:boto:Token: None 2023-09-29T22:46:44.574 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.575 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.575 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.581 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bdf8aefe2b71a11e-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.582 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.582 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.582 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.582 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.582 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:46:44.583 DEBUG:boto:Data: 2023-09-29T22:46:44.583 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.583 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.583 DEBUG:boto:Params: {} 2023-09-29T22:46:44.583 DEBUG:boto:Token: None 2023-09-29T22:46:44.583 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29 2023-09-29T22:46:44.583 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zbo0avB2C9HsuQhT85E2saBr5u0= 2023-09-29T22:46:44.583 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zbo0avB2C9HsuQhT85E2saBr5u0=', 'Content-Length': '0'} 2023-09-29T22:46:44.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d53e986b2c1a7ccc-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.586 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.586 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.586 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.586 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:46:44.586 DEBUG:boto:Data: 2023-09-29T22:46:44.586 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.586 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.587 DEBUG:boto:Params: {} 2023-09-29T22:46:44.587 DEBUG:boto:Token: None 2023-09-29T22:46:44.587 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29 2023-09-29T22:46:44.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zbo0avB2C9HsuQhT85E2saBr5u0= 2023-09-29T22:46:44.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zbo0avB2C9HsuQhT85E2saBr5u0=', 'Content-Length': '0'} 2023-09-29T22:46:44.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c9cc899e00d5b5-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:46:44.589 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.589 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.590 DEBUG:boto:Method: GET 2023-09-29T22:46:44.590 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:46:44.590 DEBUG:boto:Data: 2023-09-29T22:46:44.590 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.590 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.590 DEBUG:boto:Params: {} 2023-09-29T22:46:44.590 DEBUG:boto:Token: None 2023-09-29T22:46:44.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29 2023-09-29T22:46:44.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X14Vr/VNo1EQM+Bc2qZKnTPi8B8= 2023-09-29T22:46:44.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X14Vr/VNo1EQM+Bc2qZKnTPi8B8=', 'Content-Length': '0'} 2023-09-29T22:46:44.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000757cb59ac6fc908f-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.634 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:46:44.634 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:46:44.635 DEBUG:boto:Method: GET 2023-09-29T22:46:44.635 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:46:44.635 DEBUG:boto:Data: 2023-09-29T22:46:44.635 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.635 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.635 DEBUG:boto:Params: {} 2023-09-29T22:46:44.635 DEBUG:boto:Token: None 2023-09-29T22:46:44.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj29 2023-09-29T22:46:44.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X14Vr/VNo1EQM+Bc2qZKnTPi8B8= 2023-09-29T22:46:44.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X14Vr/VNo1EQM+Bc2qZKnTPi8B8=', 'Content-Length': '0'} 2023-09-29T22:46:44.639 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004dbf50d3dfab2fa0-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.639 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.639 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.639 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.640 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.640 DEBUG:boto:Method: GET 2023-09-29T22:46:44.640 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.640 DEBUG:boto:Data: 2023-09-29T22:46:44.640 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.640 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.640 DEBUG:boto:Params: {} 2023-09-29T22:46:44.640 DEBUG:boto:Token: None 2023-09-29T22:46:44.641 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.646 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013e68c9b3a988d83-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.646 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.646 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.646 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.647 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.647 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.647 DEBUG:boto:Method: GET 2023-09-29T22:46:44.647 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:46:44.647 DEBUG:boto:Data: 2023-09-29T22:46:44.647 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.647 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.647 DEBUG:boto:Params: {} 2023-09-29T22:46:44.647 DEBUG:boto:Token: None 2023-09-29T22:46:44.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.654 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000df8e619e7b2217f-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.694 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:46:44.695 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.695 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.695 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.695 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.695 DEBUG:boto:Method: GET 2023-09-29T22:46:44.695 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.696 DEBUG:boto:Data: 2023-09-29T22:46:44.696 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.696 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.696 DEBUG:boto:Params: {} 2023-09-29T22:46:44.696 DEBUG:boto:Token: None 2023-09-29T22:46:44.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0= 2023-09-29T22:46:44.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0=', 'Content-Length': '0'} 2023-09-29T22:46:44.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003967059c4fa772ea-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.700 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.700 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.700 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.700 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.700 DEBUG:boto:Method: GET 2023-09-29T22:46:44.700 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.701 DEBUG:boto:Data: 2023-09-29T22:46:44.701 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.701 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.701 DEBUG:boto:Params: {} 2023-09-29T22:46:44.701 DEBUG:boto:Token: None 2023-09-29T22:46:44.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0= 2023-09-29T22:46:44.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0=', 'Content-Length': '0'} 2023-09-29T22:46:44.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000264b6d4c2abc268b-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.705 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.705 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.705 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.705 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.705 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.705 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.705 DEBUG:boto:Data: 2023-09-29T22:46:44.705 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.705 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.705 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.706 DEBUG:boto:Params: {} 2023-09-29T22:46:44.706 DEBUG:boto:Token: None 2023-09-29T22:46:44.706 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.706 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k/kJz0RsAC1nuSc6t5usdHw8W20= 2023-09-29T22:46:44.706 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k/kJz0RsAC1nuSc6t5usdHw8W20=', 'Content-Length': '0'} 2023-09-29T22:46:44.708 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fa3f19d2a07e28b-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.708 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.709 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.709 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.709 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.709 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.709 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.709 DEBUG:boto:Data: 2023-09-29T22:46:44.709 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.709 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.709 DEBUG:boto:Params: {} 2023-09-29T22:46:44.710 DEBUG:boto:Token: None 2023-09-29T22:46:44.710 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k/kJz0RsAC1nuSc6t5usdHw8W20= 2023-09-29T22:46:44.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k/kJz0RsAC1nuSc6t5usdHw8W20=', 'Content-Length': '0'} 2023-09-29T22:46:44.712 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014d182beba77df37-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:46:44.712 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.712 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.713 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.713 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.713 DEBUG:boto:Method: GET 2023-09-29T22:46:44.713 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.713 DEBUG:boto:Data: 2023-09-29T22:46:44.713 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.713 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.713 DEBUG:boto:Params: {} 2023-09-29T22:46:44.713 DEBUG:boto:Token: None 2023-09-29T22:46:44.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0= 2023-09-29T22:46:44.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0=', 'Content-Length': '0'} 2023-09-29T22:46:44.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656f3d893ea694e3-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.716 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.716 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.716 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.717 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.717 DEBUG:boto:Method: GET 2023-09-29T22:46:44.717 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.717 DEBUG:boto:Data: 2023-09-29T22:46:44.717 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.717 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.717 DEBUG:boto:Params: {} 2023-09-29T22:46:44.717 DEBUG:boto:Token: None 2023-09-29T22:46:44.717 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:46:44.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0= 2023-09-29T22:46:44.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pz14qETqiq6WTt6JYjpZ6IL0gr0=', 'Content-Length': '0'} 2023-09-29T22:46:44.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c6925b344b081a6-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.720 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.721 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.721 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.721 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.721 DEBUG:boto:Method: GET 2023-09-29T22:46:44.721 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.721 DEBUG:boto:Data: 2023-09-29T22:46:44.721 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.721 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.721 DEBUG:boto:Params: {} 2023-09-29T22:46:44.722 DEBUG:boto:Token: None 2023-09-29T22:46:44.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f00aedddfae52487-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.725 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.726 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.726 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.726 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.726 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.726 DEBUG:boto:Method: GET 2023-09-29T22:46:44.726 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.726 DEBUG:boto:Data: 2023-09-29T22:46:44.727 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.727 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.727 DEBUG:boto:Params: {} 2023-09-29T22:46:44.727 DEBUG:boto:Token: None 2023-09-29T22:46:44.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024712da1f311bf81-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.734 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.735 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.735 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.735 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.735 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:46:44.735 DEBUG:boto:Data: 2023-09-29T22:46:44.735 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.736 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.736 DEBUG:boto:Params: {} 2023-09-29T22:46:44.736 DEBUG:boto:Token: None 2023-09-29T22:46:44.736 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33 2023-09-29T22:46:44.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tg0bA+a8rpHMeiG3IqGqajIjels= 2023-09-29T22:46:44.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tg0bA+a8rpHMeiG3IqGqajIjels=', 'Content-Length': '0'} 2023-09-29T22:46:44.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d360ae4d13273290-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.739 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.739 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.739 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.739 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:46:44.739 DEBUG:boto:Data: 2023-09-29T22:46:44.739 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.739 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.740 DEBUG:boto:Params: {} 2023-09-29T22:46:44.740 DEBUG:boto:Token: None 2023-09-29T22:46:44.740 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33 2023-09-29T22:46:44.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tg0bA+a8rpHMeiG3IqGqajIjels= 2023-09-29T22:46:44.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tg0bA+a8rpHMeiG3IqGqajIjels=', 'Content-Length': '0'} 2023-09-29T22:46:44.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ff8928c50d99abd-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:46:44.742 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.743 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.743 DEBUG:boto:Method: GET 2023-09-29T22:46:44.743 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:46:44.743 DEBUG:boto:Data: 2023-09-29T22:46:44.743 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.743 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.743 DEBUG:boto:Params: {} 2023-09-29T22:46:44.743 DEBUG:boto:Token: None 2023-09-29T22:46:44.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33 2023-09-29T22:46:44.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:px3P89Ygv9c9ke7hEonxi9YREjM= 2023-09-29T22:46:44.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:px3P89Ygv9c9ke7hEonxi9YREjM=', 'Content-Length': '0'} 2023-09-29T22:46:44.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344def9b80248e65-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.786 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:46:44.786 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:46:44.787 DEBUG:boto:Method: GET 2023-09-29T22:46:44.787 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:46:44.787 DEBUG:boto:Data: 2023-09-29T22:46:44.787 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.787 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.787 DEBUG:boto:Params: {} 2023-09-29T22:46:44.787 DEBUG:boto:Token: None 2023-09-29T22:46:44.787 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj33 2023-09-29T22:46:44.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:px3P89Ygv9c9ke7hEonxi9YREjM= 2023-09-29T22:46:44.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:px3P89Ygv9c9ke7hEonxi9YREjM=', 'Content-Length': '0'} 2023-09-29T22:46:44.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087874d8269c885c8-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.791 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.791 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.791 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.791 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.792 DEBUG:boto:Method: GET 2023-09-29T22:46:44.792 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.792 DEBUG:boto:Data: 2023-09-29T22:46:44.792 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.792 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.792 DEBUG:boto:Params: {} 2023-09-29T22:46:44.792 DEBUG:boto:Token: None 2023-09-29T22:46:44.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000031e99b8a781f8c07-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.797 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.798 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.798 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.798 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.798 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.798 DEBUG:boto:Method: GET 2023-09-29T22:46:44.798 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:46:44.798 DEBUG:boto:Data: 2023-09-29T22:46:44.799 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.799 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.799 DEBUG:boto:Params: {} 2023-09-29T22:46:44.799 DEBUG:boto:Token: None 2023-09-29T22:46:44.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.799 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cbb842722d57d93-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.806 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:46:44.807 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.807 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.807 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.807 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.807 DEBUG:boto:Method: GET 2023-09-29T22:46:44.807 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.807 DEBUG:boto:Data: 2023-09-29T22:46:44.807 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.807 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.808 DEBUG:boto:Params: {} 2023-09-29T22:46:44.808 DEBUG:boto:Token: None 2023-09-29T22:46:44.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc= 2023-09-29T22:46:44.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc=', 'Content-Length': '0'} 2023-09-29T22:46:44.810 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab0f3ade3d92962-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.810 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.811 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.811 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.811 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.811 DEBUG:boto:Method: GET 2023-09-29T22:46:44.811 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.811 DEBUG:boto:Data: 2023-09-29T22:46:44.811 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.811 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.812 DEBUG:boto:Params: {} 2023-09-29T22:46:44.812 DEBUG:boto:Token: None 2023-09-29T22:46:44.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc= 2023-09-29T22:46:44.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc=', 'Content-Length': '0'} 2023-09-29T22:46:44.814 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2a61778420888d5-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.814 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.815 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.815 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.815 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.815 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.815 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.815 DEBUG:boto:Data: 2023-09-29T22:46:44.815 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.816 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.816 DEBUG:boto:Params: {} 2023-09-29T22:46:44.816 DEBUG:boto:Token: None 2023-09-29T22:46:44.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fV8zZ1JLKWoxn6AZs1S26V7WEFE= 2023-09-29T22:46:44.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fV8zZ1JLKWoxn6AZs1S26V7WEFE=', 'Content-Length': '0'} 2023-09-29T22:46:44.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e0718068f5949f2-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.818 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.819 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.819 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.819 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.819 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.819 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.819 DEBUG:boto:Data: 2023-09-29T22:46:44.819 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.820 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.820 DEBUG:boto:Params: {} 2023-09-29T22:46:44.820 DEBUG:boto:Token: None 2023-09-29T22:46:44.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fV8zZ1JLKWoxn6AZs1S26V7WEFE= 2023-09-29T22:46:44.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fV8zZ1JLKWoxn6AZs1S26V7WEFE=', 'Content-Length': '0'} 2023-09-29T22:46:44.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e00b9ef3b5b644d5-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:46:44.823 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.823 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.823 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.823 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.823 DEBUG:boto:Method: GET 2023-09-29T22:46:44.823 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.823 DEBUG:boto:Data: 2023-09-29T22:46:44.824 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.824 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.824 DEBUG:boto:Params: {} 2023-09-29T22:46:44.824 DEBUG:boto:Token: None 2023-09-29T22:46:44.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc= 2023-09-29T22:46:44.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc=', 'Content-Length': '0'} 2023-09-29T22:46:44.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793e89338c4ac7fe-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.870 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.870 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.871 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.871 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.871 DEBUG:boto:Method: GET 2023-09-29T22:46:44.871 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.871 DEBUG:boto:Data: 2023-09-29T22:46:44.871 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.871 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.871 DEBUG:boto:Params: {} 2023-09-29T22:46:44.872 DEBUG:boto:Token: None 2023-09-29T22:46:44.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:46:44.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc= 2023-09-29T22:46:44.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KJtDr14ez/RMtUKULEmT8pj0Ivc=', 'Content-Length': '0'} 2023-09-29T22:46:44.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e88fb9d6c7187b-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.875 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.875 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.875 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.875 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.875 DEBUG:boto:Method: GET 2023-09-29T22:46:44.875 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.876 DEBUG:boto:Data: 2023-09-29T22:46:44.876 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.876 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.876 DEBUG:boto:Params: {} 2023-09-29T22:46:44.876 DEBUG:boto:Token: None 2023-09-29T22:46:44.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ea0a1284ce81956-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.882 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.882 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.882 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.883 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.883 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.883 DEBUG:boto:Method: GET 2023-09-29T22:46:44.883 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.883 DEBUG:boto:Data: 2023-09-29T22:46:44.883 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.883 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.883 DEBUG:boto:Params: {} 2023-09-29T22:46:44.884 DEBUG:boto:Token: None 2023-09-29T22:46:44.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004233b84c9a43b61d-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.891 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.891 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.892 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.892 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.892 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:46:44.892 DEBUG:boto:Data: 2023-09-29T22:46:44.892 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.892 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.892 DEBUG:boto:Params: {} 2023-09-29T22:46:44.892 DEBUG:boto:Token: None 2023-09-29T22:46:44.893 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34 2023-09-29T22:46:44.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mf7acM3tdBPw4xQYU8JsyoTdqfM= 2023-09-29T22:46:44.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mf7acM3tdBPw4xQYU8JsyoTdqfM=', 'Content-Length': '0'} 2023-09-29T22:46:44.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bb1abca6c46b347-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.895 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.895 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.895 DEBUG:boto:Method: HEAD 2023-09-29T22:46:44.896 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:46:44.896 DEBUG:boto:Data: 2023-09-29T22:46:44.896 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.896 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.896 DEBUG:boto:Params: {} 2023-09-29T22:46:44.896 DEBUG:boto:Token: None 2023-09-29T22:46:44.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34 2023-09-29T22:46:44.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Mf7acM3tdBPw4xQYU8JsyoTdqfM= 2023-09-29T22:46:44.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mf7acM3tdBPw4xQYU8JsyoTdqfM=', 'Content-Length': '0'} 2023-09-29T22:46:44.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca6f19795e29e9c1-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:46:44.899 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.899 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.899 DEBUG:boto:Method: GET 2023-09-29T22:46:44.899 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:46:44.899 DEBUG:boto:Data: 2023-09-29T22:46:44.899 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.900 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.900 DEBUG:boto:Params: {} 2023-09-29T22:46:44.900 DEBUG:boto:Token: None 2023-09-29T22:46:44.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34 2023-09-29T22:46:44.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+wZSmh/orUnZCeOKoYy7YWI6y+g= 2023-09-29T22:46:44.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+wZSmh/orUnZCeOKoYy7YWI6y+g=', 'Content-Length': '0'} 2023-09-29T22:46:44.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcddeff729f6d98e-00651753d4-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.947 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:46:44.947 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:46:44.947 DEBUG:boto:Method: GET 2023-09-29T22:46:44.947 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:46:44.947 DEBUG:boto:Data: 2023-09-29T22:46:44.948 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.948 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.948 DEBUG:boto:Params: {} 2023-09-29T22:46:44.948 DEBUG:boto:Token: None 2023-09-29T22:46:44.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/obj34 2023-09-29T22:46:44.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+wZSmh/orUnZCeOKoYy7YWI6y+g= 2023-09-29T22:46:44.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+wZSmh/orUnZCeOKoYy7YWI6y+g=', 'Content-Length': '0'} 2023-09-29T22:46:44.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5eaf9d973c9e0df-00651753d4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.952 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.952 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.953 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.953 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.953 DEBUG:boto:Method: GET 2023-09-29T22:46:44.953 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.953 DEBUG:boto:Data: 2023-09-29T22:46:44.953 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:44.953 DEBUG:boto:Port: 8001 2023-09-29T22:46:44.953 DEBUG:boto:Params: {} 2023-09-29T22:46:44.954 DEBUG:boto:Token: None 2023-09-29T22:46:44.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.954 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4475321dc2bd80f-00651753d4-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:44.959 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:44.960 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:44.960 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:44.960 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.960 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.960 DEBUG:boto:Method: GET 2023-09-29T22:46:44.960 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:46:44.960 DEBUG:boto:Data: 2023-09-29T22:46:44.960 DEBUG:boto:Headers: {} 2023-09-29T22:46:44.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:44.961 DEBUG:boto:Port: 8000 2023-09-29T22:46:44.961 DEBUG:boto:Params: {} 2023-09-29T22:46:44.961 DEBUG:boto:Token: None 2023-09-29T22:46:44.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:44 GMT /hcnphz-59/ 2023-09-29T22:46:44.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg= 2023-09-29T22:46:44.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7J7id6bsU/mE7NMpLlNgJRm1fg=', 'Content-Length': '0'} 2023-09-29T22:46:44.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e9d7a13ceffe39c-00651753d4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.010 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:46:45.011 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.011 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.011 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.012 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.012 DEBUG:boto:Method: GET 2023-09-29T22:46:45.012 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.012 DEBUG:boto:Data: 2023-09-29T22:46:45.012 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.012 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.012 DEBUG:boto:Params: {} 2023-09-29T22:46:45.012 DEBUG:boto:Token: None 2023-09-29T22:46:45.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E= 2023-09-29T22:46:45.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E=', 'Content-Length': '0'} 2023-09-29T22:46:45.016 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bad690c9ce210bd-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.016 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.017 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.017 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.017 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.017 DEBUG:boto:Method: GET 2023-09-29T22:46:45.017 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.017 DEBUG:boto:Data: 2023-09-29T22:46:45.017 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.018 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.018 DEBUG:boto:Params: {} 2023-09-29T22:46:45.018 DEBUG:boto:Token: None 2023-09-29T22:46:45.018 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.018 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E= 2023-09-29T22:46:45.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E=', 'Content-Length': '0'} 2023-09-29T22:46:45.021 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663e97cd95c7bb05-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.022 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.022 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.022 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.022 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.022 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.022 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.022 DEBUG:boto:Data: 2023-09-29T22:46:45.023 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.023 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.023 DEBUG:boto:Params: {} 2023-09-29T22:46:45.023 DEBUG:boto:Token: None 2023-09-29T22:46:45.023 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eE1jikg+tJfMdbGjIHGuGLosrWQ= 2023-09-29T22:46:45.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eE1jikg+tJfMdbGjIHGuGLosrWQ=', 'Content-Length': '0'} 2023-09-29T22:46:45.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae0a3f470d5a5f3b-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.027 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.027 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.027 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.027 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.027 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.027 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.028 DEBUG:boto:Data: 2023-09-29T22:46:45.028 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.028 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.028 DEBUG:boto:Params: {} 2023-09-29T22:46:45.028 DEBUG:boto:Token: None 2023-09-29T22:46:45.028 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eE1jikg+tJfMdbGjIHGuGLosrWQ= 2023-09-29T22:46:45.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eE1jikg+tJfMdbGjIHGuGLosrWQ=', 'Content-Length': '0'} 2023-09-29T22:46:45.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053e9cc2d6511aa52-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:46:45.032 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.032 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.032 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.032 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.032 DEBUG:boto:Method: GET 2023-09-29T22:46:45.032 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.032 DEBUG:boto:Data: 2023-09-29T22:46:45.032 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.033 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.033 DEBUG:boto:Params: {} 2023-09-29T22:46:45.033 DEBUG:boto:Token: None 2023-09-29T22:46:45.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.033 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E= 2023-09-29T22:46:45.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E=', 'Content-Length': '0'} 2023-09-29T22:46:45.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003455fb582079686a-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.078 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.079 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.079 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.079 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.079 DEBUG:boto:Method: GET 2023-09-29T22:46:45.079 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.079 DEBUG:boto:Data: 2023-09-29T22:46:45.079 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.079 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.080 DEBUG:boto:Params: {} 2023-09-29T22:46:45.080 DEBUG:boto:Token: None 2023-09-29T22:46:45.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:46:45.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E= 2023-09-29T22:46:45.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS0t4eOOU+/02TFfwkijLe/Vj6E=', 'Content-Length': '0'} 2023-09-29T22:46:45.084 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005a11d62129bebc0-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.084 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.084 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.084 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.084 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.085 DEBUG:boto:Method: GET 2023-09-29T22:46:45.085 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.085 DEBUG:boto:Data: 2023-09-29T22:46:45.085 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.085 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.086 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.086 DEBUG:boto:Params: {} 2023-09-29T22:46:45.086 DEBUG:boto:Token: None 2023-09-29T22:46:45.086 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.086 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.086 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074b4dfaecac0f58a-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.091 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.091 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.091 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.092 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.092 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.092 DEBUG:boto:Method: GET 2023-09-29T22:46:45.092 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.092 DEBUG:boto:Data: 2023-09-29T22:46:45.092 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.092 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.092 DEBUG:boto:Params: {} 2023-09-29T22:46:45.093 DEBUG:boto:Token: None 2023-09-29T22:46:45.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.100 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c23e864da4cfd2a2-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.100 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.101 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.101 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.101 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.101 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:46:45.101 DEBUG:boto:Data: 2023-09-29T22:46:45.101 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.102 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.102 DEBUG:boto:Params: {} 2023-09-29T22:46:45.102 DEBUG:boto:Token: None 2023-09-29T22:46:45.102 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38 2023-09-29T22:46:45.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+rE9Z8VzD3c7xIkpkg/Zp+VYDlk= 2023-09-29T22:46:45.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+rE9Z8VzD3c7xIkpkg/Zp+VYDlk=', 'Content-Length': '0'} 2023-09-29T22:46:45.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be737d282670b7c0-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.104 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.104 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.105 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.105 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:46:45.105 DEBUG:boto:Data: 2023-09-29T22:46:45.105 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.105 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.105 DEBUG:boto:Params: {} 2023-09-29T22:46:45.105 DEBUG:boto:Token: None 2023-09-29T22:46:45.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38 2023-09-29T22:46:45.106 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+rE9Z8VzD3c7xIkpkg/Zp+VYDlk= 2023-09-29T22:46:45.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+rE9Z8VzD3c7xIkpkg/Zp+VYDlk=', 'Content-Length': '0'} 2023-09-29T22:46:45.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009059f40fc1e96b1-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:46:45.108 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.108 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.108 DEBUG:boto:Method: GET 2023-09-29T22:46:45.108 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:46:45.109 DEBUG:boto:Data: 2023-09-29T22:46:45.109 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.109 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.109 DEBUG:boto:Params: {} 2023-09-29T22:46:45.109 DEBUG:boto:Token: None 2023-09-29T22:46:45.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38 2023-09-29T22:46:45.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7oqPvYvYrJZBGaklV43HYHsowNg= 2023-09-29T22:46:45.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7oqPvYvYrJZBGaklV43HYHsowNg=', 'Content-Length': '0'} 2023-09-29T22:46:45.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e9e1819c009ad2a-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.112 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:46:45.112 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:46:45.112 DEBUG:boto:Method: GET 2023-09-29T22:46:45.112 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:46:45.113 DEBUG:boto:Data: 2023-09-29T22:46:45.113 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.113 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.113 DEBUG:boto:Params: {} 2023-09-29T22:46:45.113 DEBUG:boto:Token: None 2023-09-29T22:46:45.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj38 2023-09-29T22:46:45.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7oqPvYvYrJZBGaklV43HYHsowNg= 2023-09-29T22:46:45.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7oqPvYvYrJZBGaklV43HYHsowNg=', 'Content-Length': '0'} 2023-09-29T22:46:45.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dd773ee409ca9d4-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.162 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.163 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.163 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.163 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.163 DEBUG:boto:Method: GET 2023-09-29T22:46:45.163 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.163 DEBUG:boto:Data: 2023-09-29T22:46:45.163 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.163 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.163 DEBUG:boto:Params: {} 2023-09-29T22:46:45.164 DEBUG:boto:Token: None 2023-09-29T22:46:45.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.169 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000378f30bed556907f-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.169 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.170 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.170 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.170 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.170 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.170 DEBUG:boto:Method: GET 2023-09-29T22:46:45.170 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:46:45.171 DEBUG:boto:Data: 2023-09-29T22:46:45.171 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.171 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.171 DEBUG:boto:Params: {} 2023-09-29T22:46:45.171 DEBUG:boto:Token: None 2023-09-29T22:46:45.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.171 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000057a8bb4bae9f687b-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.178 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:46:45.179 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.179 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.179 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.179 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.179 DEBUG:boto:Method: GET 2023-09-29T22:46:45.179 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.179 DEBUG:boto:Data: 2023-09-29T22:46:45.179 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.180 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.180 DEBUG:boto:Params: {} 2023-09-29T22:46:45.180 DEBUG:boto:Token: None 2023-09-29T22:46:45.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4= 2023-09-29T22:46:45.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4=', 'Content-Length': '0'} 2023-09-29T22:46:45.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000869ecc7d2c8d68c2-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.226 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.227 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.227 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.227 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.227 DEBUG:boto:Method: GET 2023-09-29T22:46:45.227 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.227 DEBUG:boto:Data: 2023-09-29T22:46:45.227 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.227 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.228 DEBUG:boto:Params: {} 2023-09-29T22:46:45.228 DEBUG:boto:Token: None 2023-09-29T22:46:45.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.228 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4= 2023-09-29T22:46:45.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4=', 'Content-Length': '0'} 2023-09-29T22:46:45.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5a44d20f4f9ec3a-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.231 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.232 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.232 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.232 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.232 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.232 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.232 DEBUG:boto:Data: 2023-09-29T22:46:45.232 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.232 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.232 DEBUG:boto:Params: {} 2023-09-29T22:46:45.233 DEBUG:boto:Token: None 2023-09-29T22:46:45.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tn+3Td6kkU8Qh/jRLGyYnQfDkHc= 2023-09-29T22:46:45.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tn+3Td6kkU8Qh/jRLGyYnQfDkHc=', 'Content-Length': '0'} 2023-09-29T22:46:45.236 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094b68fdfe8679ad5-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.236 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.236 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.236 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.236 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.236 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.237 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.237 DEBUG:boto:Data: 2023-09-29T22:46:45.237 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.237 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.237 DEBUG:boto:Params: {} 2023-09-29T22:46:45.237 DEBUG:boto:Token: None 2023-09-29T22:46:45.237 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.237 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tn+3Td6kkU8Qh/jRLGyYnQfDkHc= 2023-09-29T22:46:45.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tn+3Td6kkU8Qh/jRLGyYnQfDkHc=', 'Content-Length': '0'} 2023-09-29T22:46:45.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079d7f3d3ba33bc81-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:46:45.240 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.241 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.241 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.241 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.241 DEBUG:boto:Method: GET 2023-09-29T22:46:45.241 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.241 DEBUG:boto:Data: 2023-09-29T22:46:45.241 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.241 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.241 DEBUG:boto:Params: {} 2023-09-29T22:46:45.242 DEBUG:boto:Token: None 2023-09-29T22:46:45.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4= 2023-09-29T22:46:45.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4=', 'Content-Length': '0'} 2023-09-29T22:46:45.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d6b7121dc170e6-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.245 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.245 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.245 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.245 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.245 DEBUG:boto:Method: GET 2023-09-29T22:46:45.245 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.246 DEBUG:boto:Data: 2023-09-29T22:46:45.246 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.246 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.246 DEBUG:boto:Params: {} 2023-09-29T22:46:45.246 DEBUG:boto:Token: None 2023-09-29T22:46:45.247 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:46:45.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4= 2023-09-29T22:46:45.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fxt1e+RQnRNxm+xJNkLYwJiCDV4=', 'Content-Length': '0'} 2023-09-29T22:46:45.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8da9458a91ae578-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.290 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.291 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.291 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.291 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.291 DEBUG:boto:Method: GET 2023-09-29T22:46:45.291 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.291 DEBUG:boto:Data: 2023-09-29T22:46:45.291 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.292 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.292 DEBUG:boto:Params: {} 2023-09-29T22:46:45.292 DEBUG:boto:Token: None 2023-09-29T22:46:45.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c154e14551d2c3bd-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.298 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.299 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.299 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.299 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.299 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.299 DEBUG:boto:Method: GET 2023-09-29T22:46:45.299 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.299 DEBUG:boto:Data: 2023-09-29T22:46:45.300 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.300 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.300 DEBUG:boto:Params: {} 2023-09-29T22:46:45.300 DEBUG:boto:Token: None 2023-09-29T22:46:45.300 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.307 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006c904ec6ba08107a-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.307 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.307 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.308 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.308 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.308 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:46:45.308 DEBUG:boto:Data: 2023-09-29T22:46:45.308 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.308 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.309 DEBUG:boto:Params: {} 2023-09-29T22:46:45.309 DEBUG:boto:Token: None 2023-09-29T22:46:45.309 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44 2023-09-29T22:46:45.309 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aB2tlH4zq4rpNPlDTLkeGvX6tEM= 2023-09-29T22:46:45.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aB2tlH4zq4rpNPlDTLkeGvX6tEM=', 'Content-Length': '0'} 2023-09-29T22:46:45.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000572e4f63e6047eed-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.312 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.312 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.312 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.312 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:46:45.312 DEBUG:boto:Data: 2023-09-29T22:46:45.313 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.313 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.313 DEBUG:boto:Params: {} 2023-09-29T22:46:45.313 DEBUG:boto:Token: None 2023-09-29T22:46:45.313 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44 2023-09-29T22:46:45.314 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aB2tlH4zq4rpNPlDTLkeGvX6tEM= 2023-09-29T22:46:45.314 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aB2tlH4zq4rpNPlDTLkeGvX6tEM=', 'Content-Length': '0'} 2023-09-29T22:46:45.316 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a7140a1d8d15aaa-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:46:45.316 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.316 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.317 DEBUG:boto:Method: GET 2023-09-29T22:46:45.317 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:46:45.317 DEBUG:boto:Data: 2023-09-29T22:46:45.317 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.317 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.317 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.317 DEBUG:boto:Params: {} 2023-09-29T22:46:45.318 DEBUG:boto:Token: None 2023-09-29T22:46:45.318 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44 2023-09-29T22:46:45.318 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4KyJRiT1GZWfU17ZWjr0tqN9/go= 2023-09-29T22:46:45.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4KyJRiT1GZWfU17ZWjr0tqN9/go=', 'Content-Length': '0'} 2023-09-29T22:46:45.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6279eaecda9e18f-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.366 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:46:45.367 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:46:45.367 DEBUG:boto:Method: GET 2023-09-29T22:46:45.367 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:46:45.367 DEBUG:boto:Data: 2023-09-29T22:46:45.367 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.367 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.367 DEBUG:boto:Params: {} 2023-09-29T22:46:45.368 DEBUG:boto:Token: None 2023-09-29T22:46:45.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj44 2023-09-29T22:46:45.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4KyJRiT1GZWfU17ZWjr0tqN9/go= 2023-09-29T22:46:45.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4KyJRiT1GZWfU17ZWjr0tqN9/go=', 'Content-Length': '0'} 2023-09-29T22:46:45.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b346b6d28201556d-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.371 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.371 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.371 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.371 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.371 DEBUG:boto:Method: GET 2023-09-29T22:46:45.372 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.372 DEBUG:boto:Data: 2023-09-29T22:46:45.372 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.372 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.372 DEBUG:boto:Params: {} 2023-09-29T22:46:45.372 DEBUG:boto:Token: None 2023-09-29T22:46:45.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c277c54580b99a4-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.377 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.378 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.378 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.378 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.378 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.379 DEBUG:boto:Method: GET 2023-09-29T22:46:45.379 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:46:45.379 DEBUG:boto:Data: 2023-09-29T22:46:45.379 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.379 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.379 DEBUG:boto:Params: {} 2023-09-29T22:46:45.379 DEBUG:boto:Token: None 2023-09-29T22:46:45.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.387 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036f536db5df4e99e-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.430 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:46:45.431 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.432 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.432 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.432 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.432 DEBUG:boto:Method: GET 2023-09-29T22:46:45.432 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.432 DEBUG:boto:Data: 2023-09-29T22:46:45.433 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.433 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.433 DEBUG:boto:Params: {} 2023-09-29T22:46:45.433 DEBUG:boto:Token: None 2023-09-29T22:46:45.433 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg= 2023-09-29T22:46:45.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg=', 'Content-Length': '0'} 2023-09-29T22:46:45.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006365a249bd6e33f1-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.437 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.437 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.437 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.437 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.437 DEBUG:boto:Method: GET 2023-09-29T22:46:45.437 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.437 DEBUG:boto:Data: 2023-09-29T22:46:45.438 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.438 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.438 DEBUG:boto:Params: {} 2023-09-29T22:46:45.438 DEBUG:boto:Token: None 2023-09-29T22:46:45.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg= 2023-09-29T22:46:45.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg=', 'Content-Length': '0'} 2023-09-29T22:46:45.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e149f0f7f5cb28c-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.441 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.442 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.442 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.442 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.442 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.442 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.442 DEBUG:boto:Data: 2023-09-29T22:46:45.442 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.443 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.443 DEBUG:boto:Params: {} 2023-09-29T22:46:45.443 DEBUG:boto:Token: None 2023-09-29T22:46:45.443 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fjy4olUI+CheOtvrpA8cfj+Pvtw= 2023-09-29T22:46:45.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fjy4olUI+CheOtvrpA8cfj+Pvtw=', 'Content-Length': '0'} 2023-09-29T22:46:45.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8ea54ab5d1306f0-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.446 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.446 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.446 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.446 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.447 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.447 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.447 DEBUG:boto:Data: 2023-09-29T22:46:45.447 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.447 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.447 DEBUG:boto:Params: {} 2023-09-29T22:46:45.447 DEBUG:boto:Token: None 2023-09-29T22:46:45.447 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fjy4olUI+CheOtvrpA8cfj+Pvtw= 2023-09-29T22:46:45.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fjy4olUI+CheOtvrpA8cfj+Pvtw=', 'Content-Length': '0'} 2023-09-29T22:46:45.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0e6218405af9282-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:46:45.451 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.451 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.451 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.451 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.451 DEBUG:boto:Method: GET 2023-09-29T22:46:45.451 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.451 DEBUG:boto:Data: 2023-09-29T22:46:45.451 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.452 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.452 DEBUG:boto:Params: {} 2023-09-29T22:46:45.452 DEBUG:boto:Token: None 2023-09-29T22:46:45.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg= 2023-09-29T22:46:45.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg=', 'Content-Length': '0'} 2023-09-29T22:46:45.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059f78d1baec580f9-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.455 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.455 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.455 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.456 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.456 DEBUG:boto:Method: GET 2023-09-29T22:46:45.456 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.456 DEBUG:boto:Data: 2023-09-29T22:46:45.456 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.456 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.456 DEBUG:boto:Params: {} 2023-09-29T22:46:45.456 DEBUG:boto:Token: None 2023-09-29T22:46:45.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:46:45.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg= 2023-09-29T22:46:45.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hYx8YrnUq7w0fIG9hWwYqHqyAhg=', 'Content-Length': '0'} 2023-09-29T22:46:45.460 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d132fe9b91b390-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.460 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.460 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.460 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.460 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.461 DEBUG:boto:Method: GET 2023-09-29T22:46:45.461 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.461 DEBUG:boto:Data: 2023-09-29T22:46:45.461 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.461 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.461 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.461 DEBUG:boto:Params: {} 2023-09-29T22:46:45.461 DEBUG:boto:Token: None 2023-09-29T22:46:45.462 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.462 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068500018972e43d2-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.467 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.467 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.467 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.467 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.468 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.468 DEBUG:boto:Method: GET 2023-09-29T22:46:45.468 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.468 DEBUG:boto:Data: 2023-09-29T22:46:45.468 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.468 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.468 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.468 DEBUG:boto:Params: {} 2023-09-29T22:46:45.468 DEBUG:boto:Token: None 2023-09-29T22:46:45.469 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.469 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.469 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.476 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a87599d05fb6e1c1-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.518 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.519 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.519 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.519 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.519 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:46:45.519 DEBUG:boto:Data: 2023-09-29T22:46:45.520 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.520 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.520 DEBUG:boto:Params: {} 2023-09-29T22:46:45.520 DEBUG:boto:Token: None 2023-09-29T22:46:45.520 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46 2023-09-29T22:46:45.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PuF8c1S9DoW5z17iO0ifcfM60lA= 2023-09-29T22:46:45.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PuF8c1S9DoW5z17iO0ifcfM60lA=', 'Content-Length': '0'} 2023-09-29T22:46:45.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b483d333a3316f6-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.524 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.524 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.524 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.524 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:46:45.524 DEBUG:boto:Data: 2023-09-29T22:46:45.524 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.525 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.525 DEBUG:boto:Params: {} 2023-09-29T22:46:45.525 DEBUG:boto:Token: None 2023-09-29T22:46:45.525 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46 2023-09-29T22:46:45.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PuF8c1S9DoW5z17iO0ifcfM60lA= 2023-09-29T22:46:45.525 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PuF8c1S9DoW5z17iO0ifcfM60lA=', 'Content-Length': '0'} 2023-09-29T22:46:45.528 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aaccfc48447f616-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:46:45.528 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.528 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.528 DEBUG:boto:Method: GET 2023-09-29T22:46:45.529 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:46:45.529 DEBUG:boto:Data: 2023-09-29T22:46:45.529 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.529 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.529 DEBUG:boto:Params: {} 2023-09-29T22:46:45.529 DEBUG:boto:Token: None 2023-09-29T22:46:45.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46 2023-09-29T22:46:45.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NO9sXZZe1cfZzAjLDkl7JMLuc1c= 2023-09-29T22:46:45.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NO9sXZZe1cfZzAjLDkl7JMLuc1c=', 'Content-Length': '0'} 2023-09-29T22:46:45.532 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cbb2f07772cbd60-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.574 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:46:45.575 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:46:45.575 DEBUG:boto:Method: GET 2023-09-29T22:46:45.575 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:46:45.575 DEBUG:boto:Data: 2023-09-29T22:46:45.575 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.575 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.575 DEBUG:boto:Params: {} 2023-09-29T22:46:45.576 DEBUG:boto:Token: None 2023-09-29T22:46:45.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj46 2023-09-29T22:46:45.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NO9sXZZe1cfZzAjLDkl7JMLuc1c= 2023-09-29T22:46:45.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NO9sXZZe1cfZzAjLDkl7JMLuc1c=', 'Content-Length': '0'} 2023-09-29T22:46:45.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098d6eb3da883223d-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.579 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.579 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.579 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.579 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.580 DEBUG:boto:Method: GET 2023-09-29T22:46:45.580 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.580 DEBUG:boto:Data: 2023-09-29T22:46:45.580 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.580 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.580 DEBUG:boto:Params: {} 2023-09-29T22:46:45.580 DEBUG:boto:Token: None 2023-09-29T22:46:45.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.585 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005f4b633e1f9513e9-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.585 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.586 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.586 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.586 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.586 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.586 DEBUG:boto:Method: GET 2023-09-29T22:46:45.587 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:46:45.587 DEBUG:boto:Data: 2023-09-29T22:46:45.587 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.587 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.587 DEBUG:boto:Params: {} 2023-09-29T22:46:45.587 DEBUG:boto:Token: None 2023-09-29T22:46:45.587 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e17a1d382a43d39a-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.594 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:46:45.595 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.595 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.595 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.595 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.595 DEBUG:boto:Method: GET 2023-09-29T22:46:45.595 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.596 DEBUG:boto:Data: 2023-09-29T22:46:45.596 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.596 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.596 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.596 DEBUG:boto:Params: {} 2023-09-29T22:46:45.596 DEBUG:boto:Token: None 2023-09-29T22:46:45.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI= 2023-09-29T22:46:45.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI=', 'Content-Length': '0'} 2023-09-29T22:46:45.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097531d94c62c32d5-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.599 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.599 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.599 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.599 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.599 DEBUG:boto:Method: GET 2023-09-29T22:46:45.600 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.600 DEBUG:boto:Data: 2023-09-29T22:46:45.600 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.600 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.600 DEBUG:boto:Params: {} 2023-09-29T22:46:45.600 DEBUG:boto:Token: None 2023-09-29T22:46:45.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI= 2023-09-29T22:46:45.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI=', 'Content-Length': '0'} 2023-09-29T22:46:45.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005682d0e172b52504-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.650 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.651 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.651 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.651 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.651 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.651 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.651 DEBUG:boto:Data: 2023-09-29T22:46:45.651 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.651 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.652 DEBUG:boto:Params: {} 2023-09-29T22:46:45.652 DEBUG:boto:Token: None 2023-09-29T22:46:45.652 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZoOJupNvp+tuVWWNmSevmb6hONE= 2023-09-29T22:46:45.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZoOJupNvp+tuVWWNmSevmb6hONE=', 'Content-Length': '0'} 2023-09-29T22:46:45.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8479d9c0a5dc104-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.655 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.655 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.655 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.655 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.655 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.656 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.656 DEBUG:boto:Data: 2023-09-29T22:46:45.656 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.656 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.656 DEBUG:boto:Params: {} 2023-09-29T22:46:45.656 DEBUG:boto:Token: None 2023-09-29T22:46:45.656 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZoOJupNvp+tuVWWNmSevmb6hONE= 2023-09-29T22:46:45.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZoOJupNvp+tuVWWNmSevmb6hONE=', 'Content-Length': '0'} 2023-09-29T22:46:45.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa2b39d86f1c54d3-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:46:45.660 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.660 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.660 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.660 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.660 DEBUG:boto:Method: GET 2023-09-29T22:46:45.660 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.660 DEBUG:boto:Data: 2023-09-29T22:46:45.660 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.661 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.661 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.661 DEBUG:boto:Params: {} 2023-09-29T22:46:45.661 DEBUG:boto:Token: None 2023-09-29T22:46:45.661 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI= 2023-09-29T22:46:45.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI=', 'Content-Length': '0'} 2023-09-29T22:46:45.664 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004477b8b48e210e80-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.710 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.711 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.711 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.711 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.711 DEBUG:boto:Method: GET 2023-09-29T22:46:45.711 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.711 DEBUG:boto:Data: 2023-09-29T22:46:45.711 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.712 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.712 DEBUG:boto:Params: {} 2023-09-29T22:46:45.712 DEBUG:boto:Token: None 2023-09-29T22:46:45.712 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:46:45.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI= 2023-09-29T22:46:45.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I0dcoUaxG2Auyh+TCctjCH51oiI=', 'Content-Length': '0'} 2023-09-29T22:46:45.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009da087acb5d72c3c-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.716 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.716 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.717 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.717 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.717 DEBUG:boto:Method: GET 2023-09-29T22:46:45.717 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.717 DEBUG:boto:Data: 2023-09-29T22:46:45.717 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.717 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.717 DEBUG:boto:Params: {} 2023-09-29T22:46:45.718 DEBUG:boto:Token: None 2023-09-29T22:46:45.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.722 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000683d77c1960e5599-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.723 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.723 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.723 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.723 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.723 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.724 DEBUG:boto:Method: GET 2023-09-29T22:46:45.724 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.724 DEBUG:boto:Data: 2023-09-29T22:46:45.724 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.724 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.724 DEBUG:boto:Params: {} 2023-09-29T22:46:45.724 DEBUG:boto:Token: None 2023-09-29T22:46:45.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d08c7e02ad1b1aa0-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.732 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.732 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.732 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.733 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.733 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:46:45.733 DEBUG:boto:Data: 2023-09-29T22:46:45.733 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.733 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.733 DEBUG:boto:Params: {} 2023-09-29T22:46:45.733 DEBUG:boto:Token: None 2023-09-29T22:46:45.734 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51 2023-09-29T22:46:45.734 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C7XjTQFBaL//CmWlmlOpK0Q2yME= 2023-09-29T22:46:45.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C7XjTQFBaL//CmWlmlOpK0Q2yME=', 'Content-Length': '0'} 2023-09-29T22:46:45.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076012277b0f91ff9-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.736 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.736 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.737 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.737 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:46:45.737 DEBUG:boto:Data: 2023-09-29T22:46:45.737 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.737 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.737 DEBUG:boto:Params: {} 2023-09-29T22:46:45.737 DEBUG:boto:Token: None 2023-09-29T22:46:45.738 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51 2023-09-29T22:46:45.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C7XjTQFBaL//CmWlmlOpK0Q2yME= 2023-09-29T22:46:45.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C7XjTQFBaL//CmWlmlOpK0Q2yME=', 'Content-Length': '0'} 2023-09-29T22:46:45.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029b60cca372bbe78-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:46:45.740 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.741 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.741 DEBUG:boto:Method: GET 2023-09-29T22:46:45.741 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:46:45.741 DEBUG:boto:Data: 2023-09-29T22:46:45.741 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.741 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.741 DEBUG:boto:Params: {} 2023-09-29T22:46:45.742 DEBUG:boto:Token: None 2023-09-29T22:46:45.742 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51 2023-09-29T22:46:45.742 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0QJJvj0595LH8mC+eYIeUyjAGJE= 2023-09-29T22:46:45.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0QJJvj0595LH8mC+eYIeUyjAGJE=', 'Content-Length': '0'} 2023-09-29T22:46:45.744 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016199b3afdd6685f-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.745 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:46:45.745 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:46:45.745 DEBUG:boto:Method: GET 2023-09-29T22:46:45.745 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:46:45.745 DEBUG:boto:Data: 2023-09-29T22:46:45.745 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.746 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.746 DEBUG:boto:Params: {} 2023-09-29T22:46:45.746 DEBUG:boto:Token: None 2023-09-29T22:46:45.746 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj51 2023-09-29T22:46:45.746 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0QJJvj0595LH8mC+eYIeUyjAGJE= 2023-09-29T22:46:45.746 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0QJJvj0595LH8mC+eYIeUyjAGJE=', 'Content-Length': '0'} 2023-09-29T22:46:45.748 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000feea43ff10873df8-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.790 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.791 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.791 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.791 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.791 DEBUG:boto:Method: GET 2023-09-29T22:46:45.791 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.791 DEBUG:boto:Data: 2023-09-29T22:46:45.791 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.791 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.792 DEBUG:boto:Params: {} 2023-09-29T22:46:45.792 DEBUG:boto:Token: None 2023-09-29T22:46:45.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bbf071865db4d58-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.797 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.797 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.798 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.798 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.798 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.798 DEBUG:boto:Method: GET 2023-09-29T22:46:45.798 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:46:45.798 DEBUG:boto:Data: 2023-09-29T22:46:45.798 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.798 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.799 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.799 DEBUG:boto:Params: {} 2023-09-29T22:46:45.799 DEBUG:boto:Token: None 2023-09-29T22:46:45.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.799 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f12475c95cf0278-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.807 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:46:45.807 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.808 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.808 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.808 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.808 DEBUG:boto:Method: GET 2023-09-29T22:46:45.808 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.808 DEBUG:boto:Data: 2023-09-29T22:46:45.808 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.808 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.808 DEBUG:boto:Params: {} 2023-09-29T22:46:45.809 DEBUG:boto:Token: None 2023-09-29T22:46:45.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM= 2023-09-29T22:46:45.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM=', 'Content-Length': '0'} 2023-09-29T22:46:45.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006d511518173feb7-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.858 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.859 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.859 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.859 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.859 DEBUG:boto:Method: GET 2023-09-29T22:46:45.859 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.859 DEBUG:boto:Data: 2023-09-29T22:46:45.859 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.859 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.859 DEBUG:boto:Params: {} 2023-09-29T22:46:45.860 DEBUG:boto:Token: None 2023-09-29T22:46:45.860 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM= 2023-09-29T22:46:45.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM=', 'Content-Length': '0'} 2023-09-29T22:46:45.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000158f631e1b2b586f-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.863 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.864 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.864 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.864 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.864 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.864 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.864 DEBUG:boto:Data: 2023-09-29T22:46:45.864 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.864 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.865 DEBUG:boto:Params: {} 2023-09-29T22:46:45.865 DEBUG:boto:Token: None 2023-09-29T22:46:45.865 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iYuGSZZGa/yv1xVV1emyaSb4L6c= 2023-09-29T22:46:45.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iYuGSZZGa/yv1xVV1emyaSb4L6c=', 'Content-Length': '0'} 2023-09-29T22:46:45.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007736d36504078cb8-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.868 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.868 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.868 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.868 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.868 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.868 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.868 DEBUG:boto:Data: 2023-09-29T22:46:45.869 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.869 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.869 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.869 DEBUG:boto:Params: {} 2023-09-29T22:46:45.869 DEBUG:boto:Token: None 2023-09-29T22:46:45.869 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.870 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iYuGSZZGa/yv1xVV1emyaSb4L6c= 2023-09-29T22:46:45.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iYuGSZZGa/yv1xVV1emyaSb4L6c=', 'Content-Length': '0'} 2023-09-29T22:46:45.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049fc875a9c43c225-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:46:45.873 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.873 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.873 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.874 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.874 DEBUG:boto:Method: GET 2023-09-29T22:46:45.874 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.874 DEBUG:boto:Data: 2023-09-29T22:46:45.874 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.874 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.874 DEBUG:boto:Params: {} 2023-09-29T22:46:45.875 DEBUG:boto:Token: None 2023-09-29T22:46:45.875 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.875 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM= 2023-09-29T22:46:45.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM=', 'Content-Length': '0'} 2023-09-29T22:46:45.878 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8d13b867df4d9bc-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.878 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.879 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.879 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.879 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.879 DEBUG:boto:Method: GET 2023-09-29T22:46:45.879 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.879 DEBUG:boto:Data: 2023-09-29T22:46:45.880 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.880 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.880 DEBUG:boto:Params: {} 2023-09-29T22:46:45.880 DEBUG:boto:Token: None 2023-09-29T22:46:45.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:46:45.880 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM= 2023-09-29T22:46:45.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HhApiQHva+FkgDrQIkwnAIpIYIM=', 'Content-Length': '0'} 2023-09-29T22:46:45.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000022ad709cc6c3499a-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.884 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.884 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.884 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:45.884 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:45.885 DEBUG:boto:Method: GET 2023-09-29T22:46:45.885 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:46:45.885 DEBUG:boto:Data: 2023-09-29T22:46:45.885 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.885 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.886 DEBUG:boto:Params: {} 2023-09-29T22:46:45.886 DEBUG:boto:Token: None 2023-09-29T22:46:45.886 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.891 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aae9a895f8d239dc-00651753d5-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.891 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.891 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:45.891 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:45.892 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:45.892 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:45.892 DEBUG:boto:Method: GET 2023-09-29T22:46:45.892 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:46:45.892 DEBUG:boto:Data: 2023-09-29T22:46:45.892 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.892 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.892 DEBUG:boto:Params: {} 2023-09-29T22:46:45.892 DEBUG:boto:Token: None 2023-09-29T22:46:45.893 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/ 2023-09-29T22:46:45.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8= 2023-09-29T22:46:45.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+qVAlvhoc6j6aiMt24L3x9UnaD8=', 'Content-Length': '0'} 2023-09-29T22:46:45.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba91215cb2090871-00651753d5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.946 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:45.947 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.947 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.947 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.947 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:46:45.947 DEBUG:boto:Data: 2023-09-29T22:46:45.947 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.948 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.948 DEBUG:boto:Params: {} 2023-09-29T22:46:45.948 DEBUG:boto:Token: None 2023-09-29T22:46:45.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55 2023-09-29T22:46:45.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:es2NMj6+jPAamDRaVkDqHPCZROM= 2023-09-29T22:46:45.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:es2NMj6+jPAamDRaVkDqHPCZROM=', 'Content-Length': '0'} 2023-09-29T22:46:45.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a82e37f64401c8ce-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.952 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.952 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.952 DEBUG:boto:Method: HEAD 2023-09-29T22:46:45.952 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:46:45.952 DEBUG:boto:Data: 2023-09-29T22:46:45.952 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:45.952 DEBUG:boto:Port: 8000 2023-09-29T22:46:45.953 DEBUG:boto:Params: {} 2023-09-29T22:46:45.953 DEBUG:boto:Token: None 2023-09-29T22:46:45.953 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55 2023-09-29T22:46:45.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:es2NMj6+jPAamDRaVkDqHPCZROM= 2023-09-29T22:46:45.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:es2NMj6+jPAamDRaVkDqHPCZROM=', 'Content-Length': '0'} 2023-09-29T22:46:45.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd5ea0e1a9b0f6c3-00651753d5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:45.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:46:45.956 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:45.957 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:45.957 DEBUG:boto:Method: GET 2023-09-29T22:46:45.957 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:46:45.957 DEBUG:boto:Data: 2023-09-29T22:46:45.957 DEBUG:boto:Headers: {} 2023-09-29T22:46:45.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:45.957 DEBUG:boto:Port: 8001 2023-09-29T22:46:45.958 DEBUG:boto:Params: {} 2023-09-29T22:46:45.958 DEBUG:boto:Token: None 2023-09-29T22:46:45.958 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:45 GMT /hcnphz-59/obj55 2023-09-29T22:46:45.958 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cA99sUEpMELNqs9uFU7PVvj6HQw= 2023-09-29T22:46:45.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:45 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cA99sUEpMELNqs9uFU7PVvj6HQw=', 'Content-Length': '0'} 2023-09-29T22:46:45.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008100c5db2f7a922b-00651753d5-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:45 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.006 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:46:46.007 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:46:46.007 DEBUG:boto:Method: GET 2023-09-29T22:46:46.007 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:46:46.007 DEBUG:boto:Data: 2023-09-29T22:46:46.007 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.007 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.007 DEBUG:boto:Params: {} 2023-09-29T22:46:46.007 DEBUG:boto:Token: None 2023-09-29T22:46:46.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj55 2023-09-29T22:46:46.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AUFYL3Y5Bhp0HhgrAK1xjsIYO54= 2023-09-29T22:46:46.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AUFYL3Y5Bhp0HhgrAK1xjsIYO54=', 'Content-Length': '0'} 2023-09-29T22:46:46.011 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008500f676706835b9-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.012 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.012 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.012 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:46.012 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:46.012 DEBUG:boto:Method: GET 2023-09-29T22:46:46.012 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:46:46.012 DEBUG:boto:Data: 2023-09-29T22:46:46.013 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.013 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.013 DEBUG:boto:Params: {} 2023-09-29T22:46:46.013 DEBUG:boto:Token: None 2023-09-29T22:46:46.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.018 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000949b12d3cf785141-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.018 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.019 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.019 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.019 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:46.019 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:46:46.019 DEBUG:boto:Method: GET 2023-09-29T22:46:46.020 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:46:46.020 DEBUG:boto:Data: 2023-09-29T22:46:46.020 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.020 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.020 DEBUG:boto:Params: {} 2023-09-29T22:46:46.020 DEBUG:boto:Token: None 2023-09-29T22:46:46.021 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.028 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008de5599de12a1ec3-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.028 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:46:46.029 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.029 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.029 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.029 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.030 DEBUG:boto:Method: GET 2023-09-29T22:46:46.030 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.030 DEBUG:boto:Data: 2023-09-29T22:46:46.030 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.030 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.030 DEBUG:boto:Params: {} 2023-09-29T22:46:46.030 DEBUG:boto:Token: None 2023-09-29T22:46:46.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4= 2023-09-29T22:46:46.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4=', 'Content-Length': '0'} 2023-09-29T22:46:46.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000794e5725a73f67a8-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.034 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.034 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.034 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.034 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.034 DEBUG:boto:Method: GET 2023-09-29T22:46:46.034 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.034 DEBUG:boto:Data: 2023-09-29T22:46:46.035 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.035 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.035 DEBUG:boto:Params: {} 2023-09-29T22:46:46.035 DEBUG:boto:Token: None 2023-09-29T22:46:46.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4= 2023-09-29T22:46:46.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4=', 'Content-Length': '0'} 2023-09-29T22:46:46.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007595dd412d8c3a4a-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.078 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.078 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.078 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.079 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.079 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.079 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.079 DEBUG:boto:Data: 2023-09-29T22:46:46.079 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.079 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.079 DEBUG:boto:Params: {} 2023-09-29T22:46:46.080 DEBUG:boto:Token: None 2023-09-29T22:46:46.080 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9qzXyXzZQCKkUfM8wstZ3OvVdsQ= 2023-09-29T22:46:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9qzXyXzZQCKkUfM8wstZ3OvVdsQ=', 'Content-Length': '0'} 2023-09-29T22:46:46.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fefeaa5106c8c7c1-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.083 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.083 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.083 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.083 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.083 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.084 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.084 DEBUG:boto:Data: 2023-09-29T22:46:46.084 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.084 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.084 DEBUG:boto:Params: {} 2023-09-29T22:46:46.084 DEBUG:boto:Token: None 2023-09-29T22:46:46.084 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9qzXyXzZQCKkUfM8wstZ3OvVdsQ= 2023-09-29T22:46:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9qzXyXzZQCKkUfM8wstZ3OvVdsQ=', 'Content-Length': '0'} 2023-09-29T22:46:46.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c76a811324f5178-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:46:46.087 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.088 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.088 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.088 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.088 DEBUG:boto:Method: GET 2023-09-29T22:46:46.088 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.088 DEBUG:boto:Data: 2023-09-29T22:46:46.088 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.088 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.089 DEBUG:boto:Params: {} 2023-09-29T22:46:46.089 DEBUG:boto:Token: None 2023-09-29T22:46:46.089 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4= 2023-09-29T22:46:46.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4=', 'Content-Length': '0'} 2023-09-29T22:46:46.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067e416c23f958ce2-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.134 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.134 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.135 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.135 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.135 DEBUG:boto:Method: GET 2023-09-29T22:46:46.135 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.135 DEBUG:boto:Data: 2023-09-29T22:46:46.135 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.135 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.135 DEBUG:boto:Params: {} 2023-09-29T22:46:46.136 DEBUG:boto:Token: None 2023-09-29T22:46:46.136 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:46:46.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4= 2023-09-29T22:46:46.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7ze0PV0JgpEyMJGFEk5ANvnmDb4=', 'Content-Length': '0'} 2023-09-29T22:46:46.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000889f6fb11441ff6-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.142 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.142 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.142 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.142 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.143 DEBUG:boto:Method: GET 2023-09-29T22:46:46.143 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.143 DEBUG:boto:Data: 2023-09-29T22:46:46.143 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.143 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.143 DEBUG:boto:Params: {} 2023-09-29T22:46:46.143 DEBUG:boto:Token: None 2023-09-29T22:46:46.143 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.150 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003048b0fda7959699-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.150 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.150 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.150 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.150 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.151 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.151 DEBUG:boto:Method: GET 2023-09-29T22:46:46.151 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.151 DEBUG:boto:Data: 2023-09-29T22:46:46.151 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.151 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.151 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.151 DEBUG:boto:Params: {} 2023-09-29T22:46:46.151 DEBUG:boto:Token: None 2023-09-29T22:46:46.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.159 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089027b69dfe142b8-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.159 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.159 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.159 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.160 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.160 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:46:46.160 DEBUG:boto:Data: 2023-09-29T22:46:46.160 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.160 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.161 DEBUG:boto:Params: {} 2023-09-29T22:46:46.161 DEBUG:boto:Token: None 2023-09-29T22:46:46.161 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57 2023-09-29T22:46:46.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kpbN9wWFwLkMZp8Nb/kigS3zlns= 2023-09-29T22:46:46.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kpbN9wWFwLkMZp8Nb/kigS3zlns=', 'Content-Length': '0'} 2023-09-29T22:46:46.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd859d9fc84b91e7-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.163 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.163 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.164 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.164 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:46:46.164 DEBUG:boto:Data: 2023-09-29T22:46:46.164 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.164 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.164 DEBUG:boto:Params: {} 2023-09-29T22:46:46.164 DEBUG:boto:Token: None 2023-09-29T22:46:46.165 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57 2023-09-29T22:46:46.165 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kpbN9wWFwLkMZp8Nb/kigS3zlns= 2023-09-29T22:46:46.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kpbN9wWFwLkMZp8Nb/kigS3zlns=', 'Content-Length': '0'} 2023-09-29T22:46:46.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c18d572dc67ad2ac-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:46:46.167 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.168 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.168 DEBUG:boto:Method: GET 2023-09-29T22:46:46.168 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:46:46.168 DEBUG:boto:Data: 2023-09-29T22:46:46.168 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.168 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.168 DEBUG:boto:Params: {} 2023-09-29T22:46:46.169 DEBUG:boto:Token: None 2023-09-29T22:46:46.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57 2023-09-29T22:46:46.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dL4XT7pUbPZgZisRmRq054t6gB4= 2023-09-29T22:46:46.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dL4XT7pUbPZgZisRmRq054t6gB4=', 'Content-Length': '0'} 2023-09-29T22:46:46.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000816a0d6e2d897c15-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.214 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:46:46.215 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:46:46.215 DEBUG:boto:Method: GET 2023-09-29T22:46:46.215 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:46:46.215 DEBUG:boto:Data: 2023-09-29T22:46:46.215 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.215 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.216 DEBUG:boto:Params: {} 2023-09-29T22:46:46.216 DEBUG:boto:Token: None 2023-09-29T22:46:46.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj57 2023-09-29T22:46:46.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dL4XT7pUbPZgZisRmRq054t6gB4= 2023-09-29T22:46:46.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dL4XT7pUbPZgZisRmRq054t6gB4=', 'Content-Length': '0'} 2023-09-29T22:46:46.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f79deecae4dcb7ce-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.220 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.220 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.220 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.220 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.220 DEBUG:boto:Method: GET 2023-09-29T22:46:46.221 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.221 DEBUG:boto:Data: 2023-09-29T22:46:46.221 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.221 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.221 DEBUG:boto:Params: {} 2023-09-29T22:46:46.221 DEBUG:boto:Token: None 2023-09-29T22:46:46.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.222 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008eaecedfb2668900-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.227 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.227 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.227 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.227 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.227 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.228 DEBUG:boto:Method: GET 2023-09-29T22:46:46.228 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:46:46.228 DEBUG:boto:Data: 2023-09-29T22:46:46.228 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.228 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.228 DEBUG:boto:Params: {} 2023-09-29T22:46:46.229 DEBUG:boto:Token: None 2023-09-29T22:46:46.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.236 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000efd179368397b561-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.236 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:46:46.237 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.237 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.237 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.237 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.237 DEBUG:boto:Method: GET 2023-09-29T22:46:46.238 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.238 DEBUG:boto:Data: 2023-09-29T22:46:46.238 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.238 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.238 DEBUG:boto:Params: {} 2023-09-29T22:46:46.238 DEBUG:boto:Token: None 2023-09-29T22:46:46.239 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM= 2023-09-29T22:46:46.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM=', 'Content-Length': '0'} 2023-09-29T22:46:46.241 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a2483bf5df9c261-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.241 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.242 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.242 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.242 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.242 DEBUG:boto:Method: GET 2023-09-29T22:46:46.242 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.242 DEBUG:boto:Data: 2023-09-29T22:46:46.242 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.243 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.243 DEBUG:boto:Params: {} 2023-09-29T22:46:46.243 DEBUG:boto:Token: None 2023-09-29T22:46:46.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM= 2023-09-29T22:46:46.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM=', 'Content-Length': '0'} 2023-09-29T22:46:46.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000beeab288493645fa-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.246 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.246 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.246 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.246 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.246 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.246 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.246 DEBUG:boto:Data: 2023-09-29T22:46:46.247 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.247 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.247 DEBUG:boto:Params: {} 2023-09-29T22:46:46.247 DEBUG:boto:Token: None 2023-09-29T22:46:46.247 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hyILk6w54WObgLAOaqo8Nm6ih/E= 2023-09-29T22:46:46.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyILk6w54WObgLAOaqo8Nm6ih/E=', 'Content-Length': '0'} 2023-09-29T22:46:46.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb2e361c610f9aa3-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.250 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.250 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.250 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.251 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.251 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.251 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.251 DEBUG:boto:Data: 2023-09-29T22:46:46.251 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.251 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.251 DEBUG:boto:Params: {} 2023-09-29T22:46:46.252 DEBUG:boto:Token: None 2023-09-29T22:46:46.252 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hyILk6w54WObgLAOaqo8Nm6ih/E= 2023-09-29T22:46:46.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyILk6w54WObgLAOaqo8Nm6ih/E=', 'Content-Length': '0'} 2023-09-29T22:46:46.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e40417950143cc17-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:46:46.255 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.255 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.255 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.255 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.256 DEBUG:boto:Method: GET 2023-09-29T22:46:46.256 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.256 DEBUG:boto:Data: 2023-09-29T22:46:46.256 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.256 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.256 DEBUG:boto:Params: {} 2023-09-29T22:46:46.256 DEBUG:boto:Token: None 2023-09-29T22:46:46.257 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM= 2023-09-29T22:46:46.257 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM=', 'Content-Length': '0'} 2023-09-29T22:46:46.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab557edbc6a3bae1-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.260 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.260 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.260 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.260 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.260 DEBUG:boto:Method: GET 2023-09-29T22:46:46.260 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.260 DEBUG:boto:Data: 2023-09-29T22:46:46.260 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.261 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.261 DEBUG:boto:Params: {} 2023-09-29T22:46:46.261 DEBUG:boto:Token: None 2023-09-29T22:46:46.261 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:46:46.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM= 2023-09-29T22:46:46.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2APa+ugMuCzIqJ0zllqbLwttEdM=', 'Content-Length': '0'} 2023-09-29T22:46:46.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef03136baa08201-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.310 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.310 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.311 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.311 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.311 DEBUG:boto:Method: GET 2023-09-29T22:46:46.311 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.311 DEBUG:boto:Data: 2023-09-29T22:46:46.311 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.312 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.312 DEBUG:boto:Params: {} 2023-09-29T22:46:46.312 DEBUG:boto:Token: None 2023-09-29T22:46:46.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.317 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000851ee0c847f5a5e6-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.317 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.318 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.318 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.318 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.318 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.318 DEBUG:boto:Method: GET 2023-09-29T22:46:46.318 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.318 DEBUG:boto:Data: 2023-09-29T22:46:46.318 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.319 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.319 DEBUG:boto:Params: {} 2023-09-29T22:46:46.319 DEBUG:boto:Token: None 2023-09-29T22:46:46.319 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.319 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.319 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.326 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fb0e989f0a6eb1c-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.327 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.327 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.327 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.327 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.327 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:46:46.327 DEBUG:boto:Data: 2023-09-29T22:46:46.328 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.328 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.328 DEBUG:boto:Params: {} 2023-09-29T22:46:46.328 DEBUG:boto:Token: None 2023-09-29T22:46:46.328 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58 2023-09-29T22:46:46.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CPWM2VNleC/5VbvFLoK2WjGlXZE= 2023-09-29T22:46:46.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CPWM2VNleC/5VbvFLoK2WjGlXZE=', 'Content-Length': '0'} 2023-09-29T22:46:46.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000172a7d0f70476e64-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.331 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.331 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.331 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.331 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:46:46.331 DEBUG:boto:Data: 2023-09-29T22:46:46.332 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.332 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.332 DEBUG:boto:Params: {} 2023-09-29T22:46:46.332 DEBUG:boto:Token: None 2023-09-29T22:46:46.332 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58 2023-09-29T22:46:46.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CPWM2VNleC/5VbvFLoK2WjGlXZE= 2023-09-29T22:46:46.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CPWM2VNleC/5VbvFLoK2WjGlXZE=', 'Content-Length': '0'} 2023-09-29T22:46:46.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e1afd1be353d4ce-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:46:46.335 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.335 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.335 DEBUG:boto:Method: GET 2023-09-29T22:46:46.335 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:46:46.335 DEBUG:boto:Data: 2023-09-29T22:46:46.336 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.336 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.336 DEBUG:boto:Params: {} 2023-09-29T22:46:46.336 DEBUG:boto:Token: None 2023-09-29T22:46:46.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58 2023-09-29T22:46:46.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5hg3ZCjBH0NJFmlE19SAziol17Q= 2023-09-29T22:46:46.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5hg3ZCjBH0NJFmlE19SAziol17Q=', 'Content-Length': '0'} 2023-09-29T22:46:46.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036a56e0b73226d2c-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.386 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:46:46.387 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:46:46.387 DEBUG:boto:Method: GET 2023-09-29T22:46:46.387 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:46:46.387 DEBUG:boto:Data: 2023-09-29T22:46:46.387 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.387 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.387 DEBUG:boto:Params: {} 2023-09-29T22:46:46.388 DEBUG:boto:Token: None 2023-09-29T22:46:46.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj58 2023-09-29T22:46:46.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5hg3ZCjBH0NJFmlE19SAziol17Q= 2023-09-29T22:46:46.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5hg3ZCjBH0NJFmlE19SAziol17Q=', 'Content-Length': '0'} 2023-09-29T22:46:46.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009679ed8363cb5867-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.392 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.392 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.392 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.392 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.392 DEBUG:boto:Method: GET 2023-09-29T22:46:46.392 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.393 DEBUG:boto:Data: 2023-09-29T22:46:46.393 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.393 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.393 DEBUG:boto:Params: {} 2023-09-29T22:46:46.393 DEBUG:boto:Token: None 2023-09-29T22:46:46.393 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.398 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000189200f2d0b0fa18-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.398 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.399 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.399 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.399 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.399 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.399 DEBUG:boto:Method: GET 2023-09-29T22:46:46.399 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:46:46.399 DEBUG:boto:Data: 2023-09-29T22:46:46.399 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.400 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.400 DEBUG:boto:Params: {} 2023-09-29T22:46:46.400 DEBUG:boto:Token: None 2023-09-29T22:46:46.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076b40fea74335b49-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.450 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:46:46.451 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.452 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.452 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.452 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.452 DEBUG:boto:Method: GET 2023-09-29T22:46:46.452 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.452 DEBUG:boto:Data: 2023-09-29T22:46:46.452 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.453 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.453 DEBUG:boto:Params: {} 2023-09-29T22:46:46.453 DEBUG:boto:Token: None 2023-09-29T22:46:46.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0= 2023-09-29T22:46:46.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0=', 'Content-Length': '0'} 2023-09-29T22:46:46.456 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005409a13cb84c2f6e-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.457 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.457 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.457 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.457 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.457 DEBUG:boto:Method: GET 2023-09-29T22:46:46.457 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.457 DEBUG:boto:Data: 2023-09-29T22:46:46.457 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.458 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.458 DEBUG:boto:Params: {} 2023-09-29T22:46:46.458 DEBUG:boto:Token: None 2023-09-29T22:46:46.458 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.458 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0= 2023-09-29T22:46:46.458 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0=', 'Content-Length': '0'} 2023-09-29T22:46:46.461 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fcf01087f5df7d2-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.461 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.461 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.461 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.462 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.462 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.462 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.462 DEBUG:boto:Data: 2023-09-29T22:46:46.462 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.462 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.462 DEBUG:boto:Params: {} 2023-09-29T22:46:46.462 DEBUG:boto:Token: None 2023-09-29T22:46:46.462 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.463 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFeWDDTfroaXxs+UhFZ9dCgRMMo= 2023-09-29T22:46:46.463 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFeWDDTfroaXxs+UhFZ9dCgRMMo=', 'Content-Length': '0'} 2023-09-29T22:46:46.465 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef4c2e5127bc9e01-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.466 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.466 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.466 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.466 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.466 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.466 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.466 DEBUG:boto:Data: 2023-09-29T22:46:46.466 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.467 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.467 DEBUG:boto:Params: {} 2023-09-29T22:46:46.467 DEBUG:boto:Token: None 2023-09-29T22:46:46.467 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.467 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TFeWDDTfroaXxs+UhFZ9dCgRMMo= 2023-09-29T22:46:46.467 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TFeWDDTfroaXxs+UhFZ9dCgRMMo=', 'Content-Length': '0'} 2023-09-29T22:46:46.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b245c1ff3cd52b-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:46:46.470 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.470 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.470 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.471 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.471 DEBUG:boto:Method: GET 2023-09-29T22:46:46.471 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.471 DEBUG:boto:Data: 2023-09-29T22:46:46.471 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.471 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.471 DEBUG:boto:Params: {} 2023-09-29T22:46:46.471 DEBUG:boto:Token: None 2023-09-29T22:46:46.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0= 2023-09-29T22:46:46.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0=', 'Content-Length': '0'} 2023-09-29T22:46:46.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ddb19c69f8656bd-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.475 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.475 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.475 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.475 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.476 DEBUG:boto:Method: GET 2023-09-29T22:46:46.476 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.476 DEBUG:boto:Data: 2023-09-29T22:46:46.476 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.476 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.476 DEBUG:boto:Params: {} 2023-09-29T22:46:46.476 DEBUG:boto:Token: None 2023-09-29T22:46:46.477 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:46:46.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0= 2023-09-29T22:46:46.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oH9TbCSJzHq20FAnOyfWBQag9c0=', 'Content-Length': '0'} 2023-09-29T22:46:46.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097dd783e3a8f24e2-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.480 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.480 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.480 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.480 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.480 DEBUG:boto:Method: GET 2023-09-29T22:46:46.480 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.480 DEBUG:boto:Data: 2023-09-29T22:46:46.480 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.481 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.481 DEBUG:boto:Params: {} 2023-09-29T22:46:46.481 DEBUG:boto:Token: None 2023-09-29T22:46:46.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000585a0d351ecb9330-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.530 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.531 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.531 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.531 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.531 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.532 DEBUG:boto:Method: GET 2023-09-29T22:46:46.532 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.532 DEBUG:boto:Data: 2023-09-29T22:46:46.532 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.532 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.532 DEBUG:boto:Params: {} 2023-09-29T22:46:46.533 DEBUG:boto:Token: None 2023-09-29T22:46:46.533 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff15d22ce6df9fcb-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.542 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.542 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.542 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.542 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.543 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:46:46.543 DEBUG:boto:Data: 2023-09-29T22:46:46.543 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.543 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.543 DEBUG:boto:Params: {} 2023-09-29T22:46:46.543 DEBUG:boto:Token: None 2023-09-29T22:46:46.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68 2023-09-29T22:46:46.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbtc7nlB0jKP4oDh/KdUtwlHLJE= 2023-09-29T22:46:46.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbtc7nlB0jKP4oDh/KdUtwlHLJE=', 'Content-Length': '0'} 2023-09-29T22:46:46.546 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f05b3936a8a935ee-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.547 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.547 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.547 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.547 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:46:46.547 DEBUG:boto:Data: 2023-09-29T22:46:46.547 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.547 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.547 DEBUG:boto:Params: {} 2023-09-29T22:46:46.548 DEBUG:boto:Token: None 2023-09-29T22:46:46.548 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68 2023-09-29T22:46:46.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lbtc7nlB0jKP4oDh/KdUtwlHLJE= 2023-09-29T22:46:46.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lbtc7nlB0jKP4oDh/KdUtwlHLJE=', 'Content-Length': '0'} 2023-09-29T22:46:46.550 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f876feacba59f6-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:46:46.550 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.550 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.551 DEBUG:boto:Method: GET 2023-09-29T22:46:46.551 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:46:46.551 DEBUG:boto:Data: 2023-09-29T22:46:46.551 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.551 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.551 DEBUG:boto:Params: {} 2023-09-29T22:46:46.551 DEBUG:boto:Token: None 2023-09-29T22:46:46.551 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68 2023-09-29T22:46:46.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:emFdPn1EMBmpjbI2HAbaIe7MQ4k= 2023-09-29T22:46:46.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:emFdPn1EMBmpjbI2HAbaIe7MQ4k=', 'Content-Length': '0'} 2023-09-29T22:46:46.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074bb4ff0180eab5a-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.554 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:46:46.554 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:46:46.554 DEBUG:boto:Method: GET 2023-09-29T22:46:46.554 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:46:46.554 DEBUG:boto:Data: 2023-09-29T22:46:46.554 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.555 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.555 DEBUG:boto:Params: {} 2023-09-29T22:46:46.555 DEBUG:boto:Token: None 2023-09-29T22:46:46.555 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj68 2023-09-29T22:46:46.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:emFdPn1EMBmpjbI2HAbaIe7MQ4k= 2023-09-29T22:46:46.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:emFdPn1EMBmpjbI2HAbaIe7MQ4k=', 'Content-Length': '0'} 2023-09-29T22:46:46.557 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006135301ad0ff231c-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.558 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.558 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.558 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.558 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.558 DEBUG:boto:Method: GET 2023-09-29T22:46:46.558 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.559 DEBUG:boto:Data: 2023-09-29T22:46:46.559 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.559 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.559 DEBUG:boto:Params: {} 2023-09-29T22:46:46.559 DEBUG:boto:Token: None 2023-09-29T22:46:46.559 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.563 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000064bc51e17c59cc7-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.564 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.564 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.564 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.564 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.564 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.564 DEBUG:boto:Method: GET 2023-09-29T22:46:46.565 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:46:46.565 DEBUG:boto:Data: 2023-09-29T22:46:46.565 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.565 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.565 DEBUG:boto:Params: {} 2023-09-29T22:46:46.565 DEBUG:boto:Token: None 2023-09-29T22:46:46.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.571 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3a649f385ecc6f1-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.572 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:46:46.572 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.572 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.572 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.573 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.573 DEBUG:boto:Method: GET 2023-09-29T22:46:46.573 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.573 DEBUG:boto:Data: 2023-09-29T22:46:46.573 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.573 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.573 DEBUG:boto:Params: {} 2023-09-29T22:46:46.573 DEBUG:boto:Token: None 2023-09-29T22:46:46.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII= 2023-09-29T22:46:46.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII=', 'Content-Length': '0'} 2023-09-29T22:46:46.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000470e183cc07b4677-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.622 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.623 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.623 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.623 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.623 DEBUG:boto:Method: GET 2023-09-29T22:46:46.623 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.623 DEBUG:boto:Data: 2023-09-29T22:46:46.623 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.624 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.624 DEBUG:boto:Params: {} 2023-09-29T22:46:46.624 DEBUG:boto:Token: None 2023-09-29T22:46:46.624 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.624 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII= 2023-09-29T22:46:46.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII=', 'Content-Length': '0'} 2023-09-29T22:46:46.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dea035fb55f2c20b-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.627 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.627 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.628 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.628 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.628 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.628 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.628 DEBUG:boto:Data: 2023-09-29T22:46:46.628 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.628 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.628 DEBUG:boto:Params: {} 2023-09-29T22:46:46.628 DEBUG:boto:Token: None 2023-09-29T22:46:46.629 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e8KRYjSq2loShUO9RS2UsUrwuDU= 2023-09-29T22:46:46.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e8KRYjSq2loShUO9RS2UsUrwuDU=', 'Content-Length': '0'} 2023-09-29T22:46:46.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000293b7fa45d84c0a4-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.631 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.632 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.632 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.632 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.632 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.632 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.632 DEBUG:boto:Data: 2023-09-29T22:46:46.632 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.632 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.633 DEBUG:boto:Params: {} 2023-09-29T22:46:46.633 DEBUG:boto:Token: None 2023-09-29T22:46:46.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e8KRYjSq2loShUO9RS2UsUrwuDU= 2023-09-29T22:46:46.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e8KRYjSq2loShUO9RS2UsUrwuDU=', 'Content-Length': '0'} 2023-09-29T22:46:46.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000958668e08087974f-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:46:46.636 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.636 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.636 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.636 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.637 DEBUG:boto:Method: GET 2023-09-29T22:46:46.637 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.637 DEBUG:boto:Data: 2023-09-29T22:46:46.637 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.637 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.637 DEBUG:boto:Params: {} 2023-09-29T22:46:46.637 DEBUG:boto:Token: None 2023-09-29T22:46:46.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII= 2023-09-29T22:46:46.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII=', 'Content-Length': '0'} 2023-09-29T22:46:46.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008d0c3b9e0810c75-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.641 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.641 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.641 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.641 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.641 DEBUG:boto:Method: GET 2023-09-29T22:46:46.641 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.641 DEBUG:boto:Data: 2023-09-29T22:46:46.641 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.642 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.642 DEBUG:boto:Params: {} 2023-09-29T22:46:46.642 DEBUG:boto:Token: None 2023-09-29T22:46:46.642 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:46:46.642 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII= 2023-09-29T22:46:46.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sYFaTd6QSoCu87WKFpqezgegBII=', 'Content-Length': '0'} 2023-09-29T22:46:46.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc73d5c9bfe5ff7a-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.690 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.691 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.691 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.691 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.691 DEBUG:boto:Method: GET 2023-09-29T22:46:46.691 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.691 DEBUG:boto:Data: 2023-09-29T22:46:46.691 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.692 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.692 DEBUG:boto:Params: {} 2023-09-29T22:46:46.692 DEBUG:boto:Token: None 2023-09-29T22:46:46.692 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.697 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045f4cc617d5a4709-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.697 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.697 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.697 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.698 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.698 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.698 DEBUG:boto:Method: GET 2023-09-29T22:46:46.698 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.698 DEBUG:boto:Data: 2023-09-29T22:46:46.698 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.698 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.698 DEBUG:boto:Params: {} 2023-09-29T22:46:46.699 DEBUG:boto:Token: None 2023-09-29T22:46:46.699 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.705 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908e6b872a3389d7-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.706 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.706 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.706 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.706 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.706 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:46:46.707 DEBUG:boto:Data: 2023-09-29T22:46:46.707 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.707 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.707 DEBUG:boto:Params: {} 2023-09-29T22:46:46.707 DEBUG:boto:Token: None 2023-09-29T22:46:46.707 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69 2023-09-29T22:46:46.707 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WyINx4XJgR3YaznHlsaJPevJs9M= 2023-09-29T22:46:46.707 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WyINx4XJgR3YaznHlsaJPevJs9M=', 'Content-Length': '0'} 2023-09-29T22:46:46.709 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006306d8dea0f639a2-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.710 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.710 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.710 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.710 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:46:46.710 DEBUG:boto:Data: 2023-09-29T22:46:46.710 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.711 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.711 DEBUG:boto:Params: {} 2023-09-29T22:46:46.711 DEBUG:boto:Token: None 2023-09-29T22:46:46.711 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69 2023-09-29T22:46:46.711 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WyINx4XJgR3YaznHlsaJPevJs9M= 2023-09-29T22:46:46.711 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WyINx4XJgR3YaznHlsaJPevJs9M=', 'Content-Length': '0'} 2023-09-29T22:46:46.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f7d514c9ad3f726-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:46:46.714 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.714 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.714 DEBUG:boto:Method: GET 2023-09-29T22:46:46.715 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:46:46.715 DEBUG:boto:Data: 2023-09-29T22:46:46.715 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.715 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.715 DEBUG:boto:Params: {} 2023-09-29T22:46:46.715 DEBUG:boto:Token: None 2023-09-29T22:46:46.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69 2023-09-29T22:46:46.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/1LPfvpYIsTD/IJ4fSN645SauT8= 2023-09-29T22:46:46.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/1LPfvpYIsTD/IJ4fSN645SauT8=', 'Content-Length': '0'} 2023-09-29T22:46:46.717 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bacc96e0305b4b31-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.758 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:46:46.759 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:46:46.759 DEBUG:boto:Method: GET 2023-09-29T22:46:46.759 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:46:46.759 DEBUG:boto:Data: 2023-09-29T22:46:46.759 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.759 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.759 DEBUG:boto:Params: {} 2023-09-29T22:46:46.759 DEBUG:boto:Token: None 2023-09-29T22:46:46.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj69 2023-09-29T22:46:46.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/1LPfvpYIsTD/IJ4fSN645SauT8= 2023-09-29T22:46:46.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/1LPfvpYIsTD/IJ4fSN645SauT8=', 'Content-Length': '0'} 2023-09-29T22:46:46.763 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000252b2c9ed650aabc-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.763 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.763 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.763 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.763 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.763 DEBUG:boto:Method: GET 2023-09-29T22:46:46.764 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.764 DEBUG:boto:Data: 2023-09-29T22:46:46.764 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.764 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.764 DEBUG:boto:Params: {} 2023-09-29T22:46:46.764 DEBUG:boto:Token: None 2023-09-29T22:46:46.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.765 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.769 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d938eca4eb9045ff-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.769 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.770 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.770 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.770 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.770 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.770 DEBUG:boto:Method: GET 2023-09-29T22:46:46.770 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:46:46.770 DEBUG:boto:Data: 2023-09-29T22:46:46.770 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.771 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.771 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.771 DEBUG:boto:Params: {} 2023-09-29T22:46:46.771 DEBUG:boto:Token: None 2023-09-29T22:46:46.771 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.771 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.771 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a61de1134ed8245-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.818 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:46:46.819 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.819 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.819 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.819 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.820 DEBUG:boto:Method: GET 2023-09-29T22:46:46.820 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.820 DEBUG:boto:Data: 2023-09-29T22:46:46.820 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.820 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.820 DEBUG:boto:Params: {} 2023-09-29T22:46:46.820 DEBUG:boto:Token: None 2023-09-29T22:46:46.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI= 2023-09-29T22:46:46.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI=', 'Content-Length': '0'} 2023-09-29T22:46:46.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033d63c719cdc45c6-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.824 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.824 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.824 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.824 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.824 DEBUG:boto:Method: GET 2023-09-29T22:46:46.824 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.824 DEBUG:boto:Data: 2023-09-29T22:46:46.825 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.825 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.825 DEBUG:boto:Params: {} 2023-09-29T22:46:46.825 DEBUG:boto:Token: None 2023-09-29T22:46:46.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI= 2023-09-29T22:46:46.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI=', 'Content-Length': '0'} 2023-09-29T22:46:46.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000637326e72c332206-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.829 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.829 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.829 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.829 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.829 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.829 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.829 DEBUG:boto:Data: 2023-09-29T22:46:46.829 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.830 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.830 DEBUG:boto:Params: {} 2023-09-29T22:46:46.830 DEBUG:boto:Token: None 2023-09-29T22:46:46.830 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GzqEiUKTNkIPY+SzQ1J7j/QB3dE= 2023-09-29T22:46:46.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GzqEiUKTNkIPY+SzQ1J7j/QB3dE=', 'Content-Length': '0'} 2023-09-29T22:46:46.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d64362346b9c7e-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.833 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.833 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.833 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.833 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.833 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.834 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.834 DEBUG:boto:Data: 2023-09-29T22:46:46.834 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.834 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.834 DEBUG:boto:Params: {} 2023-09-29T22:46:46.834 DEBUG:boto:Token: None 2023-09-29T22:46:46.834 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.835 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GzqEiUKTNkIPY+SzQ1J7j/QB3dE= 2023-09-29T22:46:46.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GzqEiUKTNkIPY+SzQ1J7j/QB3dE=', 'Content-Length': '0'} 2023-09-29T22:46:46.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb1881a4ff70abfa-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:46:46.838 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.838 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.838 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.838 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.838 DEBUG:boto:Method: GET 2023-09-29T22:46:46.839 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.839 DEBUG:boto:Data: 2023-09-29T22:46:46.839 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.839 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.839 DEBUG:boto:Params: {} 2023-09-29T22:46:46.839 DEBUG:boto:Token: None 2023-09-29T22:46:46.839 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.839 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI= 2023-09-29T22:46:46.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI=', 'Content-Length': '0'} 2023-09-29T22:46:46.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000445e94b3a970231c-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.843 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.843 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.843 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.843 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.843 DEBUG:boto:Method: GET 2023-09-29T22:46:46.843 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.843 DEBUG:boto:Data: 2023-09-29T22:46:46.843 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.844 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.844 DEBUG:boto:Params: {} 2023-09-29T22:46:46.844 DEBUG:boto:Token: None 2023-09-29T22:46:46.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:46:46.844 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI= 2023-09-29T22:46:46.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Irlf8t7jAoqafulTDBjhzaelLjI=', 'Content-Length': '0'} 2023-09-29T22:46:46.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc9a46b4f6cae48f-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.848 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.848 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.848 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.848 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.848 DEBUG:boto:Method: GET 2023-09-29T22:46:46.848 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.848 DEBUG:boto:Data: 2023-09-29T22:46:46.848 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.849 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.849 DEBUG:boto:Params: {} 2023-09-29T22:46:46.849 DEBUG:boto:Token: None 2023-09-29T22:46:46.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000058b340915ddc8023-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.898 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.899 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.899 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.899 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.899 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.899 DEBUG:boto:Method: GET 2023-09-29T22:46:46.899 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.899 DEBUG:boto:Data: 2023-09-29T22:46:46.899 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.900 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.900 DEBUG:boto:Params: {} 2023-09-29T22:46:46.900 DEBUG:boto:Token: None 2023-09-29T22:46:46.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632f2c7f4f6987b0-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.908 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.909 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.909 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.909 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.909 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:46:46.909 DEBUG:boto:Data: 2023-09-29T22:46:46.909 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.910 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.910 DEBUG:boto:Params: {} 2023-09-29T22:46:46.910 DEBUG:boto:Token: None 2023-09-29T22:46:46.910 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76 2023-09-29T22:46:46.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/bKX0Gt/0/k1AMSq4jIrF2NycU= 2023-09-29T22:46:46.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/bKX0Gt/0/k1AMSq4jIrF2NycU=', 'Content-Length': '0'} 2023-09-29T22:46:46.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a34aea34899eb5b-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.913 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.913 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.913 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.913 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:46:46.913 DEBUG:boto:Data: 2023-09-29T22:46:46.913 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.914 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.914 DEBUG:boto:Params: {} 2023-09-29T22:46:46.914 DEBUG:boto:Token: None 2023-09-29T22:46:46.914 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76 2023-09-29T22:46:46.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/bKX0Gt/0/k1AMSq4jIrF2NycU= 2023-09-29T22:46:46.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/bKX0Gt/0/k1AMSq4jIrF2NycU=', 'Content-Length': '0'} 2023-09-29T22:46:46.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000743e6815894ed63-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:46:46.917 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.917 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.917 DEBUG:boto:Method: GET 2023-09-29T22:46:46.917 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:46:46.917 DEBUG:boto:Data: 2023-09-29T22:46:46.917 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.917 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.917 DEBUG:boto:Params: {} 2023-09-29T22:46:46.918 DEBUG:boto:Token: None 2023-09-29T22:46:46.918 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76 2023-09-29T22:46:46.918 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8XCuPOcoS1Jix6crC4TqATi3lEQ= 2023-09-29T22:46:46.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8XCuPOcoS1Jix6crC4TqATi3lEQ=', 'Content-Length': '0'} 2023-09-29T22:46:46.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ac559e01a0bf4a2-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.920 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:46:46.920 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:46:46.920 DEBUG:boto:Method: GET 2023-09-29T22:46:46.921 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:46:46.921 DEBUG:boto:Data: 2023-09-29T22:46:46.921 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.921 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.921 DEBUG:boto:Params: {} 2023-09-29T22:46:46.921 DEBUG:boto:Token: None 2023-09-29T22:46:46.921 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj76 2023-09-29T22:46:46.921 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8XCuPOcoS1Jix6crC4TqATi3lEQ= 2023-09-29T22:46:46.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8XCuPOcoS1Jix6crC4TqATi3lEQ=', 'Content-Length': '0'} 2023-09-29T22:46:46.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e4cfc21a748b5bf-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.970 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.971 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.971 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.971 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.971 DEBUG:boto:Method: GET 2023-09-29T22:46:46.971 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.971 DEBUG:boto:Data: 2023-09-29T22:46:46.971 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.971 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.971 DEBUG:boto:Params: {} 2023-09-29T22:46:46.972 DEBUG:boto:Token: None 2023-09-29T22:46:46.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.976 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000277f5171917b4b1e-00651753d6-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.977 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.977 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:46.977 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:46.977 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.977 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.978 DEBUG:boto:Method: GET 2023-09-29T22:46:46.978 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:46:46.978 DEBUG:boto:Data: 2023-09-29T22:46:46.978 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.978 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.978 DEBUG:boto:Params: {} 2023-09-29T22:46:46.978 DEBUG:boto:Token: None 2023-09-29T22:46:46.978 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/ 2023-09-29T22:46:46.979 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc= 2023-09-29T22:46:46.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzMzBLuPmCPHJ/35KNR29rtM3Gc=', 'Content-Length': '0'} 2023-09-29T22:46:46.985 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ffcc198f623a076-00651753d6-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.986 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:46.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:46:46.986 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:46.986 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:46.986 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.986 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.987 DEBUG:boto:Method: GET 2023-09-29T22:46:46.987 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.987 DEBUG:boto:Data: 2023-09-29T22:46:46.987 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.987 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.987 DEBUG:boto:Params: {} 2023-09-29T22:46:46.987 DEBUG:boto:Token: None 2023-09-29T22:46:46.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kl0K8oa7cdEQHmM25Lcb1CCOapU= 2023-09-29T22:46:46.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kl0K8oa7cdEQHmM25Lcb1CCOapU=', 'Content-Length': '0'} 2023-09-29T22:46:46.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f03900259b9103e1-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.990 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:46.990 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:46.990 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.990 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.990 DEBUG:boto:Method: GET 2023-09-29T22:46:46.990 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.990 DEBUG:boto:Data: 2023-09-29T22:46:46.991 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.991 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.991 DEBUG:boto:Params: {} 2023-09-29T22:46:46.991 DEBUG:boto:Token: None 2023-09-29T22:46:46.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Kl0K8oa7cdEQHmM25Lcb1CCOapU= 2023-09-29T22:46:46.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Kl0K8oa7cdEQHmM25Lcb1CCOapU=', 'Content-Length': '0'} 2023-09-29T22:46:46.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2c8d616139e7ca5-00651753d6-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.994 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:46.994 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:46.994 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.994 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.994 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.994 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.994 DEBUG:boto:Data: 2023-09-29T22:46:46.994 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.994 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:46.995 DEBUG:boto:Port: 8001 2023-09-29T22:46:46.995 DEBUG:boto:Params: {} 2023-09-29T22:46:46.995 DEBUG:boto:Token: None 2023-09-29T22:46:46.995 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.995 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n4d+OhOrT7xd4GZqqZYdh+6cDcI= 2023-09-29T22:46:46.995 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n4d+OhOrT7xd4GZqqZYdh+6cDcI=', 'Content-Length': '0'} 2023-09-29T22:46:46.997 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cc2372c1261d2a4-00651753d6-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:46.997 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:46.998 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:46.998 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.998 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.998 DEBUG:boto:Method: HEAD 2023-09-29T22:46:46.998 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.998 DEBUG:boto:Data: 2023-09-29T22:46:46.998 DEBUG:boto:Headers: {} 2023-09-29T22:46:46.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:46.998 DEBUG:boto:Port: 8000 2023-09-29T22:46:46.998 DEBUG:boto:Params: {} 2023-09-29T22:46:46.999 DEBUG:boto:Token: None 2023-09-29T22:46:46.999 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:46 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:46.999 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n4d+OhOrT7xd4GZqqZYdh+6cDcI= 2023-09-29T22:46:46.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n4d+OhOrT7xd4GZqqZYdh+6cDcI=', 'Content-Length': '0'} 2023-09-29T22:46:47.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b2109a87f7fcabf-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:46:47.002 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:47.002 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:47.002 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.002 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.002 DEBUG:boto:Method: GET 2023-09-29T22:46:47.002 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.002 DEBUG:boto:Data: 2023-09-29T22:46:47.003 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.003 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.003 DEBUG:boto:Params: {} 2023-09-29T22:46:47.003 DEBUG:boto:Token: None 2023-09-29T22:46:47.003 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SUiUKi1Z0aTa7reMDCLUYl3TdvA= 2023-09-29T22:46:47.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SUiUKi1Z0aTa7reMDCLUYl3TdvA=', 'Content-Length': '0'} 2023-09-29T22:46:47.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b52fed5be65377a-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.006 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:47.006 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:47.006 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.006 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.006 DEBUG:boto:Method: GET 2023-09-29T22:46:47.006 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.006 DEBUG:boto:Data: 2023-09-29T22:46:47.006 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.007 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.007 DEBUG:boto:Params: {} 2023-09-29T22:46:47.007 DEBUG:boto:Token: None 2023-09-29T22:46:47.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:46:47.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SUiUKi1Z0aTa7reMDCLUYl3TdvA= 2023-09-29T22:46:47.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SUiUKi1Z0aTa7reMDCLUYl3TdvA=', 'Content-Length': '0'} 2023-09-29T22:46:47.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e52f3799691ddf6-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.010 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.010 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.011 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.011 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.011 DEBUG:boto:Method: GET 2023-09-29T22:46:47.011 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.011 DEBUG:boto:Data: 2023-09-29T22:46:47.011 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.011 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.011 DEBUG:boto:Params: {} 2023-09-29T22:46:47.011 DEBUG:boto:Token: None 2023-09-29T22:46:47.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.016 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000590661557fa84520-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.016 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.017 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.017 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.017 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.017 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.017 DEBUG:boto:Method: GET 2023-09-29T22:46:47.017 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.018 DEBUG:boto:Data: 2023-09-29T22:46:47.018 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.018 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.018 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.018 DEBUG:boto:Params: {} 2023-09-29T22:46:47.018 DEBUG:boto:Token: None 2023-09-29T22:46:47.018 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.018 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.024 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000310e61908a19305-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.024 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.025 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:47.025 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:47.025 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.025 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:46:47.025 DEBUG:boto:Data: 2023-09-29T22:46:47.025 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.026 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.026 DEBUG:boto:Params: {} 2023-09-29T22:46:47.026 DEBUG:boto:Token: None 2023-09-29T22:46:47.026 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj85 2023-09-29T22:46:47.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x/ahut7n8u5XZv4ySrTJ9w+dNDY= 2023-09-29T22:46:47.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x/ahut7n8u5XZv4ySrTJ9w+dNDY=', 'Content-Length': '0'} 2023-09-29T22:46:47.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a8edc844823e8b1-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.028 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:47.028 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:47.029 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.029 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:46:47.029 DEBUG:boto:Data: 2023-09-29T22:46:47.029 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.029 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.029 DEBUG:boto:Params: {} 2023-09-29T22:46:47.029 DEBUG:boto:Token: None 2023-09-29T22:46:47.029 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj85 2023-09-29T22:46:47.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:x/ahut7n8u5XZv4ySrTJ9w+dNDY= 2023-09-29T22:46:47.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:x/ahut7n8u5XZv4ySrTJ9w+dNDY=', 'Content-Length': '0'} 2023-09-29T22:46:47.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f343322534197b00-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:46:47.032 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:47.032 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:47.033 DEBUG:boto:Method: GET 2023-09-29T22:46:47.033 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:46:47.033 DEBUG:boto:Data: 2023-09-29T22:46:47.033 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.033 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.033 DEBUG:boto:Params: {} 2023-09-29T22:46:47.033 DEBUG:boto:Token: None 2023-09-29T22:46:47.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj85 2023-09-29T22:46:47.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Czyl22g63YQHBUFg08J6f3WdQb0= 2023-09-29T22:46:47.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Czyl22g63YQHBUFg08J6f3WdQb0=', 'Content-Length': '0'} 2023-09-29T22:46:47.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab4427afe20545bd-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.078 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:46:47.079 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:46:47.079 DEBUG:boto:Method: GET 2023-09-29T22:46:47.079 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:46:47.079 DEBUG:boto:Data: 2023-09-29T22:46:47.079 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.079 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.079 DEBUG:boto:Params: {} 2023-09-29T22:46:47.080 DEBUG:boto:Token: None 2023-09-29T22:46:47.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj85 2023-09-29T22:46:47.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Czyl22g63YQHBUFg08J6f3WdQb0= 2023-09-29T22:46:47.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Czyl22g63YQHBUFg08J6f3WdQb0=', 'Content-Length': '0'} 2023-09-29T22:46:47.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1c159f3d2eea00d-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.083 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.083 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.083 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.083 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.084 DEBUG:boto:Method: GET 2023-09-29T22:46:47.084 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.084 DEBUG:boto:Data: 2023-09-29T22:46:47.084 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.084 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.084 DEBUG:boto:Params: {} 2023-09-29T22:46:47.084 DEBUG:boto:Token: None 2023-09-29T22:46:47.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8e838bf482e374e-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.089 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.090 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.090 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.090 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.090 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.090 DEBUG:boto:Method: GET 2023-09-29T22:46:47.090 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:46:47.091 DEBUG:boto:Data: 2023-09-29T22:46:47.091 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.091 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.091 DEBUG:boto:Params: {} 2023-09-29T22:46:47.091 DEBUG:boto:Token: None 2023-09-29T22:46:47.091 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.098 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007aa1fe5057f87acd-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.098 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:46:47.098 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.099 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.099 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.099 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.099 DEBUG:boto:Method: GET 2023-09-29T22:46:47.099 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.099 DEBUG:boto:Data: 2023-09-29T22:46:47.099 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.099 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.099 DEBUG:boto:Params: {} 2023-09-29T22:46:47.100 DEBUG:boto:Token: None 2023-09-29T22:46:47.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0= 2023-09-29T22:46:47.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0=', 'Content-Length': '0'} 2023-09-29T22:46:47.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047eca3e4e246df91-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.103 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.103 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.103 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.103 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.103 DEBUG:boto:Method: GET 2023-09-29T22:46:47.104 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.104 DEBUG:boto:Data: 2023-09-29T22:46:47.104 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.104 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.104 DEBUG:boto:Params: {} 2023-09-29T22:46:47.104 DEBUG:boto:Token: None 2023-09-29T22:46:47.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0= 2023-09-29T22:46:47.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0=', 'Content-Length': '0'} 2023-09-29T22:46:47.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cff9bd36ea073cb7-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.107 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.107 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.107 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.107 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.108 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.108 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.108 DEBUG:boto:Data: 2023-09-29T22:46:47.108 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.108 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.108 DEBUG:boto:Params: {} 2023-09-29T22:46:47.108 DEBUG:boto:Token: None 2023-09-29T22:46:47.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RH6K9s7bCk8iQ7VfV7jUFqr7YYM= 2023-09-29T22:46:47.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RH6K9s7bCk8iQ7VfV7jUFqr7YYM=', 'Content-Length': '0'} 2023-09-29T22:46:47.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d37f20643055f255-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.111 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.111 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.111 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.111 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.112 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.112 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.112 DEBUG:boto:Data: 2023-09-29T22:46:47.112 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.112 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.112 DEBUG:boto:Params: {} 2023-09-29T22:46:47.112 DEBUG:boto:Token: None 2023-09-29T22:46:47.112 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RH6K9s7bCk8iQ7VfV7jUFqr7YYM= 2023-09-29T22:46:47.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RH6K9s7bCk8iQ7VfV7jUFqr7YYM=', 'Content-Length': '0'} 2023-09-29T22:46:47.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099053a53685dff23-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:46:47.115 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.115 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.115 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.115 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.115 DEBUG:boto:Method: GET 2023-09-29T22:46:47.115 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.116 DEBUG:boto:Data: 2023-09-29T22:46:47.116 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.116 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.116 DEBUG:boto:Params: {} 2023-09-29T22:46:47.116 DEBUG:boto:Token: None 2023-09-29T22:46:47.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0= 2023-09-29T22:46:47.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0=', 'Content-Length': '0'} 2023-09-29T22:46:47.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006071cf2c1deb11e3-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.162 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.163 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.163 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.163 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.163 DEBUG:boto:Method: GET 2023-09-29T22:46:47.163 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.163 DEBUG:boto:Data: 2023-09-29T22:46:47.164 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.164 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.164 DEBUG:boto:Params: {} 2023-09-29T22:46:47.164 DEBUG:boto:Token: None 2023-09-29T22:46:47.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:46:47.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0= 2023-09-29T22:46:47.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GPi8WXx7aTeh1KZ4HkDMVuQsoV0=', 'Content-Length': '0'} 2023-09-29T22:46:47.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bdb7bafbdbcdd0a-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.168 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.168 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.168 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.169 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.169 DEBUG:boto:Method: GET 2023-09-29T22:46:47.169 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.169 DEBUG:boto:Data: 2023-09-29T22:46:47.169 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.169 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.169 DEBUG:boto:Params: {} 2023-09-29T22:46:47.170 DEBUG:boto:Token: None 2023-09-29T22:46:47.170 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.170 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.170 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001372436b9787d52b-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.175 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.175 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.175 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.175 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.176 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.176 DEBUG:boto:Method: GET 2023-09-29T22:46:47.176 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.176 DEBUG:boto:Data: 2023-09-29T22:46:47.176 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.176 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.177 DEBUG:boto:Params: {} 2023-09-29T22:46:47.177 DEBUG:boto:Token: None 2023-09-29T22:46:47.177 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.177 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e402ce3a585b06c7-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.185 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.185 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.185 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.185 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.185 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:46:47.186 DEBUG:boto:Data: 2023-09-29T22:46:47.186 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.186 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.186 DEBUG:boto:Params: {} 2023-09-29T22:46:47.186 DEBUG:boto:Token: None 2023-09-29T22:46:47.186 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87 2023-09-29T22:46:47.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QIx0xY5CVI+ctYadMerqYKL0RDI= 2023-09-29T22:46:47.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QIx0xY5CVI+ctYadMerqYKL0RDI=', 'Content-Length': '0'} 2023-09-29T22:46:47.188 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005214e5186b49ef8-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.189 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.189 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.189 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.189 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:46:47.189 DEBUG:boto:Data: 2023-09-29T22:46:47.189 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.190 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.190 DEBUG:boto:Params: {} 2023-09-29T22:46:47.190 DEBUG:boto:Token: None 2023-09-29T22:46:47.193 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87 2023-09-29T22:46:47.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QIx0xY5CVI+ctYadMerqYKL0RDI= 2023-09-29T22:46:47.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QIx0xY5CVI+ctYadMerqYKL0RDI=', 'Content-Length': '0'} 2023-09-29T22:46:47.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b840c638682b0004-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:46:47.196 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.196 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.197 DEBUG:boto:Method: GET 2023-09-29T22:46:47.197 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:46:47.197 DEBUG:boto:Data: 2023-09-29T22:46:47.197 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.197 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.197 DEBUG:boto:Params: {} 2023-09-29T22:46:47.197 DEBUG:boto:Token: None 2023-09-29T22:46:47.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87 2023-09-29T22:46:47.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IYHZtd2STRqzQyNIFP57vJ/Smbw= 2023-09-29T22:46:47.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IYHZtd2STRqzQyNIFP57vJ/Smbw=', 'Content-Length': '0'} 2023-09-29T22:46:47.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004379f7856aebe740-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.200 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:46:47.200 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:46:47.200 DEBUG:boto:Method: GET 2023-09-29T22:46:47.201 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:46:47.201 DEBUG:boto:Data: 2023-09-29T22:46:47.201 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.201 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.201 DEBUG:boto:Params: {} 2023-09-29T22:46:47.201 DEBUG:boto:Token: None 2023-09-29T22:46:47.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj87 2023-09-29T22:46:47.202 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IYHZtd2STRqzQyNIFP57vJ/Smbw= 2023-09-29T22:46:47.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IYHZtd2STRqzQyNIFP57vJ/Smbw=', 'Content-Length': '0'} 2023-09-29T22:46:47.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c593428b77c28b5-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.246 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.247 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.247 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.247 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.247 DEBUG:boto:Method: GET 2023-09-29T22:46:47.247 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.247 DEBUG:boto:Data: 2023-09-29T22:46:47.248 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.248 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.248 DEBUG:boto:Params: {} 2023-09-29T22:46:47.248 DEBUG:boto:Token: None 2023-09-29T22:46:47.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.253 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f80fc734c78384e7-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.254 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.254 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.254 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.254 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.255 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.255 DEBUG:boto:Method: GET 2023-09-29T22:46:47.255 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:46:47.255 DEBUG:boto:Data: 2023-09-29T22:46:47.255 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.255 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.255 DEBUG:boto:Params: {} 2023-09-29T22:46:47.255 DEBUG:boto:Token: None 2023-09-29T22:46:47.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006eaa4ceba2987854-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.262 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:46:47.263 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.263 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.263 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.263 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.264 DEBUG:boto:Method: GET 2023-09-29T22:46:47.264 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.264 DEBUG:boto:Data: 2023-09-29T22:46:47.264 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.264 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.264 DEBUG:boto:Params: {} 2023-09-29T22:46:47.264 DEBUG:boto:Token: None 2023-09-29T22:46:47.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc= 2023-09-29T22:46:47.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc=', 'Content-Length': '0'} 2023-09-29T22:46:47.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097902d4c16dd62f9-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.310 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.311 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.311 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.311 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.311 DEBUG:boto:Method: GET 2023-09-29T22:46:47.311 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.311 DEBUG:boto:Data: 2023-09-29T22:46:47.311 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.311 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.312 DEBUG:boto:Params: {} 2023-09-29T22:46:47.312 DEBUG:boto:Token: None 2023-09-29T22:46:47.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc= 2023-09-29T22:46:47.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc=', 'Content-Length': '0'} 2023-09-29T22:46:47.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000790015de8538f22c-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.315 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.316 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.316 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.316 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.316 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.316 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.316 DEBUG:boto:Data: 2023-09-29T22:46:47.316 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.317 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.317 DEBUG:boto:Params: {} 2023-09-29T22:46:47.317 DEBUG:boto:Token: None 2023-09-29T22:46:47.317 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rYeLMyV+ruFjTFvlgIvfqh+fPos= 2023-09-29T22:46:47.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rYeLMyV+ruFjTFvlgIvfqh+fPos=', 'Content-Length': '0'} 2023-09-29T22:46:47.320 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5500d03bfd04adc-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.320 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.320 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.320 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.320 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.320 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.320 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.321 DEBUG:boto:Data: 2023-09-29T22:46:47.321 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.321 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.321 DEBUG:boto:Params: {} 2023-09-29T22:46:47.321 DEBUG:boto:Token: None 2023-09-29T22:46:47.321 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rYeLMyV+ruFjTFvlgIvfqh+fPos= 2023-09-29T22:46:47.322 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rYeLMyV+ruFjTFvlgIvfqh+fPos=', 'Content-Length': '0'} 2023-09-29T22:46:47.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb963bda7a11e072-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:46:47.324 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.324 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.325 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.325 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.325 DEBUG:boto:Method: GET 2023-09-29T22:46:47.325 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.325 DEBUG:boto:Data: 2023-09-29T22:46:47.325 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.325 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.325 DEBUG:boto:Params: {} 2023-09-29T22:46:47.325 DEBUG:boto:Token: None 2023-09-29T22:46:47.326 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.326 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc= 2023-09-29T22:46:47.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc=', 'Content-Length': '0'} 2023-09-29T22:46:47.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d8685497f17d5bd-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.329 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.329 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.329 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.329 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.329 DEBUG:boto:Method: GET 2023-09-29T22:46:47.329 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.329 DEBUG:boto:Data: 2023-09-29T22:46:47.329 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.330 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.330 DEBUG:boto:Params: {} 2023-09-29T22:46:47.330 DEBUG:boto:Token: None 2023-09-29T22:46:47.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:46:47.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc= 2023-09-29T22:46:47.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bY+jsI2O/8jSRAeQx7hYj1ewBmc=', 'Content-Length': '0'} 2023-09-29T22:46:47.333 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019888cc541821d6e-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.374 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.375 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.375 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.375 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.375 DEBUG:boto:Method: GET 2023-09-29T22:46:47.375 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.375 DEBUG:boto:Data: 2023-09-29T22:46:47.375 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.375 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.375 DEBUG:boto:Params: {} 2023-09-29T22:46:47.376 DEBUG:boto:Token: None 2023-09-29T22:46:47.376 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.381 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2031b68ebf01fab-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.381 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.381 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.382 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.382 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.382 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.382 DEBUG:boto:Method: GET 2023-09-29T22:46:47.382 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.382 DEBUG:boto:Data: 2023-09-29T22:46:47.382 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.382 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.382 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.382 DEBUG:boto:Params: {} 2023-09-29T22:46:47.383 DEBUG:boto:Token: None 2023-09-29T22:46:47.383 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.383 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.383 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.389 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a5f24370708cc76d-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.389 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.390 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.390 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.390 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.390 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:46:47.390 DEBUG:boto:Data: 2023-09-29T22:46:47.390 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.391 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.391 DEBUG:boto:Params: {} 2023-09-29T22:46:47.391 DEBUG:boto:Token: None 2023-09-29T22:46:47.391 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92 2023-09-29T22:46:47.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yIAk6dvA5F+TKO37GE5SzDxsKeY= 2023-09-29T22:46:47.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yIAk6dvA5F+TKO37GE5SzDxsKeY=', 'Content-Length': '0'} 2023-09-29T22:46:47.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d46d59a95ca2a9d-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.394 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.394 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.394 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.394 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:46:47.394 DEBUG:boto:Data: 2023-09-29T22:46:47.394 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.395 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.395 DEBUG:boto:Params: {} 2023-09-29T22:46:47.395 DEBUG:boto:Token: None 2023-09-29T22:46:47.395 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92 2023-09-29T22:46:47.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yIAk6dvA5F+TKO37GE5SzDxsKeY= 2023-09-29T22:46:47.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yIAk6dvA5F+TKO37GE5SzDxsKeY=', 'Content-Length': '0'} 2023-09-29T22:46:47.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6f85dbf5c0189c-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:46:47.397 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.398 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.398 DEBUG:boto:Method: GET 2023-09-29T22:46:47.398 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:46:47.398 DEBUG:boto:Data: 2023-09-29T22:46:47.398 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.398 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.398 DEBUG:boto:Params: {} 2023-09-29T22:46:47.399 DEBUG:boto:Token: None 2023-09-29T22:46:47.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92 2023-09-29T22:46:47.399 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YkgJ1ExQiTtazpzoRdwoa6s2MkE= 2023-09-29T22:46:47.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YkgJ1ExQiTtazpzoRdwoa6s2MkE=', 'Content-Length': '0'} 2023-09-29T22:46:47.401 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036524cb7d10d1034-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.442 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:46:47.443 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:46:47.443 DEBUG:boto:Method: GET 2023-09-29T22:46:47.443 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:46:47.443 DEBUG:boto:Data: 2023-09-29T22:46:47.443 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.443 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.443 DEBUG:boto:Params: {} 2023-09-29T22:46:47.444 DEBUG:boto:Token: None 2023-09-29T22:46:47.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj92 2023-09-29T22:46:47.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YkgJ1ExQiTtazpzoRdwoa6s2MkE= 2023-09-29T22:46:47.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YkgJ1ExQiTtazpzoRdwoa6s2MkE=', 'Content-Length': '0'} 2023-09-29T22:46:47.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027993a5c5b289e81-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.447 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.447 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.447 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.447 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.447 DEBUG:boto:Method: GET 2023-09-29T22:46:47.447 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.447 DEBUG:boto:Data: 2023-09-29T22:46:47.448 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.448 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.448 DEBUG:boto:Params: {} 2023-09-29T22:46:47.448 DEBUG:boto:Token: None 2023-09-29T22:46:47.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5ec7086d3f107f1-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.453 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.454 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.454 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.454 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.454 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.454 DEBUG:boto:Method: GET 2023-09-29T22:46:47.454 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:46:47.454 DEBUG:boto:Data: 2023-09-29T22:46:47.454 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.455 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.455 DEBUG:boto:Params: {} 2023-09-29T22:46:47.455 DEBUG:boto:Token: None 2023-09-29T22:46:47.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.461 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006621b8f447660388-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.502 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:46:47.503 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.503 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.503 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.503 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.504 DEBUG:boto:Method: GET 2023-09-29T22:46:47.504 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.504 DEBUG:boto:Data: 2023-09-29T22:46:47.504 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.504 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.504 DEBUG:boto:Params: {} 2023-09-29T22:46:47.504 DEBUG:boto:Token: None 2023-09-29T22:46:47.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q= 2023-09-29T22:46:47.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q=', 'Content-Length': '0'} 2023-09-29T22:46:47.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d2837968223a30-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.508 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.508 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.508 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.508 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.508 DEBUG:boto:Method: GET 2023-09-29T22:46:47.509 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.509 DEBUG:boto:Data: 2023-09-29T22:46:47.509 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.509 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.509 DEBUG:boto:Params: {} 2023-09-29T22:46:47.509 DEBUG:boto:Token: None 2023-09-29T22:46:47.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q= 2023-09-29T22:46:47.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q=', 'Content-Length': '0'} 2023-09-29T22:46:47.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cf1d97600308d09-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.513 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.513 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.513 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.513 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.513 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.513 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.513 DEBUG:boto:Data: 2023-09-29T22:46:47.513 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.514 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.514 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.514 DEBUG:boto:Params: {} 2023-09-29T22:46:47.514 DEBUG:boto:Token: None 2023-09-29T22:46:47.514 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IPONPp9WFzjfc9Hzo8CdvehEZyU= 2023-09-29T22:46:47.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IPONPp9WFzjfc9Hzo8CdvehEZyU=', 'Content-Length': '0'} 2023-09-29T22:46:47.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae01a06b2191df9a-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.517 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.517 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.517 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.517 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.518 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.518 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.518 DEBUG:boto:Data: 2023-09-29T22:46:47.518 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.518 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.518 DEBUG:boto:Params: {} 2023-09-29T22:46:47.518 DEBUG:boto:Token: None 2023-09-29T22:46:47.519 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.519 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IPONPp9WFzjfc9Hzo8CdvehEZyU= 2023-09-29T22:46:47.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IPONPp9WFzjfc9Hzo8CdvehEZyU=', 'Content-Length': '0'} 2023-09-29T22:46:47.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adcb1e6a2d49c818-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:46:47.522 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.522 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.522 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.522 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.522 DEBUG:boto:Method: GET 2023-09-29T22:46:47.522 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.522 DEBUG:boto:Data: 2023-09-29T22:46:47.523 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.523 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.523 DEBUG:boto:Params: {} 2023-09-29T22:46:47.523 DEBUG:boto:Token: None 2023-09-29T22:46:47.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q= 2023-09-29T22:46:47.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q=', 'Content-Length': '0'} 2023-09-29T22:46:47.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721c33d66aaf7ccb-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.526 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.526 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.526 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.527 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.527 DEBUG:boto:Method: GET 2023-09-29T22:46:47.527 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.527 DEBUG:boto:Data: 2023-09-29T22:46:47.527 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.527 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.527 DEBUG:boto:Params: {} 2023-09-29T22:46:47.527 DEBUG:boto:Token: None 2023-09-29T22:46:47.527 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:46:47.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q= 2023-09-29T22:46:47.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFPulWbkwD8PoxX81PV805n6V/Q=', 'Content-Length': '0'} 2023-09-29T22:46:47.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c0018b684aee366-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.531 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.531 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.531 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.531 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.531 DEBUG:boto:Method: GET 2023-09-29T22:46:47.531 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.532 DEBUG:boto:Data: 2023-09-29T22:46:47.532 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.532 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.532 DEBUG:boto:Params: {} 2023-09-29T22:46:47.532 DEBUG:boto:Token: None 2023-09-29T22:46:47.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.537 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fb79799bda719b4-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.538 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.538 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.538 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.539 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.539 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.539 DEBUG:boto:Method: GET 2023-09-29T22:46:47.539 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.539 DEBUG:boto:Data: 2023-09-29T22:46:47.539 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.539 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.540 DEBUG:boto:Params: {} 2023-09-29T22:46:47.540 DEBUG:boto:Token: None 2023-09-29T22:46:47.540 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.540 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.547 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000000747a8fbc631e18-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.590 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.591 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.591 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.591 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.591 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:46:47.591 DEBUG:boto:Data: 2023-09-29T22:46:47.591 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.592 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.592 DEBUG:boto:Params: {} 2023-09-29T22:46:47.592 DEBUG:boto:Token: None 2023-09-29T22:46:47.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95 2023-09-29T22:46:47.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YLhCEpPg/YmJm0nzpMxNZlzoWko= 2023-09-29T22:46:47.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YLhCEpPg/YmJm0nzpMxNZlzoWko=', 'Content-Length': '0'} 2023-09-29T22:46:47.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae89e09f81cf4cd3-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.598 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.598 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.598 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.598 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:46:47.598 DEBUG:boto:Data: 2023-09-29T22:46:47.599 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.599 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.599 DEBUG:boto:Params: {} 2023-09-29T22:46:47.599 DEBUG:boto:Token: None 2023-09-29T22:46:47.599 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95 2023-09-29T22:46:47.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YLhCEpPg/YmJm0nzpMxNZlzoWko= 2023-09-29T22:46:47.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YLhCEpPg/YmJm0nzpMxNZlzoWko=', 'Content-Length': '0'} 2023-09-29T22:46:47.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb074c5aec32040c-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:46:47.603 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.603 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.603 DEBUG:boto:Method: GET 2023-09-29T22:46:47.603 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:46:47.603 DEBUG:boto:Data: 2023-09-29T22:46:47.603 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.604 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.604 DEBUG:boto:Params: {} 2023-09-29T22:46:47.604 DEBUG:boto:Token: None 2023-09-29T22:46:47.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95 2023-09-29T22:46:47.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xz9R6DBW202YY/hRffFkSQeRffE= 2023-09-29T22:46:47.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xz9R6DBW202YY/hRffFkSQeRffE=', 'Content-Length': '0'} 2023-09-29T22:46:47.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088c4fe2ed7059cc5-00651753d7-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.650 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:46:47.651 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:46:47.651 DEBUG:boto:Method: GET 2023-09-29T22:46:47.651 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:46:47.651 DEBUG:boto:Data: 2023-09-29T22:46:47.651 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.651 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.651 DEBUG:boto:Params: {} 2023-09-29T22:46:47.651 DEBUG:boto:Token: None 2023-09-29T22:46:47.652 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj95 2023-09-29T22:46:47.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xz9R6DBW202YY/hRffFkSQeRffE= 2023-09-29T22:46:47.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xz9R6DBW202YY/hRffFkSQeRffE=', 'Content-Length': '0'} 2023-09-29T22:46:47.655 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646cac7623abeb5c-00651753d7-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.655 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.655 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.655 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.656 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.656 DEBUG:boto:Method: GET 2023-09-29T22:46:47.656 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.656 DEBUG:boto:Data: 2023-09-29T22:46:47.656 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:46:47.656 DEBUG:boto:Port: 8001 2023-09-29T22:46:47.656 DEBUG:boto:Params: {} 2023-09-29T22:46:47.656 DEBUG:boto:Token: None 2023-09-29T22:46:47.657 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.657 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.662 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088963123d691b667-00651753d7-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.662 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.662 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.662 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.663 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.663 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.663 DEBUG:boto:Method: GET 2023-09-29T22:46:47.663 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:46:47.663 DEBUG:boto:Data: 2023-09-29T22:46:47.663 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.663 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.663 DEBUG:boto:Params: {} 2023-09-29T22:46:47.664 DEBUG:boto:Token: None 2023-09-29T22:46:47.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM= 2023-09-29T22:46:47.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/yrKMgpODZ7LUGkRjEE7amerDLM=', 'Content-Length': '0'} 2023-09-29T22:46:47.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2cedc356a7e8119-00651753d7-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.671 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:46:47.672 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:46:47.672 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:46:47.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:46:47.673 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:46:47.718 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:47.718 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:47.802 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:46:47.802 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-59 2023-09-29T22:46:47.824 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-09-29T22:46:47.940 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.941 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.941 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.941 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:47.941 DEBUG:boto:Data: 2023-09-29T22:46:47.941 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.941 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.941 DEBUG:boto:Params: {} 2023-09-29T22:46:47.942 DEBUG:boto:Token: None 2023-09-29T22:46:47.942 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.942 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XvLrRxEvOzUaOrDIqacoMhb04/g= 2023-09-29T22:46:47.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XvLrRxEvOzUaOrDIqacoMhb04/g=', 'Content-Length': '0'} 2023-09-29T22:46:47.948 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000034112a9bd5c4ddcc-00651753d7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.948 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:46:47.948 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:46:47.948 DEBUG:boto:Method: PUT 2023-09-29T22:46:47.949 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:46:47.949 DEBUG:boto:Data: 2023-09-29T22:46:47.949 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:46:47.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.949 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.949 DEBUG:boto:Params: {} 2023-09-29T22:46:47.949 DEBUG:boto:Token: None 2023-09-29T22:46:47.949 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/obj31 2023-09-29T22:46:47.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d7VZTc3yrFszDT7F83ygu6PTRyU= 2023-09-29T22:46:47.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d7VZTc3yrFszDT7F83ygu6PTRyU='} 2023-09-29T22:46:47.994 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037765b7c1907a207-00651753d7-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:47.995 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:47.995 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:47.995 DEBUG:boto:Method: HEAD 2023-09-29T22:46:47.995 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:47.995 DEBUG:boto:Data: 2023-09-29T22:46:47.995 DEBUG:boto:Headers: {} 2023-09-29T22:46:47.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:47.995 DEBUG:boto:Port: 8000 2023-09-29T22:46:47.996 DEBUG:boto:Params: {} 2023-09-29T22:46:47.996 DEBUG:boto:Token: None 2023-09-29T22:46:47.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:47 GMT /hcnphz-59/ 2023-09-29T22:46:47.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XvLrRxEvOzUaOrDIqacoMhb04/g= 2023-09-29T22:46:47.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XvLrRxEvOzUaOrDIqacoMhb04/g=', 'Content-Length': '0'} 2023-09-29T22:46:48.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064a4cc506c31ae99-00651753d7-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.006 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:46:48.006 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:46:48.007 DEBUG:boto:Method: PUT 2023-09-29T22:46:48.007 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:46:48.007 DEBUG:boto:Data: 2023-09-29T22:46:48.007 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:46:48.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.007 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.007 DEBUG:boto:Params: {} 2023-09-29T22:46:48.007 DEBUG:boto:Token: None 2023-09-29T22:46:48.007 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/obj62 2023-09-29T22:46:48.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lMPg/tl+JMUKxNI/ctaMW1h6xJI= 2023-09-29T22:46:48.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lMPg/tl+JMUKxNI/ctaMW1h6xJI='} 2023-09-29T22:46:48.018 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000490af26eb739f0f5-00651753d8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.018 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:48.018 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:48.019 DEBUG:boto:Method: HEAD 2023-09-29T22:46:48.019 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:48.019 DEBUG:boto:Data: 2023-09-29T22:46:48.019 DEBUG:boto:Headers: {} 2023-09-29T22:46:48.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.019 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.019 DEBUG:boto:Params: {} 2023-09-29T22:46:48.019 DEBUG:boto:Token: None 2023-09-29T22:46:48.019 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/ 2023-09-29T22:46:48.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:seLjuVgu3PMxW6pGdHxHV+Azd7o= 2023-09-29T22:46:48.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:seLjuVgu3PMxW6pGdHxHV+Azd7o=', 'Content-Length': '0'} 2023-09-29T22:46:48.025 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bef8d85d0aa9349b-00651753d8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.025 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:46:48.025 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:46:48.025 DEBUG:boto:Method: PUT 2023-09-29T22:46:48.025 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:46:48.026 DEBUG:boto:Data: 2023-09-29T22:46:48.026 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:46:48.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.026 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.026 DEBUG:boto:Params: {} 2023-09-29T22:46:48.026 DEBUG:boto:Token: None 2023-09-29T22:46:48.026 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/obj93 2023-09-29T22:46:48.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OziOl6QrGo5kr9McUUNlFJ/2KHs= 2023-09-29T22:46:48.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OziOl6QrGo5kr9McUUNlFJ/2KHs='} 2023-09-29T22:46:48.070 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002ec1a5a81bbf8dd1-00651753d8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.071 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:48.071 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:48.071 DEBUG:boto:Method: HEAD 2023-09-29T22:46:48.071 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:48.071 DEBUG:boto:Data: 2023-09-29T22:46:48.071 DEBUG:boto:Headers: {} 2023-09-29T22:46:48.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.072 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.072 DEBUG:boto:Params: {} 2023-09-29T22:46:48.072 DEBUG:boto:Token: None 2023-09-29T22:46:48.072 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/ 2023-09-29T22:46:48.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:seLjuVgu3PMxW6pGdHxHV+Azd7o= 2023-09-29T22:46:48.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:seLjuVgu3PMxW6pGdHxHV+Azd7o=', 'Content-Length': '0'} 2023-09-29T22:46:48.078 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fb801b464100c9fd-00651753d8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.078 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:46:48.078 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:46:48.079 DEBUG:boto:Method: PUT 2023-09-29T22:46:48.079 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:46:48.079 DEBUG:boto:Data: 2023-09-29T22:46:48.079 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:46:48.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.079 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.079 DEBUG:boto:Params: {} 2023-09-29T22:46:48.079 DEBUG:boto:Token: None 2023-09-29T22:46:48.079 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/obj124 2023-09-29T22:46:48.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QiMakfBzlzxzkBM2VsbvXZNGtKI= 2023-09-29T22:46:48.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QiMakfBzlzxzkBM2VsbvXZNGtKI='} 2023-09-29T22:46:48.090 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a2399942de16084f-00651753d8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.090 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:46:48.090 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:46:48.090 DEBUG:boto:Method: HEAD 2023-09-29T22:46:48.090 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:46:48.090 DEBUG:boto:Data: 2023-09-29T22:46:48.090 DEBUG:boto:Headers: {} 2023-09-29T22:46:48.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.091 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.091 DEBUG:boto:Params: {} 2023-09-29T22:46:48.091 DEBUG:boto:Token: None 2023-09-29T22:46:48.091 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/ 2023-09-29T22:46:48.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:seLjuVgu3PMxW6pGdHxHV+Azd7o= 2023-09-29T22:46:48.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:seLjuVgu3PMxW6pGdHxHV+Azd7o=', 'Content-Length': '0'} 2023-09-29T22:46:48.095 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5153769b381102b-00651753d8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.097 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:46:48.097 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:46:48.097 DEBUG:boto:Method: PUT 2023-09-29T22:46:48.097 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:46:48.097 DEBUG:boto:Data: 2023-09-29T22:46:48.097 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:46:48.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:46:48.097 DEBUG:boto:Port: 8000 2023-09-29T22:46:48.097 DEBUG:boto:Params: {} 2023-09-29T22:46:48.098 DEBUG:boto:Token: None 2023-09-29T22:46:48.098 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:46:48 GMT /hcnphz-59/obj155 2023-09-29T22:46:48.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/f/iZHMCvOxxltijBC0DIFKIA5s= 2023-09-29T22:46:48.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:46:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/f/iZHMCvOxxltijBC0DIFKIA5s='} 2023-09-29T22:46:48.142 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000042007bd1538107b5-00651753d8-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:46:48.143 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:46:48.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:46:48.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:46:48.143 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:46:48.195 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:46:48.196 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:46:48.266 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:48.265+0000 7f721ee96f40 1 waiting to reach latest gen 5, on 4.. 2023-09-29T22:46:53.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:53.264+0000 7f721ee96f40 1 waiting to reach latest gen 5, on 4.. 2023-09-29T22:46:58.268 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:46:58.268+0000 7f721ee96f40 1 waiting to reach latest gen 5, on 4.. 2023-09-29T22:47:03.273 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:03.272+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:03.274 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:47:03.274 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:08.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:08.276+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:08.278 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:47:08.278 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:10.197 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:47:10.192+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:47:13.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:13.280+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:13.283 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:47:13.283 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:18.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:18.288+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:18.289 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:47:18.289 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:23.293 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:23.292+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:23.294 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:47:23.294 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:28.298 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:28.296+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:28.298 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:47:28.299 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:33.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:33.299+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:33.303 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , 00000000001.75.6, , , , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , ] 2023-09-29T22:47:33.303 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:38.307 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:38.307+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:38.308 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , 00000000001.75.6, , , , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , ] 2023-09-29T22:47:38.308 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:43.313 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:43.311+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:43.313 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , 00000000001.75.6, , , , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , ] 2023-09-29T22:47:43.313 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:48.317 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:48.315+0000 7f721ee96f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:48.317 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , 00000000001.75.6, , , , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , ] 2023-09-29T22:47:48.318 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:53.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:53.319+0000 7f721ee96f40 1 bucket sync caught up with source: 2023-09-29T22:47:53.322 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:53.322 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , 00000000002.97.6, , 00000000001.75.6, , 00000000001.90.6, , , , , , , , , , 00000000001.83.6, , , , , , , , , , , , , , 00000000001.100.6] 2023-09-29T22:47:53.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:53.319+0000 7f721ee96f40 0 bucket checkpoint complete 2023-09-29T22:47:53.349 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:47:53.349 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:47:53.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:47:53.350 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:47:53.399 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:47:53.400 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:47:53.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:53.463+0000 7f75a43e0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:53.465 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000002.586.6, 00000000002.254.6, 00000000003.197.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:47:53.465 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000004.210.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000006.211.6, 00000000004.316.6] 2023-09-29T22:47:58.467 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:47:58.467+0000 7f75a43e0f40 1 waiting for incremental sync to catch up: 2023-09-29T22:47:58.468 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000002.261.6, 00000000002.586.6, 00000000002.254.6, 00000000003.197.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000005.191.6, 00000000004.316.6] 2023-09-29T22:47:58.468 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000004.210.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000006.211.6, 00000000004.316.6] 2023-09-29T22:48:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:03.467+0000 7f75a43e0f40 1 bucket sync caught up with source: 2023-09-29T22:48:03.470 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000004.210.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000006.211.6, 00000000004.316.6] 2023-09-29T22:48:03.470 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000004.210.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000006.211.6, 00000000004.316.6] 2023-09-29T22:48:03.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:03.467+0000 7f75a43e0f40 0 bucket checkpoint complete 2023-09-29T22:48:03.500 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:48:03.500 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.500 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.501 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.501 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:03.501 DEBUG:boto:Data: 2023-09-29T22:48:03.501 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.501 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.501 DEBUG:boto:Params: {} 2023-09-29T22:48:03.502 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:48:03.502 DEBUG:boto:Token: None 2023-09-29T22:48:03.502 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.502 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BFTGA7US2GQ+MnlrxSMOOJDA0JA= 2023-09-29T22:48:03.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BFTGA7US2GQ+MnlrxSMOOJDA0JA=', 'Content-Length': '0'} 2023-09-29T22:48:03.508 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f71f21395d03d6bd-0065175423-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.509 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.509 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.509 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.509 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:03.509 DEBUG:boto:Data: 2023-09-29T22:48:03.509 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.509 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.510 DEBUG:boto:Params: {} 2023-09-29T22:48:03.510 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:48:03.510 DEBUG:boto:Token: None 2023-09-29T22:48:03.510 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.510 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BFTGA7US2GQ+MnlrxSMOOJDA0JA= 2023-09-29T22:48:03.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BFTGA7US2GQ+MnlrxSMOOJDA0JA=', 'Content-Length': '0'} 2023-09-29T22:48:03.517 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000524ed72aa2e64cce-0065175423-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.517 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:48:03.517 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.517 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.518 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.518 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.518 DEBUG:boto:Method: GET 2023-09-29T22:48:03.518 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.518 DEBUG:boto:Data: 2023-09-29T22:48:03.518 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.518 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.518 DEBUG:boto:Params: {} 2023-09-29T22:48:03.519 DEBUG:boto:Token: None 2023-09-29T22:48:03.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/?versions 2023-09-29T22:48:03.519 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg= 2023-09-29T22:48:03.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg=', 'Content-Length': '0'} 2023-09-29T22:48:03.522 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4026165808a81cb-0065175423-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.522 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:48:03.524 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:48:03.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:48:03.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:48:03.527 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:48:03.527 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.527 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.528 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.528 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.528 DEBUG:boto:Method: GET 2023-09-29T22:48:03.528 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.528 DEBUG:boto:Data: 2023-09-29T22:48:03.528 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.528 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.528 DEBUG:boto:Params: {} 2023-09-29T22:48:03.528 DEBUG:boto:Token: None 2023-09-29T22:48:03.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/?versions 2023-09-29T22:48:03.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg= 2023-09-29T22:48:03.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg=', 'Content-Length': '0'} 2023-09-29T22:48:03.535 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bde78f30633ff89-0065175423-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.535 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:48:03.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:48:03.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:48:03.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:48:03.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:48:03.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:48:03.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:48:03.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:48:03.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:48:03.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:48:03.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:48:03.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:48:03.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:48:03.540 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.540 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.540 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.540 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.541 DEBUG:boto:Method: GET 2023-09-29T22:48:03.541 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.541 DEBUG:boto:Data: 2023-09-29T22:48:03.541 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.541 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.541 DEBUG:boto:Params: {} 2023-09-29T22:48:03.541 DEBUG:boto:Token: None 2023-09-29T22:48:03.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/?versions 2023-09-29T22:48:03.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg= 2023-09-29T22:48:03.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg=', 'Content-Length': '0'} 2023-09-29T22:48:03.546 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da1333fbdd2a6b05-0065175423-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.546 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.548 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.548 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.548 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.548 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.548 DEBUG:boto:Method: GET 2023-09-29T22:48:03.548 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:48:03.548 DEBUG:boto:Data: 2023-09-29T22:48:03.548 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.549 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.549 DEBUG:boto:Params: {} 2023-09-29T22:48:03.549 DEBUG:boto:Token: None 2023-09-29T22:48:03.549 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/?versions 2023-09-29T22:48:03.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg= 2023-09-29T22:48:03.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zGoTKNPxkhTBlUwBdmFHIn5gFg=', 'Content-Length': '0'} 2023-09-29T22:48:03.556 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066241f19025c793d-0065175423-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.556 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:48:03.558 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.558 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.558 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.558 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.558 DEBUG:boto:Method: GET 2023-09-29T22:48:03.558 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.558 DEBUG:boto:Data: 2023-09-29T22:48:03.558 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.558 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.559 DEBUG:boto:Params: {} 2023-09-29T22:48:03.559 DEBUG:boto:Token: None 2023-09-29T22:48:03.559 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.559 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4= 2023-09-29T22:48:03.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4=', 'Content-Length': '0'} 2023-09-29T22:48:03.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e68afe55c47b6de-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.602 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.602 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.602 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.603 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.603 DEBUG:boto:Method: GET 2023-09-29T22:48:03.603 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.603 DEBUG:boto:Data: 2023-09-29T22:48:03.603 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.603 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.603 DEBUG:boto:Params: {} 2023-09-29T22:48:03.603 DEBUG:boto:Token: None 2023-09-29T22:48:03.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4= 2023-09-29T22:48:03.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4=', 'Content-Length': '0'} 2023-09-29T22:48:03.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5e9769bcd318900-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.607 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.607 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.607 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.607 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.607 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.608 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.608 DEBUG:boto:Data: 2023-09-29T22:48:03.608 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.608 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.608 DEBUG:boto:Params: {} 2023-09-29T22:48:03.608 DEBUG:boto:Token: None 2023-09-29T22:48:03.608 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j8da8/FFIxV1wyVJfaweTUsoZGE= 2023-09-29T22:48:03.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j8da8/FFIxV1wyVJfaweTUsoZGE=', 'Content-Length': '0'} 2023-09-29T22:48:03.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b322970dfb4e98fb-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.611 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.612 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.612 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.612 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.612 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.612 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.612 DEBUG:boto:Data: 2023-09-29T22:48:03.612 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.612 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.613 DEBUG:boto:Params: {} 2023-09-29T22:48:03.613 DEBUG:boto:Token: None 2023-09-29T22:48:03.613 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.613 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j8da8/FFIxV1wyVJfaweTUsoZGE= 2023-09-29T22:48:03.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j8da8/FFIxV1wyVJfaweTUsoZGE=', 'Content-Length': '0'} 2023-09-29T22:48:03.616 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5d138193e325d9f-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:48:03.616 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.616 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.617 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.617 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.617 DEBUG:boto:Method: GET 2023-09-29T22:48:03.617 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.617 DEBUG:boto:Data: 2023-09-29T22:48:03.617 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.617 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.617 DEBUG:boto:Params: {} 2023-09-29T22:48:03.617 DEBUG:boto:Token: None 2023-09-29T22:48:03.618 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.618 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4= 2023-09-29T22:48:03.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4=', 'Content-Length': '0'} 2023-09-29T22:48:03.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92475432c8f41b4-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.621 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.621 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.621 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.621 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.621 DEBUG:boto:Method: GET 2023-09-29T22:48:03.622 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.622 DEBUG:boto:Data: 2023-09-29T22:48:03.622 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.622 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.622 DEBUG:boto:Params: {} 2023-09-29T22:48:03.622 DEBUG:boto:Token: None 2023-09-29T22:48:03.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:48:03.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4= 2023-09-29T22:48:03.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:49KhP88qhLpihOFaKo7Jp/cwLJ4=', 'Content-Length': '0'} 2023-09-29T22:48:03.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcbb495ee4b2a7d5-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.666 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.667 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.667 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.667 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.667 DEBUG:boto:Method: GET 2023-09-29T22:48:03.667 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.667 DEBUG:boto:Data: 2023-09-29T22:48:03.667 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.667 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.667 DEBUG:boto:Params: {} 2023-09-29T22:48:03.668 DEBUG:boto:Token: None 2023-09-29T22:48:03.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000491e9eefce3326c-0065175423-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.673 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.673 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.673 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.673 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.674 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.674 DEBUG:boto:Method: GET 2023-09-29T22:48:03.674 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.674 DEBUG:boto:Data: 2023-09-29T22:48:03.674 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.674 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.674 DEBUG:boto:Params: {} 2023-09-29T22:48:03.674 DEBUG:boto:Token: None 2023-09-29T22:48:03.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006fd0678df73d53fb-0065175423-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.681 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.681 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.681 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.682 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.682 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:48:03.682 DEBUG:boto:Data: 2023-09-29T22:48:03.682 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.682 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.682 DEBUG:boto:Params: {} 2023-09-29T22:48:03.682 DEBUG:boto:Token: None 2023-09-29T22:48:03.682 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11 2023-09-29T22:48:03.683 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wiuL+hutJBpahzaCi1iWxOsIkMc= 2023-09-29T22:48:03.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wiuL+hutJBpahzaCi1iWxOsIkMc=', 'Content-Length': '0'} 2023-09-29T22:48:03.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6bb73413e2c9e0f-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.685 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.685 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.686 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.686 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:48:03.686 DEBUG:boto:Data: 2023-09-29T22:48:03.686 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.686 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.686 DEBUG:boto:Params: {} 2023-09-29T22:48:03.686 DEBUG:boto:Token: None 2023-09-29T22:48:03.687 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11 2023-09-29T22:48:03.687 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wiuL+hutJBpahzaCi1iWxOsIkMc= 2023-09-29T22:48:03.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wiuL+hutJBpahzaCi1iWxOsIkMc=', 'Content-Length': '0'} 2023-09-29T22:48:03.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019e7862ff697b618-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:48:03.689 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.689 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.689 DEBUG:boto:Method: GET 2023-09-29T22:48:03.689 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:48:03.690 DEBUG:boto:Data: 2023-09-29T22:48:03.690 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.690 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.690 DEBUG:boto:Params: {} 2023-09-29T22:48:03.690 DEBUG:boto:Token: None 2023-09-29T22:48:03.690 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11 2023-09-29T22:48:03.690 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wOkI0L0iZLJxcnjH5SStHEBE4j4= 2023-09-29T22:48:03.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wOkI0L0iZLJxcnjH5SStHEBE4j4=', 'Content-Length': '0'} 2023-09-29T22:48:03.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c81b79fee7621e75-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.734 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:48:03.735 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:48:03.735 DEBUG:boto:Method: GET 2023-09-29T22:48:03.735 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:48:03.735 DEBUG:boto:Data: 2023-09-29T22:48:03.735 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.735 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.736 DEBUG:boto:Params: {} 2023-09-29T22:48:03.736 DEBUG:boto:Token: None 2023-09-29T22:48:03.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj11 2023-09-29T22:48:03.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wOkI0L0iZLJxcnjH5SStHEBE4j4= 2023-09-29T22:48:03.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wOkI0L0iZLJxcnjH5SStHEBE4j4=', 'Content-Length': '0'} 2023-09-29T22:48:03.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000311f1a161d57820a-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.739 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.739 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.739 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.739 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.739 DEBUG:boto:Method: GET 2023-09-29T22:48:03.740 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.740 DEBUG:boto:Data: 2023-09-29T22:48:03.740 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.740 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.740 DEBUG:boto:Params: {} 2023-09-29T22:48:03.740 DEBUG:boto:Token: None 2023-09-29T22:48:03.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.742 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.747 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041704671de81bb06-0065175423-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.748 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.749 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.749 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.749 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.749 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.749 DEBUG:boto:Method: GET 2023-09-29T22:48:03.750 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:48:03.750 DEBUG:boto:Data: 2023-09-29T22:48:03.750 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.750 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.750 DEBUG:boto:Params: {} 2023-09-29T22:48:03.751 DEBUG:boto:Token: None 2023-09-29T22:48:03.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.751 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062e772f106ba3a5b-0065175423-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.798 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:48:03.799 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.799 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.799 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.800 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.800 DEBUG:boto:Method: GET 2023-09-29T22:48:03.800 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.800 DEBUG:boto:Data: 2023-09-29T22:48:03.800 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.801 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.801 DEBUG:boto:Params: {} 2023-09-29T22:48:03.801 DEBUG:boto:Token: None 2023-09-29T22:48:03.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ= 2023-09-29T22:48:03.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ=', 'Content-Length': '0'} 2023-09-29T22:48:03.805 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af8d644e1935a9ce-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.805 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.805 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.805 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.806 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.806 DEBUG:boto:Method: GET 2023-09-29T22:48:03.806 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.806 DEBUG:boto:Data: 2023-09-29T22:48:03.806 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.807 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.807 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.807 DEBUG:boto:Params: {} 2023-09-29T22:48:03.807 DEBUG:boto:Token: None 2023-09-29T22:48:03.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ= 2023-09-29T22:48:03.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ=', 'Content-Length': '0'} 2023-09-29T22:48:03.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003717e026a49b7c15-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.811 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.811 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.811 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.812 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.812 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.812 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.812 DEBUG:boto:Data: 2023-09-29T22:48:03.812 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.813 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.813 DEBUG:boto:Params: {} 2023-09-29T22:48:03.813 DEBUG:boto:Token: None 2023-09-29T22:48:03.813 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vyDtkiY2HthWC8LYtLLNPSjgQgg= 2023-09-29T22:48:03.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vyDtkiY2HthWC8LYtLLNPSjgQgg=', 'Content-Length': '0'} 2023-09-29T22:48:03.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eec8b03bb83feff-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.817 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.817 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.817 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.817 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.817 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.817 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.817 DEBUG:boto:Data: 2023-09-29T22:48:03.818 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.818 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.818 DEBUG:boto:Params: {} 2023-09-29T22:48:03.818 DEBUG:boto:Token: None 2023-09-29T22:48:03.818 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.818 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vyDtkiY2HthWC8LYtLLNPSjgQgg= 2023-09-29T22:48:03.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vyDtkiY2HthWC8LYtLLNPSjgQgg=', 'Content-Length': '0'} 2023-09-29T22:48:03.821 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad592eb33eafd7bd-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:48:03.822 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.822 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.822 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.822 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.822 DEBUG:boto:Method: GET 2023-09-29T22:48:03.822 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.823 DEBUG:boto:Data: 2023-09-29T22:48:03.823 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.823 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.823 DEBUG:boto:Params: {} 2023-09-29T22:48:03.823 DEBUG:boto:Token: None 2023-09-29T22:48:03.823 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ= 2023-09-29T22:48:03.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ=', 'Content-Length': '0'} 2023-09-29T22:48:03.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031ba037e0f8f32b9-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.826 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.827 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.827 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.827 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.827 DEBUG:boto:Method: GET 2023-09-29T22:48:03.827 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.827 DEBUG:boto:Data: 2023-09-29T22:48:03.827 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.827 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.827 DEBUG:boto:Params: {} 2023-09-29T22:48:03.828 DEBUG:boto:Token: None 2023-09-29T22:48:03.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:48:03.828 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ= 2023-09-29T22:48:03.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UAnXY0dviJeXBg2p8qpThqL3FbQ=', 'Content-Length': '0'} 2023-09-29T22:48:03.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062e458a0ef5d6d92-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.831 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.832 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.832 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.832 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.832 DEBUG:boto:Method: GET 2023-09-29T22:48:03.832 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.832 DEBUG:boto:Data: 2023-09-29T22:48:03.832 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.832 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.833 DEBUG:boto:Params: {} 2023-09-29T22:48:03.833 DEBUG:boto:Token: None 2023-09-29T22:48:03.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.837 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6d4578e6232b7f3-0065175423-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.838 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.838 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.838 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.838 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.838 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.839 DEBUG:boto:Method: GET 2023-09-29T22:48:03.839 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.839 DEBUG:boto:Data: 2023-09-29T22:48:03.839 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.839 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.839 DEBUG:boto:Params: {} 2023-09-29T22:48:03.839 DEBUG:boto:Token: None 2023-09-29T22:48:03.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000506a9b49cc491a79-0065175423-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.890 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.891 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.891 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.891 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.891 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:48:03.892 DEBUG:boto:Data: 2023-09-29T22:48:03.892 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.892 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.892 DEBUG:boto:Params: {} 2023-09-29T22:48:03.892 DEBUG:boto:Token: None 2023-09-29T22:48:03.893 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115 2023-09-29T22:48:03.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KSyZOJRhwtNPC1rSdXo4FBuYnuw= 2023-09-29T22:48:03.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KSyZOJRhwtNPC1rSdXo4FBuYnuw=', 'Content-Length': '0'} 2023-09-29T22:48:03.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b0221ff2604522e-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.896 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.896 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.896 DEBUG:boto:Method: HEAD 2023-09-29T22:48:03.896 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:48:03.896 DEBUG:boto:Data: 2023-09-29T22:48:03.896 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.896 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.897 DEBUG:boto:Params: {} 2023-09-29T22:48:03.897 DEBUG:boto:Token: None 2023-09-29T22:48:03.897 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115 2023-09-29T22:48:03.897 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KSyZOJRhwtNPC1rSdXo4FBuYnuw= 2023-09-29T22:48:03.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KSyZOJRhwtNPC1rSdXo4FBuYnuw=', 'Content-Length': '0'} 2023-09-29T22:48:03.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a00efc486f70762d-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:48:03.900 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.900 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.900 DEBUG:boto:Method: GET 2023-09-29T22:48:03.900 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:48:03.901 DEBUG:boto:Data: 2023-09-29T22:48:03.901 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.901 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.901 DEBUG:boto:Params: {} 2023-09-29T22:48:03.901 DEBUG:boto:Token: None 2023-09-29T22:48:03.901 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115 2023-09-29T22:48:03.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tvqSHg1JMuIMhDphTqlWu1vYOUA= 2023-09-29T22:48:03.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tvqSHg1JMuIMhDphTqlWu1vYOUA=', 'Content-Length': '0'} 2023-09-29T22:48:03.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f618405f8c1aac2-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.950 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:48:03.951 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:48:03.951 DEBUG:boto:Method: GET 2023-09-29T22:48:03.951 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:48:03.951 DEBUG:boto:Data: 2023-09-29T22:48:03.951 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.952 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.952 DEBUG:boto:Params: {} 2023-09-29T22:48:03.952 DEBUG:boto:Token: None 2023-09-29T22:48:03.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj115 2023-09-29T22:48:03.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tvqSHg1JMuIMhDphTqlWu1vYOUA= 2023-09-29T22:48:03.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tvqSHg1JMuIMhDphTqlWu1vYOUA=', 'Content-Length': '0'} 2023-09-29T22:48:03.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cee53cff1c85b82-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.955 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.955 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.955 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.955 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.955 DEBUG:boto:Method: GET 2023-09-29T22:48:03.956 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.956 DEBUG:boto:Data: 2023-09-29T22:48:03.956 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.956 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.956 DEBUG:boto:Params: {} 2023-09-29T22:48:03.956 DEBUG:boto:Token: None 2023-09-29T22:48:03.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.961 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000056539ed8be7654df-0065175423-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.961 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.962 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:03.962 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:03.962 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.962 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.962 DEBUG:boto:Method: GET 2023-09-29T22:48:03.962 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:48:03.963 DEBUG:boto:Data: 2023-09-29T22:48:03.963 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.963 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.963 DEBUG:boto:Params: {} 2023-09-29T22:48:03.963 DEBUG:boto:Token: None 2023-09-29T22:48:03.963 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/ 2023-09-29T22:48:03.963 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk= 2023-09-29T22:48:03.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u99WBVCG5HMlcrvIgHyn52mtTxk=', 'Content-Length': '0'} 2023-09-29T22:48:03.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082a07fa3e3b7e14a-0065175423-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.971 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:03.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:48:03.972 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:03.972 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:03.972 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.972 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.972 DEBUG:boto:Method: GET 2023-09-29T22:48:03.973 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.973 DEBUG:boto:Data: 2023-09-29T22:48:03.973 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:03.973 DEBUG:boto:Port: 8001 2023-09-29T22:48:03.973 DEBUG:boto:Params: {} 2023-09-29T22:48:03.974 DEBUG:boto:Token: None 2023-09-29T22:48:03.974 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.974 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tjXQjDBuoEIblIt/cNXHUmPQsAU= 2023-09-29T22:48:03.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tjXQjDBuoEIblIt/cNXHUmPQsAU=', 'Content-Length': '0'} 2023-09-29T22:48:03.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a5b6ad68619543f-0065175423-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:03.977 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:03.977 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:03.977 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.978 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.978 DEBUG:boto:Method: GET 2023-09-29T22:48:03.978 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.978 DEBUG:boto:Data: 2023-09-29T22:48:03.978 DEBUG:boto:Headers: {} 2023-09-29T22:48:03.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:03.979 DEBUG:boto:Port: 8000 2023-09-29T22:48:03.979 DEBUG:boto:Params: {} 2023-09-29T22:48:03.979 DEBUG:boto:Token: None 2023-09-29T22:48:03.979 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:03 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:03.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tjXQjDBuoEIblIt/cNXHUmPQsAU= 2023-09-29T22:48:03.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tjXQjDBuoEIblIt/cNXHUmPQsAU=', 'Content-Length': '0'} 2023-09-29T22:48:03.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002057b84abcaa0981-0065175423-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.022 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.023 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.023 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.023 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.023 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.023 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.023 DEBUG:boto:Data: 2023-09-29T22:48:04.023 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.023 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.024 DEBUG:boto:Params: {} 2023-09-29T22:48:04.024 DEBUG:boto:Token: None 2023-09-29T22:48:04.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1iWpXxhFIGiTIeNO7mSEOQl7Dz8= 2023-09-29T22:48:04.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1iWpXxhFIGiTIeNO7mSEOQl7Dz8=', 'Content-Length': '0'} 2023-09-29T22:48:04.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000017978b185dfc186e-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.027 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.027 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.027 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.028 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.028 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.028 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.028 DEBUG:boto:Data: 2023-09-29T22:48:04.028 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.029 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.029 DEBUG:boto:Params: {} 2023-09-29T22:48:04.029 DEBUG:boto:Token: None 2023-09-29T22:48:04.029 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1iWpXxhFIGiTIeNO7mSEOQl7Dz8= 2023-09-29T22:48:04.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1iWpXxhFIGiTIeNO7mSEOQl7Dz8=', 'Content-Length': '0'} 2023-09-29T22:48:04.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015d75e63058e49ab-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:48:04.033 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.033 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.033 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.033 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.034 DEBUG:boto:Method: GET 2023-09-29T22:48:04.034 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.034 DEBUG:boto:Data: 2023-09-29T22:48:04.034 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.035 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.035 DEBUG:boto:Params: {} 2023-09-29T22:48:04.035 DEBUG:boto:Token: None 2023-09-29T22:48:04.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HrlUA/4s9AFhY8WFtSQSEdEFA8o= 2023-09-29T22:48:04.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HrlUA/4s9AFhY8WFtSQSEdEFA8o=', 'Content-Length': '0'} 2023-09-29T22:48:04.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b5b6a208fbca2c9-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.082 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.082 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.083 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.083 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.083 DEBUG:boto:Method: GET 2023-09-29T22:48:04.083 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.083 DEBUG:boto:Data: 2023-09-29T22:48:04.083 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.084 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.084 DEBUG:boto:Params: {} 2023-09-29T22:48:04.084 DEBUG:boto:Token: None 2023-09-29T22:48:04.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:48:04.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HrlUA/4s9AFhY8WFtSQSEdEFA8o= 2023-09-29T22:48:04.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HrlUA/4s9AFhY8WFtSQSEdEFA8o=', 'Content-Length': '0'} 2023-09-29T22:48:04.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000663e2ff1e084216c-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.088 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.088 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.088 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.088 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.089 DEBUG:boto:Method: GET 2023-09-29T22:48:04.089 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.089 DEBUG:boto:Data: 2023-09-29T22:48:04.089 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.089 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.090 DEBUG:boto:Params: {} 2023-09-29T22:48:04.090 DEBUG:boto:Token: None 2023-09-29T22:48:04.090 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.090 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048d249b42881bb52-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.095 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.095 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.096 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.096 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.096 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.096 DEBUG:boto:Method: GET 2023-09-29T22:48:04.096 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.096 DEBUG:boto:Data: 2023-09-29T22:48:04.097 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.097 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.097 DEBUG:boto:Params: {} 2023-09-29T22:48:04.097 DEBUG:boto:Token: None 2023-09-29T22:48:04.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.105 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b860ce69d59eb2f4-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.105 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.106 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.106 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.106 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.107 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:48:04.107 DEBUG:boto:Data: 2023-09-29T22:48:04.107 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.107 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.108 DEBUG:boto:Params: {} 2023-09-29T22:48:04.108 DEBUG:boto:Token: None 2023-09-29T22:48:04.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116 2023-09-29T22:48:04.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s9DOpOVch8u3l15Ww27++cId3Kc= 2023-09-29T22:48:04.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s9DOpOVch8u3l15Ww27++cId3Kc=', 'Content-Length': '0'} 2023-09-29T22:48:04.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003609d95ba913a352-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.112 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.112 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.112 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.112 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:48:04.112 DEBUG:boto:Data: 2023-09-29T22:48:04.113 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.113 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.113 DEBUG:boto:Params: {} 2023-09-29T22:48:04.114 DEBUG:boto:Token: None 2023-09-29T22:48:04.114 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116 2023-09-29T22:48:04.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s9DOpOVch8u3l15Ww27++cId3Kc= 2023-09-29T22:48:04.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s9DOpOVch8u3l15Ww27++cId3Kc=', 'Content-Length': '0'} 2023-09-29T22:48:04.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c60841c17822b17-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:48:04.117 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.118 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.118 DEBUG:boto:Method: GET 2023-09-29T22:48:04.118 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:48:04.118 DEBUG:boto:Data: 2023-09-29T22:48:04.119 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.119 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.119 DEBUG:boto:Params: {} 2023-09-29T22:48:04.120 DEBUG:boto:Token: None 2023-09-29T22:48:04.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116 2023-09-29T22:48:04.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DoTE0dV3Ofq5HYTGltR28NdCxX8= 2023-09-29T22:48:04.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DoTE0dV3Ofq5HYTGltR28NdCxX8=', 'Content-Length': '0'} 2023-09-29T22:48:04.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d90e96b3357e8bd-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.166 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:48:04.167 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:48:04.167 DEBUG:boto:Method: GET 2023-09-29T22:48:04.167 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:48:04.167 DEBUG:boto:Data: 2023-09-29T22:48:04.168 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.168 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.168 DEBUG:boto:Params: {} 2023-09-29T22:48:04.168 DEBUG:boto:Token: None 2023-09-29T22:48:04.169 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj116 2023-09-29T22:48:04.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DoTE0dV3Ofq5HYTGltR28NdCxX8= 2023-09-29T22:48:04.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DoTE0dV3Ofq5HYTGltR28NdCxX8=', 'Content-Length': '0'} 2023-09-29T22:48:04.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6e72c3eaa7ad69b-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.172 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.172 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.173 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.173 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.173 DEBUG:boto:Method: GET 2023-09-29T22:48:04.173 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.173 DEBUG:boto:Data: 2023-09-29T22:48:04.174 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.174 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.174 DEBUG:boto:Params: {} 2023-09-29T22:48:04.175 DEBUG:boto:Token: None 2023-09-29T22:48:04.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.175 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008abc0d7a5543eec2-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.179 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.180 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.180 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.180 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.181 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.181 DEBUG:boto:Method: GET 2023-09-29T22:48:04.181 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:48:04.181 DEBUG:boto:Data: 2023-09-29T22:48:04.181 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.182 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.182 DEBUG:boto:Params: {} 2023-09-29T22:48:04.182 DEBUG:boto:Token: None 2023-09-29T22:48:04.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005efd61ee9e5d041e-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.190 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:48:04.190 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.191 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.191 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.191 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.191 DEBUG:boto:Method: GET 2023-09-29T22:48:04.192 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.192 DEBUG:boto:Data: 2023-09-29T22:48:04.192 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.192 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.193 DEBUG:boto:Params: {} 2023-09-29T22:48:04.193 DEBUG:boto:Token: None 2023-09-29T22:48:04.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54= 2023-09-29T22:48:04.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54=', 'Content-Length': '0'} 2023-09-29T22:48:04.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a3fc1e516a78df3-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.196 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.197 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.197 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.197 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.197 DEBUG:boto:Method: GET 2023-09-29T22:48:04.198 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.198 DEBUG:boto:Data: 2023-09-29T22:48:04.198 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.199 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.199 DEBUG:boto:Params: {} 2023-09-29T22:48:04.199 DEBUG:boto:Token: None 2023-09-29T22:48:04.199 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54= 2023-09-29T22:48:04.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54=', 'Content-Length': '0'} 2023-09-29T22:48:04.202 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000012e766fce097912c-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.203 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.203 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.203 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.203 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.204 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.204 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.204 DEBUG:boto:Data: 2023-09-29T22:48:04.204 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.205 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.205 DEBUG:boto:Params: {} 2023-09-29T22:48:04.205 DEBUG:boto:Token: None 2023-09-29T22:48:04.206 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.206 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UxAXmOfFDAJ89wRUyuKj6kWv/HI= 2023-09-29T22:48:04.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UxAXmOfFDAJ89wRUyuKj6kWv/HI=', 'Content-Length': '0'} 2023-09-29T22:48:04.208 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a075b0c49b81376-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.209 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.209 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.209 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.210 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.210 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.210 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.210 DEBUG:boto:Data: 2023-09-29T22:48:04.210 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.211 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.211 DEBUG:boto:Params: {} 2023-09-29T22:48:04.212 DEBUG:boto:Token: None 2023-09-29T22:48:04.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UxAXmOfFDAJ89wRUyuKj6kWv/HI= 2023-09-29T22:48:04.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UxAXmOfFDAJ89wRUyuKj6kWv/HI=', 'Content-Length': '0'} 2023-09-29T22:48:04.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8cf57db05e7410f-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:48:04.216 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.216 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.216 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.216 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.216 DEBUG:boto:Method: GET 2023-09-29T22:48:04.217 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.217 DEBUG:boto:Data: 2023-09-29T22:48:04.217 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.218 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.218 DEBUG:boto:Params: {} 2023-09-29T22:48:04.218 DEBUG:boto:Token: None 2023-09-29T22:48:04.218 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.218 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54= 2023-09-29T22:48:04.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54=', 'Content-Length': '0'} 2023-09-29T22:48:04.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a415df741fada907-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.222 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.222 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.222 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.223 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.223 DEBUG:boto:Method: GET 2023-09-29T22:48:04.223 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.223 DEBUG:boto:Data: 2023-09-29T22:48:04.223 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.223 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.223 DEBUG:boto:Params: {} 2023-09-29T22:48:04.224 DEBUG:boto:Token: None 2023-09-29T22:48:04.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:48:04.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54= 2023-09-29T22:48:04.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2fQnv5OzgbIkySlXUelqQHJcs54=', 'Content-Length': '0'} 2023-09-29T22:48:04.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013a21f29a2cf9448-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.270 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.271 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.271 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.271 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.271 DEBUG:boto:Method: GET 2023-09-29T22:48:04.272 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.272 DEBUG:boto:Data: 2023-09-29T22:48:04.272 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.272 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.273 DEBUG:boto:Params: {} 2023-09-29T22:48:04.273 DEBUG:boto:Token: None 2023-09-29T22:48:04.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.279 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000153f51d98b304cd0-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.279 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.280 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.280 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.280 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.281 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.281 DEBUG:boto:Method: GET 2023-09-29T22:48:04.281 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.281 DEBUG:boto:Data: 2023-09-29T22:48:04.281 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.282 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.282 DEBUG:boto:Params: {} 2023-09-29T22:48:04.282 DEBUG:boto:Token: None 2023-09-29T22:48:04.283 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.283 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.290 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018635c846510e677-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.290 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.291 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.291 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.291 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.291 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:48:04.291 DEBUG:boto:Data: 2023-09-29T22:48:04.292 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.292 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.292 DEBUG:boto:Params: {} 2023-09-29T22:48:04.292 DEBUG:boto:Token: None 2023-09-29T22:48:04.293 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124 2023-09-29T22:48:04.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A+dWbsEJwyze9MTZ0/6sEbcRGT4= 2023-09-29T22:48:04.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A+dWbsEJwyze9MTZ0/6sEbcRGT4=', 'Content-Length': '0'} 2023-09-29T22:48:04.295 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000157085a746ebb08f-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.296 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.296 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.296 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.296 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:48:04.296 DEBUG:boto:Data: 2023-09-29T22:48:04.297 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.297 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.297 DEBUG:boto:Params: {} 2023-09-29T22:48:04.298 DEBUG:boto:Token: None 2023-09-29T22:48:04.298 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124 2023-09-29T22:48:04.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A+dWbsEJwyze9MTZ0/6sEbcRGT4= 2023-09-29T22:48:04.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A+dWbsEJwyze9MTZ0/6sEbcRGT4=', 'Content-Length': '0'} 2023-09-29T22:48:04.301 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3e37d4688e0a34c-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:48:04.301 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.301 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.302 DEBUG:boto:Method: GET 2023-09-29T22:48:04.302 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:48:04.302 DEBUG:boto:Data: 2023-09-29T22:48:04.302 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.303 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.303 DEBUG:boto:Params: {} 2023-09-29T22:48:04.303 DEBUG:boto:Token: None 2023-09-29T22:48:04.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124 2023-09-29T22:48:04.304 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5NLK3TtymCsncTc7PrFmbx4jFwQ= 2023-09-29T22:48:04.304 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5NLK3TtymCsncTc7PrFmbx4jFwQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055e719c216fd16c9-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.346 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:48:04.347 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:48:04.347 DEBUG:boto:Method: GET 2023-09-29T22:48:04.347 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:48:04.347 DEBUG:boto:Data: 2023-09-29T22:48:04.347 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.348 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.348 DEBUG:boto:Params: {} 2023-09-29T22:48:04.348 DEBUG:boto:Token: None 2023-09-29T22:48:04.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj124 2023-09-29T22:48:04.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5NLK3TtymCsncTc7PrFmbx4jFwQ= 2023-09-29T22:48:04.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5NLK3TtymCsncTc7PrFmbx4jFwQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e486e88097fcb330-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.353 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.353 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.353 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.354 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.354 DEBUG:boto:Method: GET 2023-09-29T22:48:04.354 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.354 DEBUG:boto:Data: 2023-09-29T22:48:04.354 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.355 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.355 DEBUG:boto:Params: {} 2023-09-29T22:48:04.355 DEBUG:boto:Token: None 2023-09-29T22:48:04.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee00ea2d441c7e45-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.361 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.361 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.362 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.362 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.362 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.362 DEBUG:boto:Method: GET 2023-09-29T22:48:04.362 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:48:04.363 DEBUG:boto:Data: 2023-09-29T22:48:04.363 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.363 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.363 DEBUG:boto:Params: {} 2023-09-29T22:48:04.364 DEBUG:boto:Token: None 2023-09-29T22:48:04.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d629da5052d3b8ef-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.414 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:48:04.415 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.415 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.415 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.415 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.416 DEBUG:boto:Method: GET 2023-09-29T22:48:04.416 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.416 DEBUG:boto:Data: 2023-09-29T22:48:04.416 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.416 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.416 DEBUG:boto:Params: {} 2023-09-29T22:48:04.417 DEBUG:boto:Token: None 2023-09-29T22:48:04.417 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M= 2023-09-29T22:48:04.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M=', 'Content-Length': '0'} 2023-09-29T22:48:04.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d65c2d6121e12d67-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.421 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.421 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.421 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.421 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.421 DEBUG:boto:Method: GET 2023-09-29T22:48:04.421 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.421 DEBUG:boto:Data: 2023-09-29T22:48:04.421 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.422 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.422 DEBUG:boto:Params: {} 2023-09-29T22:48:04.422 DEBUG:boto:Token: None 2023-09-29T22:48:04.422 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M= 2023-09-29T22:48:04.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M=', 'Content-Length': '0'} 2023-09-29T22:48:04.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3d7ac2d39a0548-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.425 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.425 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.426 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.426 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.426 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.426 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.426 DEBUG:boto:Data: 2023-09-29T22:48:04.426 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.426 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.427 DEBUG:boto:Params: {} 2023-09-29T22:48:04.427 DEBUG:boto:Token: None 2023-09-29T22:48:04.427 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9RNJ8FzBLfbFVEXEnUn1PmwXGrQ= 2023-09-29T22:48:04.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9RNJ8FzBLfbFVEXEnUn1PmwXGrQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce15be47b2aafb0b-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.430 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.430 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.430 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.430 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.430 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.431 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.431 DEBUG:boto:Data: 2023-09-29T22:48:04.431 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.431 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.431 DEBUG:boto:Params: {} 2023-09-29T22:48:04.431 DEBUG:boto:Token: None 2023-09-29T22:48:04.432 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9RNJ8FzBLfbFVEXEnUn1PmwXGrQ= 2023-09-29T22:48:04.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9RNJ8FzBLfbFVEXEnUn1PmwXGrQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a5e1ffa11925fee1-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:48:04.435 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.435 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.435 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.435 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.435 DEBUG:boto:Method: GET 2023-09-29T22:48:04.436 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.436 DEBUG:boto:Data: 2023-09-29T22:48:04.436 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.436 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.436 DEBUG:boto:Params: {} 2023-09-29T22:48:04.437 DEBUG:boto:Token: None 2023-09-29T22:48:04.437 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.437 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M= 2023-09-29T22:48:04.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M=', 'Content-Length': '0'} 2023-09-29T22:48:04.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f691be7dda9f5979-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.440 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.440 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.440 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.441 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.441 DEBUG:boto:Method: GET 2023-09-29T22:48:04.441 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.441 DEBUG:boto:Data: 2023-09-29T22:48:04.441 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.441 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.441 DEBUG:boto:Params: {} 2023-09-29T22:48:04.442 DEBUG:boto:Token: None 2023-09-29T22:48:04.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:48:04.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M= 2023-09-29T22:48:04.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PG9mWHXT+UHu851yLWm51UGkV2M=', 'Content-Length': '0'} 2023-09-29T22:48:04.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ac1936a8a64dbd9-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.445 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.445 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.445 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.445 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.446 DEBUG:boto:Method: GET 2023-09-29T22:48:04.446 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.446 DEBUG:boto:Data: 2023-09-29T22:48:04.446 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.446 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.446 DEBUG:boto:Params: {} 2023-09-29T22:48:04.446 DEBUG:boto:Token: None 2023-09-29T22:48:04.446 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d2e95678deb652ce-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.494 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.495 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.495 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.495 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.495 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.495 DEBUG:boto:Method: GET 2023-09-29T22:48:04.496 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.496 DEBUG:boto:Data: 2023-09-29T22:48:04.496 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.496 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.496 DEBUG:boto:Params: {} 2023-09-29T22:48:04.496 DEBUG:boto:Token: None 2023-09-29T22:48:04.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b44dc7c16bb68891-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.505 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.505 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.505 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.505 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.506 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:48:04.506 DEBUG:boto:Data: 2023-09-29T22:48:04.506 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.506 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.506 DEBUG:boto:Params: {} 2023-09-29T22:48:04.506 DEBUG:boto:Token: None 2023-09-29T22:48:04.506 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145 2023-09-29T22:48:04.506 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9pK9QE01hgmbQvBkIr3lD0KcJcs= 2023-09-29T22:48:04.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9pK9QE01hgmbQvBkIr3lD0KcJcs=', 'Content-Length': '0'} 2023-09-29T22:48:04.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7c6a39aa5261371-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.509 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.509 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.509 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.509 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:48:04.509 DEBUG:boto:Data: 2023-09-29T22:48:04.509 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.510 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.510 DEBUG:boto:Params: {} 2023-09-29T22:48:04.510 DEBUG:boto:Token: None 2023-09-29T22:48:04.510 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145 2023-09-29T22:48:04.510 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9pK9QE01hgmbQvBkIr3lD0KcJcs= 2023-09-29T22:48:04.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9pK9QE01hgmbQvBkIr3lD0KcJcs=', 'Content-Length': '0'} 2023-09-29T22:48:04.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b14cc2bd8908e1-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:48:04.512 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.512 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.512 DEBUG:boto:Method: GET 2023-09-29T22:48:04.513 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:48:04.513 DEBUG:boto:Data: 2023-09-29T22:48:04.513 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.513 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.513 DEBUG:boto:Params: {} 2023-09-29T22:48:04.513 DEBUG:boto:Token: None 2023-09-29T22:48:04.513 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145 2023-09-29T22:48:04.513 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tLZpSwbEKyYHc1d4OZJ+0BE1dVI= 2023-09-29T22:48:04.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tLZpSwbEKyYHc1d4OZJ+0BE1dVI=', 'Content-Length': '0'} 2023-09-29T22:48:04.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e009535d88820cf-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.516 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:48:04.516 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:48:04.516 DEBUG:boto:Method: GET 2023-09-29T22:48:04.516 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:48:04.516 DEBUG:boto:Data: 2023-09-29T22:48:04.516 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.517 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.517 DEBUG:boto:Params: {} 2023-09-29T22:48:04.517 DEBUG:boto:Token: None 2023-09-29T22:48:04.517 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj145 2023-09-29T22:48:04.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tLZpSwbEKyYHc1d4OZJ+0BE1dVI= 2023-09-29T22:48:04.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tLZpSwbEKyYHc1d4OZJ+0BE1dVI=', 'Content-Length': '0'} 2023-09-29T22:48:04.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000624c828590e9b829-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.562 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.563 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.563 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.563 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.563 DEBUG:boto:Method: GET 2023-09-29T22:48:04.563 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.563 DEBUG:boto:Data: 2023-09-29T22:48:04.563 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.563 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.564 DEBUG:boto:Params: {} 2023-09-29T22:48:04.564 DEBUG:boto:Token: None 2023-09-29T22:48:04.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000adcb3dbebd5b1405-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.568 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.569 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.569 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.569 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.569 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.569 DEBUG:boto:Method: GET 2023-09-29T22:48:04.570 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:48:04.570 DEBUG:boto:Data: 2023-09-29T22:48:04.570 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.570 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.570 DEBUG:boto:Params: {} 2023-09-29T22:48:04.570 DEBUG:boto:Token: None 2023-09-29T22:48:04.570 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.570 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004187378d22847f91-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.579 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:48:04.579 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.579 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.579 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.579 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.580 DEBUG:boto:Method: GET 2023-09-29T22:48:04.580 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.580 DEBUG:boto:Data: 2023-09-29T22:48:04.580 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.580 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.580 DEBUG:boto:Params: {} 2023-09-29T22:48:04.580 DEBUG:boto:Token: None 2023-09-29T22:48:04.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A= 2023-09-29T22:48:04.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A=', 'Content-Length': '0'} 2023-09-29T22:48:04.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001998346ec8d450fd-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.583 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.583 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.583 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.583 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.584 DEBUG:boto:Method: GET 2023-09-29T22:48:04.584 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.584 DEBUG:boto:Data: 2023-09-29T22:48:04.584 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.584 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.584 DEBUG:boto:Params: {} 2023-09-29T22:48:04.584 DEBUG:boto:Token: None 2023-09-29T22:48:04.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A= 2023-09-29T22:48:04.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A=', 'Content-Length': '0'} 2023-09-29T22:48:04.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e11f4fee190b5cc-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.587 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.587 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.587 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.587 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.587 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.587 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.587 DEBUG:boto:Data: 2023-09-29T22:48:04.587 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.588 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.588 DEBUG:boto:Params: {} 2023-09-29T22:48:04.588 DEBUG:boto:Token: None 2023-09-29T22:48:04.588 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zKhlpVMBGvoYmqr2eLj/TB7IRQ= 2023-09-29T22:48:04.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zKhlpVMBGvoYmqr2eLj/TB7IRQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052a480cb9176b8b0-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.590 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.591 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.591 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.591 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.591 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.591 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.591 DEBUG:boto:Data: 2023-09-29T22:48:04.591 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.591 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.592 DEBUG:boto:Params: {} 2023-09-29T22:48:04.592 DEBUG:boto:Token: None 2023-09-29T22:48:04.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8zKhlpVMBGvoYmqr2eLj/TB7IRQ= 2023-09-29T22:48:04.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8zKhlpVMBGvoYmqr2eLj/TB7IRQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3fd3e7f7e3204c1-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:48:04.594 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.594 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.594 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.595 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.595 DEBUG:boto:Method: GET 2023-09-29T22:48:04.595 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.595 DEBUG:boto:Data: 2023-09-29T22:48:04.595 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.595 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.595 DEBUG:boto:Params: {} 2023-09-29T22:48:04.595 DEBUG:boto:Token: None 2023-09-29T22:48:04.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A= 2023-09-29T22:48:04.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A=', 'Content-Length': '0'} 2023-09-29T22:48:04.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000870b11e1b35d6fa-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.638 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.638 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.639 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.639 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.639 DEBUG:boto:Method: GET 2023-09-29T22:48:04.639 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.639 DEBUG:boto:Data: 2023-09-29T22:48:04.639 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.639 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.639 DEBUG:boto:Params: {} 2023-09-29T22:48:04.640 DEBUG:boto:Token: None 2023-09-29T22:48:04.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:48:04.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A= 2023-09-29T22:48:04.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ThIGN1DrfpVQTJQWA+PfEnngA3A=', 'Content-Length': '0'} 2023-09-29T22:48:04.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031565b19739517c6-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.643 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.643 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.644 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.644 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.644 DEBUG:boto:Method: GET 2023-09-29T22:48:04.644 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.644 DEBUG:boto:Data: 2023-09-29T22:48:04.644 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.644 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.644 DEBUG:boto:Params: {} 2023-09-29T22:48:04.644 DEBUG:boto:Token: None 2023-09-29T22:48:04.645 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.649 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085bec883c9c3953d-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.650 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.650 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.650 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.650 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.650 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.651 DEBUG:boto:Method: GET 2023-09-29T22:48:04.651 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.651 DEBUG:boto:Data: 2023-09-29T22:48:04.651 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.651 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.651 DEBUG:boto:Params: {} 2023-09-29T22:48:04.651 DEBUG:boto:Token: None 2023-09-29T22:48:04.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.658 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc841d78b9489099-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.659 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.659 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.659 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.659 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.659 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:48:04.660 DEBUG:boto:Data: 2023-09-29T22:48:04.660 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.660 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.660 DEBUG:boto:Params: {} 2023-09-29T22:48:04.660 DEBUG:boto:Token: None 2023-09-29T22:48:04.660 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155 2023-09-29T22:48:04.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D0zQVcDiCKAOsc/9m8CWO8p+vqQ= 2023-09-29T22:48:04.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D0zQVcDiCKAOsc/9m8CWO8p+vqQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.662 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003462da27ba364098-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.663 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.663 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.663 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.663 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:48:04.663 DEBUG:boto:Data: 2023-09-29T22:48:04.663 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.664 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.664 DEBUG:boto:Params: {} 2023-09-29T22:48:04.664 DEBUG:boto:Token: None 2023-09-29T22:48:04.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155 2023-09-29T22:48:04.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D0zQVcDiCKAOsc/9m8CWO8p+vqQ= 2023-09-29T22:48:04.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D0zQVcDiCKAOsc/9m8CWO8p+vqQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.666 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020c87e1c127d6d04-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:48:04.666 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.667 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.667 DEBUG:boto:Method: GET 2023-09-29T22:48:04.667 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:48:04.667 DEBUG:boto:Data: 2023-09-29T22:48:04.667 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.667 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.667 DEBUG:boto:Params: {} 2023-09-29T22:48:04.667 DEBUG:boto:Token: None 2023-09-29T22:48:04.668 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155 2023-09-29T22:48:04.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:spNZKAmdswz3YtVzTHPYIkbzGRI= 2023-09-29T22:48:04.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:spNZKAmdswz3YtVzTHPYIkbzGRI=', 'Content-Length': '0'} 2023-09-29T22:48:04.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004da17ddd1417cf1b-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.714 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:48:04.714 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:48:04.715 DEBUG:boto:Method: GET 2023-09-29T22:48:04.715 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:48:04.715 DEBUG:boto:Data: 2023-09-29T22:48:04.715 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.715 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.715 DEBUG:boto:Params: {} 2023-09-29T22:48:04.715 DEBUG:boto:Token: None 2023-09-29T22:48:04.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj155 2023-09-29T22:48:04.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:spNZKAmdswz3YtVzTHPYIkbzGRI= 2023-09-29T22:48:04.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:spNZKAmdswz3YtVzTHPYIkbzGRI=', 'Content-Length': '0'} 2023-09-29T22:48:04.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000350b0ca47caa8ec1-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.719 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.719 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.719 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.719 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.719 DEBUG:boto:Method: GET 2023-09-29T22:48:04.719 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.719 DEBUG:boto:Data: 2023-09-29T22:48:04.720 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.720 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.720 DEBUG:boto:Params: {} 2023-09-29T22:48:04.720 DEBUG:boto:Token: None 2023-09-29T22:48:04.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.725 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f7f9746c54c67aa-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.725 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.726 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.726 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.726 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.726 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.726 DEBUG:boto:Method: GET 2023-09-29T22:48:04.726 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:48:04.727 DEBUG:boto:Data: 2023-09-29T22:48:04.727 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.727 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.727 DEBUG:boto:Params: {} 2023-09-29T22:48:04.727 DEBUG:boto:Token: None 2023-09-29T22:48:04.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.734 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b22a5c752d915cc2-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.778 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:48:04.779 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.779 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.779 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.779 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.779 DEBUG:boto:Method: GET 2023-09-29T22:48:04.780 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.780 DEBUG:boto:Data: 2023-09-29T22:48:04.780 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.780 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.780 DEBUG:boto:Params: {} 2023-09-29T22:48:04.780 DEBUG:boto:Token: None 2023-09-29T22:48:04.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc= 2023-09-29T22:48:04.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc=', 'Content-Length': '0'} 2023-09-29T22:48:04.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f936a128fe5cb20-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.784 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.784 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.784 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.784 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.784 DEBUG:boto:Method: GET 2023-09-29T22:48:04.784 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.784 DEBUG:boto:Data: 2023-09-29T22:48:04.785 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.785 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.785 DEBUG:boto:Params: {} 2023-09-29T22:48:04.785 DEBUG:boto:Token: None 2023-09-29T22:48:04.785 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.785 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc= 2023-09-29T22:48:04.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc=', 'Content-Length': '0'} 2023-09-29T22:48:04.788 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02a523c3453cef1-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.789 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.789 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.789 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.789 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.789 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.789 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.789 DEBUG:boto:Data: 2023-09-29T22:48:04.790 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.790 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.790 DEBUG:boto:Params: {} 2023-09-29T22:48:04.790 DEBUG:boto:Token: None 2023-09-29T22:48:04.790 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gHZugBjykTHluwflCvx1zkW/kBQ= 2023-09-29T22:48:04.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gHZugBjykTHluwflCvx1zkW/kBQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.793 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007299bfbdcd91231-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.793 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.793 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.793 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.793 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.794 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.794 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.794 DEBUG:boto:Data: 2023-09-29T22:48:04.794 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.794 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.794 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.794 DEBUG:boto:Params: {} 2023-09-29T22:48:04.794 DEBUG:boto:Token: None 2023-09-29T22:48:04.794 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.795 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gHZugBjykTHluwflCvx1zkW/kBQ= 2023-09-29T22:48:04.795 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gHZugBjykTHluwflCvx1zkW/kBQ=', 'Content-Length': '0'} 2023-09-29T22:48:04.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000216d1518fb5489cf-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:48:04.798 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.798 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.799 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.799 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.799 DEBUG:boto:Method: GET 2023-09-29T22:48:04.799 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.799 DEBUG:boto:Data: 2023-09-29T22:48:04.799 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.799 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.799 DEBUG:boto:Params: {} 2023-09-29T22:48:04.799 DEBUG:boto:Token: None 2023-09-29T22:48:04.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc= 2023-09-29T22:48:04.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc=', 'Content-Length': '0'} 2023-09-29T22:48:04.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f936a7e701de2e1-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.804 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.804 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.804 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.804 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.804 DEBUG:boto:Method: GET 2023-09-29T22:48:04.804 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.804 DEBUG:boto:Data: 2023-09-29T22:48:04.804 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.805 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.805 DEBUG:boto:Params: {} 2023-09-29T22:48:04.805 DEBUG:boto:Token: None 2023-09-29T22:48:04.805 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:48:04.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc= 2023-09-29T22:48:04.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLXc8vIy4PX1tww85OE16FyQMlc=', 'Content-Length': '0'} 2023-09-29T22:48:04.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b1c61945c24967a-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.808 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.808 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.808 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.808 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.809 DEBUG:boto:Method: GET 2023-09-29T22:48:04.809 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.809 DEBUG:boto:Data: 2023-09-29T22:48:04.809 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.809 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.809 DEBUG:boto:Params: {} 2023-09-29T22:48:04.809 DEBUG:boto:Token: None 2023-09-29T22:48:04.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db66daa40da090f1-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.854 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.855 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.855 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.855 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.855 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.855 DEBUG:boto:Method: GET 2023-09-29T22:48:04.855 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.856 DEBUG:boto:Data: 2023-09-29T22:48:04.856 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.856 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.856 DEBUG:boto:Params: {} 2023-09-29T22:48:04.856 DEBUG:boto:Token: None 2023-09-29T22:48:04.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.865 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073a7a5ba38461649-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.865 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.865 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.865 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.866 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.866 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:48:04.866 DEBUG:boto:Data: 2023-09-29T22:48:04.866 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.866 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.866 DEBUG:boto:Params: {} 2023-09-29T22:48:04.866 DEBUG:boto:Token: None 2023-09-29T22:48:04.866 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17 2023-09-29T22:48:04.867 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P762oRnp/evclaICywbcdk0XmS4= 2023-09-29T22:48:04.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P762oRnp/evclaICywbcdk0XmS4=', 'Content-Length': '0'} 2023-09-29T22:48:04.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000752a2976f169a1d2-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.869 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.869 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.870 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.870 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:48:04.870 DEBUG:boto:Data: 2023-09-29T22:48:04.870 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.870 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.870 DEBUG:boto:Params: {} 2023-09-29T22:48:04.870 DEBUG:boto:Token: None 2023-09-29T22:48:04.870 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17 2023-09-29T22:48:04.871 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P762oRnp/evclaICywbcdk0XmS4= 2023-09-29T22:48:04.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P762oRnp/evclaICywbcdk0XmS4=', 'Content-Length': '0'} 2023-09-29T22:48:04.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e542deccfd5b0b8e-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:48:04.873 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.873 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.873 DEBUG:boto:Method: GET 2023-09-29T22:48:04.874 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:48:04.874 DEBUG:boto:Data: 2023-09-29T22:48:04.874 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.874 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.874 DEBUG:boto:Params: {} 2023-09-29T22:48:04.874 DEBUG:boto:Token: None 2023-09-29T22:48:04.874 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17 2023-09-29T22:48:04.874 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+YVN2ywRglmBxwrahWquUrd0IJE= 2023-09-29T22:48:04.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+YVN2ywRglmBxwrahWquUrd0IJE=', 'Content-Length': '0'} 2023-09-29T22:48:04.877 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d0fbdc594aaa3b3-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.877 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:48:04.877 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:48:04.877 DEBUG:boto:Method: GET 2023-09-29T22:48:04.877 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:48:04.877 DEBUG:boto:Data: 2023-09-29T22:48:04.878 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.878 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.878 DEBUG:boto:Params: {} 2023-09-29T22:48:04.878 DEBUG:boto:Token: None 2023-09-29T22:48:04.878 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj17 2023-09-29T22:48:04.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+YVN2ywRglmBxwrahWquUrd0IJE= 2023-09-29T22:48:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+YVN2ywRglmBxwrahWquUrd0IJE=', 'Content-Length': '0'} 2023-09-29T22:48:04.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092405f6c37727c6c-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.922 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.922 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.923 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.923 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.923 DEBUG:boto:Method: GET 2023-09-29T22:48:04.923 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.923 DEBUG:boto:Data: 2023-09-29T22:48:04.923 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.923 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.924 DEBUG:boto:Params: {} 2023-09-29T22:48:04.924 DEBUG:boto:Token: None 2023-09-29T22:48:04.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009374b154567be2b8-0065175424-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.929 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.929 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:04.930 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:04.930 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.930 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.930 DEBUG:boto:Method: GET 2023-09-29T22:48:04.930 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:48:04.930 DEBUG:boto:Data: 2023-09-29T22:48:04.931 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.931 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.931 DEBUG:boto:Params: {} 2023-09-29T22:48:04.931 DEBUG:boto:Token: None 2023-09-29T22:48:04.931 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/ 2023-09-29T22:48:04.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU= 2023-09-29T22:48:04.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J6gxFVeTbXm/i2yenFaemVs2hYU=', 'Content-Length': '0'} 2023-09-29T22:48:04.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000632a259d3b6fe457-0065175424-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.939 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:04.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:48:04.939 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:04.939 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:04.939 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.940 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.940 DEBUG:boto:Method: GET 2023-09-29T22:48:04.940 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.940 DEBUG:boto:Data: 2023-09-29T22:48:04.940 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.941 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.941 DEBUG:boto:Params: {} 2023-09-29T22:48:04.941 DEBUG:boto:Token: None 2023-09-29T22:48:04.941 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faFx1dq7LmYvt10CSOdvB7JhRTg= 2023-09-29T22:48:04.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faFx1dq7LmYvt10CSOdvB7JhRTg=', 'Content-Length': '0'} 2023-09-29T22:48:04.944 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036325c166faedce4-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.944 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:04.944 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:04.944 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.944 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.944 DEBUG:boto:Method: GET 2023-09-29T22:48:04.944 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.945 DEBUG:boto:Data: 2023-09-29T22:48:04.945 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.945 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.945 DEBUG:boto:Params: {} 2023-09-29T22:48:04.945 DEBUG:boto:Token: None 2023-09-29T22:48:04.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faFx1dq7LmYvt10CSOdvB7JhRTg= 2023-09-29T22:48:04.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faFx1dq7LmYvt10CSOdvB7JhRTg=', 'Content-Length': '0'} 2023-09-29T22:48:04.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2f1dd725e3f13df-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.948 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:04.948 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:04.948 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.948 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.948 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.948 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.948 DEBUG:boto:Data: 2023-09-29T22:48:04.949 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.949 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.949 DEBUG:boto:Params: {} 2023-09-29T22:48:04.949 DEBUG:boto:Token: None 2023-09-29T22:48:04.949 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gfOI1ivdUcFV4/kVyXuNV9EhBkY= 2023-09-29T22:48:04.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gfOI1ivdUcFV4/kVyXuNV9EhBkY=', 'Content-Length': '0'} 2023-09-29T22:48:04.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001718b51e51d34fa8-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.953 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:04.954 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:04.954 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.954 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.954 DEBUG:boto:Method: HEAD 2023-09-29T22:48:04.954 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.954 DEBUG:boto:Data: 2023-09-29T22:48:04.954 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:04.955 DEBUG:boto:Port: 8000 2023-09-29T22:48:04.955 DEBUG:boto:Params: {} 2023-09-29T22:48:04.955 DEBUG:boto:Token: None 2023-09-29T22:48:04.955 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gfOI1ivdUcFV4/kVyXuNV9EhBkY= 2023-09-29T22:48:04.955 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gfOI1ivdUcFV4/kVyXuNV9EhBkY=', 'Content-Length': '0'} 2023-09-29T22:48:04.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ec6b9c1d681c25c-0065175424-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:04.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:48:04.958 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:04.958 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:04.958 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.958 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.958 DEBUG:boto:Method: GET 2023-09-29T22:48:04.958 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.958 DEBUG:boto:Data: 2023-09-29T22:48:04.958 DEBUG:boto:Headers: {} 2023-09-29T22:48:04.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:04.959 DEBUG:boto:Port: 8001 2023-09-29T22:48:04.959 DEBUG:boto:Params: {} 2023-09-29T22:48:04.959 DEBUG:boto:Token: None 2023-09-29T22:48:04.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:04 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:04.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:faFx1dq7LmYvt10CSOdvB7JhRTg= 2023-09-29T22:48:04.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:faFx1dq7LmYvt10CSOdvB7JhRTg=', 'Content-Length': '0'} 2023-09-29T22:48:04.961 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e01f129eaf8f70a-0065175424-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.002 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:05.002 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:05.003 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:05.003 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:48:05.003 DEBUG:boto:Method: GET 2023-09-29T22:48:05.003 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:05.003 DEBUG:boto:Data: 2023-09-29T22:48:05.003 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.003 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.003 DEBUG:boto:Params: {} 2023-09-29T22:48:05.003 DEBUG:boto:Token: None 2023-09-29T22:48:05.003 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:48:05.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RaYJjvwlfBq+zniGVljcSa0dkTU= 2023-09-29T22:48:05.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RaYJjvwlfBq+zniGVljcSa0dkTU=', 'Content-Length': '0'} 2023-09-29T22:48:05.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000885d32221b9a3b3e-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.007 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.007 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.007 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.007 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.007 DEBUG:boto:Method: GET 2023-09-29T22:48:05.007 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.008 DEBUG:boto:Data: 2023-09-29T22:48:05.008 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.008 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.008 DEBUG:boto:Params: {} 2023-09-29T22:48:05.008 DEBUG:boto:Token: None 2023-09-29T22:48:05.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9fb3fe0b45313f0-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.014 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.014 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.014 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.014 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.014 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.014 DEBUG:boto:Method: GET 2023-09-29T22:48:05.015 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.015 DEBUG:boto:Data: 2023-09-29T22:48:05.015 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.015 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.015 DEBUG:boto:Params: {} 2023-09-29T22:48:05.015 DEBUG:boto:Token: None 2023-09-29T22:48:05.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.015 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.022 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005931f18d570c9d7d-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.023 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.023 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:05.023 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:05.023 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.023 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:48:05.023 DEBUG:boto:Data: 2023-09-29T22:48:05.024 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.024 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.024 DEBUG:boto:Params: {} 2023-09-29T22:48:05.024 DEBUG:boto:Token: None 2023-09-29T22:48:05.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj19 2023-09-29T22:48:05.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fye+Hh8Q1MtalxwaKJoTm7AhqFo= 2023-09-29T22:48:05.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fye+Hh8Q1MtalxwaKJoTm7AhqFo=', 'Content-Length': '0'} 2023-09-29T22:48:05.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9b098786b157e2c-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.026 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:05.026 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:05.027 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.027 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:48:05.027 DEBUG:boto:Data: 2023-09-29T22:48:05.027 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.027 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.027 DEBUG:boto:Params: {} 2023-09-29T22:48:05.027 DEBUG:boto:Token: None 2023-09-29T22:48:05.027 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj19 2023-09-29T22:48:05.027 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fye+Hh8Q1MtalxwaKJoTm7AhqFo= 2023-09-29T22:48:05.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fye+Hh8Q1MtalxwaKJoTm7AhqFo=', 'Content-Length': '0'} 2023-09-29T22:48:05.029 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013f550524d03c578-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:48:05.030 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:05.030 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:05.030 DEBUG:boto:Method: GET 2023-09-29T22:48:05.030 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:48:05.030 DEBUG:boto:Data: 2023-09-29T22:48:05.030 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.030 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.030 DEBUG:boto:Params: {} 2023-09-29T22:48:05.031 DEBUG:boto:Token: None 2023-09-29T22:48:05.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj19 2023-09-29T22:48:05.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NVhzjwJTnhnXAlbHyobjRseDXtg= 2023-09-29T22:48:05.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NVhzjwJTnhnXAlbHyobjRseDXtg=', 'Content-Length': '0'} 2023-09-29T22:48:05.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ff130334762fddc-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.033 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:48:05.033 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:48:05.034 DEBUG:boto:Method: GET 2023-09-29T22:48:05.034 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:48:05.034 DEBUG:boto:Data: 2023-09-29T22:48:05.034 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.034 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.034 DEBUG:boto:Params: {} 2023-09-29T22:48:05.034 DEBUG:boto:Token: None 2023-09-29T22:48:05.034 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj19 2023-09-29T22:48:05.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NVhzjwJTnhnXAlbHyobjRseDXtg= 2023-09-29T22:48:05.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NVhzjwJTnhnXAlbHyobjRseDXtg=', 'Content-Length': '0'} 2023-09-29T22:48:05.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000678b839152a8ed5d-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.037 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.037 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.038 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.038 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.038 DEBUG:boto:Method: GET 2023-09-29T22:48:05.038 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.038 DEBUG:boto:Data: 2023-09-29T22:48:05.038 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.038 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.038 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.038 DEBUG:boto:Params: {} 2023-09-29T22:48:05.038 DEBUG:boto:Token: None 2023-09-29T22:48:05.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.039 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.042 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062964b91e81ef9ec-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.042 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.042 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.042 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.043 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.043 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.043 DEBUG:boto:Method: GET 2023-09-29T22:48:05.043 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:48:05.043 DEBUG:boto:Data: 2023-09-29T22:48:05.043 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.043 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.043 DEBUG:boto:Params: {} 2023-09-29T22:48:05.043 DEBUG:boto:Token: None 2023-09-29T22:48:05.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.044 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.050 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c8b0fb0893a8b81-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.050 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:48:05.051 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.051 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.051 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.051 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.051 DEBUG:boto:Method: GET 2023-09-29T22:48:05.051 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.051 DEBUG:boto:Data: 2023-09-29T22:48:05.051 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.052 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.052 DEBUG:boto:Params: {} 2023-09-29T22:48:05.052 DEBUG:boto:Token: None 2023-09-29T22:48:05.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss= 2023-09-29T22:48:05.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss=', 'Content-Length': '0'} 2023-09-29T22:48:05.054 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a8e2d35ee3e1daf-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.055 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.055 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.055 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.055 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.055 DEBUG:boto:Method: GET 2023-09-29T22:48:05.055 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.055 DEBUG:boto:Data: 2023-09-29T22:48:05.055 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.056 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.056 DEBUG:boto:Params: {} 2023-09-29T22:48:05.056 DEBUG:boto:Token: None 2023-09-29T22:48:05.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss= 2023-09-29T22:48:05.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss=', 'Content-Length': '0'} 2023-09-29T22:48:05.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad1472f3febed541-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.102 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.102 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.103 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.103 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.103 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.103 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.103 DEBUG:boto:Data: 2023-09-29T22:48:05.103 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.103 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.103 DEBUG:boto:Params: {} 2023-09-29T22:48:05.103 DEBUG:boto:Token: None 2023-09-29T22:48:05.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KTfD0WN6Mif4WKn9miDXlq1MTOk= 2023-09-29T22:48:05.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KTfD0WN6Mif4WKn9miDXlq1MTOk=', 'Content-Length': '0'} 2023-09-29T22:48:05.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000776d1a2f59f1a2f9-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.107 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.107 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.107 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.107 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.107 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.107 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.108 DEBUG:boto:Data: 2023-09-29T22:48:05.108 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.108 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.108 DEBUG:boto:Params: {} 2023-09-29T22:48:05.108 DEBUG:boto:Token: None 2023-09-29T22:48:05.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KTfD0WN6Mif4WKn9miDXlq1MTOk= 2023-09-29T22:48:05.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KTfD0WN6Mif4WKn9miDXlq1MTOk=', 'Content-Length': '0'} 2023-09-29T22:48:05.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007422aa08cf0980f8-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:48:05.112 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.112 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.112 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.112 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.112 DEBUG:boto:Method: GET 2023-09-29T22:48:05.112 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.112 DEBUG:boto:Data: 2023-09-29T22:48:05.112 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.113 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.113 DEBUG:boto:Params: {} 2023-09-29T22:48:05.113 DEBUG:boto:Token: None 2023-09-29T22:48:05.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss= 2023-09-29T22:48:05.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss=', 'Content-Length': '0'} 2023-09-29T22:48:05.117 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8b48fb4c9befa90-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.158 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.158 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.159 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.159 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.159 DEBUG:boto:Method: GET 2023-09-29T22:48:05.159 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.159 DEBUG:boto:Data: 2023-09-29T22:48:05.159 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.159 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.159 DEBUG:boto:Params: {} 2023-09-29T22:48:05.159 DEBUG:boto:Token: None 2023-09-29T22:48:05.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:48:05.160 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss= 2023-09-29T22:48:05.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hC6WqyyDMnAwCmnNmNqNpF2u/ss=', 'Content-Length': '0'} 2023-09-29T22:48:05.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007442bc7741bd608a-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.164 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.164 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.164 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.164 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.164 DEBUG:boto:Method: GET 2023-09-29T22:48:05.164 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.164 DEBUG:boto:Data: 2023-09-29T22:48:05.165 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.165 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.165 DEBUG:boto:Params: {} 2023-09-29T22:48:05.165 DEBUG:boto:Token: None 2023-09-29T22:48:05.165 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.165 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7581d6cb68a63d3-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.170 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.170 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.171 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.171 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.171 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.171 DEBUG:boto:Method: GET 2023-09-29T22:48:05.171 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.171 DEBUG:boto:Data: 2023-09-29T22:48:05.171 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.171 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.171 DEBUG:boto:Params: {} 2023-09-29T22:48:05.172 DEBUG:boto:Token: None 2023-09-29T22:48:05.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.179 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000814b722dfb3edb76-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.179 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.179 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.179 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.179 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.180 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:48:05.180 DEBUG:boto:Data: 2023-09-29T22:48:05.180 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.180 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.180 DEBUG:boto:Params: {} 2023-09-29T22:48:05.180 DEBUG:boto:Token: None 2023-09-29T22:48:05.180 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22 2023-09-29T22:48:05.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXj96nahxbJvpT/xP8r1x3boPPA= 2023-09-29T22:48:05.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXj96nahxbJvpT/xP8r1x3boPPA=', 'Content-Length': '0'} 2023-09-29T22:48:05.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6823d5be740172-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.182 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.183 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.183 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.183 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:48:05.183 DEBUG:boto:Data: 2023-09-29T22:48:05.183 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.183 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.183 DEBUG:boto:Params: {} 2023-09-29T22:48:05.183 DEBUG:boto:Token: None 2023-09-29T22:48:05.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22 2023-09-29T22:48:05.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXj96nahxbJvpT/xP8r1x3boPPA= 2023-09-29T22:48:05.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXj96nahxbJvpT/xP8r1x3boPPA=', 'Content-Length': '0'} 2023-09-29T22:48:05.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf04b323df418a6-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:48:05.186 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.187 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.187 DEBUG:boto:Method: GET 2023-09-29T22:48:05.187 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:48:05.187 DEBUG:boto:Data: 2023-09-29T22:48:05.187 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.187 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.187 DEBUG:boto:Params: {} 2023-09-29T22:48:05.187 DEBUG:boto:Token: None 2023-09-29T22:48:05.187 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22 2023-09-29T22:48:05.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5PdUbsZp1hYJg1h7/qIByPZKHPI= 2023-09-29T22:48:05.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5PdUbsZp1hYJg1h7/qIByPZKHPI=', 'Content-Length': '0'} 2023-09-29T22:48:05.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d8351297a7de9633-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.230 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:48:05.230 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:48:05.230 DEBUG:boto:Method: GET 2023-09-29T22:48:05.231 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:48:05.231 DEBUG:boto:Data: 2023-09-29T22:48:05.231 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.231 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.231 DEBUG:boto:Params: {} 2023-09-29T22:48:05.231 DEBUG:boto:Token: None 2023-09-29T22:48:05.231 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj22 2023-09-29T22:48:05.231 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5PdUbsZp1hYJg1h7/qIByPZKHPI= 2023-09-29T22:48:05.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5PdUbsZp1hYJg1h7/qIByPZKHPI=', 'Content-Length': '0'} 2023-09-29T22:48:05.234 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002405852bbcbe6951-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.235 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.235 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.235 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.235 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.235 DEBUG:boto:Method: GET 2023-09-29T22:48:05.235 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.235 DEBUG:boto:Data: 2023-09-29T22:48:05.235 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.236 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.236 DEBUG:boto:Params: {} 2023-09-29T22:48:05.236 DEBUG:boto:Token: None 2023-09-29T22:48:05.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.241 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003411b2749b6febe1-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.241 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.241 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.241 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.242 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.242 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.242 DEBUG:boto:Method: GET 2023-09-29T22:48:05.242 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:48:05.242 DEBUG:boto:Data: 2023-09-29T22:48:05.242 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.242 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.242 DEBUG:boto:Params: {} 2023-09-29T22:48:05.243 DEBUG:boto:Token: None 2023-09-29T22:48:05.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.243 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.250 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000639113a584ff0cb6-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.250 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:48:05.251 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.251 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.251 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.251 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.251 DEBUG:boto:Method: GET 2023-09-29T22:48:05.252 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.252 DEBUG:boto:Data: 2023-09-29T22:48:05.252 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.252 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.252 DEBUG:boto:Params: {} 2023-09-29T22:48:05.252 DEBUG:boto:Token: None 2023-09-29T22:48:05.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q= 2023-09-29T22:48:05.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q=', 'Content-Length': '0'} 2023-09-29T22:48:05.254 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004822f795bece624d-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.255 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.255 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.255 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.255 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.255 DEBUG:boto:Method: GET 2023-09-29T22:48:05.255 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.255 DEBUG:boto:Data: 2023-09-29T22:48:05.255 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.256 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.256 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.256 DEBUG:boto:Params: {} 2023-09-29T22:48:05.256 DEBUG:boto:Token: None 2023-09-29T22:48:05.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q= 2023-09-29T22:48:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q=', 'Content-Length': '0'} 2023-09-29T22:48:05.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888a430938475332-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.259 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.259 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.259 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.259 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.259 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.259 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.259 DEBUG:boto:Data: 2023-09-29T22:48:05.259 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.259 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.260 DEBUG:boto:Params: {} 2023-09-29T22:48:05.260 DEBUG:boto:Token: None 2023-09-29T22:48:05.260 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LUEbA07DdRNmawGkF5Y6PURYmUM= 2023-09-29T22:48:05.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LUEbA07DdRNmawGkF5Y6PURYmUM=', 'Content-Length': '0'} 2023-09-29T22:48:05.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0b2fed4ac53033-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.262 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.262 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.262 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.262 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.263 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.263 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.263 DEBUG:boto:Data: 2023-09-29T22:48:05.263 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.263 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.263 DEBUG:boto:Params: {} 2023-09-29T22:48:05.263 DEBUG:boto:Token: None 2023-09-29T22:48:05.263 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LUEbA07DdRNmawGkF5Y6PURYmUM= 2023-09-29T22:48:05.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LUEbA07DdRNmawGkF5Y6PURYmUM=', 'Content-Length': '0'} 2023-09-29T22:48:05.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002294c11d1dd25026-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:48:05.266 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.266 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.267 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.267 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.267 DEBUG:boto:Method: GET 2023-09-29T22:48:05.267 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.267 DEBUG:boto:Data: 2023-09-29T22:48:05.267 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.267 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.267 DEBUG:boto:Params: {} 2023-09-29T22:48:05.267 DEBUG:boto:Token: None 2023-09-29T22:48:05.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q= 2023-09-29T22:48:05.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q=', 'Content-Length': '0'} 2023-09-29T22:48:05.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000155b1758494760a3-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.310 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.310 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.311 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.311 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.311 DEBUG:boto:Method: GET 2023-09-29T22:48:05.311 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.311 DEBUG:boto:Data: 2023-09-29T22:48:05.311 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.311 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.311 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.311 DEBUG:boto:Params: {} 2023-09-29T22:48:05.311 DEBUG:boto:Token: None 2023-09-29T22:48:05.312 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:48:05.312 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q= 2023-09-29T22:48:05.312 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q7qElG5auFGREg1ibImif88dd0Q=', 'Content-Length': '0'} 2023-09-29T22:48:05.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003021a735224b28fc-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.315 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.315 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.315 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.315 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.316 DEBUG:boto:Method: GET 2023-09-29T22:48:05.316 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.316 DEBUG:boto:Data: 2023-09-29T22:48:05.316 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.316 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.316 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.316 DEBUG:boto:Params: {} 2023-09-29T22:48:05.316 DEBUG:boto:Token: None 2023-09-29T22:48:05.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.317 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.322 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000acf4fa125aca7f72-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.322 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.322 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.323 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.323 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.323 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.323 DEBUG:boto:Method: GET 2023-09-29T22:48:05.323 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.323 DEBUG:boto:Data: 2023-09-29T22:48:05.323 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.323 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.323 DEBUG:boto:Params: {} 2023-09-29T22:48:05.324 DEBUG:boto:Token: None 2023-09-29T22:48:05.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001ebb3997b097cf1c-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.331 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.332 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.332 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.332 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.332 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:48:05.332 DEBUG:boto:Data: 2023-09-29T22:48:05.332 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.332 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.332 DEBUG:boto:Params: {} 2023-09-29T22:48:05.333 DEBUG:boto:Token: None 2023-09-29T22:48:05.333 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23 2023-09-29T22:48:05.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pdm9bD8WYpyRE+dMdS0+7Ty1j5c= 2023-09-29T22:48:05.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pdm9bD8WYpyRE+dMdS0+7Ty1j5c=', 'Content-Length': '0'} 2023-09-29T22:48:05.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e318e619631bc23-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.335 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.336 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.336 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.336 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:48:05.336 DEBUG:boto:Data: 2023-09-29T22:48:05.336 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.336 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.336 DEBUG:boto:Params: {} 2023-09-29T22:48:05.336 DEBUG:boto:Token: None 2023-09-29T22:48:05.337 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23 2023-09-29T22:48:05.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Pdm9bD8WYpyRE+dMdS0+7Ty1j5c= 2023-09-29T22:48:05.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Pdm9bD8WYpyRE+dMdS0+7Ty1j5c=', 'Content-Length': '0'} 2023-09-29T22:48:05.338 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1c27cb6c80d29f5-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:48:05.339 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.339 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.339 DEBUG:boto:Method: GET 2023-09-29T22:48:05.339 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:48:05.339 DEBUG:boto:Data: 2023-09-29T22:48:05.339 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.340 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.340 DEBUG:boto:Params: {} 2023-09-29T22:48:05.340 DEBUG:boto:Token: None 2023-09-29T22:48:05.340 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23 2023-09-29T22:48:05.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vVv2ttrnb35eSRxxLXWQSWONfrA= 2023-09-29T22:48:05.340 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vVv2ttrnb35eSRxxLXWQSWONfrA=', 'Content-Length': '0'} 2023-09-29T22:48:05.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030429bef890d5687-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.386 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:48:05.387 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:48:05.387 DEBUG:boto:Method: GET 2023-09-29T22:48:05.387 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:48:05.387 DEBUG:boto:Data: 2023-09-29T22:48:05.387 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.387 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.387 DEBUG:boto:Params: {} 2023-09-29T22:48:05.387 DEBUG:boto:Token: None 2023-09-29T22:48:05.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj23 2023-09-29T22:48:05.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vVv2ttrnb35eSRxxLXWQSWONfrA= 2023-09-29T22:48:05.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vVv2ttrnb35eSRxxLXWQSWONfrA=', 'Content-Length': '0'} 2023-09-29T22:48:05.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cc874c5a5b4db79-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.391 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.391 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.391 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.391 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.392 DEBUG:boto:Method: GET 2023-09-29T22:48:05.392 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.392 DEBUG:boto:Data: 2023-09-29T22:48:05.392 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.392 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.392 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.392 DEBUG:boto:Params: {} 2023-09-29T22:48:05.392 DEBUG:boto:Token: None 2023-09-29T22:48:05.393 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.397 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a53be789eacf00eb-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.397 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.398 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.398 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.398 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.398 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.398 DEBUG:boto:Method: GET 2023-09-29T22:48:05.398 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:48:05.399 DEBUG:boto:Data: 2023-09-29T22:48:05.399 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.399 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.399 DEBUG:boto:Params: {} 2023-09-29T22:48:05.399 DEBUG:boto:Token: None 2023-09-29T22:48:05.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.399 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.406 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b0b54259d5c8b9-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.454 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:48:05.455 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.455 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.455 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.455 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.455 DEBUG:boto:Method: GET 2023-09-29T22:48:05.455 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.455 DEBUG:boto:Data: 2023-09-29T22:48:05.456 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.456 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.456 DEBUG:boto:Params: {} 2023-09-29T22:48:05.456 DEBUG:boto:Token: None 2023-09-29T22:48:05.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA= 2023-09-29T22:48:05.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA=', 'Content-Length': '0'} 2023-09-29T22:48:05.459 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014b54b00c1c9882e-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.460 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.460 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.460 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.460 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.460 DEBUG:boto:Method: GET 2023-09-29T22:48:05.460 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.460 DEBUG:boto:Data: 2023-09-29T22:48:05.460 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.461 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.461 DEBUG:boto:Params: {} 2023-09-29T22:48:05.461 DEBUG:boto:Token: None 2023-09-29T22:48:05.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA= 2023-09-29T22:48:05.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA=', 'Content-Length': '0'} 2023-09-29T22:48:05.464 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000589add265bf0021c-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.464 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.464 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.464 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.465 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.465 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.465 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.465 DEBUG:boto:Data: 2023-09-29T22:48:05.465 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.465 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.465 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.465 DEBUG:boto:Params: {} 2023-09-29T22:48:05.465 DEBUG:boto:Token: None 2023-09-29T22:48:05.466 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.466 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YWth0gcQaDyaN7leB3nBhJSvKdo= 2023-09-29T22:48:05.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YWth0gcQaDyaN7leB3nBhJSvKdo=', 'Content-Length': '0'} 2023-09-29T22:48:05.468 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d06bc11ce2fb0fa8-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.469 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.469 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.469 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.469 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.469 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.469 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.469 DEBUG:boto:Data: 2023-09-29T22:48:05.470 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.470 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.470 DEBUG:boto:Params: {} 2023-09-29T22:48:05.470 DEBUG:boto:Token: None 2023-09-29T22:48:05.470 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.470 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YWth0gcQaDyaN7leB3nBhJSvKdo= 2023-09-29T22:48:05.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YWth0gcQaDyaN7leB3nBhJSvKdo=', 'Content-Length': '0'} 2023-09-29T22:48:05.473 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f171479642647ecc-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:48:05.473 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.473 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.474 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.474 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.474 DEBUG:boto:Method: GET 2023-09-29T22:48:05.474 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.474 DEBUG:boto:Data: 2023-09-29T22:48:05.474 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.474 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.474 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.474 DEBUG:boto:Params: {} 2023-09-29T22:48:05.475 DEBUG:boto:Token: None 2023-09-29T22:48:05.475 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.475 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA= 2023-09-29T22:48:05.475 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA=', 'Content-Length': '0'} 2023-09-29T22:48:05.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8fdd6c65e4b7a6a-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.478 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.478 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.478 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.478 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.478 DEBUG:boto:Method: GET 2023-09-29T22:48:05.478 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.478 DEBUG:boto:Data: 2023-09-29T22:48:05.479 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.479 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.479 DEBUG:boto:Params: {} 2023-09-29T22:48:05.479 DEBUG:boto:Token: None 2023-09-29T22:48:05.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:48:05.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA= 2023-09-29T22:48:05.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:svBHPmyf4CihbJ0d1Nh3ak6gYQA=', 'Content-Length': '0'} 2023-09-29T22:48:05.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4d10021977de46e-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.483 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.483 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.483 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.483 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.483 DEBUG:boto:Method: GET 2023-09-29T22:48:05.483 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.483 DEBUG:boto:Data: 2023-09-29T22:48:05.483 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.484 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.484 DEBUG:boto:Params: {} 2023-09-29T22:48:05.484 DEBUG:boto:Token: None 2023-09-29T22:48:05.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4930d6e3264d7fa-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.489 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.489 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.489 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.490 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.490 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.490 DEBUG:boto:Method: GET 2023-09-29T22:48:05.490 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.490 DEBUG:boto:Data: 2023-09-29T22:48:05.490 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.490 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.490 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.490 DEBUG:boto:Params: {} 2023-09-29T22:48:05.491 DEBUG:boto:Token: None 2023-09-29T22:48:05.491 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.491 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.491 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d36df2d46118dbd-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.538 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.539 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.539 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.539 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.539 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:48:05.539 DEBUG:boto:Data: 2023-09-29T22:48:05.539 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.539 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.539 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.540 DEBUG:boto:Params: {} 2023-09-29T22:48:05.540 DEBUG:boto:Token: None 2023-09-29T22:48:05.540 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29 2023-09-29T22:48:05.540 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C+tG9xYQz1Nzn7FsAHosnw3MGQM= 2023-09-29T22:48:05.540 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C+tG9xYQz1Nzn7FsAHosnw3MGQM=', 'Content-Length': '0'} 2023-09-29T22:48:05.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bf607b9fb13ceb0-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.543 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.543 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.543 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.543 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:48:05.543 DEBUG:boto:Data: 2023-09-29T22:48:05.543 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.544 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.544 DEBUG:boto:Params: {} 2023-09-29T22:48:05.544 DEBUG:boto:Token: None 2023-09-29T22:48:05.544 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29 2023-09-29T22:48:05.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C+tG9xYQz1Nzn7FsAHosnw3MGQM= 2023-09-29T22:48:05.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C+tG9xYQz1Nzn7FsAHosnw3MGQM=', 'Content-Length': '0'} 2023-09-29T22:48:05.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dd77850e2dba144-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:48:05.547 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.547 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.547 DEBUG:boto:Method: GET 2023-09-29T22:48:05.547 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:48:05.548 DEBUG:boto:Data: 2023-09-29T22:48:05.548 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.548 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.548 DEBUG:boto:Params: {} 2023-09-29T22:48:05.548 DEBUG:boto:Token: None 2023-09-29T22:48:05.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29 2023-09-29T22:48:05.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:It3woU3T3wQEep9AE0qGvocpyv8= 2023-09-29T22:48:05.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:It3woU3T3wQEep9AE0qGvocpyv8=', 'Content-Length': '0'} 2023-09-29T22:48:05.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3bc6aad3f1118bd-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.594 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:48:05.595 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:48:05.595 DEBUG:boto:Method: GET 2023-09-29T22:48:05.595 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:48:05.595 DEBUG:boto:Data: 2023-09-29T22:48:05.595 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.595 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.595 DEBUG:boto:Params: {} 2023-09-29T22:48:05.596 DEBUG:boto:Token: None 2023-09-29T22:48:05.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj29 2023-09-29T22:48:05.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:It3woU3T3wQEep9AE0qGvocpyv8= 2023-09-29T22:48:05.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:It3woU3T3wQEep9AE0qGvocpyv8=', 'Content-Length': '0'} 2023-09-29T22:48:05.599 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001685f4338eaf4a23-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.599 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.599 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.599 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.599 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.599 DEBUG:boto:Method: GET 2023-09-29T22:48:05.600 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.600 DEBUG:boto:Data: 2023-09-29T22:48:05.600 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.600 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.600 DEBUG:boto:Params: {} 2023-09-29T22:48:05.600 DEBUG:boto:Token: None 2023-09-29T22:48:05.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e632bbf580dcb3b9-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.605 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.606 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.606 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.606 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.606 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.606 DEBUG:boto:Method: GET 2023-09-29T22:48:05.606 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:48:05.606 DEBUG:boto:Data: 2023-09-29T22:48:05.606 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.607 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.607 DEBUG:boto:Params: {} 2023-09-29T22:48:05.607 DEBUG:boto:Token: None 2023-09-29T22:48:05.607 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.614 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e84dac00e7ca90-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.615 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:48:05.615 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.615 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.615 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.616 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.616 DEBUG:boto:Method: GET 2023-09-29T22:48:05.616 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.616 DEBUG:boto:Data: 2023-09-29T22:48:05.616 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.616 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.616 DEBUG:boto:Params: {} 2023-09-29T22:48:05.616 DEBUG:boto:Token: None 2023-09-29T22:48:05.617 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.617 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10= 2023-09-29T22:48:05.617 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10=', 'Content-Length': '0'} 2023-09-29T22:48:05.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146c61855d3bd7b0-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.619 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.619 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.619 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.620 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.620 DEBUG:boto:Method: GET 2023-09-29T22:48:05.620 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.620 DEBUG:boto:Data: 2023-09-29T22:48:05.620 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.620 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.620 DEBUG:boto:Params: {} 2023-09-29T22:48:05.621 DEBUG:boto:Token: None 2023-09-29T22:48:05.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.621 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10= 2023-09-29T22:48:05.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10=', 'Content-Length': '0'} 2023-09-29T22:48:05.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e9122a03ac6ff1-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.666 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.667 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.667 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.667 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.667 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.667 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.667 DEBUG:boto:Data: 2023-09-29T22:48:05.667 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.668 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.668 DEBUG:boto:Params: {} 2023-09-29T22:48:05.668 DEBUG:boto:Token: None 2023-09-29T22:48:05.668 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PuKuL2f0CAmtHDqeX7n6C8OYMqo= 2023-09-29T22:48:05.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PuKuL2f0CAmtHDqeX7n6C8OYMqo=', 'Content-Length': '0'} 2023-09-29T22:48:05.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f623c3fff3a6769c-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.671 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.671 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.671 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.671 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.671 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.671 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.671 DEBUG:boto:Data: 2023-09-29T22:48:05.672 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.672 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.672 DEBUG:boto:Params: {} 2023-09-29T22:48:05.672 DEBUG:boto:Token: None 2023-09-29T22:48:05.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PuKuL2f0CAmtHDqeX7n6C8OYMqo= 2023-09-29T22:48:05.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PuKuL2f0CAmtHDqeX7n6C8OYMqo=', 'Content-Length': '0'} 2023-09-29T22:48:05.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dfba9c5129f9468-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:48:05.675 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.676 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.676 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.676 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.676 DEBUG:boto:Method: GET 2023-09-29T22:48:05.676 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.676 DEBUG:boto:Data: 2023-09-29T22:48:05.676 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.676 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.676 DEBUG:boto:Params: {} 2023-09-29T22:48:05.677 DEBUG:boto:Token: None 2023-09-29T22:48:05.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10= 2023-09-29T22:48:05.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10=', 'Content-Length': '0'} 2023-09-29T22:48:05.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000989431f3f068b494-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.722 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.723 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.723 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.723 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.723 DEBUG:boto:Method: GET 2023-09-29T22:48:05.723 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.723 DEBUG:boto:Data: 2023-09-29T22:48:05.723 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.724 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.724 DEBUG:boto:Params: {} 2023-09-29T22:48:05.724 DEBUG:boto:Token: None 2023-09-29T22:48:05.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:48:05.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10= 2023-09-29T22:48:05.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GWUjRB4YRfQbj62kGiW/Ck0be10=', 'Content-Length': '0'} 2023-09-29T22:48:05.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bee4d68d47f33e66-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.727 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.728 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.728 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.728 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.728 DEBUG:boto:Method: GET 2023-09-29T22:48:05.728 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.728 DEBUG:boto:Data: 2023-09-29T22:48:05.728 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.729 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.729 DEBUG:boto:Params: {} 2023-09-29T22:48:05.729 DEBUG:boto:Token: None 2023-09-29T22:48:05.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.733 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052b09d8d47ae4ed1-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.734 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.734 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.734 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.734 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.734 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.735 DEBUG:boto:Method: GET 2023-09-29T22:48:05.735 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.735 DEBUG:boto:Data: 2023-09-29T22:48:05.735 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.735 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.735 DEBUG:boto:Params: {} 2023-09-29T22:48:05.735 DEBUG:boto:Token: None 2023-09-29T22:48:05.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.743 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000535ffbe7b7a7e52e-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.743 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.743 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.744 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.744 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.744 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:48:05.744 DEBUG:boto:Data: 2023-09-29T22:48:05.744 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.744 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.744 DEBUG:boto:Params: {} 2023-09-29T22:48:05.745 DEBUG:boto:Token: None 2023-09-29T22:48:05.745 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31 2023-09-29T22:48:05.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0IcJe1CFbwJ/NIFwt6HsoCewgzY= 2023-09-29T22:48:05.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0IcJe1CFbwJ/NIFwt6HsoCewgzY=', 'Content-Length': '0'} 2023-09-29T22:48:05.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa0fa5702a30de00-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.747 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.747 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.748 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.748 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:48:05.748 DEBUG:boto:Data: 2023-09-29T22:48:05.748 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.748 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.748 DEBUG:boto:Params: {} 2023-09-29T22:48:05.748 DEBUG:boto:Token: None 2023-09-29T22:48:05.749 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31 2023-09-29T22:48:05.749 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0IcJe1CFbwJ/NIFwt6HsoCewgzY= 2023-09-29T22:48:05.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0IcJe1CFbwJ/NIFwt6HsoCewgzY=', 'Content-Length': '0'} 2023-09-29T22:48:05.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060328ea52d9d559c-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:48:05.751 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.752 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.752 DEBUG:boto:Method: GET 2023-09-29T22:48:05.752 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:48:05.752 DEBUG:boto:Data: 2023-09-29T22:48:05.752 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.752 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.752 DEBUG:boto:Params: {} 2023-09-29T22:48:05.752 DEBUG:boto:Token: None 2023-09-29T22:48:05.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31 2023-09-29T22:48:05.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bySFCfbrgweUlGele/sT9PIWX7k= 2023-09-29T22:48:05.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bySFCfbrgweUlGele/sT9PIWX7k=', 'Content-Length': '0'} 2023-09-29T22:48:05.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b0ca48229d6608f-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.798 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:48:05.799 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:48:05.799 DEBUG:boto:Method: GET 2023-09-29T22:48:05.799 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:48:05.799 DEBUG:boto:Data: 2023-09-29T22:48:05.799 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.800 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.800 DEBUG:boto:Params: {} 2023-09-29T22:48:05.800 DEBUG:boto:Token: None 2023-09-29T22:48:05.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj31 2023-09-29T22:48:05.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bySFCfbrgweUlGele/sT9PIWX7k= 2023-09-29T22:48:05.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bySFCfbrgweUlGele/sT9PIWX7k=', 'Content-Length': '0'} 2023-09-29T22:48:05.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033ed4ff27282ec28-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.803 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.803 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.803 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.803 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.804 DEBUG:boto:Method: GET 2023-09-29T22:48:05.804 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.804 DEBUG:boto:Data: 2023-09-29T22:48:05.804 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.804 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.804 DEBUG:boto:Params: {} 2023-09-29T22:48:05.804 DEBUG:boto:Token: None 2023-09-29T22:48:05.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b02b031cef0f238-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.810 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.810 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.810 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.810 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.810 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.811 DEBUG:boto:Method: GET 2023-09-29T22:48:05.811 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:48:05.811 DEBUG:boto:Data: 2023-09-29T22:48:05.811 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.811 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.811 DEBUG:boto:Params: {} 2023-09-29T22:48:05.811 DEBUG:boto:Token: None 2023-09-29T22:48:05.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000048f6851f3d62a965-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.819 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:48:05.820 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.820 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.820 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.820 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.820 DEBUG:boto:Method: GET 2023-09-29T22:48:05.820 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.820 DEBUG:boto:Data: 2023-09-29T22:48:05.820 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.821 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.821 DEBUG:boto:Params: {} 2023-09-29T22:48:05.821 DEBUG:boto:Token: None 2023-09-29T22:48:05.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0= 2023-09-29T22:48:05.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0=', 'Content-Length': '0'} 2023-09-29T22:48:05.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ab6383bf7755f94-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.824 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.824 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.824 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.824 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.824 DEBUG:boto:Method: GET 2023-09-29T22:48:05.824 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.825 DEBUG:boto:Data: 2023-09-29T22:48:05.825 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.825 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.825 DEBUG:boto:Params: {} 2023-09-29T22:48:05.825 DEBUG:boto:Token: None 2023-09-29T22:48:05.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0= 2023-09-29T22:48:05.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0=', 'Content-Length': '0'} 2023-09-29T22:48:05.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072359b0c71d7f481-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.828 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.828 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.828 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.829 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.829 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.829 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.829 DEBUG:boto:Data: 2023-09-29T22:48:05.829 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.829 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.829 DEBUG:boto:Params: {} 2023-09-29T22:48:05.830 DEBUG:boto:Token: None 2023-09-29T22:48:05.830 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lommqpiw9PSqA/vQQNTQLhHyOHU= 2023-09-29T22:48:05.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lommqpiw9PSqA/vQQNTQLhHyOHU=', 'Content-Length': '0'} 2023-09-29T22:48:05.832 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051c8f7eefaf85292-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.832 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.832 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.832 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.833 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.833 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.833 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.833 DEBUG:boto:Data: 2023-09-29T22:48:05.833 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.833 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.833 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.833 DEBUG:boto:Params: {} 2023-09-29T22:48:05.833 DEBUG:boto:Token: None 2023-09-29T22:48:05.834 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.834 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lommqpiw9PSqA/vQQNTQLhHyOHU= 2023-09-29T22:48:05.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lommqpiw9PSqA/vQQNTQLhHyOHU=', 'Content-Length': '0'} 2023-09-29T22:48:05.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f0c608c4efd3634-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:48:05.836 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.836 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.836 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.836 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.837 DEBUG:boto:Method: GET 2023-09-29T22:48:05.837 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.837 DEBUG:boto:Data: 2023-09-29T22:48:05.837 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.837 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.837 DEBUG:boto:Params: {} 2023-09-29T22:48:05.837 DEBUG:boto:Token: None 2023-09-29T22:48:05.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0= 2023-09-29T22:48:05.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0=', 'Content-Length': '0'} 2023-09-29T22:48:05.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f28e1280d08d2739-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.882 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.883 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.883 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.883 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.883 DEBUG:boto:Method: GET 2023-09-29T22:48:05.883 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.883 DEBUG:boto:Data: 2023-09-29T22:48:05.883 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.883 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.884 DEBUG:boto:Params: {} 2023-09-29T22:48:05.884 DEBUG:boto:Token: None 2023-09-29T22:48:05.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:48:05.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0= 2023-09-29T22:48:05.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+dFagiY3Dd9qIoZALJIBSY6Yrx0=', 'Content-Length': '0'} 2023-09-29T22:48:05.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e80a63d36419d5c9-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.887 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.888 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.888 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.888 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.888 DEBUG:boto:Method: GET 2023-09-29T22:48:05.888 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.888 DEBUG:boto:Data: 2023-09-29T22:48:05.888 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.888 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.889 DEBUG:boto:Params: {} 2023-09-29T22:48:05.889 DEBUG:boto:Token: None 2023-09-29T22:48:05.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0236b4aea27db13-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.894 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.894 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.894 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.895 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.895 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.895 DEBUG:boto:Method: GET 2023-09-29T22:48:05.895 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.895 DEBUG:boto:Data: 2023-09-29T22:48:05.895 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.895 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.895 DEBUG:boto:Params: {} 2023-09-29T22:48:05.896 DEBUG:boto:Token: None 2023-09-29T22:48:05.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.902 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d9ce9053a993a0e-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.902 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.903 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.903 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.903 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.903 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:48:05.903 DEBUG:boto:Data: 2023-09-29T22:48:05.903 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.904 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.904 DEBUG:boto:Params: {} 2023-09-29T22:48:05.904 DEBUG:boto:Token: None 2023-09-29T22:48:05.904 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33 2023-09-29T22:48:05.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JH/83j9PWiz07iQIKp72LvhqBRg= 2023-09-29T22:48:05.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JH/83j9PWiz07iQIKp72LvhqBRg=', 'Content-Length': '0'} 2023-09-29T22:48:05.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0f0d826c43de77e-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.907 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.907 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.907 DEBUG:boto:Method: HEAD 2023-09-29T22:48:05.907 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:48:05.907 DEBUG:boto:Data: 2023-09-29T22:48:05.907 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.907 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.907 DEBUG:boto:Params: {} 2023-09-29T22:48:05.908 DEBUG:boto:Token: None 2023-09-29T22:48:05.908 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33 2023-09-29T22:48:05.908 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JH/83j9PWiz07iQIKp72LvhqBRg= 2023-09-29T22:48:05.908 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JH/83j9PWiz07iQIKp72LvhqBRg=', 'Content-Length': '0'} 2023-09-29T22:48:05.910 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007804d59eaab7e050-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:48:05.910 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.911 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.911 DEBUG:boto:Method: GET 2023-09-29T22:48:05.911 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:48:05.911 DEBUG:boto:Data: 2023-09-29T22:48:05.911 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.911 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.911 DEBUG:boto:Params: {} 2023-09-29T22:48:05.912 DEBUG:boto:Token: None 2023-09-29T22:48:05.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33 2023-09-29T22:48:05.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:02qiu7yO/JH7bePbMb/RkmoHEMA= 2023-09-29T22:48:05.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:02qiu7yO/JH7bePbMb/RkmoHEMA=', 'Content-Length': '0'} 2023-09-29T22:48:05.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f2c800d2863fbf8-0065175425-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.954 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:48:05.955 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:48:05.955 DEBUG:boto:Method: GET 2023-09-29T22:48:05.955 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:48:05.955 DEBUG:boto:Data: 2023-09-29T22:48:05.955 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.955 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.955 DEBUG:boto:Params: {} 2023-09-29T22:48:05.956 DEBUG:boto:Token: None 2023-09-29T22:48:05.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/obj33 2023-09-29T22:48:05.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:02qiu7yO/JH7bePbMb/RkmoHEMA= 2023-09-29T22:48:05.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:02qiu7yO/JH7bePbMb/RkmoHEMA=', 'Content-Length': '0'} 2023-09-29T22:48:05.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068766219466cd99e-0065175425-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.960 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.960 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.960 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.960 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.960 DEBUG:boto:Method: GET 2023-09-29T22:48:05.960 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.961 DEBUG:boto:Data: 2023-09-29T22:48:05.961 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:05.961 DEBUG:boto:Port: 8001 2023-09-29T22:48:05.961 DEBUG:boto:Params: {} 2023-09-29T22:48:05.961 DEBUG:boto:Token: None 2023-09-29T22:48:05.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.962 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003857596b7c0298a9-0065175425-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:05.966 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:05.966 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:05.967 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:05.967 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.967 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.967 DEBUG:boto:Method: GET 2023-09-29T22:48:05.967 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:48:05.967 DEBUG:boto:Data: 2023-09-29T22:48:05.967 DEBUG:boto:Headers: {} 2023-09-29T22:48:05.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:05.968 DEBUG:boto:Port: 8000 2023-09-29T22:48:05.968 DEBUG:boto:Params: {} 2023-09-29T22:48:05.968 DEBUG:boto:Token: None 2023-09-29T22:48:05.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:05 GMT /hcnphz-59/ 2023-09-29T22:48:05.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g= 2023-09-29T22:48:05.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGH52E7MK2YXEAEkdK7w6XqUa7g=', 'Content-Length': '0'} 2023-09-29T22:48:05.975 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df5971ed552aecbd-0065175425-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.018 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:48:06.019 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.019 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.019 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.019 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.019 DEBUG:boto:Method: GET 2023-09-29T22:48:06.020 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.020 DEBUG:boto:Data: 2023-09-29T22:48:06.020 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.020 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.020 DEBUG:boto:Params: {} 2023-09-29T22:48:06.020 DEBUG:boto:Token: None 2023-09-29T22:48:06.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs= 2023-09-29T22:48:06.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs=', 'Content-Length': '0'} 2023-09-29T22:48:06.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c0ca68cd4e30ca6-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.024 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.024 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.024 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.024 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.024 DEBUG:boto:Method: GET 2023-09-29T22:48:06.025 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.025 DEBUG:boto:Data: 2023-09-29T22:48:06.025 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.025 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.025 DEBUG:boto:Params: {} 2023-09-29T22:48:06.025 DEBUG:boto:Token: None 2023-09-29T22:48:06.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs= 2023-09-29T22:48:06.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs=', 'Content-Length': '0'} 2023-09-29T22:48:06.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de380d0ceae4e12f-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.029 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.029 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.029 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.029 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.029 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.029 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.029 DEBUG:boto:Data: 2023-09-29T22:48:06.030 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.030 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.030 DEBUG:boto:Params: {} 2023-09-29T22:48:06.030 DEBUG:boto:Token: None 2023-09-29T22:48:06.030 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TvVNcgS80QNmfLTP0G5wyqLXatQ= 2023-09-29T22:48:06.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TvVNcgS80QNmfLTP0G5wyqLXatQ=', 'Content-Length': '0'} 2023-09-29T22:48:06.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987086d9a58af0ec-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.033 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.033 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.033 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.034 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.034 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.034 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.034 DEBUG:boto:Data: 2023-09-29T22:48:06.034 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.034 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.034 DEBUG:boto:Params: {} 2023-09-29T22:48:06.034 DEBUG:boto:Token: None 2023-09-29T22:48:06.035 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TvVNcgS80QNmfLTP0G5wyqLXatQ= 2023-09-29T22:48:06.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TvVNcgS80QNmfLTP0G5wyqLXatQ=', 'Content-Length': '0'} 2023-09-29T22:48:06.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000008c4db354513c6f6-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:48:06.038 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.038 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.038 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.038 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.038 DEBUG:boto:Method: GET 2023-09-29T22:48:06.038 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.038 DEBUG:boto:Data: 2023-09-29T22:48:06.038 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.039 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.039 DEBUG:boto:Params: {} 2023-09-29T22:48:06.039 DEBUG:boto:Token: None 2023-09-29T22:48:06.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.039 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs= 2023-09-29T22:48:06.039 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs=', 'Content-Length': '0'} 2023-09-29T22:48:06.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042122c50e5ef4b22-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.042 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.042 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.042 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.043 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.043 DEBUG:boto:Method: GET 2023-09-29T22:48:06.043 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.043 DEBUG:boto:Data: 2023-09-29T22:48:06.043 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.043 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.043 DEBUG:boto:Params: {} 2023-09-29T22:48:06.043 DEBUG:boto:Token: None 2023-09-29T22:48:06.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:48:06.044 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs= 2023-09-29T22:48:06.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3UzbTTK7ueOQqmJ9Ps7zAYFnFrs=', 'Content-Length': '0'} 2023-09-29T22:48:06.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025042713d332ef66-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.047 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.047 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.047 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.047 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.048 DEBUG:boto:Method: GET 2023-09-29T22:48:06.048 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.048 DEBUG:boto:Data: 2023-09-29T22:48:06.048 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.048 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.048 DEBUG:boto:Params: {} 2023-09-29T22:48:06.048 DEBUG:boto:Token: None 2023-09-29T22:48:06.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000004bac4ec70c2385d-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.053 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.054 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.054 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.054 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.054 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.054 DEBUG:boto:Method: GET 2023-09-29T22:48:06.054 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.054 DEBUG:boto:Data: 2023-09-29T22:48:06.054 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.055 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.055 DEBUG:boto:Params: {} 2023-09-29T22:48:06.055 DEBUG:boto:Token: None 2023-09-29T22:48:06.055 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.055 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a3bb84d44946277-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.102 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.103 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.103 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.103 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.103 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:48:06.103 DEBUG:boto:Data: 2023-09-29T22:48:06.103 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.104 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.104 DEBUG:boto:Params: {} 2023-09-29T22:48:06.104 DEBUG:boto:Token: None 2023-09-29T22:48:06.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34 2023-09-29T22:48:06.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V2IDaOOl+bb0qb68wyDnIRFjooI= 2023-09-29T22:48:06.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V2IDaOOl+bb0qb68wyDnIRFjooI=', 'Content-Length': '0'} 2023-09-29T22:48:06.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1e1b5ea0b58ab00-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.108 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.108 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.108 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.108 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:48:06.108 DEBUG:boto:Data: 2023-09-29T22:48:06.108 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.108 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.108 DEBUG:boto:Params: {} 2023-09-29T22:48:06.109 DEBUG:boto:Token: None 2023-09-29T22:48:06.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34 2023-09-29T22:48:06.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:V2IDaOOl+bb0qb68wyDnIRFjooI= 2023-09-29T22:48:06.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V2IDaOOl+bb0qb68wyDnIRFjooI=', 'Content-Length': '0'} 2023-09-29T22:48:06.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b89cdeec236e406b-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:48:06.112 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.112 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.112 DEBUG:boto:Method: GET 2023-09-29T22:48:06.112 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:48:06.112 DEBUG:boto:Data: 2023-09-29T22:48:06.112 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.113 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.113 DEBUG:boto:Params: {} 2023-09-29T22:48:06.113 DEBUG:boto:Token: None 2023-09-29T22:48:06.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34 2023-09-29T22:48:06.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EwCnftqFYdjuFCL8HHqbXQjkrog= 2023-09-29T22:48:06.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EwCnftqFYdjuFCL8HHqbXQjkrog=', 'Content-Length': '0'} 2023-09-29T22:48:06.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0d2f1f36ecd01fc-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.162 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:48:06.163 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:48:06.163 DEBUG:boto:Method: GET 2023-09-29T22:48:06.163 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:48:06.163 DEBUG:boto:Data: 2023-09-29T22:48:06.163 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.163 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.163 DEBUG:boto:Params: {} 2023-09-29T22:48:06.164 DEBUG:boto:Token: None 2023-09-29T22:48:06.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj34 2023-09-29T22:48:06.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EwCnftqFYdjuFCL8HHqbXQjkrog= 2023-09-29T22:48:06.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EwCnftqFYdjuFCL8HHqbXQjkrog=', 'Content-Length': '0'} 2023-09-29T22:48:06.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d9b0c03c5f9a04d-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.166 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.167 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.167 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.167 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.167 DEBUG:boto:Method: GET 2023-09-29T22:48:06.167 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.167 DEBUG:boto:Data: 2023-09-29T22:48:06.167 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.167 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.168 DEBUG:boto:Params: {} 2023-09-29T22:48:06.168 DEBUG:boto:Token: None 2023-09-29T22:48:06.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.171 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b407e6a31550d9f8-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.171 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.171 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.172 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.172 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.172 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.172 DEBUG:boto:Method: GET 2023-09-29T22:48:06.172 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:48:06.172 DEBUG:boto:Data: 2023-09-29T22:48:06.172 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.172 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.172 DEBUG:boto:Params: {} 2023-09-29T22:48:06.173 DEBUG:boto:Token: None 2023-09-29T22:48:06.173 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.173 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.178 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000517bd7336cece990-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.178 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:48:06.179 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.179 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.179 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.179 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.179 DEBUG:boto:Method: GET 2023-09-29T22:48:06.180 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.180 DEBUG:boto:Data: 2023-09-29T22:48:06.180 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.180 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.180 DEBUG:boto:Params: {} 2023-09-29T22:48:06.180 DEBUG:boto:Token: None 2023-09-29T22:48:06.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o= 2023-09-29T22:48:06.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o=', 'Content-Length': '0'} 2023-09-29T22:48:06.183 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d9acb0ba4a6ea05-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.183 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.183 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.183 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.183 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.183 DEBUG:boto:Method: GET 2023-09-29T22:48:06.184 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.184 DEBUG:boto:Data: 2023-09-29T22:48:06.184 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.184 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.184 DEBUG:boto:Params: {} 2023-09-29T22:48:06.184 DEBUG:boto:Token: None 2023-09-29T22:48:06.184 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.185 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o= 2023-09-29T22:48:06.185 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o=', 'Content-Length': '0'} 2023-09-29T22:48:06.187 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c1de76bee1187b5-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.187 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.187 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.187 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.187 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.187 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.188 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.188 DEBUG:boto:Data: 2023-09-29T22:48:06.188 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.188 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.188 DEBUG:boto:Params: {} 2023-09-29T22:48:06.188 DEBUG:boto:Token: None 2023-09-29T22:48:06.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DYTDWpiiATcZqf/mb62uwJUX4m0= 2023-09-29T22:48:06.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DYTDWpiiATcZqf/mb62uwJUX4m0=', 'Content-Length': '0'} 2023-09-29T22:48:06.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6e36102d861aeb7-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.191 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.191 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.191 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.191 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.191 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.192 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.192 DEBUG:boto:Data: 2023-09-29T22:48:06.192 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.192 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.192 DEBUG:boto:Params: {} 2023-09-29T22:48:06.192 DEBUG:boto:Token: None 2023-09-29T22:48:06.192 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DYTDWpiiATcZqf/mb62uwJUX4m0= 2023-09-29T22:48:06.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DYTDWpiiATcZqf/mb62uwJUX4m0=', 'Content-Length': '0'} 2023-09-29T22:48:06.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091b9eb6543355cdc-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:48:06.195 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.195 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.195 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.195 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.196 DEBUG:boto:Method: GET 2023-09-29T22:48:06.196 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.196 DEBUG:boto:Data: 2023-09-29T22:48:06.196 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.196 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.196 DEBUG:boto:Params: {} 2023-09-29T22:48:06.196 DEBUG:boto:Token: None 2023-09-29T22:48:06.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o= 2023-09-29T22:48:06.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o=', 'Content-Length': '0'} 2023-09-29T22:48:06.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d77b121c43f004c-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.242 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.242 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.243 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.243 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.243 DEBUG:boto:Method: GET 2023-09-29T22:48:06.243 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.243 DEBUG:boto:Data: 2023-09-29T22:48:06.243 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.243 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.243 DEBUG:boto:Params: {} 2023-09-29T22:48:06.244 DEBUG:boto:Token: None 2023-09-29T22:48:06.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:48:06.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o= 2023-09-29T22:48:06.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tlwl+tIPKKIIYOe3j71a4t+P7/o=', 'Content-Length': '0'} 2023-09-29T22:48:06.247 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0ae5b9f6b4c2f16-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.247 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.247 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.248 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.248 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.248 DEBUG:boto:Method: GET 2023-09-29T22:48:06.248 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.248 DEBUG:boto:Data: 2023-09-29T22:48:06.248 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.248 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.248 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.248 DEBUG:boto:Params: {} 2023-09-29T22:48:06.249 DEBUG:boto:Token: None 2023-09-29T22:48:06.249 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cbfda23bc163c04-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.254 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.254 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.254 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.255 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.255 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.255 DEBUG:boto:Method: GET 2023-09-29T22:48:06.255 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.255 DEBUG:boto:Data: 2023-09-29T22:48:06.255 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.255 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.255 DEBUG:boto:Params: {} 2023-09-29T22:48:06.256 DEBUG:boto:Token: None 2023-09-29T22:48:06.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.263 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000071ac17fa6c05dac0-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.264 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.264 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.264 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.264 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.264 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:48:06.264 DEBUG:boto:Data: 2023-09-29T22:48:06.265 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.265 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.265 DEBUG:boto:Params: {} 2023-09-29T22:48:06.265 DEBUG:boto:Token: None 2023-09-29T22:48:06.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38 2023-09-29T22:48:06.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tSkOYKR6BCehLVvtZeo0vfOtAaY= 2023-09-29T22:48:06.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tSkOYKR6BCehLVvtZeo0vfOtAaY=', 'Content-Length': '0'} 2023-09-29T22:48:06.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088834dcb26c5b53f-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.267 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.268 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.268 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.268 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:48:06.268 DEBUG:boto:Data: 2023-09-29T22:48:06.268 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.268 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.268 DEBUG:boto:Params: {} 2023-09-29T22:48:06.269 DEBUG:boto:Token: None 2023-09-29T22:48:06.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38 2023-09-29T22:48:06.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tSkOYKR6BCehLVvtZeo0vfOtAaY= 2023-09-29T22:48:06.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tSkOYKR6BCehLVvtZeo0vfOtAaY=', 'Content-Length': '0'} 2023-09-29T22:48:06.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008facac0ae3f5bb18-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:48:06.271 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.271 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.272 DEBUG:boto:Method: GET 2023-09-29T22:48:06.272 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:48:06.272 DEBUG:boto:Data: 2023-09-29T22:48:06.272 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.272 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.272 DEBUG:boto:Params: {} 2023-09-29T22:48:06.272 DEBUG:boto:Token: None 2023-09-29T22:48:06.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38 2023-09-29T22:48:06.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+f+t15gwh88r9RkaGFNm7BlFGCY= 2023-09-29T22:48:06.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+f+t15gwh88r9RkaGFNm7BlFGCY=', 'Content-Length': '0'} 2023-09-29T22:48:06.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a66bb98f3b2446d-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.318 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:48:06.319 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:48:06.319 DEBUG:boto:Method: GET 2023-09-29T22:48:06.319 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:48:06.319 DEBUG:boto:Data: 2023-09-29T22:48:06.319 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.319 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.319 DEBUG:boto:Params: {} 2023-09-29T22:48:06.320 DEBUG:boto:Token: None 2023-09-29T22:48:06.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj38 2023-09-29T22:48:06.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+f+t15gwh88r9RkaGFNm7BlFGCY= 2023-09-29T22:48:06.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+f+t15gwh88r9RkaGFNm7BlFGCY=', 'Content-Length': '0'} 2023-09-29T22:48:06.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bcd2ca77f28cbd8f-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.323 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.323 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.323 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.323 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.324 DEBUG:boto:Method: GET 2023-09-29T22:48:06.324 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.324 DEBUG:boto:Data: 2023-09-29T22:48:06.324 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.324 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.324 DEBUG:boto:Params: {} 2023-09-29T22:48:06.324 DEBUG:boto:Token: None 2023-09-29T22:48:06.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.330 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005ee29420df4d1c76-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.330 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.330 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.330 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.331 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.331 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.331 DEBUG:boto:Method: GET 2023-09-29T22:48:06.331 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:48:06.331 DEBUG:boto:Data: 2023-09-29T22:48:06.331 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.331 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.331 DEBUG:boto:Params: {} 2023-09-29T22:48:06.331 DEBUG:boto:Token: None 2023-09-29T22:48:06.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.339 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a9f753db316a09d5-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.382 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:48:06.383 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.383 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.383 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.383 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.383 DEBUG:boto:Method: GET 2023-09-29T22:48:06.385 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.385 DEBUG:boto:Data: 2023-09-29T22:48:06.385 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.385 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.385 DEBUG:boto:Params: {} 2023-09-29T22:48:06.385 DEBUG:boto:Token: None 2023-09-29T22:48:06.386 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8= 2023-09-29T22:48:06.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8=', 'Content-Length': '0'} 2023-09-29T22:48:06.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d3e162fd94eb22c-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.389 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.389 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.389 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.390 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.390 DEBUG:boto:Method: GET 2023-09-29T22:48:06.390 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.390 DEBUG:boto:Data: 2023-09-29T22:48:06.390 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.390 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.390 DEBUG:boto:Params: {} 2023-09-29T22:48:06.390 DEBUG:boto:Token: None 2023-09-29T22:48:06.391 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8= 2023-09-29T22:48:06.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8=', 'Content-Length': '0'} 2023-09-29T22:48:06.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041a1bbeb3beadcd8-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.394 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.395 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.395 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.395 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.395 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.395 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.395 DEBUG:boto:Data: 2023-09-29T22:48:06.395 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.395 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.396 DEBUG:boto:Params: {} 2023-09-29T22:48:06.396 DEBUG:boto:Token: None 2023-09-29T22:48:06.396 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.396 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fEgWWMhx/rxB2uTRtsONAvUofeE= 2023-09-29T22:48:06.396 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fEgWWMhx/rxB2uTRtsONAvUofeE=', 'Content-Length': '0'} 2023-09-29T22:48:06.399 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009be1497c57bc5eec-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.399 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.399 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.399 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.399 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.400 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.400 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.400 DEBUG:boto:Data: 2023-09-29T22:48:06.400 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.400 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.400 DEBUG:boto:Params: {} 2023-09-29T22:48:06.401 DEBUG:boto:Token: None 2023-09-29T22:48:06.401 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fEgWWMhx/rxB2uTRtsONAvUofeE= 2023-09-29T22:48:06.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fEgWWMhx/rxB2uTRtsONAvUofeE=', 'Content-Length': '0'} 2023-09-29T22:48:06.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033315058885f3ae3-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:48:06.404 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.404 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.404 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.404 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.404 DEBUG:boto:Method: GET 2023-09-29T22:48:06.404 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.405 DEBUG:boto:Data: 2023-09-29T22:48:06.405 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.405 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.405 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.405 DEBUG:boto:Params: {} 2023-09-29T22:48:06.405 DEBUG:boto:Token: None 2023-09-29T22:48:06.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8= 2023-09-29T22:48:06.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8=', 'Content-Length': '0'} 2023-09-29T22:48:06.408 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000281c887ab216cf59-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.408 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.408 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.409 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.409 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.409 DEBUG:boto:Method: GET 2023-09-29T22:48:06.409 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.409 DEBUG:boto:Data: 2023-09-29T22:48:06.409 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.409 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.409 DEBUG:boto:Params: {} 2023-09-29T22:48:06.409 DEBUG:boto:Token: None 2023-09-29T22:48:06.410 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:48:06.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8= 2023-09-29T22:48:06.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mfKPt6rufUAKWZ5vtex0NDhSLj8=', 'Content-Length': '0'} 2023-09-29T22:48:06.413 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48982702d825e75-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.413 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.413 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.413 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.413 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.413 DEBUG:boto:Method: GET 2023-09-29T22:48:06.414 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.414 DEBUG:boto:Data: 2023-09-29T22:48:06.414 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.414 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.414 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.414 DEBUG:boto:Params: {} 2023-09-29T22:48:06.414 DEBUG:boto:Token: None 2023-09-29T22:48:06.415 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.415 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.415 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.419 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001878ded0ecf5b62a-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.462 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.463 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.463 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.463 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.463 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.464 DEBUG:boto:Method: GET 2023-09-29T22:48:06.464 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.464 DEBUG:boto:Data: 2023-09-29T22:48:06.464 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.464 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.464 DEBUG:boto:Params: {} 2023-09-29T22:48:06.464 DEBUG:boto:Token: None 2023-09-29T22:48:06.465 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.465 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.465 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.473 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b61acd505aeaaf5-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.474 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.474 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.474 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.474 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.474 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:48:06.475 DEBUG:boto:Data: 2023-09-29T22:48:06.475 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.475 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.475 DEBUG:boto:Params: {} 2023-09-29T22:48:06.475 DEBUG:boto:Token: None 2023-09-29T22:48:06.475 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44 2023-09-29T22:48:06.475 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QDZ4ehaPdL4z7Sz9nUO5sI04myc= 2023-09-29T22:48:06.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QDZ4ehaPdL4z7Sz9nUO5sI04myc=', 'Content-Length': '0'} 2023-09-29T22:48:06.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067cbe681d7e698a4-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.478 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.478 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.479 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.479 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:48:06.479 DEBUG:boto:Data: 2023-09-29T22:48:06.479 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.479 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.479 DEBUG:boto:Params: {} 2023-09-29T22:48:06.479 DEBUG:boto:Token: None 2023-09-29T22:48:06.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44 2023-09-29T22:48:06.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QDZ4ehaPdL4z7Sz9nUO5sI04myc= 2023-09-29T22:48:06.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QDZ4ehaPdL4z7Sz9nUO5sI04myc=', 'Content-Length': '0'} 2023-09-29T22:48:06.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000145d634c2cceee81-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:48:06.483 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.483 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.483 DEBUG:boto:Method: GET 2023-09-29T22:48:06.483 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:48:06.483 DEBUG:boto:Data: 2023-09-29T22:48:06.484 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.484 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.484 DEBUG:boto:Params: {} 2023-09-29T22:48:06.484 DEBUG:boto:Token: None 2023-09-29T22:48:06.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44 2023-09-29T22:48:06.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLt4EZeD/RTHYP1Kh67s29TdU7U= 2023-09-29T22:48:06.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLt4EZeD/RTHYP1Kh67s29TdU7U=', 'Content-Length': '0'} 2023-09-29T22:48:06.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d9a9342c80a4acb-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.487 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:48:06.487 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:48:06.488 DEBUG:boto:Method: GET 2023-09-29T22:48:06.488 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:48:06.488 DEBUG:boto:Data: 2023-09-29T22:48:06.488 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.488 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.488 DEBUG:boto:Params: {} 2023-09-29T22:48:06.488 DEBUG:boto:Token: None 2023-09-29T22:48:06.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj44 2023-09-29T22:48:06.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dLt4EZeD/RTHYP1Kh67s29TdU7U= 2023-09-29T22:48:06.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dLt4EZeD/RTHYP1Kh67s29TdU7U=', 'Content-Length': '0'} 2023-09-29T22:48:06.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000540885c2a6ea8e83-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.534 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.534 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.535 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.535 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.535 DEBUG:boto:Method: GET 2023-09-29T22:48:06.535 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.535 DEBUG:boto:Data: 2023-09-29T22:48:06.535 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.535 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.535 DEBUG:boto:Params: {} 2023-09-29T22:48:06.536 DEBUG:boto:Token: None 2023-09-29T22:48:06.536 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.541 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d25fa1243f1daa54-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.541 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.541 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.541 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.542 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.542 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.542 DEBUG:boto:Method: GET 2023-09-29T22:48:06.542 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:48:06.542 DEBUG:boto:Data: 2023-09-29T22:48:06.542 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.543 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.543 DEBUG:boto:Params: {} 2023-09-29T22:48:06.543 DEBUG:boto:Token: None 2023-09-29T22:48:06.543 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.543 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.551 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093446ab3946314fc-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.551 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:48:06.552 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.552 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.552 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.552 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.552 DEBUG:boto:Method: GET 2023-09-29T22:48:06.552 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.552 DEBUG:boto:Data: 2023-09-29T22:48:06.552 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.553 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.553 DEBUG:boto:Params: {} 2023-09-29T22:48:06.553 DEBUG:boto:Token: None 2023-09-29T22:48:06.553 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc= 2023-09-29T22:48:06.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc=', 'Content-Length': '0'} 2023-09-29T22:48:06.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac74866afa7a5a03-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.555 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.556 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.556 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.556 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.556 DEBUG:boto:Method: GET 2023-09-29T22:48:06.556 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.556 DEBUG:boto:Data: 2023-09-29T22:48:06.556 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.557 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.557 DEBUG:boto:Params: {} 2023-09-29T22:48:06.557 DEBUG:boto:Token: None 2023-09-29T22:48:06.557 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc= 2023-09-29T22:48:06.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc=', 'Content-Length': '0'} 2023-09-29T22:48:06.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dfec0f92864ec0b-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.560 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.560 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.560 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.560 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.560 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.560 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.560 DEBUG:boto:Data: 2023-09-29T22:48:06.560 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.560 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.561 DEBUG:boto:Params: {} 2023-09-29T22:48:06.561 DEBUG:boto:Token: None 2023-09-29T22:48:06.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS5X3OwK2oh34HpHUIiP6XI4piU= 2023-09-29T22:48:06.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS5X3OwK2oh34HpHUIiP6XI4piU=', 'Content-Length': '0'} 2023-09-29T22:48:06.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018a7fb7c963fc26d-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.563 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.564 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.564 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.564 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.564 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.564 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.564 DEBUG:boto:Data: 2023-09-29T22:48:06.564 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.565 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.565 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.565 DEBUG:boto:Params: {} 2023-09-29T22:48:06.565 DEBUG:boto:Token: None 2023-09-29T22:48:06.565 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS5X3OwK2oh34HpHUIiP6XI4piU= 2023-09-29T22:48:06.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS5X3OwK2oh34HpHUIiP6XI4piU=', 'Content-Length': '0'} 2023-09-29T22:48:06.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c892e65da65bcdc0-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:48:06.568 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.568 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.568 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.569 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.569 DEBUG:boto:Method: GET 2023-09-29T22:48:06.569 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.569 DEBUG:boto:Data: 2023-09-29T22:48:06.569 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.569 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.569 DEBUG:boto:Params: {} 2023-09-29T22:48:06.570 DEBUG:boto:Token: None 2023-09-29T22:48:06.570 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.570 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc= 2023-09-29T22:48:06.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc=', 'Content-Length': '0'} 2023-09-29T22:48:06.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005dc6335f87961ee3-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.618 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.619 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.619 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.619 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.619 DEBUG:boto:Method: GET 2023-09-29T22:48:06.619 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.619 DEBUG:boto:Data: 2023-09-29T22:48:06.619 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.619 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.619 DEBUG:boto:Params: {} 2023-09-29T22:48:06.620 DEBUG:boto:Token: None 2023-09-29T22:48:06.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:48:06.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc= 2023-09-29T22:48:06.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/izO9EcDYAeRcuKT+7js33Xsvoc=', 'Content-Length': '0'} 2023-09-29T22:48:06.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000982732bb8a4d044f-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.623 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.624 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.624 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.624 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.624 DEBUG:boto:Method: GET 2023-09-29T22:48:06.624 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.624 DEBUG:boto:Data: 2023-09-29T22:48:06.624 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.624 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.625 DEBUG:boto:Params: {} 2023-09-29T22:48:06.625 DEBUG:boto:Token: None 2023-09-29T22:48:06.625 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.625 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004f9784118a6279cb-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.630 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.630 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.631 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.631 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.631 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.631 DEBUG:boto:Method: GET 2023-09-29T22:48:06.631 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.631 DEBUG:boto:Data: 2023-09-29T22:48:06.631 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.632 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.632 DEBUG:boto:Params: {} 2023-09-29T22:48:06.632 DEBUG:boto:Token: None 2023-09-29T22:48:06.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.640 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006da34942bfb1ed6c-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.640 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.640 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.641 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.641 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.641 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:48:06.641 DEBUG:boto:Data: 2023-09-29T22:48:06.641 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.641 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.641 DEBUG:boto:Params: {} 2023-09-29T22:48:06.642 DEBUG:boto:Token: None 2023-09-29T22:48:06.642 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46 2023-09-29T22:48:06.642 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tsR4qoROt9fMb1I2ZdE8/osaGSQ= 2023-09-29T22:48:06.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tsR4qoROt9fMb1I2ZdE8/osaGSQ=', 'Content-Length': '0'} 2023-09-29T22:48:06.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be11c7418a1d2f3b-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.644 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.645 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.645 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.645 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:48:06.645 DEBUG:boto:Data: 2023-09-29T22:48:06.646 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.646 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.646 DEBUG:boto:Params: {} 2023-09-29T22:48:06.646 DEBUG:boto:Token: None 2023-09-29T22:48:06.646 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46 2023-09-29T22:48:06.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tsR4qoROt9fMb1I2ZdE8/osaGSQ= 2023-09-29T22:48:06.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tsR4qoROt9fMb1I2ZdE8/osaGSQ=', 'Content-Length': '0'} 2023-09-29T22:48:06.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d40eb9aad78f3e-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:48:06.649 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.649 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.649 DEBUG:boto:Method: GET 2023-09-29T22:48:06.649 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:48:06.650 DEBUG:boto:Data: 2023-09-29T22:48:06.650 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.650 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.650 DEBUG:boto:Params: {} 2023-09-29T22:48:06.650 DEBUG:boto:Token: None 2023-09-29T22:48:06.650 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46 2023-09-29T22:48:06.650 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mrOZj7VNZm6vDXMt3yzw/UVQwbc= 2023-09-29T22:48:06.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mrOZj7VNZm6vDXMt3yzw/UVQwbc=', 'Content-Length': '0'} 2023-09-29T22:48:06.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a102925edbc0c05-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.653 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:48:06.653 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:48:06.654 DEBUG:boto:Method: GET 2023-09-29T22:48:06.654 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:48:06.654 DEBUG:boto:Data: 2023-09-29T22:48:06.654 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.654 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.654 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.654 DEBUG:boto:Params: {} 2023-09-29T22:48:06.655 DEBUG:boto:Token: None 2023-09-29T22:48:06.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj46 2023-09-29T22:48:06.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mrOZj7VNZm6vDXMt3yzw/UVQwbc= 2023-09-29T22:48:06.655 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mrOZj7VNZm6vDXMt3yzw/UVQwbc=', 'Content-Length': '0'} 2023-09-29T22:48:06.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a738664032a30736-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.658 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.658 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.658 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.658 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.658 DEBUG:boto:Method: GET 2023-09-29T22:48:06.659 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.659 DEBUG:boto:Data: 2023-09-29T22:48:06.659 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.659 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.659 DEBUG:boto:Params: {} 2023-09-29T22:48:06.659 DEBUG:boto:Token: None 2023-09-29T22:48:06.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c2610fc1db48a33-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.664 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.664 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.664 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.665 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.665 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.665 DEBUG:boto:Method: GET 2023-09-29T22:48:06.665 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:48:06.665 DEBUG:boto:Data: 2023-09-29T22:48:06.665 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.666 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.666 DEBUG:boto:Params: {} 2023-09-29T22:48:06.666 DEBUG:boto:Token: None 2023-09-29T22:48:06.666 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a63736784cd7de5e-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.673 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:48:06.674 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.674 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.674 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.674 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.674 DEBUG:boto:Method: GET 2023-09-29T22:48:06.674 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.674 DEBUG:boto:Data: 2023-09-29T22:48:06.675 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.675 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.675 DEBUG:boto:Params: {} 2023-09-29T22:48:06.675 DEBUG:boto:Token: None 2023-09-29T22:48:06.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0= 2023-09-29T22:48:06.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0=', 'Content-Length': '0'} 2023-09-29T22:48:06.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0b13ceca56c5a79-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.718 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.718 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.719 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.719 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.719 DEBUG:boto:Method: GET 2023-09-29T22:48:06.719 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.719 DEBUG:boto:Data: 2023-09-29T22:48:06.719 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.719 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.719 DEBUG:boto:Params: {} 2023-09-29T22:48:06.720 DEBUG:boto:Token: None 2023-09-29T22:48:06.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0= 2023-09-29T22:48:06.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0=', 'Content-Length': '0'} 2023-09-29T22:48:06.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c972eaf98978f7be-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.725 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.725 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.725 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.725 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.725 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.726 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.726 DEBUG:boto:Data: 2023-09-29T22:48:06.726 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.726 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.726 DEBUG:boto:Params: {} 2023-09-29T22:48:06.726 DEBUG:boto:Token: None 2023-09-29T22:48:06.726 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qwplHnvOG1E08216qKBLSH5XS98= 2023-09-29T22:48:06.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qwplHnvOG1E08216qKBLSH5XS98=', 'Content-Length': '0'} 2023-09-29T22:48:06.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000257f8fe41d81edaa-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.730 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.730 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.730 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.730 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.730 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.731 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.731 DEBUG:boto:Data: 2023-09-29T22:48:06.731 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.731 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.731 DEBUG:boto:Params: {} 2023-09-29T22:48:06.731 DEBUG:boto:Token: None 2023-09-29T22:48:06.731 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qwplHnvOG1E08216qKBLSH5XS98= 2023-09-29T22:48:06.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qwplHnvOG1E08216qKBLSH5XS98=', 'Content-Length': '0'} 2023-09-29T22:48:06.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec15eee8f79c0ea6-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:48:06.734 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.735 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.735 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.735 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.735 DEBUG:boto:Method: GET 2023-09-29T22:48:06.735 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.735 DEBUG:boto:Data: 2023-09-29T22:48:06.735 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.735 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.736 DEBUG:boto:Params: {} 2023-09-29T22:48:06.736 DEBUG:boto:Token: None 2023-09-29T22:48:06.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0= 2023-09-29T22:48:06.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0=', 'Content-Length': '0'} 2023-09-29T22:48:06.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d614595a4f2db0-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.738 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.738 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.739 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.739 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.739 DEBUG:boto:Method: GET 2023-09-29T22:48:06.739 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.739 DEBUG:boto:Data: 2023-09-29T22:48:06.739 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.739 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.739 DEBUG:boto:Params: {} 2023-09-29T22:48:06.740 DEBUG:boto:Token: None 2023-09-29T22:48:06.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:48:06.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0= 2023-09-29T22:48:06.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5Zf2kTpUxKNhhj9Fo5o3Edc25Q0=', 'Content-Length': '0'} 2023-09-29T22:48:06.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001138dc9c1353ec68-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.782 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.783 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.783 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.783 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.783 DEBUG:boto:Method: GET 2023-09-29T22:48:06.783 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.783 DEBUG:boto:Data: 2023-09-29T22:48:06.783 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.784 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.784 DEBUG:boto:Params: {} 2023-09-29T22:48:06.784 DEBUG:boto:Token: None 2023-09-29T22:48:06.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5ab5c1c93aef374-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.789 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.789 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.790 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.790 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.790 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.790 DEBUG:boto:Method: GET 2023-09-29T22:48:06.790 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.790 DEBUG:boto:Data: 2023-09-29T22:48:06.790 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.790 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.791 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.791 DEBUG:boto:Params: {} 2023-09-29T22:48:06.791 DEBUG:boto:Token: None 2023-09-29T22:48:06.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.791 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000059c4c32922c0092c-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.800 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.800 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.800 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.800 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.801 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:48:06.801 DEBUG:boto:Data: 2023-09-29T22:48:06.801 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.801 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.801 DEBUG:boto:Params: {} 2023-09-29T22:48:06.801 DEBUG:boto:Token: None 2023-09-29T22:48:06.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51 2023-09-29T22:48:06.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SC0we6VuBWnmLFhonUrqjFahV28= 2023-09-29T22:48:06.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SC0we6VuBWnmLFhonUrqjFahV28=', 'Content-Length': '0'} 2023-09-29T22:48:06.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdb47070abb52371-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.804 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.804 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.804 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.804 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:48:06.805 DEBUG:boto:Data: 2023-09-29T22:48:06.805 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.805 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.805 DEBUG:boto:Params: {} 2023-09-29T22:48:06.805 DEBUG:boto:Token: None 2023-09-29T22:48:06.805 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51 2023-09-29T22:48:06.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SC0we6VuBWnmLFhonUrqjFahV28= 2023-09-29T22:48:06.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SC0we6VuBWnmLFhonUrqjFahV28=', 'Content-Length': '0'} 2023-09-29T22:48:06.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f7fd01a2279e82-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:48:06.808 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.808 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.808 DEBUG:boto:Method: GET 2023-09-29T22:48:06.808 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:48:06.809 DEBUG:boto:Data: 2023-09-29T22:48:06.809 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.809 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.809 DEBUG:boto:Params: {} 2023-09-29T22:48:06.809 DEBUG:boto:Token: None 2023-09-29T22:48:06.809 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51 2023-09-29T22:48:06.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BSlg4O7pnK8V57a/SLGE15pNgPo= 2023-09-29T22:48:06.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BSlg4O7pnK8V57a/SLGE15pNgPo=', 'Content-Length': '0'} 2023-09-29T22:48:06.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d124a9f237a449c-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.812 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:48:06.812 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:48:06.812 DEBUG:boto:Method: GET 2023-09-29T22:48:06.812 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:48:06.812 DEBUG:boto:Data: 2023-09-29T22:48:06.813 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.813 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.813 DEBUG:boto:Params: {} 2023-09-29T22:48:06.813 DEBUG:boto:Token: None 2023-09-29T22:48:06.813 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj51 2023-09-29T22:48:06.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BSlg4O7pnK8V57a/SLGE15pNgPo= 2023-09-29T22:48:06.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BSlg4O7pnK8V57a/SLGE15pNgPo=', 'Content-Length': '0'} 2023-09-29T22:48:06.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007063333a3fe579ef-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.862 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.863 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.863 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.863 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.863 DEBUG:boto:Method: GET 2023-09-29T22:48:06.863 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.863 DEBUG:boto:Data: 2023-09-29T22:48:06.863 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.863 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.863 DEBUG:boto:Params: {} 2023-09-29T22:48:06.864 DEBUG:boto:Token: None 2023-09-29T22:48:06.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d3dee0ee4e4960-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.870 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.870 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.870 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.870 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.870 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.871 DEBUG:boto:Method: GET 2023-09-29T22:48:06.871 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:48:06.871 DEBUG:boto:Data: 2023-09-29T22:48:06.871 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.871 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.871 DEBUG:boto:Params: {} 2023-09-29T22:48:06.871 DEBUG:boto:Token: None 2023-09-29T22:48:06.871 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd6c73f240b16b16-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.879 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:48:06.879 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:06.880 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:06.880 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.880 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.880 DEBUG:boto:Method: GET 2023-09-29T22:48:06.880 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.880 DEBUG:boto:Data: 2023-09-29T22:48:06.880 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.880 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.880 DEBUG:boto:Params: {} 2023-09-29T22:48:06.881 DEBUG:boto:Token: None 2023-09-29T22:48:06.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk= 2023-09-29T22:48:06.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk=', 'Content-Length': '0'} 2023-09-29T22:48:06.883 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047eed0b6ffd74a47-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.926 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:06.927 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:06.927 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.927 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.927 DEBUG:boto:Method: GET 2023-09-29T22:48:06.927 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.927 DEBUG:boto:Data: 2023-09-29T22:48:06.927 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.928 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.928 DEBUG:boto:Params: {} 2023-09-29T22:48:06.928 DEBUG:boto:Token: None 2023-09-29T22:48:06.928 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk= 2023-09-29T22:48:06.928 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk=', 'Content-Length': '0'} 2023-09-29T22:48:06.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004a475a64fbe322fb-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.932 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:06.932 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:06.932 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.932 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.932 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.932 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.932 DEBUG:boto:Data: 2023-09-29T22:48:06.932 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.933 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.933 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.933 DEBUG:boto:Params: {} 2023-09-29T22:48:06.933 DEBUG:boto:Token: None 2023-09-29T22:48:06.933 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.934 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z6OuGXqUqepT/nvwLhwEf+MSjsY= 2023-09-29T22:48:06.934 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z6OuGXqUqepT/nvwLhwEf+MSjsY=', 'Content-Length': '0'} 2023-09-29T22:48:06.937 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff485a0150fca0ec-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.937 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:06.937 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:06.937 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.938 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.938 DEBUG:boto:Method: HEAD 2023-09-29T22:48:06.938 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.938 DEBUG:boto:Data: 2023-09-29T22:48:06.938 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.938 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.938 DEBUG:boto:Params: {} 2023-09-29T22:48:06.939 DEBUG:boto:Token: None 2023-09-29T22:48:06.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.939 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Z6OuGXqUqepT/nvwLhwEf+MSjsY= 2023-09-29T22:48:06.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Z6OuGXqUqepT/nvwLhwEf+MSjsY=', 'Content-Length': '0'} 2023-09-29T22:48:06.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffba2ee8b1d6a2cc-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:48:06.942 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:06.943 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:06.943 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.943 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.943 DEBUG:boto:Method: GET 2023-09-29T22:48:06.943 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.943 DEBUG:boto:Data: 2023-09-29T22:48:06.943 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.944 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.944 DEBUG:boto:Params: {} 2023-09-29T22:48:06.944 DEBUG:boto:Token: None 2023-09-29T22:48:06.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk= 2023-09-29T22:48:06.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk=', 'Content-Length': '0'} 2023-09-29T22:48:06.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f336dc1b6091644-0065175426-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.947 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:06.948 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:06.948 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.948 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.948 DEBUG:boto:Method: GET 2023-09-29T22:48:06.948 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.948 DEBUG:boto:Data: 2023-09-29T22:48:06.948 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.949 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.949 DEBUG:boto:Params: {} 2023-09-29T22:48:06.949 DEBUG:boto:Token: None 2023-09-29T22:48:06.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:48:06.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk= 2023-09-29T22:48:06.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ecl+xxWIn3mqxfhhSNmm6ft91gk=', 'Content-Length': '0'} 2023-09-29T22:48:06.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2313a4d93143d0d-0065175426-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.953 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.953 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.953 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:06.953 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:06.953 DEBUG:boto:Method: GET 2023-09-29T22:48:06.953 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:48:06.953 DEBUG:boto:Data: 2023-09-29T22:48:06.953 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:06.954 DEBUG:boto:Port: 8001 2023-09-29T22:48:06.954 DEBUG:boto:Params: {} 2023-09-29T22:48:06.954 DEBUG:boto:Token: None 2023-09-29T22:48:06.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.954 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.959 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005253c0cfa7e14585-0065175426-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:06.959 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:06.960 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:06.960 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:06.960 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:06.960 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:06.960 DEBUG:boto:Method: GET 2023-09-29T22:48:06.960 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:48:06.961 DEBUG:boto:Data: 2023-09-29T22:48:06.961 DEBUG:boto:Headers: {} 2023-09-29T22:48:06.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:06.961 DEBUG:boto:Port: 8000 2023-09-29T22:48:06.961 DEBUG:boto:Params: {} 2023-09-29T22:48:06.961 DEBUG:boto:Token: None 2023-09-29T22:48:06.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:06 GMT /hcnphz-59/ 2023-09-29T22:48:06.962 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q= 2023-09-29T22:48:06.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tbztuiV8wEDIUuuxWJUFNmt1y9Q=', 'Content-Length': '0'} 2023-09-29T22:48:06.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c8d64cbe373ac76a-0065175426-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.010 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.011 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:07.011 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:07.011 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.011 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:48:07.011 DEBUG:boto:Data: 2023-09-29T22:48:07.011 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.012 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.012 DEBUG:boto:Params: {} 2023-09-29T22:48:07.012 DEBUG:boto:Token: None 2023-09-29T22:48:07.012 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj55 2023-09-29T22:48:07.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TqrwNwxQ4KqsQ3ZhjetnYk5W1ZA= 2023-09-29T22:48:07.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TqrwNwxQ4KqsQ3ZhjetnYk5W1ZA=', 'Content-Length': '0'} 2023-09-29T22:48:07.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bb22b0c0dafe44-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.015 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:07.015 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:07.015 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.015 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:48:07.016 DEBUG:boto:Data: 2023-09-29T22:48:07.016 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.016 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.016 DEBUG:boto:Params: {} 2023-09-29T22:48:07.016 DEBUG:boto:Token: None 2023-09-29T22:48:07.016 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj55 2023-09-29T22:48:07.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TqrwNwxQ4KqsQ3ZhjetnYk5W1ZA= 2023-09-29T22:48:07.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TqrwNwxQ4KqsQ3ZhjetnYk5W1ZA=', 'Content-Length': '0'} 2023-09-29T22:48:07.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f54407b8b5a1b431-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:48:07.020 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:07.020 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:07.020 DEBUG:boto:Method: GET 2023-09-29T22:48:07.020 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:48:07.020 DEBUG:boto:Data: 2023-09-29T22:48:07.020 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.020 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.020 DEBUG:boto:Params: {} 2023-09-29T22:48:07.021 DEBUG:boto:Token: None 2023-09-29T22:48:07.021 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj55 2023-09-29T22:48:07.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bk92F5SEtcPJ6Iber2txQJ2j4uU= 2023-09-29T22:48:07.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bk92F5SEtcPJ6Iber2txQJ2j4uU=', 'Content-Length': '0'} 2023-09-29T22:48:07.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d99a7a04f31b4423-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.066 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:48:07.066 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:48:07.067 DEBUG:boto:Method: GET 2023-09-29T22:48:07.067 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:48:07.067 DEBUG:boto:Data: 2023-09-29T22:48:07.067 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.067 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.067 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.067 DEBUG:boto:Params: {} 2023-09-29T22:48:07.067 DEBUG:boto:Token: None 2023-09-29T22:48:07.068 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj55 2023-09-29T22:48:07.068 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bk92F5SEtcPJ6Iber2txQJ2j4uU= 2023-09-29T22:48:07.068 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bk92F5SEtcPJ6Iber2txQJ2j4uU=', 'Content-Length': '0'} 2023-09-29T22:48:07.071 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f121267f2a4be72e-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.071 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.071 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.071 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:07.071 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:07.071 DEBUG:boto:Method: GET 2023-09-29T22:48:07.072 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:48:07.072 DEBUG:boto:Data: 2023-09-29T22:48:07.072 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.072 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.072 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.072 DEBUG:boto:Params: {} 2023-09-29T22:48:07.072 DEBUG:boto:Token: None 2023-09-29T22:48:07.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.073 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.077 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042cba40703ce4a22-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.078 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.078 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.078 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.078 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:07.078 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:48:07.078 DEBUG:boto:Method: GET 2023-09-29T22:48:07.079 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:48:07.079 DEBUG:boto:Data: 2023-09-29T22:48:07.079 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.079 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.079 DEBUG:boto:Params: {} 2023-09-29T22:48:07.079 DEBUG:boto:Token: None 2023-09-29T22:48:07.079 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.079 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.087 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc212b785ad90f16-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.087 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:48:07.087 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.087 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.088 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.088 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.088 DEBUG:boto:Method: GET 2023-09-29T22:48:07.088 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.088 DEBUG:boto:Data: 2023-09-29T22:48:07.088 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.088 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.088 DEBUG:boto:Params: {} 2023-09-29T22:48:07.088 DEBUG:boto:Token: None 2023-09-29T22:48:07.089 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s= 2023-09-29T22:48:07.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s=', 'Content-Length': '0'} 2023-09-29T22:48:07.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d74752eb3a0fe7e2-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.091 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.091 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.091 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.091 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.091 DEBUG:boto:Method: GET 2023-09-29T22:48:07.091 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.092 DEBUG:boto:Data: 2023-09-29T22:48:07.092 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.092 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.092 DEBUG:boto:Params: {} 2023-09-29T22:48:07.092 DEBUG:boto:Token: None 2023-09-29T22:48:07.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s= 2023-09-29T22:48:07.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s=', 'Content-Length': '0'} 2023-09-29T22:48:07.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a05201624e93388-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.143 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.143 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.143 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.143 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.143 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.144 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.144 DEBUG:boto:Data: 2023-09-29T22:48:07.144 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.144 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.144 DEBUG:boto:Params: {} 2023-09-29T22:48:07.144 DEBUG:boto:Token: None 2023-09-29T22:48:07.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zb4x4ipzvnWda8e6CMGB6VsSIlU= 2023-09-29T22:48:07.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zb4x4ipzvnWda8e6CMGB6VsSIlU=', 'Content-Length': '0'} 2023-09-29T22:48:07.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000969c268770dcdfb2-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.147 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.147 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.148 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.148 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.148 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.148 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.148 DEBUG:boto:Data: 2023-09-29T22:48:07.148 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.148 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.148 DEBUG:boto:Params: {} 2023-09-29T22:48:07.149 DEBUG:boto:Token: None 2023-09-29T22:48:07.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zb4x4ipzvnWda8e6CMGB6VsSIlU= 2023-09-29T22:48:07.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zb4x4ipzvnWda8e6CMGB6VsSIlU=', 'Content-Length': '0'} 2023-09-29T22:48:07.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ffa0f665c2a143d4-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:48:07.152 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.152 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.152 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.152 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.152 DEBUG:boto:Method: GET 2023-09-29T22:48:07.152 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.152 DEBUG:boto:Data: 2023-09-29T22:48:07.153 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.153 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.153 DEBUG:boto:Params: {} 2023-09-29T22:48:07.153 DEBUG:boto:Token: None 2023-09-29T22:48:07.153 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s= 2023-09-29T22:48:07.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s=', 'Content-Length': '0'} 2023-09-29T22:48:07.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000487be41dd98c59f8-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.198 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.198 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.199 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.199 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.199 DEBUG:boto:Method: GET 2023-09-29T22:48:07.199 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.199 DEBUG:boto:Data: 2023-09-29T22:48:07.199 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.199 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.199 DEBUG:boto:Params: {} 2023-09-29T22:48:07.200 DEBUG:boto:Token: None 2023-09-29T22:48:07.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:48:07.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s= 2023-09-29T22:48:07.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WCiGH63KEVjn33njwLoj61mDF4s=', 'Content-Length': '0'} 2023-09-29T22:48:07.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085046192b1347fd4-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.203 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.203 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.203 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.203 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.203 DEBUG:boto:Method: GET 2023-09-29T22:48:07.204 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.204 DEBUG:boto:Data: 2023-09-29T22:48:07.204 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.204 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.204 DEBUG:boto:Params: {} 2023-09-29T22:48:07.204 DEBUG:boto:Token: None 2023-09-29T22:48:07.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006a0fc7d23bd1edbc-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.209 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.209 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.210 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.210 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.210 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.210 DEBUG:boto:Method: GET 2023-09-29T22:48:07.210 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.210 DEBUG:boto:Data: 2023-09-29T22:48:07.210 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.210 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.211 DEBUG:boto:Params: {} 2023-09-29T22:48:07.211 DEBUG:boto:Token: None 2023-09-29T22:48:07.211 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.218 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001df52b5e9e1086d7-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.218 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.218 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.218 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.218 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.219 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:48:07.219 DEBUG:boto:Data: 2023-09-29T22:48:07.219 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.219 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.219 DEBUG:boto:Params: {} 2023-09-29T22:48:07.219 DEBUG:boto:Token: None 2023-09-29T22:48:07.219 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57 2023-09-29T22:48:07.219 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFtCVH4i6owzs49/w8iaZ+/c5xk= 2023-09-29T22:48:07.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFtCVH4i6owzs49/w8iaZ+/c5xk=', 'Content-Length': '0'} 2023-09-29T22:48:07.221 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b25413551234d9ad-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.222 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.222 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.222 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.222 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:48:07.222 DEBUG:boto:Data: 2023-09-29T22:48:07.222 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.222 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.222 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.222 DEBUG:boto:Params: {} 2023-09-29T22:48:07.223 DEBUG:boto:Token: None 2023-09-29T22:48:07.223 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57 2023-09-29T22:48:07.223 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFtCVH4i6owzs49/w8iaZ+/c5xk= 2023-09-29T22:48:07.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFtCVH4i6owzs49/w8iaZ+/c5xk=', 'Content-Length': '0'} 2023-09-29T22:48:07.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e656b0d1f18e511b-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:48:07.225 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.225 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.225 DEBUG:boto:Method: GET 2023-09-29T22:48:07.225 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:48:07.225 DEBUG:boto:Data: 2023-09-29T22:48:07.225 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.226 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.226 DEBUG:boto:Params: {} 2023-09-29T22:48:07.226 DEBUG:boto:Token: None 2023-09-29T22:48:07.226 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57 2023-09-29T22:48:07.226 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g99MKxLDh1W2rvA1AU82p+644pU= 2023-09-29T22:48:07.226 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g99MKxLDh1W2rvA1AU82p+644pU=', 'Content-Length': '0'} 2023-09-29T22:48:07.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa7a2003c5ee8e3-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.229 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:48:07.229 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:48:07.229 DEBUG:boto:Method: GET 2023-09-29T22:48:07.229 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:48:07.229 DEBUG:boto:Data: 2023-09-29T22:48:07.229 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.229 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.229 DEBUG:boto:Params: {} 2023-09-29T22:48:07.230 DEBUG:boto:Token: None 2023-09-29T22:48:07.230 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj57 2023-09-29T22:48:07.230 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g99MKxLDh1W2rvA1AU82p+644pU= 2023-09-29T22:48:07.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g99MKxLDh1W2rvA1AU82p+644pU=', 'Content-Length': '0'} 2023-09-29T22:48:07.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014f2fd4a65daae2c-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.274 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.274 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.275 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.275 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.275 DEBUG:boto:Method: GET 2023-09-29T22:48:07.275 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.275 DEBUG:boto:Data: 2023-09-29T22:48:07.275 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.275 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.275 DEBUG:boto:Params: {} 2023-09-29T22:48:07.275 DEBUG:boto:Token: None 2023-09-29T22:48:07.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.280 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009b013f1c71aa3e46-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.281 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.281 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.281 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.281 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.281 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.281 DEBUG:boto:Method: GET 2023-09-29T22:48:07.282 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:48:07.282 DEBUG:boto:Data: 2023-09-29T22:48:07.282 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.282 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.282 DEBUG:boto:Params: {} 2023-09-29T22:48:07.282 DEBUG:boto:Token: None 2023-09-29T22:48:07.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.283 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002905b19e4589df0b-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.289 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:48:07.290 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.290 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.290 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.290 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.290 DEBUG:boto:Method: GET 2023-09-29T22:48:07.290 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.291 DEBUG:boto:Data: 2023-09-29T22:48:07.291 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.291 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.291 DEBUG:boto:Params: {} 2023-09-29T22:48:07.291 DEBUG:boto:Token: None 2023-09-29T22:48:07.291 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y= 2023-09-29T22:48:07.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y=', 'Content-Length': '0'} 2023-09-29T22:48:07.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009542bc29cb9646c9-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.334 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.335 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.335 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.335 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.335 DEBUG:boto:Method: GET 2023-09-29T22:48:07.335 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.335 DEBUG:boto:Data: 2023-09-29T22:48:07.335 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.335 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.335 DEBUG:boto:Params: {} 2023-09-29T22:48:07.336 DEBUG:boto:Token: None 2023-09-29T22:48:07.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y= 2023-09-29T22:48:07.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y=', 'Content-Length': '0'} 2023-09-29T22:48:07.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000188261d7cdc22b3a-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.339 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.339 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.339 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.339 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.339 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.340 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.340 DEBUG:boto:Data: 2023-09-29T22:48:07.340 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.340 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.340 DEBUG:boto:Params: {} 2023-09-29T22:48:07.340 DEBUG:boto:Token: None 2023-09-29T22:48:07.340 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.340 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SGPeb4XHdwMvRxWDggGuG5DnCvM= 2023-09-29T22:48:07.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SGPeb4XHdwMvRxWDggGuG5DnCvM=', 'Content-Length': '0'} 2023-09-29T22:48:07.343 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f004eb3565cc264c-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.343 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.343 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.344 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.344 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.344 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.344 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.344 DEBUG:boto:Data: 2023-09-29T22:48:07.344 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.344 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.344 DEBUG:boto:Params: {} 2023-09-29T22:48:07.344 DEBUG:boto:Token: None 2023-09-29T22:48:07.345 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SGPeb4XHdwMvRxWDggGuG5DnCvM= 2023-09-29T22:48:07.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SGPeb4XHdwMvRxWDggGuG5DnCvM=', 'Content-Length': '0'} 2023-09-29T22:48:07.347 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc034a2507e83254-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:48:07.348 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.348 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.348 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.348 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.348 DEBUG:boto:Method: GET 2023-09-29T22:48:07.348 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.348 DEBUG:boto:Data: 2023-09-29T22:48:07.348 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.349 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.349 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.349 DEBUG:boto:Params: {} 2023-09-29T22:48:07.349 DEBUG:boto:Token: None 2023-09-29T22:48:07.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y= 2023-09-29T22:48:07.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y=', 'Content-Length': '0'} 2023-09-29T22:48:07.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e29298e2829fd38-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.352 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.352 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.352 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.353 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.353 DEBUG:boto:Method: GET 2023-09-29T22:48:07.353 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.353 DEBUG:boto:Data: 2023-09-29T22:48:07.353 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.353 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.353 DEBUG:boto:Params: {} 2023-09-29T22:48:07.353 DEBUG:boto:Token: None 2023-09-29T22:48:07.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:48:07.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y= 2023-09-29T22:48:07.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:srLwj36lsxdHtikDNyEKTldro4Y=', 'Content-Length': '0'} 2023-09-29T22:48:07.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042060c662e1911cd-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.398 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.399 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.399 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.399 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.399 DEBUG:boto:Method: GET 2023-09-29T22:48:07.399 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.399 DEBUG:boto:Data: 2023-09-29T22:48:07.399 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.399 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.399 DEBUG:boto:Params: {} 2023-09-29T22:48:07.400 DEBUG:boto:Token: None 2023-09-29T22:48:07.400 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.405 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006662285ad307aaeb-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.405 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.405 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.405 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.406 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.406 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.406 DEBUG:boto:Method: GET 2023-09-29T22:48:07.406 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.406 DEBUG:boto:Data: 2023-09-29T22:48:07.406 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.406 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.406 DEBUG:boto:Params: {} 2023-09-29T22:48:07.407 DEBUG:boto:Token: None 2023-09-29T22:48:07.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.407 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000098e070e20aa6afae-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.416 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.416 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.416 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.417 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.417 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:48:07.417 DEBUG:boto:Data: 2023-09-29T22:48:07.417 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.417 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.417 DEBUG:boto:Params: {} 2023-09-29T22:48:07.417 DEBUG:boto:Token: None 2023-09-29T22:48:07.417 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58 2023-09-29T22:48:07.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HSNaOYjfs+Y8ccQZUWKrkBWxegw= 2023-09-29T22:48:07.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HSNaOYjfs+Y8ccQZUWKrkBWxegw=', 'Content-Length': '0'} 2023-09-29T22:48:07.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013d6f1223986b179-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.420 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.421 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.421 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.421 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:48:07.421 DEBUG:boto:Data: 2023-09-29T22:48:07.421 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.421 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.421 DEBUG:boto:Params: {} 2023-09-29T22:48:07.421 DEBUG:boto:Token: None 2023-09-29T22:48:07.422 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58 2023-09-29T22:48:07.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HSNaOYjfs+Y8ccQZUWKrkBWxegw= 2023-09-29T22:48:07.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HSNaOYjfs+Y8ccQZUWKrkBWxegw=', 'Content-Length': '0'} 2023-09-29T22:48:07.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebaf334947e6f1fc-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:48:07.424 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.424 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.424 DEBUG:boto:Method: GET 2023-09-29T22:48:07.424 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:48:07.425 DEBUG:boto:Data: 2023-09-29T22:48:07.425 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.425 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.425 DEBUG:boto:Params: {} 2023-09-29T22:48:07.425 DEBUG:boto:Token: None 2023-09-29T22:48:07.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58 2023-09-29T22:48:07.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:70Omfp6/zBpIRaQ1S2Q6CSZO9Nc= 2023-09-29T22:48:07.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:70Omfp6/zBpIRaQ1S2Q6CSZO9Nc=', 'Content-Length': '0'} 2023-09-29T22:48:07.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000821f3c1d18cc8865-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.470 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:48:07.471 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:48:07.471 DEBUG:boto:Method: GET 2023-09-29T22:48:07.471 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:48:07.471 DEBUG:boto:Data: 2023-09-29T22:48:07.471 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.471 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.471 DEBUG:boto:Params: {} 2023-09-29T22:48:07.471 DEBUG:boto:Token: None 2023-09-29T22:48:07.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj58 2023-09-29T22:48:07.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:70Omfp6/zBpIRaQ1S2Q6CSZO9Nc= 2023-09-29T22:48:07.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:70Omfp6/zBpIRaQ1S2Q6CSZO9Nc=', 'Content-Length': '0'} 2023-09-29T22:48:07.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ea1f4071ab17646-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.475 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.475 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.475 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.475 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.475 DEBUG:boto:Method: GET 2023-09-29T22:48:07.475 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.475 DEBUG:boto:Data: 2023-09-29T22:48:07.475 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.476 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.476 DEBUG:boto:Params: {} 2023-09-29T22:48:07.476 DEBUG:boto:Token: None 2023-09-29T22:48:07.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.481 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c997a5b2dfa3de03-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.481 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.482 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.482 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.482 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.482 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.482 DEBUG:boto:Method: GET 2023-09-29T22:48:07.482 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:48:07.482 DEBUG:boto:Data: 2023-09-29T22:48:07.482 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.483 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.483 DEBUG:boto:Params: {} 2023-09-29T22:48:07.483 DEBUG:boto:Token: None 2023-09-29T22:48:07.483 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.483 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.489 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005209c4d3368c38bc-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.530 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:48:07.531 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.531 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.531 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.531 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.531 DEBUG:boto:Method: GET 2023-09-29T22:48:07.531 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.531 DEBUG:boto:Data: 2023-09-29T22:48:07.531 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.532 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.532 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.532 DEBUG:boto:Params: {} 2023-09-29T22:48:07.532 DEBUG:boto:Token: None 2023-09-29T22:48:07.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess= 2023-09-29T22:48:07.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess=', 'Content-Length': '0'} 2023-09-29T22:48:07.535 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b279c21de5a4e65-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.536 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.536 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.536 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.536 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.536 DEBUG:boto:Method: GET 2023-09-29T22:48:07.536 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.536 DEBUG:boto:Data: 2023-09-29T22:48:07.536 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.537 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.537 DEBUG:boto:Params: {} 2023-09-29T22:48:07.537 DEBUG:boto:Token: None 2023-09-29T22:48:07.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess= 2023-09-29T22:48:07.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess=', 'Content-Length': '0'} 2023-09-29T22:48:07.539 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002980119b080815ab-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.540 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.540 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.540 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.540 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.540 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.540 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.540 DEBUG:boto:Data: 2023-09-29T22:48:07.541 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.541 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.541 DEBUG:boto:Params: {} 2023-09-29T22:48:07.541 DEBUG:boto:Token: None 2023-09-29T22:48:07.541 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uxejc+GS/HJL+cF1Z7lNEb9x7w4= 2023-09-29T22:48:07.541 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uxejc+GS/HJL+cF1Z7lNEb9x7w4=', 'Content-Length': '0'} 2023-09-29T22:48:07.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6115d2c23b5ce5f-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.544 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.544 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.544 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.544 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.545 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.545 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.545 DEBUG:boto:Data: 2023-09-29T22:48:07.545 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.545 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.545 DEBUG:boto:Params: {} 2023-09-29T22:48:07.545 DEBUG:boto:Token: None 2023-09-29T22:48:07.545 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uxejc+GS/HJL+cF1Z7lNEb9x7w4= 2023-09-29T22:48:07.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uxejc+GS/HJL+cF1Z7lNEb9x7w4=', 'Content-Length': '0'} 2023-09-29T22:48:07.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b309370cf7d85597-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:48:07.548 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.548 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.548 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.548 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.548 DEBUG:boto:Method: GET 2023-09-29T22:48:07.549 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.549 DEBUG:boto:Data: 2023-09-29T22:48:07.549 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.549 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.549 DEBUG:boto:Params: {} 2023-09-29T22:48:07.549 DEBUG:boto:Token: None 2023-09-29T22:48:07.549 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess= 2023-09-29T22:48:07.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess=', 'Content-Length': '0'} 2023-09-29T22:48:07.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ea03198d3b20f7a-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.598 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.599 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.599 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.599 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.599 DEBUG:boto:Method: GET 2023-09-29T22:48:07.599 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.599 DEBUG:boto:Data: 2023-09-29T22:48:07.599 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.599 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.599 DEBUG:boto:Params: {} 2023-09-29T22:48:07.600 DEBUG:boto:Token: None 2023-09-29T22:48:07.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:48:07.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess= 2023-09-29T22:48:07.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vCdCm7q1qAyN7yYAPj7SvhZaess=', 'Content-Length': '0'} 2023-09-29T22:48:07.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003691fc867a6c8da6-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.603 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.603 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.603 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.604 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.604 DEBUG:boto:Method: GET 2023-09-29T22:48:07.604 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.604 DEBUG:boto:Data: 2023-09-29T22:48:07.604 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.604 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.604 DEBUG:boto:Params: {} 2023-09-29T22:48:07.604 DEBUG:boto:Token: None 2023-09-29T22:48:07.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.605 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001395751f24230db2-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.609 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.610 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.610 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.610 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.610 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.610 DEBUG:boto:Method: GET 2023-09-29T22:48:07.610 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.610 DEBUG:boto:Data: 2023-09-29T22:48:07.611 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.611 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.611 DEBUG:boto:Params: {} 2023-09-29T22:48:07.611 DEBUG:boto:Token: None 2023-09-29T22:48:07.611 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000544dcb27ad734cfc-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.658 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.659 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.659 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.659 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.659 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:48:07.659 DEBUG:boto:Data: 2023-09-29T22:48:07.659 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.660 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.660 DEBUG:boto:Params: {} 2023-09-29T22:48:07.660 DEBUG:boto:Token: None 2023-09-29T22:48:07.660 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62 2023-09-29T22:48:07.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PPGKs6VXJIv58n5vEuiejwE52k8= 2023-09-29T22:48:07.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PPGKs6VXJIv58n5vEuiejwE52k8=', 'Content-Length': '0'} 2023-09-29T22:48:07.663 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0067acb32b92804-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.663 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.663 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.664 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.664 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:48:07.664 DEBUG:boto:Data: 2023-09-29T22:48:07.664 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.664 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.664 DEBUG:boto:Params: {} 2023-09-29T22:48:07.664 DEBUG:boto:Token: None 2023-09-29T22:48:07.664 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62 2023-09-29T22:48:07.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PPGKs6VXJIv58n5vEuiejwE52k8= 2023-09-29T22:48:07.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PPGKs6VXJIv58n5vEuiejwE52k8=', 'Content-Length': '0'} 2023-09-29T22:48:07.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d78136706a38bb8f-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:48:07.667 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.668 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.668 DEBUG:boto:Method: GET 2023-09-29T22:48:07.668 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:48:07.668 DEBUG:boto:Data: 2023-09-29T22:48:07.668 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.668 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.668 DEBUG:boto:Params: {} 2023-09-29T22:48:07.668 DEBUG:boto:Token: None 2023-09-29T22:48:07.669 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62 2023-09-29T22:48:07.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3KObp+oyoQp9O8sY9QRKA/AzSsY= 2023-09-29T22:48:07.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3KObp+oyoQp9O8sY9QRKA/AzSsY=', 'Content-Length': '0'} 2023-09-29T22:48:07.671 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e94618614cc07073-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.671 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:48:07.672 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:48:07.672 DEBUG:boto:Method: GET 2023-09-29T22:48:07.672 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:48:07.672 DEBUG:boto:Data: 2023-09-29T22:48:07.672 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.672 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.672 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.672 DEBUG:boto:Params: {} 2023-09-29T22:48:07.672 DEBUG:boto:Token: None 2023-09-29T22:48:07.673 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj62 2023-09-29T22:48:07.673 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3KObp+oyoQp9O8sY9QRKA/AzSsY= 2023-09-29T22:48:07.673 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3KObp+oyoQp9O8sY9QRKA/AzSsY=', 'Content-Length': '0'} 2023-09-29T22:48:07.675 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008096a54a4222f4d4-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.675 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.676 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.676 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.676 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.676 DEBUG:boto:Method: GET 2023-09-29T22:48:07.676 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.676 DEBUG:boto:Data: 2023-09-29T22:48:07.676 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.676 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.677 DEBUG:boto:Params: {} 2023-09-29T22:48:07.677 DEBUG:boto:Token: None 2023-09-29T22:48:07.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.681 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005453197a66106a00-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.722 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.723 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.723 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.723 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.723 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.723 DEBUG:boto:Method: GET 2023-09-29T22:48:07.723 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:48:07.723 DEBUG:boto:Data: 2023-09-29T22:48:07.723 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.724 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.724 DEBUG:boto:Params: {} 2023-09-29T22:48:07.724 DEBUG:boto:Token: None 2023-09-29T22:48:07.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.732 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007e9e902b3d9db932-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.733 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:48:07.733 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.733 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.733 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.733 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.734 DEBUG:boto:Method: GET 2023-09-29T22:48:07.734 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.734 DEBUG:boto:Data: 2023-09-29T22:48:07.734 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.734 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.734 DEBUG:boto:Params: {} 2023-09-29T22:48:07.734 DEBUG:boto:Token: None 2023-09-29T22:48:07.734 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs= 2023-09-29T22:48:07.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs=', 'Content-Length': '0'} 2023-09-29T22:48:07.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001877f9527743a418-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.737 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.737 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.738 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.738 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.738 DEBUG:boto:Method: GET 2023-09-29T22:48:07.738 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.738 DEBUG:boto:Data: 2023-09-29T22:48:07.738 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.738 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.738 DEBUG:boto:Params: {} 2023-09-29T22:48:07.738 DEBUG:boto:Token: None 2023-09-29T22:48:07.739 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs= 2023-09-29T22:48:07.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs=', 'Content-Length': '0'} 2023-09-29T22:48:07.741 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf81e0de71a04127-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.782 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.783 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.783 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.783 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.783 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.783 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.783 DEBUG:boto:Data: 2023-09-29T22:48:07.783 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.784 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.784 DEBUG:boto:Params: {} 2023-09-29T22:48:07.784 DEBUG:boto:Token: None 2023-09-29T22:48:07.784 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tdGBSipQceq5DkjQ8UkoA9W0t8g= 2023-09-29T22:48:07.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tdGBSipQceq5DkjQ8UkoA9W0t8g=', 'Content-Length': '0'} 2023-09-29T22:48:07.787 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a2444d31db582b2-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.787 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.787 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.787 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.787 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.788 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.788 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.788 DEBUG:boto:Data: 2023-09-29T22:48:07.788 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.788 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.788 DEBUG:boto:Params: {} 2023-09-29T22:48:07.788 DEBUG:boto:Token: None 2023-09-29T22:48:07.788 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.789 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tdGBSipQceq5DkjQ8UkoA9W0t8g= 2023-09-29T22:48:07.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tdGBSipQceq5DkjQ8UkoA9W0t8g=', 'Content-Length': '0'} 2023-09-29T22:48:07.791 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e70f8c643f59800c-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:48:07.792 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.792 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.792 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.792 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.792 DEBUG:boto:Method: GET 2023-09-29T22:48:07.792 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.792 DEBUG:boto:Data: 2023-09-29T22:48:07.792 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.793 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.793 DEBUG:boto:Params: {} 2023-09-29T22:48:07.793 DEBUG:boto:Token: None 2023-09-29T22:48:07.793 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs= 2023-09-29T22:48:07.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs=', 'Content-Length': '0'} 2023-09-29T22:48:07.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006850dc6388285281-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.796 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.796 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.796 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.796 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.797 DEBUG:boto:Method: GET 2023-09-29T22:48:07.797 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.797 DEBUG:boto:Data: 2023-09-29T22:48:07.797 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.797 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.797 DEBUG:boto:Params: {} 2023-09-29T22:48:07.797 DEBUG:boto:Token: None 2023-09-29T22:48:07.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:48:07.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs= 2023-09-29T22:48:07.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cRJY5cln2yY/ih5WLezjgXj0kfs=', 'Content-Length': '0'} 2023-09-29T22:48:07.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd7ff6a7c2381421-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.801 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.801 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.801 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.801 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.801 DEBUG:boto:Method: GET 2023-09-29T22:48:07.801 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.801 DEBUG:boto:Data: 2023-09-29T22:48:07.801 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.802 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.802 DEBUG:boto:Params: {} 2023-09-29T22:48:07.802 DEBUG:boto:Token: None 2023-09-29T22:48:07.802 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.807 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003247e94155297a7-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.854 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.855 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.855 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.855 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.855 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.855 DEBUG:boto:Method: GET 2023-09-29T22:48:07.855 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.856 DEBUG:boto:Data: 2023-09-29T22:48:07.856 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.856 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.856 DEBUG:boto:Params: {} 2023-09-29T22:48:07.856 DEBUG:boto:Token: None 2023-09-29T22:48:07.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.864 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2a0af683c38a6a8-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.865 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.865 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.865 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.865 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.865 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:48:07.866 DEBUG:boto:Data: 2023-09-29T22:48:07.866 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.866 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.866 DEBUG:boto:Params: {} 2023-09-29T22:48:07.866 DEBUG:boto:Token: None 2023-09-29T22:48:07.866 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68 2023-09-29T22:48:07.866 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2XBhCWxCWyLCuE/xW4TSGUpiSQM= 2023-09-29T22:48:07.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2XBhCWxCWyLCuE/xW4TSGUpiSQM=', 'Content-Length': '0'} 2023-09-29T22:48:07.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f69e7a2fe259b0fa-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.869 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.869 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.870 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.870 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:48:07.870 DEBUG:boto:Data: 2023-09-29T22:48:07.870 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.870 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.870 DEBUG:boto:Params: {} 2023-09-29T22:48:07.870 DEBUG:boto:Token: None 2023-09-29T22:48:07.870 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68 2023-09-29T22:48:07.871 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2XBhCWxCWyLCuE/xW4TSGUpiSQM= 2023-09-29T22:48:07.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2XBhCWxCWyLCuE/xW4TSGUpiSQM=', 'Content-Length': '0'} 2023-09-29T22:48:07.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff271fd1c69b808f-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:48:07.873 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.873 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.873 DEBUG:boto:Method: GET 2023-09-29T22:48:07.873 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:48:07.873 DEBUG:boto:Data: 2023-09-29T22:48:07.874 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.874 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.874 DEBUG:boto:Params: {} 2023-09-29T22:48:07.874 DEBUG:boto:Token: None 2023-09-29T22:48:07.874 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68 2023-09-29T22:48:07.874 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFmJP5LOoLzLNlFQegDzuQL12kY= 2023-09-29T22:48:07.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFmJP5LOoLzLNlFQegDzuQL12kY=', 'Content-Length': '0'} 2023-09-29T22:48:07.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f21ba9a223d0b94d-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.876 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:48:07.877 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:48:07.877 DEBUG:boto:Method: GET 2023-09-29T22:48:07.877 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:48:07.877 DEBUG:boto:Data: 2023-09-29T22:48:07.877 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.877 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.877 DEBUG:boto:Params: {} 2023-09-29T22:48:07.877 DEBUG:boto:Token: None 2023-09-29T22:48:07.878 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj68 2023-09-29T22:48:07.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFmJP5LOoLzLNlFQegDzuQL12kY= 2023-09-29T22:48:07.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFmJP5LOoLzLNlFQegDzuQL12kY=', 'Content-Length': '0'} 2023-09-29T22:48:07.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5df730a6a786847-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.922 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.923 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.923 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.923 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.923 DEBUG:boto:Method: GET 2023-09-29T22:48:07.923 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.923 DEBUG:boto:Data: 2023-09-29T22:48:07.923 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.923 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.924 DEBUG:boto:Params: {} 2023-09-29T22:48:07.924 DEBUG:boto:Token: None 2023-09-29T22:48:07.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.928 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000005762f050e3ab5f-0065175427-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.929 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.929 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:07.929 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:07.929 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.929 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.930 DEBUG:boto:Method: GET 2023-09-29T22:48:07.930 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:48:07.930 DEBUG:boto:Data: 2023-09-29T22:48:07.930 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.930 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.930 DEBUG:boto:Params: {} 2023-09-29T22:48:07.930 DEBUG:boto:Token: None 2023-09-29T22:48:07.931 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/ 2023-09-29T22:48:07.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU= 2023-09-29T22:48:07.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HnbFVEBGYzzM5k05VIuyTGGcHKU=', 'Content-Length': '0'} 2023-09-29T22:48:07.938 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002df2c75b84322560-0065175427-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.938 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:07.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:48:07.939 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:07.939 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:07.939 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.939 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.939 DEBUG:boto:Method: GET 2023-09-29T22:48:07.939 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.939 DEBUG:boto:Data: 2023-09-29T22:48:07.939 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.940 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.940 DEBUG:boto:Params: {} 2023-09-29T22:48:07.940 DEBUG:boto:Token: None 2023-09-29T22:48:07.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bildd7kt2p+1zjq0pQs4n/6FnzQ= 2023-09-29T22:48:07.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bildd7kt2p+1zjq0pQs4n/6FnzQ=', 'Content-Length': '0'} 2023-09-29T22:48:07.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e81102454917b71e-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.942 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:07.943 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:07.943 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.943 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.943 DEBUG:boto:Method: GET 2023-09-29T22:48:07.943 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.943 DEBUG:boto:Data: 2023-09-29T22:48:07.943 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.943 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.944 DEBUG:boto:Params: {} 2023-09-29T22:48:07.944 DEBUG:boto:Token: None 2023-09-29T22:48:07.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bildd7kt2p+1zjq0pQs4n/6FnzQ= 2023-09-29T22:48:07.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bildd7kt2p+1zjq0pQs4n/6FnzQ=', 'Content-Length': '0'} 2023-09-29T22:48:07.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060cf9366346f7907-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.947 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:07.947 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:07.947 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.947 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.947 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.947 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.947 DEBUG:boto:Data: 2023-09-29T22:48:07.947 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.948 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.948 DEBUG:boto:Params: {} 2023-09-29T22:48:07.948 DEBUG:boto:Token: None 2023-09-29T22:48:07.948 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0njKi/1kl1ekiEEsMBLfBFAS0SQ= 2023-09-29T22:48:07.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0njKi/1kl1ekiEEsMBLfBFAS0SQ=', 'Content-Length': '0'} 2023-09-29T22:48:07.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032d453f358569652-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.950 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:07.951 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:07.951 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.951 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.951 DEBUG:boto:Method: HEAD 2023-09-29T22:48:07.951 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.951 DEBUG:boto:Data: 2023-09-29T22:48:07.951 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:07.951 DEBUG:boto:Port: 8000 2023-09-29T22:48:07.952 DEBUG:boto:Params: {} 2023-09-29T22:48:07.952 DEBUG:boto:Token: None 2023-09-29T22:48:07.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0njKi/1kl1ekiEEsMBLfBFAS0SQ= 2023-09-29T22:48:07.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0njKi/1kl1ekiEEsMBLfBFAS0SQ=', 'Content-Length': '0'} 2023-09-29T22:48:07.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d5984e9aafe5f7-0065175427-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:07.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:48:07.955 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:07.955 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:07.955 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.955 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.955 DEBUG:boto:Method: GET 2023-09-29T22:48:07.955 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.955 DEBUG:boto:Data: 2023-09-29T22:48:07.955 DEBUG:boto:Headers: {} 2023-09-29T22:48:07.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:07.955 DEBUG:boto:Port: 8001 2023-09-29T22:48:07.956 DEBUG:boto:Params: {} 2023-09-29T22:48:07.956 DEBUG:boto:Token: None 2023-09-29T22:48:07.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:07 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:07.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bildd7kt2p+1zjq0pQs4n/6FnzQ= 2023-09-29T22:48:07.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:07 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bildd7kt2p+1zjq0pQs4n/6FnzQ=', 'Content-Length': '0'} 2023-09-29T22:48:07.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000873ac11427cb6208-0065175427-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:07 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.002 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:08.003 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:08.003 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:08.003 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:48:08.003 DEBUG:boto:Method: GET 2023-09-29T22:48:08.003 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:08.003 DEBUG:boto:Data: 2023-09-29T22:48:08.003 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.003 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.003 DEBUG:boto:Params: {} 2023-09-29T22:48:08.004 DEBUG:boto:Token: None 2023-09-29T22:48:08.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:48:08.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ovLkvgvmFriCjblVhuUQw0BIjqQ= 2023-09-29T22:48:08.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ovLkvgvmFriCjblVhuUQw0BIjqQ=', 'Content-Length': '0'} 2023-09-29T22:48:08.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b901d7e16c25b1e-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.007 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.007 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.007 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.007 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.008 DEBUG:boto:Method: GET 2023-09-29T22:48:08.008 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.008 DEBUG:boto:Data: 2023-09-29T22:48:08.008 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.008 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.008 DEBUG:boto:Params: {} 2023-09-29T22:48:08.008 DEBUG:boto:Token: None 2023-09-29T22:48:08.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000079bd71ca21ff4927-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.013 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.013 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.013 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.014 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.014 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.014 DEBUG:boto:Method: GET 2023-09-29T22:48:08.014 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.014 DEBUG:boto:Data: 2023-09-29T22:48:08.014 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.014 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.014 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.014 DEBUG:boto:Params: {} 2023-09-29T22:48:08.014 DEBUG:boto:Token: None 2023-09-29T22:48:08.015 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.015 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.015 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000148bc66376ec99fc-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.022 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.022 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:08.022 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:08.023 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.023 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:48:08.023 DEBUG:boto:Data: 2023-09-29T22:48:08.023 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.023 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.023 DEBUG:boto:Params: {} 2023-09-29T22:48:08.023 DEBUG:boto:Token: None 2023-09-29T22:48:08.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj69 2023-09-29T22:48:08.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AxXm9ASGegDpfLz7DoYn1nR5k6I= 2023-09-29T22:48:08.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AxXm9ASGegDpfLz7DoYn1nR5k6I=', 'Content-Length': '0'} 2023-09-29T22:48:08.026 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cd34ca2d54ce584-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.026 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:08.027 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:08.027 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.027 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:48:08.027 DEBUG:boto:Data: 2023-09-29T22:48:08.027 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.027 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.027 DEBUG:boto:Params: {} 2023-09-29T22:48:08.027 DEBUG:boto:Token: None 2023-09-29T22:48:08.028 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj69 2023-09-29T22:48:08.028 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AxXm9ASGegDpfLz7DoYn1nR5k6I= 2023-09-29T22:48:08.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AxXm9ASGegDpfLz7DoYn1nR5k6I=', 'Content-Length': '0'} 2023-09-29T22:48:08.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fdc72150419fa937-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:48:08.030 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:08.031 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:08.031 DEBUG:boto:Method: GET 2023-09-29T22:48:08.031 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:48:08.031 DEBUG:boto:Data: 2023-09-29T22:48:08.031 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.031 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.031 DEBUG:boto:Params: {} 2023-09-29T22:48:08.031 DEBUG:boto:Token: None 2023-09-29T22:48:08.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj69 2023-09-29T22:48:08.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gb3OGKA8k6M8f8n/b9lMCwG/1FY= 2023-09-29T22:48:08.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gb3OGKA8k6M8f8n/b9lMCwG/1FY=', 'Content-Length': '0'} 2023-09-29T22:48:08.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027837603316451b1-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.034 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:48:08.035 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:48:08.035 DEBUG:boto:Method: GET 2023-09-29T22:48:08.035 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:48:08.035 DEBUG:boto:Data: 2023-09-29T22:48:08.035 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.035 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.035 DEBUG:boto:Params: {} 2023-09-29T22:48:08.035 DEBUG:boto:Token: None 2023-09-29T22:48:08.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj69 2023-09-29T22:48:08.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gb3OGKA8k6M8f8n/b9lMCwG/1FY= 2023-09-29T22:48:08.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gb3OGKA8k6M8f8n/b9lMCwG/1FY=', 'Content-Length': '0'} 2023-09-29T22:48:08.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000457fc63d79621429-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.038 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.038 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.038 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.039 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.039 DEBUG:boto:Method: GET 2023-09-29T22:48:08.039 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.039 DEBUG:boto:Data: 2023-09-29T22:48:08.039 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.039 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.039 DEBUG:boto:Params: {} 2023-09-29T22:48:08.039 DEBUG:boto:Token: None 2023-09-29T22:48:08.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fff1fe28510609e0-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.044 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.044 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.045 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.045 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.045 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.045 DEBUG:boto:Method: GET 2023-09-29T22:48:08.045 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:48:08.045 DEBUG:boto:Data: 2023-09-29T22:48:08.045 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.045 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.046 DEBUG:boto:Params: {} 2023-09-29T22:48:08.046 DEBUG:boto:Token: None 2023-09-29T22:48:08.046 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.046 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062af450936ff3d11-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.053 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:48:08.053 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.053 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.053 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.054 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.054 DEBUG:boto:Method: GET 2023-09-29T22:48:08.054 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.054 DEBUG:boto:Data: 2023-09-29T22:48:08.054 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.054 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.054 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.054 DEBUG:boto:Params: {} 2023-09-29T22:48:08.054 DEBUG:boto:Token: None 2023-09-29T22:48:08.055 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.055 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw= 2023-09-29T22:48:08.055 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw=', 'Content-Length': '0'} 2023-09-29T22:48:08.057 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013e112216b56b839-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.098 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.099 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.099 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.099 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.099 DEBUG:boto:Method: GET 2023-09-29T22:48:08.099 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.099 DEBUG:boto:Data: 2023-09-29T22:48:08.099 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.099 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.100 DEBUG:boto:Params: {} 2023-09-29T22:48:08.100 DEBUG:boto:Token: None 2023-09-29T22:48:08.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw= 2023-09-29T22:48:08.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw=', 'Content-Length': '0'} 2023-09-29T22:48:08.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb602ec161be5c6a-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.103 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.103 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.104 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.104 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.104 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.104 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.104 DEBUG:boto:Data: 2023-09-29T22:48:08.104 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.104 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.104 DEBUG:boto:Params: {} 2023-09-29T22:48:08.105 DEBUG:boto:Token: None 2023-09-29T22:48:08.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:skCzsSs+3TDVaZCi+tbfRSQASKE= 2023-09-29T22:48:08.105 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:skCzsSs+3TDVaZCi+tbfRSQASKE=', 'Content-Length': '0'} 2023-09-29T22:48:08.107 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000658541fb894a7593-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.108 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.108 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.108 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.108 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.108 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.108 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.108 DEBUG:boto:Data: 2023-09-29T22:48:08.108 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.109 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.109 DEBUG:boto:Params: {} 2023-09-29T22:48:08.109 DEBUG:boto:Token: None 2023-09-29T22:48:08.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:skCzsSs+3TDVaZCi+tbfRSQASKE= 2023-09-29T22:48:08.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:skCzsSs+3TDVaZCi+tbfRSQASKE=', 'Content-Length': '0'} 2023-09-29T22:48:08.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003be6b0b2dd1b6d7b-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:48:08.112 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.112 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.112 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.112 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.112 DEBUG:boto:Method: GET 2023-09-29T22:48:08.113 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.113 DEBUG:boto:Data: 2023-09-29T22:48:08.113 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.113 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.113 DEBUG:boto:Params: {} 2023-09-29T22:48:08.113 DEBUG:boto:Token: None 2023-09-29T22:48:08.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw= 2023-09-29T22:48:08.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw=', 'Content-Length': '0'} 2023-09-29T22:48:08.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1b4c22cdda0f212-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.116 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.116 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.117 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.117 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.117 DEBUG:boto:Method: GET 2023-09-29T22:48:08.117 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.117 DEBUG:boto:Data: 2023-09-29T22:48:08.117 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.117 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.117 DEBUG:boto:Params: {} 2023-09-29T22:48:08.117 DEBUG:boto:Token: None 2023-09-29T22:48:08.118 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:48:08.118 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw= 2023-09-29T22:48:08.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:clyBHfecIbM+8MtRJoXav2WViNw=', 'Content-Length': '0'} 2023-09-29T22:48:08.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c733be210f650ef-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.166 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.167 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.167 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.167 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.167 DEBUG:boto:Method: GET 2023-09-29T22:48:08.167 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.167 DEBUG:boto:Data: 2023-09-29T22:48:08.167 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.167 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.168 DEBUG:boto:Params: {} 2023-09-29T22:48:08.168 DEBUG:boto:Token: None 2023-09-29T22:48:08.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5a34a10901da85d-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.174 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.175 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.175 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.175 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.175 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.175 DEBUG:boto:Method: GET 2023-09-29T22:48:08.175 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.175 DEBUG:boto:Data: 2023-09-29T22:48:08.175 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.176 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.176 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.176 DEBUG:boto:Params: {} 2023-09-29T22:48:08.176 DEBUG:boto:Token: None 2023-09-29T22:48:08.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099969be6f752aedb-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.183 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.183 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.183 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.183 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.183 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:48:08.183 DEBUG:boto:Data: 2023-09-29T22:48:08.184 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.184 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.184 DEBUG:boto:Params: {} 2023-09-29T22:48:08.184 DEBUG:boto:Token: None 2023-09-29T22:48:08.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76 2023-09-29T22:48:08.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7KW60fty6oi2Pk1wHxOHxnfPI4w= 2023-09-29T22:48:08.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7KW60fty6oi2Pk1wHxOHxnfPI4w=', 'Content-Length': '0'} 2023-09-29T22:48:08.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ad19f2ea50d7e28-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.187 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.187 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.187 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.187 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:48:08.187 DEBUG:boto:Data: 2023-09-29T22:48:08.187 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.187 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.187 DEBUG:boto:Params: {} 2023-09-29T22:48:08.188 DEBUG:boto:Token: None 2023-09-29T22:48:08.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76 2023-09-29T22:48:08.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7KW60fty6oi2Pk1wHxOHxnfPI4w= 2023-09-29T22:48:08.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7KW60fty6oi2Pk1wHxOHxnfPI4w=', 'Content-Length': '0'} 2023-09-29T22:48:08.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020e57f86f299556a-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:48:08.190 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.190 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.191 DEBUG:boto:Method: GET 2023-09-29T22:48:08.191 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:48:08.191 DEBUG:boto:Data: 2023-09-29T22:48:08.191 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.191 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.191 DEBUG:boto:Params: {} 2023-09-29T22:48:08.191 DEBUG:boto:Token: None 2023-09-29T22:48:08.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76 2023-09-29T22:48:08.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XY/Z+vd7GcFO9P0JRPOhK3VZ6BA= 2023-09-29T22:48:08.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XY/Z+vd7GcFO9P0JRPOhK3VZ6BA=', 'Content-Length': '0'} 2023-09-29T22:48:08.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3870a205f3a022c-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.234 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:48:08.235 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:48:08.235 DEBUG:boto:Method: GET 2023-09-29T22:48:08.235 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:48:08.235 DEBUG:boto:Data: 2023-09-29T22:48:08.235 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.235 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.236 DEBUG:boto:Params: {} 2023-09-29T22:48:08.236 DEBUG:boto:Token: None 2023-09-29T22:48:08.236 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj76 2023-09-29T22:48:08.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XY/Z+vd7GcFO9P0JRPOhK3VZ6BA= 2023-09-29T22:48:08.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XY/Z+vd7GcFO9P0JRPOhK3VZ6BA=', 'Content-Length': '0'} 2023-09-29T22:48:08.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000675764dc97e1af7b-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.240 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.240 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.240 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.240 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.240 DEBUG:boto:Method: GET 2023-09-29T22:48:08.240 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.241 DEBUG:boto:Data: 2023-09-29T22:48:08.241 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.241 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.241 DEBUG:boto:Params: {} 2023-09-29T22:48:08.241 DEBUG:boto:Token: None 2023-09-29T22:48:08.241 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.241 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.246 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e3b6fb8df5a21cc2-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.246 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.246 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.246 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.247 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.247 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.247 DEBUG:boto:Method: GET 2023-09-29T22:48:08.247 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:48:08.247 DEBUG:boto:Data: 2023-09-29T22:48:08.247 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.247 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.247 DEBUG:boto:Params: {} 2023-09-29T22:48:08.247 DEBUG:boto:Token: None 2023-09-29T22:48:08.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.248 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.255 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000addf5a910deb2073-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.298 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:48:08.299 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.299 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.299 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.299 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.300 DEBUG:boto:Method: GET 2023-09-29T22:48:08.300 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.300 DEBUG:boto:Data: 2023-09-29T22:48:08.300 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.300 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.300 DEBUG:boto:Params: {} 2023-09-29T22:48:08.300 DEBUG:boto:Token: None 2023-09-29T22:48:08.301 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.301 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE= 2023-09-29T22:48:08.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE=', 'Content-Length': '0'} 2023-09-29T22:48:08.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344c2e9019d9de90-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.304 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.304 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.305 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.305 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.305 DEBUG:boto:Method: GET 2023-09-29T22:48:08.305 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.305 DEBUG:boto:Data: 2023-09-29T22:48:08.305 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.305 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.306 DEBUG:boto:Params: {} 2023-09-29T22:48:08.306 DEBUG:boto:Token: None 2023-09-29T22:48:08.306 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE= 2023-09-29T22:48:08.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE=', 'Content-Length': '0'} 2023-09-29T22:48:08.309 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f60e5dd3fa2e69ce-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.309 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.309 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.309 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.310 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.310 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.310 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.310 DEBUG:boto:Data: 2023-09-29T22:48:08.310 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.310 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.310 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.310 DEBUG:boto:Params: {} 2023-09-29T22:48:08.311 DEBUG:boto:Token: None 2023-09-29T22:48:08.311 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.311 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u7ouP0kuHHyTNtRzNgETSWnil30= 2023-09-29T22:48:08.311 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u7ouP0kuHHyTNtRzNgETSWnil30=', 'Content-Length': '0'} 2023-09-29T22:48:08.314 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000eef36b7ba260882-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.314 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.314 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.314 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.314 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.315 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.315 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.315 DEBUG:boto:Data: 2023-09-29T22:48:08.315 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.315 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.315 DEBUG:boto:Params: {} 2023-09-29T22:48:08.315 DEBUG:boto:Token: None 2023-09-29T22:48:08.316 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u7ouP0kuHHyTNtRzNgETSWnil30= 2023-09-29T22:48:08.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u7ouP0kuHHyTNtRzNgETSWnil30=', 'Content-Length': '0'} 2023-09-29T22:48:08.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d060b53bb8b53bed-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:48:08.319 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.319 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.319 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.319 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.319 DEBUG:boto:Method: GET 2023-09-29T22:48:08.320 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.320 DEBUG:boto:Data: 2023-09-29T22:48:08.320 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.320 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.320 DEBUG:boto:Params: {} 2023-09-29T22:48:08.320 DEBUG:boto:Token: None 2023-09-29T22:48:08.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE= 2023-09-29T22:48:08.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE=', 'Content-Length': '0'} 2023-09-29T22:48:08.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aed0377f1498c836-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.324 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.324 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.324 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.324 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.324 DEBUG:boto:Method: GET 2023-09-29T22:48:08.324 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.325 DEBUG:boto:Data: 2023-09-29T22:48:08.325 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.325 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.325 DEBUG:boto:Params: {} 2023-09-29T22:48:08.325 DEBUG:boto:Token: None 2023-09-29T22:48:08.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:48:08.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE= 2023-09-29T22:48:08.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z8mQPUQ6ycNnLqSdD3IqVAE3ndE=', 'Content-Length': '0'} 2023-09-29T22:48:08.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbb336da16e2ca72-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.329 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.329 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.329 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.329 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.330 DEBUG:boto:Method: GET 2023-09-29T22:48:08.330 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.330 DEBUG:boto:Data: 2023-09-29T22:48:08.330 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.330 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.330 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.330 DEBUG:boto:Params: {} 2023-09-29T22:48:08.330 DEBUG:boto:Token: None 2023-09-29T22:48:08.331 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.331 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.335 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008996b8d9662b140b-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.378 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.379 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.379 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.379 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.379 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.379 DEBUG:boto:Method: GET 2023-09-29T22:48:08.379 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.379 DEBUG:boto:Data: 2023-09-29T22:48:08.380 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.380 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.380 DEBUG:boto:Params: {} 2023-09-29T22:48:08.380 DEBUG:boto:Token: None 2023-09-29T22:48:08.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3d5bb8edac529fd-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.389 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.389 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.389 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.389 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.389 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:48:08.390 DEBUG:boto:Data: 2023-09-29T22:48:08.390 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.390 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.390 DEBUG:boto:Params: {} 2023-09-29T22:48:08.390 DEBUG:boto:Token: None 2023-09-29T22:48:08.390 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85 2023-09-29T22:48:08.390 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pw3ae+TxOLUOMjfOwqMVjEFS81Q= 2023-09-29T22:48:08.390 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pw3ae+TxOLUOMjfOwqMVjEFS81Q=', 'Content-Length': '0'} 2023-09-29T22:48:08.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000a7e81f9c5179ed-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.393 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.393 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.393 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.393 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:48:08.393 DEBUG:boto:Data: 2023-09-29T22:48:08.394 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.394 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.394 DEBUG:boto:Params: {} 2023-09-29T22:48:08.394 DEBUG:boto:Token: None 2023-09-29T22:48:08.394 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85 2023-09-29T22:48:08.394 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pw3ae+TxOLUOMjfOwqMVjEFS81Q= 2023-09-29T22:48:08.394 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pw3ae+TxOLUOMjfOwqMVjEFS81Q=', 'Content-Length': '0'} 2023-09-29T22:48:08.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000717fb79ca9e1e566-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:48:08.397 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.397 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.397 DEBUG:boto:Method: GET 2023-09-29T22:48:08.397 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:48:08.397 DEBUG:boto:Data: 2023-09-29T22:48:08.397 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.397 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.397 DEBUG:boto:Params: {} 2023-09-29T22:48:08.397 DEBUG:boto:Token: None 2023-09-29T22:48:08.398 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85 2023-09-29T22:48:08.398 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2xdX9M8DEtPV9Io6bcFViKFQ8y0= 2023-09-29T22:48:08.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2xdX9M8DEtPV9Io6bcFViKFQ8y0=', 'Content-Length': '0'} 2023-09-29T22:48:08.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023823e74e99c52e4-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.400 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:48:08.400 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:48:08.400 DEBUG:boto:Method: GET 2023-09-29T22:48:08.400 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:48:08.401 DEBUG:boto:Data: 2023-09-29T22:48:08.401 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.401 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.401 DEBUG:boto:Params: {} 2023-09-29T22:48:08.401 DEBUG:boto:Token: None 2023-09-29T22:48:08.401 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj85 2023-09-29T22:48:08.401 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2xdX9M8DEtPV9Io6bcFViKFQ8y0= 2023-09-29T22:48:08.401 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2xdX9M8DEtPV9Io6bcFViKFQ8y0=', 'Content-Length': '0'} 2023-09-29T22:48:08.403 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0116fab31488e2-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.446 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.447 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.447 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.447 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.447 DEBUG:boto:Method: GET 2023-09-29T22:48:08.447 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.447 DEBUG:boto:Data: 2023-09-29T22:48:08.447 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.448 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.448 DEBUG:boto:Params: {} 2023-09-29T22:48:08.448 DEBUG:boto:Token: None 2023-09-29T22:48:08.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.453 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000913e923380e8fde9-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.453 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.453 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.454 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.454 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.454 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.454 DEBUG:boto:Method: GET 2023-09-29T22:48:08.454 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:48:08.454 DEBUG:boto:Data: 2023-09-29T22:48:08.454 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.454 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.454 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.455 DEBUG:boto:Params: {} 2023-09-29T22:48:08.455 DEBUG:boto:Token: None 2023-09-29T22:48:08.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006aeeffe7b60b2558-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.462 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:48:08.463 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.463 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.463 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.463 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.463 DEBUG:boto:Method: GET 2023-09-29T22:48:08.463 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.463 DEBUG:boto:Data: 2023-09-29T22:48:08.463 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.464 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.464 DEBUG:boto:Params: {} 2023-09-29T22:48:08.464 DEBUG:boto:Token: None 2023-09-29T22:48:08.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0= 2023-09-29T22:48:08.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0=', 'Content-Length': '0'} 2023-09-29T22:48:08.466 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd84c79b3a8a1ba1-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.466 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.467 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.467 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.467 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.467 DEBUG:boto:Method: GET 2023-09-29T22:48:08.467 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.467 DEBUG:boto:Data: 2023-09-29T22:48:08.467 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.467 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.467 DEBUG:boto:Params: {} 2023-09-29T22:48:08.467 DEBUG:boto:Token: None 2023-09-29T22:48:08.468 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0= 2023-09-29T22:48:08.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0=', 'Content-Length': '0'} 2023-09-29T22:48:08.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c65ad0a0cf05cbd7-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.470 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.471 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.471 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.471 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.471 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.471 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.471 DEBUG:boto:Data: 2023-09-29T22:48:08.471 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.471 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.471 DEBUG:boto:Params: {} 2023-09-29T22:48:08.471 DEBUG:boto:Token: None 2023-09-29T22:48:08.472 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iLT5gdakm7fYaBFvRfrHNY6iaks= 2023-09-29T22:48:08.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iLT5gdakm7fYaBFvRfrHNY6iaks=', 'Content-Length': '0'} 2023-09-29T22:48:08.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d83f079ac31ff21b-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.474 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.474 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.474 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.474 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.475 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.475 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.475 DEBUG:boto:Data: 2023-09-29T22:48:08.475 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.475 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.475 DEBUG:boto:Params: {} 2023-09-29T22:48:08.475 DEBUG:boto:Token: None 2023-09-29T22:48:08.475 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iLT5gdakm7fYaBFvRfrHNY6iaks= 2023-09-29T22:48:08.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iLT5gdakm7fYaBFvRfrHNY6iaks=', 'Content-Length': '0'} 2023-09-29T22:48:08.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2afb0676e116d6d-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:48:08.478 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.478 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.478 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.478 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.479 DEBUG:boto:Method: GET 2023-09-29T22:48:08.479 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.479 DEBUG:boto:Data: 2023-09-29T22:48:08.479 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.479 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.479 DEBUG:boto:Params: {} 2023-09-29T22:48:08.479 DEBUG:boto:Token: None 2023-09-29T22:48:08.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0= 2023-09-29T22:48:08.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0=', 'Content-Length': '0'} 2023-09-29T22:48:08.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14ed0a7759077eb-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.482 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.482 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.482 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.483 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.483 DEBUG:boto:Method: GET 2023-09-29T22:48:08.483 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.483 DEBUG:boto:Data: 2023-09-29T22:48:08.483 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.483 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.483 DEBUG:boto:Params: {} 2023-09-29T22:48:08.483 DEBUG:boto:Token: None 2023-09-29T22:48:08.483 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:48:08.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0= 2023-09-29T22:48:08.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O3hn0a/zVllBAs0JvGD0LCZrWu0=', 'Content-Length': '0'} 2023-09-29T22:48:08.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005ba4a09fbbccc6a-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.486 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.486 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.486 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.486 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.487 DEBUG:boto:Method: GET 2023-09-29T22:48:08.487 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.487 DEBUG:boto:Data: 2023-09-29T22:48:08.487 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.487 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.487 DEBUG:boto:Params: {} 2023-09-29T22:48:08.487 DEBUG:boto:Token: None 2023-09-29T22:48:08.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.492 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b313faef7d31dc9b-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.492 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.492 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.492 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.493 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.493 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.493 DEBUG:boto:Method: GET 2023-09-29T22:48:08.493 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.493 DEBUG:boto:Data: 2023-09-29T22:48:08.493 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.493 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.493 DEBUG:boto:Params: {} 2023-09-29T22:48:08.494 DEBUG:boto:Token: None 2023-09-29T22:48:08.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.494 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6c026d3076bea39-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.501 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.502 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.502 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.502 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.502 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:48:08.502 DEBUG:boto:Data: 2023-09-29T22:48:08.502 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.502 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.503 DEBUG:boto:Params: {} 2023-09-29T22:48:08.503 DEBUG:boto:Token: None 2023-09-29T22:48:08.503 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87 2023-09-29T22:48:08.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7AboHKSzHi5U4ujgV1uIurCEh24= 2023-09-29T22:48:08.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7AboHKSzHi5U4ujgV1uIurCEh24=', 'Content-Length': '0'} 2023-09-29T22:48:08.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c43f4ffb204931d-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.505 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.505 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.505 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.505 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:48:08.506 DEBUG:boto:Data: 2023-09-29T22:48:08.506 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.506 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.506 DEBUG:boto:Params: {} 2023-09-29T22:48:08.506 DEBUG:boto:Token: None 2023-09-29T22:48:08.506 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87 2023-09-29T22:48:08.506 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7AboHKSzHi5U4ujgV1uIurCEh24= 2023-09-29T22:48:08.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7AboHKSzHi5U4ujgV1uIurCEh24=', 'Content-Length': '0'} 2023-09-29T22:48:08.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009df2c22ce18b9590-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:48:08.509 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.509 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.509 DEBUG:boto:Method: GET 2023-09-29T22:48:08.509 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:48:08.509 DEBUG:boto:Data: 2023-09-29T22:48:08.509 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.509 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.509 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.509 DEBUG:boto:Params: {} 2023-09-29T22:48:08.510 DEBUG:boto:Token: None 2023-09-29T22:48:08.510 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87 2023-09-29T22:48:08.510 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:57xBI0j/Vgu8IEFgM6o3XL2ns0I= 2023-09-29T22:48:08.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:57xBI0j/Vgu8IEFgM6o3XL2ns0I=', 'Content-Length': '0'} 2023-09-29T22:48:08.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e5e9e45725813dc-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.554 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:48:08.555 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:48:08.555 DEBUG:boto:Method: GET 2023-09-29T22:48:08.555 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:48:08.555 DEBUG:boto:Data: 2023-09-29T22:48:08.555 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.555 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.555 DEBUG:boto:Params: {} 2023-09-29T22:48:08.555 DEBUG:boto:Token: None 2023-09-29T22:48:08.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj87 2023-09-29T22:48:08.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:57xBI0j/Vgu8IEFgM6o3XL2ns0I= 2023-09-29T22:48:08.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:57xBI0j/Vgu8IEFgM6o3XL2ns0I=', 'Content-Length': '0'} 2023-09-29T22:48:08.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cda385ba1ebce40-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.559 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.559 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.559 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.559 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.559 DEBUG:boto:Method: GET 2023-09-29T22:48:08.559 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.559 DEBUG:boto:Data: 2023-09-29T22:48:08.559 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.559 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.560 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.560 DEBUG:boto:Params: {} 2023-09-29T22:48:08.560 DEBUG:boto:Token: None 2023-09-29T22:48:08.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.565 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000367f41c45f714b7f-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.565 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.565 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.565 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.566 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.566 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.566 DEBUG:boto:Method: GET 2023-09-29T22:48:08.566 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:48:08.566 DEBUG:boto:Data: 2023-09-29T22:48:08.566 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.566 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.566 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.566 DEBUG:boto:Params: {} 2023-09-29T22:48:08.567 DEBUG:boto:Token: None 2023-09-29T22:48:08.567 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.567 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.567 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000520b6027897f5cab-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.574 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:48:08.574 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.575 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.575 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.575 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.575 DEBUG:boto:Method: GET 2023-09-29T22:48:08.575 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.575 DEBUG:boto:Data: 2023-09-29T22:48:08.575 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.575 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.575 DEBUG:boto:Params: {} 2023-09-29T22:48:08.576 DEBUG:boto:Token: None 2023-09-29T22:48:08.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80= 2023-09-29T22:48:08.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80=', 'Content-Length': '0'} 2023-09-29T22:48:08.578 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071772b1d47844ba8-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.578 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.578 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.578 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.579 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.579 DEBUG:boto:Method: GET 2023-09-29T22:48:08.579 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.579 DEBUG:boto:Data: 2023-09-29T22:48:08.579 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.579 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.579 DEBUG:boto:Params: {} 2023-09-29T22:48:08.579 DEBUG:boto:Token: None 2023-09-29T22:48:08.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80= 2023-09-29T22:48:08.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80=', 'Content-Length': '0'} 2023-09-29T22:48:08.582 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee8d40773cf7ecc-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.582 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.582 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.582 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.583 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.583 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.583 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.583 DEBUG:boto:Data: 2023-09-29T22:48:08.583 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.583 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.583 DEBUG:boto:Params: {} 2023-09-29T22:48:08.583 DEBUG:boto:Token: None 2023-09-29T22:48:08.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ruohe+oVwhpraLoNj1d7YaEv1Ts= 2023-09-29T22:48:08.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ruohe+oVwhpraLoNj1d7YaEv1Ts=', 'Content-Length': '0'} 2023-09-29T22:48:08.586 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000deee2c59d18e552c-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.586 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.586 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.587 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.587 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.587 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.587 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.587 DEBUG:boto:Data: 2023-09-29T22:48:08.587 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.587 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.587 DEBUG:boto:Params: {} 2023-09-29T22:48:08.587 DEBUG:boto:Token: None 2023-09-29T22:48:08.588 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.588 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ruohe+oVwhpraLoNj1d7YaEv1Ts= 2023-09-29T22:48:08.588 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ruohe+oVwhpraLoNj1d7YaEv1Ts=', 'Content-Length': '0'} 2023-09-29T22:48:08.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4c90bd2032c96eb-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:48:08.590 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.591 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.591 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.591 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.591 DEBUG:boto:Method: GET 2023-09-29T22:48:08.591 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.591 DEBUG:boto:Data: 2023-09-29T22:48:08.591 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.591 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.591 DEBUG:boto:Params: {} 2023-09-29T22:48:08.592 DEBUG:boto:Token: None 2023-09-29T22:48:08.592 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80= 2023-09-29T22:48:08.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80=', 'Content-Length': '0'} 2023-09-29T22:48:08.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6443995796edf2-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.595 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.595 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.595 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.595 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.595 DEBUG:boto:Method: GET 2023-09-29T22:48:08.595 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.595 DEBUG:boto:Data: 2023-09-29T22:48:08.595 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.595 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.596 DEBUG:boto:Params: {} 2023-09-29T22:48:08.596 DEBUG:boto:Token: None 2023-09-29T22:48:08.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:48:08.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80= 2023-09-29T22:48:08.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDJhN9Apu2cFrH6BDLNPr8qVV80=', 'Content-Length': '0'} 2023-09-29T22:48:08.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000579c27663b4e3797-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.599 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.599 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.599 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.599 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.599 DEBUG:boto:Method: GET 2023-09-29T22:48:08.599 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.599 DEBUG:boto:Data: 2023-09-29T22:48:08.600 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.600 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.600 DEBUG:boto:Params: {} 2023-09-29T22:48:08.600 DEBUG:boto:Token: None 2023-09-29T22:48:08.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a35432adcc48235f-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.646 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.647 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.647 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.647 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.647 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.647 DEBUG:boto:Method: GET 2023-09-29T22:48:08.647 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.648 DEBUG:boto:Data: 2023-09-29T22:48:08.648 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.648 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.648 DEBUG:boto:Params: {} 2023-09-29T22:48:08.648 DEBUG:boto:Token: None 2023-09-29T22:48:08.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000aa24ca2a0219b07-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.657 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.657 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.658 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.658 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.658 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:48:08.658 DEBUG:boto:Data: 2023-09-29T22:48:08.658 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.658 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.658 DEBUG:boto:Params: {} 2023-09-29T22:48:08.659 DEBUG:boto:Token: None 2023-09-29T22:48:08.659 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92 2023-09-29T22:48:08.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ceiEbjOtsuPsTQHX6Clp3KJbRBs= 2023-09-29T22:48:08.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ceiEbjOtsuPsTQHX6Clp3KJbRBs=', 'Content-Length': '0'} 2023-09-29T22:48:08.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1750e0ab21480d-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.662 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.662 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.662 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.662 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:48:08.662 DEBUG:boto:Data: 2023-09-29T22:48:08.662 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.662 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.663 DEBUG:boto:Params: {} 2023-09-29T22:48:08.663 DEBUG:boto:Token: None 2023-09-29T22:48:08.663 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92 2023-09-29T22:48:08.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ceiEbjOtsuPsTQHX6Clp3KJbRBs= 2023-09-29T22:48:08.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ceiEbjOtsuPsTQHX6Clp3KJbRBs=', 'Content-Length': '0'} 2023-09-29T22:48:08.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2967d22a18b97d8-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:48:08.666 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.666 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.666 DEBUG:boto:Method: GET 2023-09-29T22:48:08.666 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:48:08.666 DEBUG:boto:Data: 2023-09-29T22:48:08.666 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.666 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.666 DEBUG:boto:Params: {} 2023-09-29T22:48:08.667 DEBUG:boto:Token: None 2023-09-29T22:48:08.667 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92 2023-09-29T22:48:08.667 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XzXiUND6zpKIEVG8ZbMGq2ERDao= 2023-09-29T22:48:08.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XzXiUND6zpKIEVG8ZbMGq2ERDao=', 'Content-Length': '0'} 2023-09-29T22:48:08.669 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d68193d05034b82-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.669 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:48:08.669 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:48:08.669 DEBUG:boto:Method: GET 2023-09-29T22:48:08.669 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:48:08.669 DEBUG:boto:Data: 2023-09-29T22:48:08.670 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.670 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.670 DEBUG:boto:Params: {} 2023-09-29T22:48:08.670 DEBUG:boto:Token: None 2023-09-29T22:48:08.670 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj92 2023-09-29T22:48:08.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XzXiUND6zpKIEVG8ZbMGq2ERDao= 2023-09-29T22:48:08.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XzXiUND6zpKIEVG8ZbMGq2ERDao=', 'Content-Length': '0'} 2023-09-29T22:48:08.673 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb285feb62c49d73-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.714 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.715 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.715 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.715 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.715 DEBUG:boto:Method: GET 2023-09-29T22:48:08.715 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.715 DEBUG:boto:Data: 2023-09-29T22:48:08.715 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.715 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.715 DEBUG:boto:Params: {} 2023-09-29T22:48:08.716 DEBUG:boto:Token: None 2023-09-29T22:48:08.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008a17c267935b819b-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.721 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.721 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.721 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.721 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.722 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.722 DEBUG:boto:Method: GET 2023-09-29T22:48:08.722 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:48:08.722 DEBUG:boto:Data: 2023-09-29T22:48:08.722 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.722 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.722 DEBUG:boto:Params: {} 2023-09-29T22:48:08.722 DEBUG:boto:Token: None 2023-09-29T22:48:08.723 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.731 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003c7aa0c7bbb99325-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.732 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:48:08.732 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.732 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.732 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.732 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.733 DEBUG:boto:Method: GET 2023-09-29T22:48:08.733 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.733 DEBUG:boto:Data: 2023-09-29T22:48:08.733 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.733 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.733 DEBUG:boto:Params: {} 2023-09-29T22:48:08.733 DEBUG:boto:Token: None 2023-09-29T22:48:08.733 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.734 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E= 2023-09-29T22:48:08.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E=', 'Content-Length': '0'} 2023-09-29T22:48:08.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea88d02bce2c3f8-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.736 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.736 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.736 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.736 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.736 DEBUG:boto:Method: GET 2023-09-29T22:48:08.737 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.737 DEBUG:boto:Data: 2023-09-29T22:48:08.737 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.737 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.737 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.737 DEBUG:boto:Params: {} 2023-09-29T22:48:08.737 DEBUG:boto:Token: None 2023-09-29T22:48:08.737 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E= 2023-09-29T22:48:08.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E=', 'Content-Length': '0'} 2023-09-29T22:48:08.740 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000968d05acaead90fa-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.740 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.740 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.740 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.740 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.740 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.740 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.741 DEBUG:boto:Data: 2023-09-29T22:48:08.741 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.741 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.741 DEBUG:boto:Params: {} 2023-09-29T22:48:08.741 DEBUG:boto:Token: None 2023-09-29T22:48:08.741 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.741 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TbgqXOaikXMQ5afNisIq0Xc4VtU= 2023-09-29T22:48:08.742 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TbgqXOaikXMQ5afNisIq0Xc4VtU=', 'Content-Length': '0'} 2023-09-29T22:48:08.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05e361a50ee4752-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.743 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.743 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.744 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.744 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.744 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.744 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.744 DEBUG:boto:Data: 2023-09-29T22:48:08.744 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.744 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.744 DEBUG:boto:Params: {} 2023-09-29T22:48:08.744 DEBUG:boto:Token: None 2023-09-29T22:48:08.745 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TbgqXOaikXMQ5afNisIq0Xc4VtU= 2023-09-29T22:48:08.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TbgqXOaikXMQ5afNisIq0Xc4VtU=', 'Content-Length': '0'} 2023-09-29T22:48:08.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056793bc788a3bd76-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:48:08.747 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.747 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.747 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.747 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.748 DEBUG:boto:Method: GET 2023-09-29T22:48:08.748 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.748 DEBUG:boto:Data: 2023-09-29T22:48:08.748 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.748 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.748 DEBUG:boto:Params: {} 2023-09-29T22:48:08.748 DEBUG:boto:Token: None 2023-09-29T22:48:08.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E= 2023-09-29T22:48:08.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E=', 'Content-Length': '0'} 2023-09-29T22:48:08.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dafb99188c153bb0-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.794 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.795 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.795 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.795 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.795 DEBUG:boto:Method: GET 2023-09-29T22:48:08.795 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.795 DEBUG:boto:Data: 2023-09-29T22:48:08.795 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.796 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.796 DEBUG:boto:Params: {} 2023-09-29T22:48:08.796 DEBUG:boto:Token: None 2023-09-29T22:48:08.796 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:48:08.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E= 2023-09-29T22:48:08.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hkoRPy7s2ZZcKEGl391yZzgKC4E=', 'Content-Length': '0'} 2023-09-29T22:48:08.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fe1fee09cc194dd-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.800 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.800 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.800 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.800 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.800 DEBUG:boto:Method: GET 2023-09-29T22:48:08.800 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.800 DEBUG:boto:Data: 2023-09-29T22:48:08.800 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.801 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.801 DEBUG:boto:Params: {} 2023-09-29T22:48:08.801 DEBUG:boto:Token: None 2023-09-29T22:48:08.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.806 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000085f9adb77e137282-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.806 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.807 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.807 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.807 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.807 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.807 DEBUG:boto:Method: GET 2023-09-29T22:48:08.807 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.808 DEBUG:boto:Data: 2023-09-29T22:48:08.808 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.808 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.808 DEBUG:boto:Params: {} 2023-09-29T22:48:08.808 DEBUG:boto:Token: None 2023-09-29T22:48:08.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.815 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcf4880dc0514d0e-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.815 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.816 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.816 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.816 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.816 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:48:08.816 DEBUG:boto:Data: 2023-09-29T22:48:08.816 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.816 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.817 DEBUG:boto:Params: {} 2023-09-29T22:48:08.817 DEBUG:boto:Token: None 2023-09-29T22:48:08.817 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93 2023-09-29T22:48:08.817 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6F32GZvceMa2Dn9bPQ/8YGrPSyU= 2023-09-29T22:48:08.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6F32GZvceMa2Dn9bPQ/8YGrPSyU=', 'Content-Length': '0'} 2023-09-29T22:48:08.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c3456f78fa38146-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.819 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.820 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.820 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.820 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:48:08.820 DEBUG:boto:Data: 2023-09-29T22:48:08.820 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.820 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.820 DEBUG:boto:Params: {} 2023-09-29T22:48:08.820 DEBUG:boto:Token: None 2023-09-29T22:48:08.821 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93 2023-09-29T22:48:08.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6F32GZvceMa2Dn9bPQ/8YGrPSyU= 2023-09-29T22:48:08.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6F32GZvceMa2Dn9bPQ/8YGrPSyU=', 'Content-Length': '0'} 2023-09-29T22:48:08.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004593a63daac0d43d-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:48:08.824 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.824 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.824 DEBUG:boto:Method: GET 2023-09-29T22:48:08.824 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:48:08.824 DEBUG:boto:Data: 2023-09-29T22:48:08.824 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.824 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.824 DEBUG:boto:Params: {} 2023-09-29T22:48:08.825 DEBUG:boto:Token: None 2023-09-29T22:48:08.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93 2023-09-29T22:48:08.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qmfeyTELTMRNHLGtpH+tVXa9OMg= 2023-09-29T22:48:08.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qmfeyTELTMRNHLGtpH+tVXa9OMg=', 'Content-Length': '0'} 2023-09-29T22:48:08.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c88467ba375d87ee-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.870 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:48:08.871 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:48:08.871 DEBUG:boto:Method: GET 2023-09-29T22:48:08.871 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:48:08.871 DEBUG:boto:Data: 2023-09-29T22:48:08.871 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.871 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.871 DEBUG:boto:Params: {} 2023-09-29T22:48:08.871 DEBUG:boto:Token: None 2023-09-29T22:48:08.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj93 2023-09-29T22:48:08.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qmfeyTELTMRNHLGtpH+tVXa9OMg= 2023-09-29T22:48:08.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qmfeyTELTMRNHLGtpH+tVXa9OMg=', 'Content-Length': '0'} 2023-09-29T22:48:08.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ecbd71175fbe9dc-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.875 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.875 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.875 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.876 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.876 DEBUG:boto:Method: GET 2023-09-29T22:48:08.876 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.876 DEBUG:boto:Data: 2023-09-29T22:48:08.876 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.876 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.876 DEBUG:boto:Params: {} 2023-09-29T22:48:08.876 DEBUG:boto:Token: None 2023-09-29T22:48:08.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000215c98fa870d0d6d-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.882 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.882 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.882 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.882 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.883 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.883 DEBUG:boto:Method: GET 2023-09-29T22:48:08.883 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:48:08.883 DEBUG:boto:Data: 2023-09-29T22:48:08.883 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.883 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.883 DEBUG:boto:Params: {} 2023-09-29T22:48:08.883 DEBUG:boto:Token: None 2023-09-29T22:48:08.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002db5b7393349d3a8-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.934 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:48:08.935 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.935 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.935 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.936 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.936 DEBUG:boto:Method: GET 2023-09-29T22:48:08.936 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.936 DEBUG:boto:Data: 2023-09-29T22:48:08.936 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.936 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.936 DEBUG:boto:Params: {} 2023-09-29T22:48:08.936 DEBUG:boto:Token: None 2023-09-29T22:48:08.937 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.937 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU= 2023-09-29T22:48:08.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU=', 'Content-Length': '0'} 2023-09-29T22:48:08.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a377c8cf0c64adb3-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.941 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.941 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.941 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.941 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.941 DEBUG:boto:Method: GET 2023-09-29T22:48:08.941 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.941 DEBUG:boto:Data: 2023-09-29T22:48:08.941 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.941 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.942 DEBUG:boto:Params: {} 2023-09-29T22:48:08.942 DEBUG:boto:Token: None 2023-09-29T22:48:08.942 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.942 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU= 2023-09-29T22:48:08.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU=', 'Content-Length': '0'} 2023-09-29T22:48:08.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005e71a6f7754fa6f-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.945 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.946 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.946 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.946 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.946 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.946 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.946 DEBUG:boto:Data: 2023-09-29T22:48:08.946 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.946 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.947 DEBUG:boto:Params: {} 2023-09-29T22:48:08.947 DEBUG:boto:Token: None 2023-09-29T22:48:08.947 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.947 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:caqpNR1MEUZBnj8m91vb/izo/8w= 2023-09-29T22:48:08.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:caqpNR1MEUZBnj8m91vb/izo/8w=', 'Content-Length': '0'} 2023-09-29T22:48:08.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099c9c5a9b47b8a6e-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.950 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.950 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.950 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.950 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.950 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.951 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.951 DEBUG:boto:Data: 2023-09-29T22:48:08.951 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.951 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.951 DEBUG:boto:Params: {} 2023-09-29T22:48:08.951 DEBUG:boto:Token: None 2023-09-29T22:48:08.951 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.951 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:caqpNR1MEUZBnj8m91vb/izo/8w= 2023-09-29T22:48:08.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:caqpNR1MEUZBnj8m91vb/izo/8w=', 'Content-Length': '0'} 2023-09-29T22:48:08.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f58332509aaf8abc-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:48:08.954 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.954 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.954 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.954 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.954 DEBUG:boto:Method: GET 2023-09-29T22:48:08.955 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.955 DEBUG:boto:Data: 2023-09-29T22:48:08.955 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.955 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.955 DEBUG:boto:Params: {} 2023-09-29T22:48:08.955 DEBUG:boto:Token: None 2023-09-29T22:48:08.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.955 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU= 2023-09-29T22:48:08.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU=', 'Content-Length': '0'} 2023-09-29T22:48:08.958 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f71ea8681f15257-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.958 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.958 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.958 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.959 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.959 DEBUG:boto:Method: GET 2023-09-29T22:48:08.959 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.959 DEBUG:boto:Data: 2023-09-29T22:48:08.959 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.959 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.959 DEBUG:boto:Params: {} 2023-09-29T22:48:08.959 DEBUG:boto:Token: None 2023-09-29T22:48:08.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:48:08.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU= 2023-09-29T22:48:08.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QdS/YU1I6GallNfnZ7B+ghY78HU=', 'Content-Length': '0'} 2023-09-29T22:48:08.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c424a26ef1b91ac4-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.963 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.963 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.963 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:08.963 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:08.963 DEBUG:boto:Method: GET 2023-09-29T22:48:08.963 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:48:08.963 DEBUG:boto:Data: 2023-09-29T22:48:08.963 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.964 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.964 DEBUG:boto:Params: {} 2023-09-29T22:48:08.964 DEBUG:boto:Token: None 2023-09-29T22:48:08.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a694ec275be6f054-0065175428-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.968 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.969 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:08.969 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:08.969 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:08.969 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:08.969 DEBUG:boto:Method: GET 2023-09-29T22:48:08.970 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:48:08.970 DEBUG:boto:Data: 2023-09-29T22:48:08.970 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.970 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.970 DEBUG:boto:Params: {} 2023-09-29T22:48:08.970 DEBUG:boto:Token: None 2023-09-29T22:48:08.970 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/ 2023-09-29T22:48:08.970 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA= 2023-09-29T22:48:08.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NYe6bbiO35S/0qaeuuKQfa6HFQA=', 'Content-Length': '0'} 2023-09-29T22:48:08.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f1e07bc3ca3c17f-0065175428-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.978 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:08.978 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.978 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.978 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.978 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:48:08.978 DEBUG:boto:Data: 2023-09-29T22:48:08.979 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.979 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.979 DEBUG:boto:Params: {} 2023-09-29T22:48:08.979 DEBUG:boto:Token: None 2023-09-29T22:48:08.979 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95 2023-09-29T22:48:08.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/692L63NpTdMuRHLHf1DiOdK9k= 2023-09-29T22:48:08.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/692L63NpTdMuRHLHf1DiOdK9k=', 'Content-Length': '0'} 2023-09-29T22:48:08.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055ea54603eeb241e-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.982 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.982 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.982 DEBUG:boto:Method: HEAD 2023-09-29T22:48:08.983 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:48:08.983 DEBUG:boto:Data: 2023-09-29T22:48:08.983 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:08.983 DEBUG:boto:Port: 8000 2023-09-29T22:48:08.983 DEBUG:boto:Params: {} 2023-09-29T22:48:08.983 DEBUG:boto:Token: None 2023-09-29T22:48:08.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95 2023-09-29T22:48:08.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n/692L63NpTdMuRHLHf1DiOdK9k= 2023-09-29T22:48:08.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n/692L63NpTdMuRHLHf1DiOdK9k=', 'Content-Length': '0'} 2023-09-29T22:48:08.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c061a623946738e-0065175428-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:08.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:48:08.986 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:08.986 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:08.986 DEBUG:boto:Method: GET 2023-09-29T22:48:08.986 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:48:08.986 DEBUG:boto:Data: 2023-09-29T22:48:08.986 DEBUG:boto:Headers: {} 2023-09-29T22:48:08.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:08.987 DEBUG:boto:Port: 8001 2023-09-29T22:48:08.987 DEBUG:boto:Params: {} 2023-09-29T22:48:08.987 DEBUG:boto:Token: None 2023-09-29T22:48:08.987 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:08 GMT /hcnphz-59/obj95 2023-09-29T22:48:08.987 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GnJ9y14XuDG48dYqrk23lw0QxMI= 2023-09-29T22:48:08.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:08 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GnJ9y14XuDG48dYqrk23lw0QxMI=', 'Content-Length': '0'} 2023-09-29T22:48:08.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dcddf9df6f2eb14-0065175428-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:08 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.030 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:48:09.030 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:48:09.031 DEBUG:boto:Method: GET 2023-09-29T22:48:09.031 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:48:09.031 DEBUG:boto:Data: 2023-09-29T22:48:09.031 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.031 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.031 DEBUG:boto:Params: {} 2023-09-29T22:48:09.031 DEBUG:boto:Token: None 2023-09-29T22:48:09.031 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/obj95 2023-09-29T22:48:09.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yC7blgkMKLrjU+t7CJDek8cRg7g= 2023-09-29T22:48:09.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yC7blgkMKLrjU+t7CJDek8cRg7g=', 'Content-Length': '0'} 2023-09-29T22:48:09.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac86a0f8434c2dd7-0065175429-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.035 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.035 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.035 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:09.035 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:09.035 DEBUG:boto:Method: GET 2023-09-29T22:48:09.036 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:48:09.036 DEBUG:boto:Data: 2023-09-29T22:48:09.036 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.036 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:48:09.036 DEBUG:boto:Port: 8001 2023-09-29T22:48:09.036 DEBUG:boto:Params: {} 2023-09-29T22:48:09.036 DEBUG:boto:Token: None 2023-09-29T22:48:09.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q817keZv40bQsTqNt0dPSI54ol8= 2023-09-29T22:48:09.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q817keZv40bQsTqNt0dPSI54ol8=', 'Content-Length': '0'} 2023-09-29T22:48:09.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000edb0ea24bcf821fe-0065175429-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.041 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:09.042 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.042 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.042 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:09.042 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:48:09.042 DEBUG:boto:Method: GET 2023-09-29T22:48:09.042 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:48:09.043 DEBUG:boto:Data: 2023-09-29T22:48:09.043 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.043 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.043 DEBUG:boto:Params: {} 2023-09-29T22:48:09.043 DEBUG:boto:Token: None 2023-09-29T22:48:09.043 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.043 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q817keZv40bQsTqNt0dPSI54ol8= 2023-09-29T22:48:09.043 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q817keZv40bQsTqNt0dPSI54ol8=', 'Content-Length': '0'} 2023-09-29T22:48:09.049 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000015b9fd77f478049f-0065175429-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.049 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:48:09.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:48:09.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:48:09.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:48:09.051 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:48:09.097 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:09.097 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:09.183 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:48:09.183 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-59 2023-09-29T22:48:09.208 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-09-29T22:48:09.327 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.327 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.327 DEBUG:boto:Method: HEAD 2023-09-29T22:48:09.327 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:09.327 DEBUG:boto:Data: 2023-09-29T22:48:09.327 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.327 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.327 DEBUG:boto:Params: {} 2023-09-29T22:48:09.328 DEBUG:boto:Token: None 2023-09-29T22:48:09.328 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk= 2023-09-29T22:48:09.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk=', 'Content-Length': '0'} 2023-09-29T22:48:09.335 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000017e71c07ac268c7b-0065175429-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.335 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:48:09.335 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:48:09.336 DEBUG:boto:Method: PUT 2023-09-29T22:48:09.336 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:48:09.336 DEBUG:boto:Data: 2023-09-29T22:48:09.336 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:48:09.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.336 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.336 DEBUG:boto:Params: {} 2023-09-29T22:48:09.336 DEBUG:boto:Token: None 2023-09-29T22:48:09.336 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/obj37 2023-09-29T22:48:09.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YCAVQ+PsBxvEtjQqHrKHrFVSruM= 2023-09-29T22:48:09.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YCAVQ+PsBxvEtjQqHrKHrFVSruM='} 2023-09-29T22:48:09.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c9e7f4758f282a0e-0065175429-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.346 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.346 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.346 DEBUG:boto:Method: HEAD 2023-09-29T22:48:09.346 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:09.346 DEBUG:boto:Data: 2023-09-29T22:48:09.347 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.347 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.347 DEBUG:boto:Params: {} 2023-09-29T22:48:09.347 DEBUG:boto:Token: None 2023-09-29T22:48:09.347 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk= 2023-09-29T22:48:09.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk=', 'Content-Length': '0'} 2023-09-29T22:48:09.352 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000009c13bb6b369f1e8-0065175429-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.353 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:48:09.353 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:48:09.353 DEBUG:boto:Method: PUT 2023-09-29T22:48:09.353 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:48:09.353 DEBUG:boto:Data: 2023-09-29T22:48:09.353 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:48:09.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.354 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.354 DEBUG:boto:Params: {} 2023-09-29T22:48:09.354 DEBUG:boto:Token: None 2023-09-29T22:48:09.354 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/obj74 2023-09-29T22:48:09.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KXciUWrF9vC38O8lafOMOrmJ734= 2023-09-29T22:48:09.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KXciUWrF9vC38O8lafOMOrmJ734='} 2023-09-29T22:48:09.399 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094a724a78014b2f0-0065175429-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.399 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.399 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.399 DEBUG:boto:Method: HEAD 2023-09-29T22:48:09.399 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:09.399 DEBUG:boto:Data: 2023-09-29T22:48:09.399 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.400 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.400 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.400 DEBUG:boto:Params: {} 2023-09-29T22:48:09.400 DEBUG:boto:Token: None 2023-09-29T22:48:09.400 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk= 2023-09-29T22:48:09.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk=', 'Content-Length': '0'} 2023-09-29T22:48:09.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000107872de6599625c-0065175429-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.408 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:48:09.408 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:48:09.408 DEBUG:boto:Method: PUT 2023-09-29T22:48:09.408 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:48:09.408 DEBUG:boto:Data: 2023-09-29T22:48:09.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:48:09.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.409 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.409 DEBUG:boto:Params: {} 2023-09-29T22:48:09.409 DEBUG:boto:Token: None 2023-09-29T22:48:09.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/obj111 2023-09-29T22:48:09.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P/7UZ4u4ciGWHLoelbVaYiGaEB0= 2023-09-29T22:48:09.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P/7UZ4u4ciGWHLoelbVaYiGaEB0='} 2023-09-29T22:48:09.419 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000808b95a495eebe7-0065175429-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.419 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.419 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.419 DEBUG:boto:Method: HEAD 2023-09-29T22:48:09.419 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:09.420 DEBUG:boto:Data: 2023-09-29T22:48:09.420 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.420 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.420 DEBUG:boto:Params: {} 2023-09-29T22:48:09.420 DEBUG:boto:Token: None 2023-09-29T22:48:09.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk= 2023-09-29T22:48:09.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk=', 'Content-Length': '0'} 2023-09-29T22:48:09.426 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e65ce1781e552c6-0065175429-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.426 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:48:09.426 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:48:09.427 DEBUG:boto:Method: PUT 2023-09-29T22:48:09.427 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:48:09.427 DEBUG:boto:Data: 2023-09-29T22:48:09.427 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:48:09.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.427 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.427 DEBUG:boto:Params: {} 2023-09-29T22:48:09.427 DEBUG:boto:Token: None 2023-09-29T22:48:09.428 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/obj148 2023-09-29T22:48:09.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oAnVl7xgOmm/WjpLbNt+GyvhM+g= 2023-09-29T22:48:09.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oAnVl7xgOmm/WjpLbNt+GyvhM+g='} 2023-09-29T22:48:09.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eed41d2e9443020d-0065175429-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.475 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:48:09.475 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:48:09.475 DEBUG:boto:Method: HEAD 2023-09-29T22:48:09.475 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:48:09.475 DEBUG:boto:Data: 2023-09-29T22:48:09.476 DEBUG:boto:Headers: {} 2023-09-29T22:48:09.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.476 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.476 DEBUG:boto:Params: {} 2023-09-29T22:48:09.476 DEBUG:boto:Token: None 2023-09-29T22:48:09.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/ 2023-09-29T22:48:09.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk= 2023-09-29T22:48:09.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0fmA3uyMmsSfTjRqzMpUV2P9gkk=', 'Content-Length': '0'} 2023-09-29T22:48:09.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ccadac9a898a5698-0065175429-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.484 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:48:09.484 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:48:09.484 DEBUG:boto:Method: PUT 2023-09-29T22:48:09.484 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:48:09.484 DEBUG:boto:Data: 2023-09-29T22:48:09.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:48:09.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:48:09.485 DEBUG:boto:Port: 8000 2023-09-29T22:48:09.485 DEBUG:boto:Params: {} 2023-09-29T22:48:09.485 DEBUG:boto:Token: None 2023-09-29T22:48:09.485 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:48:09 GMT /hcnphz-59/obj185 2023-09-29T22:48:09.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AGSodWlzjptazOY9GokivNNZgAE= 2023-09-29T22:48:09.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:48:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AGSodWlzjptazOY9GokivNNZgAE='} 2023-09-29T22:48:09.494 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b00a9e5f521f67de-0065175429-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:48:09.494 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:48:09.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:48:09.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:48:09.495 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:48:09.543 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:48:09.543 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:48:09.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:09.602+0000 7f3f1002cf40 1 waiting to reach latest gen 6, on 5.. 2023-09-29T22:48:14.606 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:14.602+0000 7f3f1002cf40 1 waiting to reach latest gen 6, on 5.. 2023-09-29T22:48:19.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:19.606+0000 7f3f1002cf40 1 waiting to reach latest gen 6, on 5.. 2023-09-29T22:48:24.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:24.610+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:24.615 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:24.615 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:29.621 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:29.618+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:29.621 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:29.621 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:34.626 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:34.622+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:34.627 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:34.627 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:39.632 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:39.630+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:39.632 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:39.632 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:44.637 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:44.634+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:44.637 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:44.637 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:49.642 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:49.637+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:49.643 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:49.643 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:54.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:54.645+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:54.648 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.99.6, , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:54.648 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:48:59.653 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:48:59.649+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:48:59.653 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.99.6, , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:48:59.653 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:04.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:04.657+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:04.659 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.99.6, , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:49:04.659 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:09.665 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:09.665+0000 7f3f1002cf40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:09.665 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , , , , , , , , , , 00000000002.99.6, , , , , , , , , , , , , , , , , , , ] 2023-09-29T22:49:09.665 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:14.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:14.669+0000 7f3f1002cf40 1 bucket sync caught up with source: 2023-09-29T22:49:14.671 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:14.671 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:14.671 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:14.669+0000 7f3f1002cf40 0 bucket checkpoint complete 2023-09-29T22:49:14.696 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:49:14.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:49:14.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:49:14.697 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:49:14.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:14.746 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:14.810 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:14.809+0000 7eff63a87f40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:14.811 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000004.210.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000007.219.6, 00000000004.316.6] 2023-09-29T22:49:14.811 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000005.221.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000004.300.6, 00000000007.219.6, 00000000006.340.6] 2023-09-29T22:49:19.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:19.813+0000 7eff63a87f40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:19.814 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000004.210.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000003.292.6, 00000000007.219.6, 00000000004.316.6] 2023-09-29T22:49:19.814 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000005.221.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000004.300.6, 00000000007.219.6, 00000000006.340.6] 2023-09-29T22:49:24.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:24.817+0000 7eff63a87f40 1 bucket sync caught up with source: 2023-09-29T22:49:24.816 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000005.221.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000004.300.6, 00000000007.219.6, 00000000006.340.6] 2023-09-29T22:49:24.816 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.272.6, 00000000002.586.6, 00000000002.254.6, 00000000005.221.6, 00000000004.242.6, 00000000001.181.6, 00000000001.280.6, 00000000004.300.6, 00000000007.219.6, 00000000006.340.6] 2023-09-29T22:49:24.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:24.817+0000 7eff63a87f40 0 bucket checkpoint complete 2023-09-29T22:49:24.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:49:24.846 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:24.846 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:24.846 DEBUG:boto:Method: HEAD 2023-09-29T22:49:24.847 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:49:24.847 DEBUG:boto:Data: 2023-09-29T22:49:24.847 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:24.847 DEBUG:boto:Port: 8001 2023-09-29T22:49:24.847 DEBUG:boto:Params: {} 2023-09-29T22:49:24.847 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:49:24.848 DEBUG:boto:Token: None 2023-09-29T22:49:24.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/ 2023-09-29T22:49:24.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Ck2u53yp4hGvgoqB4M3klQXl/8= 2023-09-29T22:49:24.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Ck2u53yp4hGvgoqB4M3klQXl/8=', 'Content-Length': '0'} 2023-09-29T22:49:24.853 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d66d72fd7d7e6236-0065175474-1b41-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.853 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:24.853 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:24.854 DEBUG:boto:Method: HEAD 2023-09-29T22:49:24.854 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:49:24.854 DEBUG:boto:Data: 2023-09-29T22:49:24.854 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.854 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:24.854 DEBUG:boto:Port: 8000 2023-09-29T22:49:24.854 DEBUG:boto:Params: {} 2023-09-29T22:49:24.854 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:49:24.855 DEBUG:boto:Token: None 2023-09-29T22:49:24.855 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/ 2023-09-29T22:49:24.855 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Ck2u53yp4hGvgoqB4M3klQXl/8= 2023-09-29T22:49:24.855 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Ck2u53yp4hGvgoqB4M3klQXl/8=', 'Content-Length': '0'} 2023-09-29T22:49:24.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e09c35e2dcbe337f-0065175474-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.863 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:49:24.863 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:24.863 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:24.863 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.864 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.864 DEBUG:boto:Method: GET 2023-09-29T22:49:24.864 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.864 DEBUG:boto:Data: 2023-09-29T22:49:24.864 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:24.864 DEBUG:boto:Port: 8001 2023-09-29T22:49:24.864 DEBUG:boto:Params: {} 2023-09-29T22:49:24.864 DEBUG:boto:Token: None 2023-09-29T22:49:24.865 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/?versions 2023-09-29T22:49:24.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4= 2023-09-29T22:49:24.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4=', 'Content-Length': '0'} 2023-09-29T22:49:24.869 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e98eea79a555599-0065175474-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.869 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:49:24.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-29T22:49:24.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-29T22:49:24.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:49:24.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:49:24.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:49:24.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:49:24.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:49:24.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:49:24.874 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:49:24.874 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:24.874 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:24.874 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.875 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.875 DEBUG:boto:Method: GET 2023-09-29T22:49:24.875 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.875 DEBUG:boto:Data: 2023-09-29T22:49:24.875 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:24.875 DEBUG:boto:Port: 8000 2023-09-29T22:49:24.875 DEBUG:boto:Params: {} 2023-09-29T22:49:24.875 DEBUG:boto:Token: None 2023-09-29T22:49:24.875 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/?versions 2023-09-29T22:49:24.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4= 2023-09-29T22:49:24.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4=', 'Content-Length': '0'} 2023-09-29T22:49:24.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007428530d3629a9cb-0065175474-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.883 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:24.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:49:24.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-29T22:49:24.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:49:24.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:49:24.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:49:24.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:49:24.887 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:24.888 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:24.888 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.888 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.888 DEBUG:boto:Method: GET 2023-09-29T22:49:24.888 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.888 DEBUG:boto:Data: 2023-09-29T22:49:24.888 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:24.888 DEBUG:boto:Port: 8001 2023-09-29T22:49:24.888 DEBUG:boto:Params: {} 2023-09-29T22:49:24.888 DEBUG:boto:Token: None 2023-09-29T22:49:24.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/?versions 2023-09-29T22:49:24.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4= 2023-09-29T22:49:24.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4=', 'Content-Length': '0'} 2023-09-29T22:49:24.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a3041040370bf2-0065175474-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.893 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:24.895 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:24.895 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:24.895 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.895 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.895 DEBUG:boto:Method: GET 2023-09-29T22:49:24.895 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:49:24.896 DEBUG:boto:Data: 2023-09-29T22:49:24.896 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:24.896 DEBUG:boto:Port: 8000 2023-09-29T22:49:24.896 DEBUG:boto:Params: {} 2023-09-29T22:49:24.896 DEBUG:boto:Token: None 2023-09-29T22:49:24.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/?versions 2023-09-29T22:49:24.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4= 2023-09-29T22:49:24.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SLdUZABAvnI5RF6Gs8XXJ417Q/4=', 'Content-Length': '0'} 2023-09-29T22:49:24.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b259a7966eaf3922-0065175474-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.903 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:24.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:49:24.905 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:24.905 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:24.905 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.905 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.906 DEBUG:boto:Method: GET 2023-09-29T22:49:24.906 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.906 DEBUG:boto:Data: 2023-09-29T22:49:24.906 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:24.906 DEBUG:boto:Port: 8001 2023-09-29T22:49:24.906 DEBUG:boto:Params: {} 2023-09-29T22:49:24.906 DEBUG:boto:Token: None 2023-09-29T22:49:24.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.907 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM= 2023-09-29T22:49:24.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM=', 'Content-Length': '0'} 2023-09-29T22:49:24.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a38ad546d1476ed1-0065175474-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.950 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:24.950 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:24.951 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.951 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.951 DEBUG:boto:Method: GET 2023-09-29T22:49:24.951 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.951 DEBUG:boto:Data: 2023-09-29T22:49:24.951 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:24.951 DEBUG:boto:Port: 8000 2023-09-29T22:49:24.951 DEBUG:boto:Params: {} 2023-09-29T22:49:24.951 DEBUG:boto:Token: None 2023-09-29T22:49:24.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM= 2023-09-29T22:49:24.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM=', 'Content-Length': '0'} 2023-09-29T22:49:24.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fa67b5e374114df-0065175474-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.955 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:24.955 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:24.955 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.955 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.955 DEBUG:boto:Method: HEAD 2023-09-29T22:49:24.955 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.956 DEBUG:boto:Data: 2023-09-29T22:49:24.956 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:24.956 DEBUG:boto:Port: 8001 2023-09-29T22:49:24.956 DEBUG:boto:Params: {} 2023-09-29T22:49:24.956 DEBUG:boto:Token: None 2023-09-29T22:49:24.956 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G7LW7+bO1p8I2uSEwPSMP0kn4t4= 2023-09-29T22:49:24.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G7LW7+bO1p8I2uSEwPSMP0kn4t4=', 'Content-Length': '0'} 2023-09-29T22:49:24.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cba3f657c499fbb-0065175474-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.959 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:24.959 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:24.959 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.959 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.959 DEBUG:boto:Method: HEAD 2023-09-29T22:49:24.960 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.960 DEBUG:boto:Data: 2023-09-29T22:49:24.960 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:24.960 DEBUG:boto:Port: 8000 2023-09-29T22:49:24.960 DEBUG:boto:Params: {} 2023-09-29T22:49:24.960 DEBUG:boto:Token: None 2023-09-29T22:49:24.960 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G7LW7+bO1p8I2uSEwPSMP0kn4t4= 2023-09-29T22:49:24.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G7LW7+bO1p8I2uSEwPSMP0kn4t4=', 'Content-Length': '0'} 2023-09-29T22:49:24.963 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053b12b96da863b17-0065175474-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:49:24.963 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:24.963 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:24.964 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.964 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.964 DEBUG:boto:Method: GET 2023-09-29T22:49:24.964 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.964 DEBUG:boto:Data: 2023-09-29T22:49:24.964 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.964 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:24.964 DEBUG:boto:Port: 8001 2023-09-29T22:49:24.964 DEBUG:boto:Params: {} 2023-09-29T22:49:24.964 DEBUG:boto:Token: None 2023-09-29T22:49:24.965 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.965 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM= 2023-09-29T22:49:24.965 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM=', 'Content-Length': '0'} 2023-09-29T22:49:24.968 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a39c240831953752-0065175474-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:24.968 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:24.968 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:24.968 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.969 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.969 DEBUG:boto:Method: GET 2023-09-29T22:49:24.969 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.969 DEBUG:boto:Data: 2023-09-29T22:49:24.969 DEBUG:boto:Headers: {} 2023-09-29T22:49:24.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:24.969 DEBUG:boto:Port: 8000 2023-09-29T22:49:24.969 DEBUG:boto:Params: {} 2023-09-29T22:49:24.969 DEBUG:boto:Token: None 2023-09-29T22:49:24.969 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:24 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:49:24.970 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM= 2023-09-29T22:49:24.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mayT2M4wiGnXXF6aC7Q4Npl77LM=', 'Content-Length': '0'} 2023-09-29T22:49:24.972 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c46fe5a80ae0b48b-0065175474-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.014 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.015 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.015 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.015 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.015 DEBUG:boto:Method: GET 2023-09-29T22:49:25.015 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.015 DEBUG:boto:Data: 2023-09-29T22:49:25.015 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.015 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.016 DEBUG:boto:Params: {} 2023-09-29T22:49:25.016 DEBUG:boto:Token: None 2023-09-29T22:49:25.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca3fc47c17fd9fc9-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.021 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.022 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.022 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.022 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.022 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.022 DEBUG:boto:Method: GET 2023-09-29T22:49:25.022 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.022 DEBUG:boto:Data: 2023-09-29T22:49:25.023 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.023 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.023 DEBUG:boto:Params: {} 2023-09-29T22:49:25.023 DEBUG:boto:Token: None 2023-09-29T22:49:25.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.032 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000883a9a2bddd72a59-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.032 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.033 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:25.033 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:25.033 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.033 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:49:25.033 DEBUG:boto:Data: 2023-09-29T22:49:25.033 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.033 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.033 DEBUG:boto:Params: {} 2023-09-29T22:49:25.034 DEBUG:boto:Token: None 2023-09-29T22:49:25.034 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj11 2023-09-29T22:49:25.034 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VrWbA+U8XcM4hw4xpkIS9X8YBcQ= 2023-09-29T22:49:25.034 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VrWbA+U8XcM4hw4xpkIS9X8YBcQ=', 'Content-Length': '0'} 2023-09-29T22:49:25.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbf6c6af9d044cde-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.036 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:25.036 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:25.036 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.036 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:49:25.036 DEBUG:boto:Data: 2023-09-29T22:49:25.036 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.037 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.037 DEBUG:boto:Params: {} 2023-09-29T22:49:25.037 DEBUG:boto:Token: None 2023-09-29T22:49:25.037 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj11 2023-09-29T22:49:25.037 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VrWbA+U8XcM4hw4xpkIS9X8YBcQ= 2023-09-29T22:49:25.037 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VrWbA+U8XcM4hw4xpkIS9X8YBcQ=', 'Content-Length': '0'} 2023-09-29T22:49:25.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031004dccd1e719d1-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:49:25.039 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:25.040 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:25.040 DEBUG:boto:Method: GET 2023-09-29T22:49:25.040 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:49:25.040 DEBUG:boto:Data: 2023-09-29T22:49:25.040 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.040 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.040 DEBUG:boto:Params: {} 2023-09-29T22:49:25.040 DEBUG:boto:Token: None 2023-09-29T22:49:25.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj11 2023-09-29T22:49:25.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EpMUI1ZpGSVT4QRlAgkrDJ9X6As= 2023-09-29T22:49:25.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EpMUI1ZpGSVT4QRlAgkrDJ9X6As=', 'Content-Length': '0'} 2023-09-29T22:49:25.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e74925c1497a6c29-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.087 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:49:25.087 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:49:25.087 DEBUG:boto:Method: GET 2023-09-29T22:49:25.087 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:49:25.087 DEBUG:boto:Data: 2023-09-29T22:49:25.087 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.087 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.087 DEBUG:boto:Params: {} 2023-09-29T22:49:25.088 DEBUG:boto:Token: None 2023-09-29T22:49:25.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj11 2023-09-29T22:49:25.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EpMUI1ZpGSVT4QRlAgkrDJ9X6As= 2023-09-29T22:49:25.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EpMUI1ZpGSVT4QRlAgkrDJ9X6As=', 'Content-Length': '0'} 2023-09-29T22:49:25.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4f10e58a80835de-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.091 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.091 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.091 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.091 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.091 DEBUG:boto:Method: GET 2023-09-29T22:49:25.091 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.091 DEBUG:boto:Data: 2023-09-29T22:49:25.092 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.092 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.092 DEBUG:boto:Params: {} 2023-09-29T22:49:25.092 DEBUG:boto:Token: None 2023-09-29T22:49:25.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa8deda00da36363-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.097 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.098 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.098 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.098 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.098 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.098 DEBUG:boto:Method: GET 2023-09-29T22:49:25.098 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:49:25.098 DEBUG:boto:Data: 2023-09-29T22:49:25.098 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.099 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.099 DEBUG:boto:Params: {} 2023-09-29T22:49:25.099 DEBUG:boto:Token: None 2023-09-29T22:49:25.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.099 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033f71d49f96c0dfd-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.107 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:49:25.108 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.108 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.108 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.108 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.108 DEBUG:boto:Method: GET 2023-09-29T22:49:25.108 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.108 DEBUG:boto:Data: 2023-09-29T22:49:25.108 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.109 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.109 DEBUG:boto:Params: {} 2023-09-29T22:49:25.109 DEBUG:boto:Token: None 2023-09-29T22:49:25.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0= 2023-09-29T22:49:25.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0=', 'Content-Length': '0'} 2023-09-29T22:49:25.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e6b977c199f42d0-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.112 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.112 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.112 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.112 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.112 DEBUG:boto:Method: GET 2023-09-29T22:49:25.112 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.112 DEBUG:boto:Data: 2023-09-29T22:49:25.112 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.113 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.113 DEBUG:boto:Params: {} 2023-09-29T22:49:25.113 DEBUG:boto:Token: None 2023-09-29T22:49:25.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0= 2023-09-29T22:49:25.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0=', 'Content-Length': '0'} 2023-09-29T22:49:25.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062ed0d33e648d919-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.158 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.159 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.159 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.159 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.159 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.159 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.159 DEBUG:boto:Data: 2023-09-29T22:49:25.159 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.160 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.160 DEBUG:boto:Params: {} 2023-09-29T22:49:25.160 DEBUG:boto:Token: None 2023-09-29T22:49:25.160 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.160 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+hE/UNsQS6cFcEbDzCoEq6vtFs8= 2023-09-29T22:49:25.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+hE/UNsQS6cFcEbDzCoEq6vtFs8=', 'Content-Length': '0'} 2023-09-29T22:49:25.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003aae0d3c87168879-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.163 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.163 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.163 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.164 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.164 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.164 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.164 DEBUG:boto:Data: 2023-09-29T22:49:25.164 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.164 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.164 DEBUG:boto:Params: {} 2023-09-29T22:49:25.164 DEBUG:boto:Token: None 2023-09-29T22:49:25.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.165 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+hE/UNsQS6cFcEbDzCoEq6vtFs8= 2023-09-29T22:49:25.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+hE/UNsQS6cFcEbDzCoEq6vtFs8=', 'Content-Length': '0'} 2023-09-29T22:49:25.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000713936e181a0fe10-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:49:25.167 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.167 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.167 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.168 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.168 DEBUG:boto:Method: GET 2023-09-29T22:49:25.168 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.168 DEBUG:boto:Data: 2023-09-29T22:49:25.168 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.168 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.168 DEBUG:boto:Params: {} 2023-09-29T22:49:25.168 DEBUG:boto:Token: None 2023-09-29T22:49:25.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0= 2023-09-29T22:49:25.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0=', 'Content-Length': '0'} 2023-09-29T22:49:25.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000894b8742d60e7917-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.214 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.214 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.215 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.215 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.215 DEBUG:boto:Method: GET 2023-09-29T22:49:25.215 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.215 DEBUG:boto:Data: 2023-09-29T22:49:25.215 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.215 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.215 DEBUG:boto:Params: {} 2023-09-29T22:49:25.215 DEBUG:boto:Token: None 2023-09-29T22:49:25.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:49:25.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0= 2023-09-29T22:49:25.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZbQpY/oqwyf0u8NAdo2B5dHjwD0=', 'Content-Length': '0'} 2023-09-29T22:49:25.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa4847ba359ba59e-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.219 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.219 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.219 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.219 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.219 DEBUG:boto:Method: GET 2023-09-29T22:49:25.219 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.219 DEBUG:boto:Data: 2023-09-29T22:49:25.219 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.220 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.220 DEBUG:boto:Params: {} 2023-09-29T22:49:25.220 DEBUG:boto:Token: None 2023-09-29T22:49:25.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.225 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1795371c2c8ecb8-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.225 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.225 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.225 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.226 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.226 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.226 DEBUG:boto:Method: GET 2023-09-29T22:49:25.226 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.226 DEBUG:boto:Data: 2023-09-29T22:49:25.226 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.226 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.226 DEBUG:boto:Params: {} 2023-09-29T22:49:25.226 DEBUG:boto:Token: None 2023-09-29T22:49:25.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.227 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.227 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.234 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000992a5ad4f2bb679-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.235 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.235 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.235 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.235 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.235 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:49:25.235 DEBUG:boto:Data: 2023-09-29T22:49:25.236 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.236 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.236 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.236 DEBUG:boto:Params: {} 2023-09-29T22:49:25.236 DEBUG:boto:Token: None 2023-09-29T22:49:25.236 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111 2023-09-29T22:49:25.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L1ezvyd+MbHes4Rg2pFVJ9bvSAQ= 2023-09-29T22:49:25.236 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L1ezvyd+MbHes4Rg2pFVJ9bvSAQ=', 'Content-Length': '0'} 2023-09-29T22:49:25.238 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ce45c30152f84bf-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.238 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.238 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.239 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.239 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:49:25.239 DEBUG:boto:Data: 2023-09-29T22:49:25.239 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.239 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.239 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.239 DEBUG:boto:Params: {} 2023-09-29T22:49:25.239 DEBUG:boto:Token: None 2023-09-29T22:49:25.239 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111 2023-09-29T22:49:25.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L1ezvyd+MbHes4Rg2pFVJ9bvSAQ= 2023-09-29T22:49:25.240 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L1ezvyd+MbHes4Rg2pFVJ9bvSAQ=', 'Content-Length': '0'} 2023-09-29T22:49:25.242 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000224fb4643459f765-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:49:25.242 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.242 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.243 DEBUG:boto:Method: GET 2023-09-29T22:49:25.243 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:49:25.243 DEBUG:boto:Data: 2023-09-29T22:49:25.243 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.243 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.243 DEBUG:boto:Params: {} 2023-09-29T22:49:25.243 DEBUG:boto:Token: None 2023-09-29T22:49:25.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111 2023-09-29T22:49:25.243 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+6HQ1NFkLDc8f+oo+nELdQp8e1Q= 2023-09-29T22:49:25.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+6HQ1NFkLDc8f+oo+nELdQp8e1Q=', 'Content-Length': '0'} 2023-09-29T22:49:25.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d68f442cd5502894-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.246 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:49:25.246 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:49:25.246 DEBUG:boto:Method: GET 2023-09-29T22:49:25.246 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:49:25.246 DEBUG:boto:Data: 2023-09-29T22:49:25.247 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.247 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.247 DEBUG:boto:Params: {} 2023-09-29T22:49:25.247 DEBUG:boto:Token: None 2023-09-29T22:49:25.247 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj111 2023-09-29T22:49:25.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+6HQ1NFkLDc8f+oo+nELdQp8e1Q= 2023-09-29T22:49:25.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+6HQ1NFkLDc8f+oo+nELdQp8e1Q=', 'Content-Length': '0'} 2023-09-29T22:49:25.249 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051b35abfe549d5e6-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.290 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.291 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.291 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.291 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.291 DEBUG:boto:Method: GET 2023-09-29T22:49:25.291 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.291 DEBUG:boto:Data: 2023-09-29T22:49:25.291 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.291 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.291 DEBUG:boto:Params: {} 2023-09-29T22:49:25.292 DEBUG:boto:Token: None 2023-09-29T22:49:25.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.296 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000285a514461b237fd-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.297 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.297 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.297 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.297 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.297 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.297 DEBUG:boto:Method: GET 2023-09-29T22:49:25.297 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:49:25.298 DEBUG:boto:Data: 2023-09-29T22:49:25.298 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.298 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.298 DEBUG:boto:Params: {} 2023-09-29T22:49:25.298 DEBUG:boto:Token: None 2023-09-29T22:49:25.298 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.306 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ff452cd83001e72-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.306 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:49:25.307 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.307 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.307 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.307 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.307 DEBUG:boto:Method: GET 2023-09-29T22:49:25.307 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.307 DEBUG:boto:Data: 2023-09-29T22:49:25.307 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.307 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.307 DEBUG:boto:Params: {} 2023-09-29T22:49:25.308 DEBUG:boto:Token: None 2023-09-29T22:49:25.308 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.308 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI= 2023-09-29T22:49:25.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI=', 'Content-Length': '0'} 2023-09-29T22:49:25.311 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa6f46bed562033a-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.358 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.358 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.359 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.359 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.359 DEBUG:boto:Method: GET 2023-09-29T22:49:25.359 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.359 DEBUG:boto:Data: 2023-09-29T22:49:25.359 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.359 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.359 DEBUG:boto:Params: {} 2023-09-29T22:49:25.359 DEBUG:boto:Token: None 2023-09-29T22:49:25.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI= 2023-09-29T22:49:25.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI=', 'Content-Length': '0'} 2023-09-29T22:49:25.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d178c1f1582a0ba9-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.363 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.363 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.363 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.363 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.363 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.363 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.364 DEBUG:boto:Data: 2023-09-29T22:49:25.364 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.364 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.364 DEBUG:boto:Params: {} 2023-09-29T22:49:25.364 DEBUG:boto:Token: None 2023-09-29T22:49:25.364 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oqbCzUNAi68rBKqB4K1wHmMWcaU= 2023-09-29T22:49:25.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oqbCzUNAi68rBKqB4K1wHmMWcaU=', 'Content-Length': '0'} 2023-09-29T22:49:25.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038074ca56e55951b-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.368 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.368 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.368 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.368 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.369 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.369 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.369 DEBUG:boto:Data: 2023-09-29T22:49:25.369 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.369 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.369 DEBUG:boto:Params: {} 2023-09-29T22:49:25.369 DEBUG:boto:Token: None 2023-09-29T22:49:25.369 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.369 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oqbCzUNAi68rBKqB4K1wHmMWcaU= 2023-09-29T22:49:25.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oqbCzUNAi68rBKqB4K1wHmMWcaU=', 'Content-Length': '0'} 2023-09-29T22:49:25.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000810551e30473e97-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:49:25.372 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.372 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.372 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.372 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.373 DEBUG:boto:Method: GET 2023-09-29T22:49:25.373 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.373 DEBUG:boto:Data: 2023-09-29T22:49:25.373 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.373 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.373 DEBUG:boto:Params: {} 2023-09-29T22:49:25.373 DEBUG:boto:Token: None 2023-09-29T22:49:25.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.374 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI= 2023-09-29T22:49:25.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI=', 'Content-Length': '0'} 2023-09-29T22:49:25.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6563de0369a3a8e-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.377 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.377 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.377 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.377 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.377 DEBUG:boto:Method: GET 2023-09-29T22:49:25.377 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.377 DEBUG:boto:Data: 2023-09-29T22:49:25.377 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.377 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.377 DEBUG:boto:Params: {} 2023-09-29T22:49:25.378 DEBUG:boto:Token: None 2023-09-29T22:49:25.378 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:49:25.378 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI= 2023-09-29T22:49:25.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7FHLorRNW0FAnRI+TCFQCv/uziI=', 'Content-Length': '0'} 2023-09-29T22:49:25.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004ced218a14b6a564-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.426 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.427 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.427 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.427 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.427 DEBUG:boto:Method: GET 2023-09-29T22:49:25.427 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.427 DEBUG:boto:Data: 2023-09-29T22:49:25.427 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.427 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.427 DEBUG:boto:Params: {} 2023-09-29T22:49:25.428 DEBUG:boto:Token: None 2023-09-29T22:49:25.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.433 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96d5b824f73aabc-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.433 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.433 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.434 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.434 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.434 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.434 DEBUG:boto:Method: GET 2023-09-29T22:49:25.434 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.434 DEBUG:boto:Data: 2023-09-29T22:49:25.434 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.434 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.435 DEBUG:boto:Params: {} 2023-09-29T22:49:25.435 DEBUG:boto:Token: None 2023-09-29T22:49:25.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.435 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c74a41fe9b536ae-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.443 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.443 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.443 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.443 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.443 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:49:25.443 DEBUG:boto:Data: 2023-09-29T22:49:25.443 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.444 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.444 DEBUG:boto:Params: {} 2023-09-29T22:49:25.444 DEBUG:boto:Token: None 2023-09-29T22:49:25.444 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115 2023-09-29T22:49:25.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s7n4L9oCWxPAgCyWp7cn4ULImdw= 2023-09-29T22:49:25.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s7n4L9oCWxPAgCyWp7cn4ULImdw=', 'Content-Length': '0'} 2023-09-29T22:49:25.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8a0d8c30d6d2b86-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.447 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.447 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.447 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.447 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:49:25.447 DEBUG:boto:Data: 2023-09-29T22:49:25.447 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.447 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.447 DEBUG:boto:Params: {} 2023-09-29T22:49:25.447 DEBUG:boto:Token: None 2023-09-29T22:49:25.448 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115 2023-09-29T22:49:25.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s7n4L9oCWxPAgCyWp7cn4ULImdw= 2023-09-29T22:49:25.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s7n4L9oCWxPAgCyWp7cn4ULImdw=', 'Content-Length': '0'} 2023-09-29T22:49:25.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cee4053b7c9a5a67-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:49:25.450 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.450 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.450 DEBUG:boto:Method: GET 2023-09-29T22:49:25.451 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:49:25.451 DEBUG:boto:Data: 2023-09-29T22:49:25.451 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.451 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.451 DEBUG:boto:Params: {} 2023-09-29T22:49:25.451 DEBUG:boto:Token: None 2023-09-29T22:49:25.451 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115 2023-09-29T22:49:25.451 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cuAJfxsJiv4dBaJZuQx/U9z8wwA= 2023-09-29T22:49:25.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cuAJfxsJiv4dBaJZuQx/U9z8wwA=', 'Content-Length': '0'} 2023-09-29T22:49:25.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3ae97800801b49e-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.494 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:49:25.494 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:49:25.495 DEBUG:boto:Method: GET 2023-09-29T22:49:25.495 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:49:25.495 DEBUG:boto:Data: 2023-09-29T22:49:25.495 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.495 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.495 DEBUG:boto:Params: {} 2023-09-29T22:49:25.495 DEBUG:boto:Token: None 2023-09-29T22:49:25.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj115 2023-09-29T22:49:25.495 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cuAJfxsJiv4dBaJZuQx/U9z8wwA= 2023-09-29T22:49:25.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cuAJfxsJiv4dBaJZuQx/U9z8wwA=', 'Content-Length': '0'} 2023-09-29T22:49:25.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c62e5d8b036959f-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.499 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.499 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.499 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.499 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.499 DEBUG:boto:Method: GET 2023-09-29T22:49:25.499 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.499 DEBUG:boto:Data: 2023-09-29T22:49:25.500 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.500 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.500 DEBUG:boto:Params: {} 2023-09-29T22:49:25.500 DEBUG:boto:Token: None 2023-09-29T22:49:25.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd3dd1c0d6373417-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.505 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.505 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.505 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.505 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.505 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.505 DEBUG:boto:Method: GET 2023-09-29T22:49:25.506 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:49:25.506 DEBUG:boto:Data: 2023-09-29T22:49:25.506 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.506 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.506 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.506 DEBUG:boto:Params: {} 2023-09-29T22:49:25.506 DEBUG:boto:Token: None 2023-09-29T22:49:25.506 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.506 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.507 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007d230d5d1bfa7e21-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.554 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:49:25.555 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.555 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.555 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.555 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.555 DEBUG:boto:Method: GET 2023-09-29T22:49:25.555 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.555 DEBUG:boto:Data: 2023-09-29T22:49:25.555 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.556 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.556 DEBUG:boto:Params: {} 2023-09-29T22:49:25.556 DEBUG:boto:Token: None 2023-09-29T22:49:25.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg= 2023-09-29T22:49:25.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg=', 'Content-Length': '0'} 2023-09-29T22:49:25.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abce5784c070d737-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.559 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.559 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.559 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.559 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.559 DEBUG:boto:Method: GET 2023-09-29T22:49:25.560 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.560 DEBUG:boto:Data: 2023-09-29T22:49:25.560 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.560 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.560 DEBUG:boto:Params: {} 2023-09-29T22:49:25.560 DEBUG:boto:Token: None 2023-09-29T22:49:25.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg= 2023-09-29T22:49:25.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg=', 'Content-Length': '0'} 2023-09-29T22:49:25.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016f33e999b58fe0f-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.563 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.563 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.564 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.564 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.564 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.564 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.564 DEBUG:boto:Data: 2023-09-29T22:49:25.564 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.564 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.564 DEBUG:boto:Params: {} 2023-09-29T22:49:25.564 DEBUG:boto:Token: None 2023-09-29T22:49:25.565 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X8GX7spNln6CT8XXz2d/LznA7dk= 2023-09-29T22:49:25.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8GX7spNln6CT8XXz2d/LznA7dk=', 'Content-Length': '0'} 2023-09-29T22:49:25.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8448de5d8718747-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.567 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.568 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.568 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.568 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.568 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.568 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.568 DEBUG:boto:Data: 2023-09-29T22:49:25.568 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.568 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.568 DEBUG:boto:Params: {} 2023-09-29T22:49:25.569 DEBUG:boto:Token: None 2023-09-29T22:49:25.569 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X8GX7spNln6CT8XXz2d/LznA7dk= 2023-09-29T22:49:25.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8GX7spNln6CT8XXz2d/LznA7dk=', 'Content-Length': '0'} 2023-09-29T22:49:25.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098bb06815a668e23-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:49:25.572 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.572 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.572 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.572 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.572 DEBUG:boto:Method: GET 2023-09-29T22:49:25.573 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.573 DEBUG:boto:Data: 2023-09-29T22:49:25.573 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.573 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.573 DEBUG:boto:Params: {} 2023-09-29T22:49:25.573 DEBUG:boto:Token: None 2023-09-29T22:49:25.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg= 2023-09-29T22:49:25.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg=', 'Content-Length': '0'} 2023-09-29T22:49:25.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cb6999efd658688-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.622 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.622 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.623 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.623 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.623 DEBUG:boto:Method: GET 2023-09-29T22:49:25.623 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.623 DEBUG:boto:Data: 2023-09-29T22:49:25.623 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.623 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.623 DEBUG:boto:Params: {} 2023-09-29T22:49:25.623 DEBUG:boto:Token: None 2023-09-29T22:49:25.624 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:49:25.624 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg= 2023-09-29T22:49:25.624 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6j/EsAPq9tKUMuXy+CnR1NAu0Eg=', 'Content-Length': '0'} 2023-09-29T22:49:25.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983ac17d12014cd2-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.627 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.627 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.627 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.627 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.628 DEBUG:boto:Method: GET 2023-09-29T22:49:25.628 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.628 DEBUG:boto:Data: 2023-09-29T22:49:25.628 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.628 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.628 DEBUG:boto:Params: {} 2023-09-29T22:49:25.628 DEBUG:boto:Token: None 2023-09-29T22:49:25.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.629 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.633 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd36667b882ea810-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.634 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.634 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.634 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.634 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.634 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.635 DEBUG:boto:Method: GET 2023-09-29T22:49:25.635 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.635 DEBUG:boto:Data: 2023-09-29T22:49:25.635 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.635 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.635 DEBUG:boto:Params: {} 2023-09-29T22:49:25.635 DEBUG:boto:Token: None 2023-09-29T22:49:25.635 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.635 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.643 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbbdae76f51eb4df-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.686 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.687 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.687 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.687 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.687 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:49:25.687 DEBUG:boto:Data: 2023-09-29T22:49:25.687 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.687 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.687 DEBUG:boto:Params: {} 2023-09-29T22:49:25.687 DEBUG:boto:Token: None 2023-09-29T22:49:25.688 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116 2023-09-29T22:49:25.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+lwNL/o8S7XelrgBEFrObSPaQA4= 2023-09-29T22:49:25.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+lwNL/o8S7XelrgBEFrObSPaQA4=', 'Content-Length': '0'} 2023-09-29T22:49:25.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003282a71b7548178e-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.691 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.691 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.691 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.691 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:49:25.691 DEBUG:boto:Data: 2023-09-29T22:49:25.691 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.691 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.692 DEBUG:boto:Params: {} 2023-09-29T22:49:25.692 DEBUG:boto:Token: None 2023-09-29T22:49:25.692 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116 2023-09-29T22:49:25.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+lwNL/o8S7XelrgBEFrObSPaQA4= 2023-09-29T22:49:25.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+lwNL/o8S7XelrgBEFrObSPaQA4=', 'Content-Length': '0'} 2023-09-29T22:49:25.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9c19bd004b5e9f9-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:49:25.695 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.695 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.695 DEBUG:boto:Method: GET 2023-09-29T22:49:25.695 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:49:25.696 DEBUG:boto:Data: 2023-09-29T22:49:25.696 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.696 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.696 DEBUG:boto:Params: {} 2023-09-29T22:49:25.696 DEBUG:boto:Token: None 2023-09-29T22:49:25.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116 2023-09-29T22:49:25.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:55zg0y5PKd1Gzy5mNwioD7siqCA= 2023-09-29T22:49:25.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:55zg0y5PKd1Gzy5mNwioD7siqCA=', 'Content-Length': '0'} 2023-09-29T22:49:25.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d632134c8a5d25cf-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.699 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:49:25.699 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:49:25.700 DEBUG:boto:Method: GET 2023-09-29T22:49:25.700 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:49:25.700 DEBUG:boto:Data: 2023-09-29T22:49:25.700 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.700 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.700 DEBUG:boto:Params: {} 2023-09-29T22:49:25.700 DEBUG:boto:Token: None 2023-09-29T22:49:25.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj116 2023-09-29T22:49:25.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:55zg0y5PKd1Gzy5mNwioD7siqCA= 2023-09-29T22:49:25.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:55zg0y5PKd1Gzy5mNwioD7siqCA=', 'Content-Length': '0'} 2023-09-29T22:49:25.703 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3056d9c06ac4e81-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.704 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.704 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.704 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.704 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.704 DEBUG:boto:Method: GET 2023-09-29T22:49:25.704 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.704 DEBUG:boto:Data: 2023-09-29T22:49:25.704 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.705 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.705 DEBUG:boto:Params: {} 2023-09-29T22:49:25.705 DEBUG:boto:Token: None 2023-09-29T22:49:25.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007883e4d4d43ae86e-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.750 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.751 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.751 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.751 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.751 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.751 DEBUG:boto:Method: GET 2023-09-29T22:49:25.751 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:49:25.751 DEBUG:boto:Data: 2023-09-29T22:49:25.751 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.752 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.752 DEBUG:boto:Params: {} 2023-09-29T22:49:25.752 DEBUG:boto:Token: None 2023-09-29T22:49:25.752 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.752 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001bf6b04cd186bb97-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.761 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:49:25.761 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.762 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.762 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.762 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.762 DEBUG:boto:Method: GET 2023-09-29T22:49:25.762 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.762 DEBUG:boto:Data: 2023-09-29T22:49:25.762 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.762 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.762 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.762 DEBUG:boto:Params: {} 2023-09-29T22:49:25.763 DEBUG:boto:Token: None 2023-09-29T22:49:25.763 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.763 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40= 2023-09-29T22:49:25.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40=', 'Content-Length': '0'} 2023-09-29T22:49:25.765 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077183878d1e80061-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.766 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.766 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.766 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.766 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.766 DEBUG:boto:Method: GET 2023-09-29T22:49:25.766 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.766 DEBUG:boto:Data: 2023-09-29T22:49:25.767 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.767 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.767 DEBUG:boto:Params: {} 2023-09-29T22:49:25.767 DEBUG:boto:Token: None 2023-09-29T22:49:25.767 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.767 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40= 2023-09-29T22:49:25.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40=', 'Content-Length': '0'} 2023-09-29T22:49:25.769 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005653346487ddf1b1-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.810 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.811 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.811 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.811 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.811 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.811 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.811 DEBUG:boto:Data: 2023-09-29T22:49:25.811 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.811 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.812 DEBUG:boto:Params: {} 2023-09-29T22:49:25.812 DEBUG:boto:Token: None 2023-09-29T22:49:25.812 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bDSR8PV3ffxLvcmq3cGZ030dgJ8= 2023-09-29T22:49:25.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bDSR8PV3ffxLvcmq3cGZ030dgJ8=', 'Content-Length': '0'} 2023-09-29T22:49:25.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075e6736f01b0ec43-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.815 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.815 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.815 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.815 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.816 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.816 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.816 DEBUG:boto:Data: 2023-09-29T22:49:25.816 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.816 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.816 DEBUG:boto:Params: {} 2023-09-29T22:49:25.816 DEBUG:boto:Token: None 2023-09-29T22:49:25.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bDSR8PV3ffxLvcmq3cGZ030dgJ8= 2023-09-29T22:49:25.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bDSR8PV3ffxLvcmq3cGZ030dgJ8=', 'Content-Length': '0'} 2023-09-29T22:49:25.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9c95dcf3f39183d-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:49:25.820 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.820 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.820 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.820 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.820 DEBUG:boto:Method: GET 2023-09-29T22:49:25.820 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.820 DEBUG:boto:Data: 2023-09-29T22:49:25.820 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.820 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.821 DEBUG:boto:Params: {} 2023-09-29T22:49:25.821 DEBUG:boto:Token: None 2023-09-29T22:49:25.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40= 2023-09-29T22:49:25.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40=', 'Content-Length': '0'} 2023-09-29T22:49:25.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000342e7c03dee69dfa-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.824 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.824 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.824 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.825 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.825 DEBUG:boto:Method: GET 2023-09-29T22:49:25.825 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.825 DEBUG:boto:Data: 2023-09-29T22:49:25.825 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.825 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.825 DEBUG:boto:Params: {} 2023-09-29T22:49:25.825 DEBUG:boto:Token: None 2023-09-29T22:49:25.826 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:49:25.826 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40= 2023-09-29T22:49:25.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k1L/UIe9kirk8FM2VyKeYPtky40=', 'Content-Length': '0'} 2023-09-29T22:49:25.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf20694c7543fa8f-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.829 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.829 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.829 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.829 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.829 DEBUG:boto:Method: GET 2023-09-29T22:49:25.829 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.830 DEBUG:boto:Data: 2023-09-29T22:49:25.830 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.830 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.830 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.830 DEBUG:boto:Params: {} 2023-09-29T22:49:25.830 DEBUG:boto:Token: None 2023-09-29T22:49:25.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000792f6cc64c64608-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.882 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.883 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.883 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.883 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.883 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.883 DEBUG:boto:Method: GET 2023-09-29T22:49:25.883 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.883 DEBUG:boto:Data: 2023-09-29T22:49:25.884 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.884 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.884 DEBUG:boto:Params: {} 2023-09-29T22:49:25.884 DEBUG:boto:Token: None 2023-09-29T22:49:25.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.894 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292eb185cc574ea0-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.895 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.895 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.895 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.895 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.895 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:49:25.896 DEBUG:boto:Data: 2023-09-29T22:49:25.896 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.896 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.896 DEBUG:boto:Params: {} 2023-09-29T22:49:25.896 DEBUG:boto:Token: None 2023-09-29T22:49:25.896 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124 2023-09-29T22:49:25.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5qp1vqX8+dpiqSohraw+7nWRqa4= 2023-09-29T22:49:25.897 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5qp1vqX8+dpiqSohraw+7nWRqa4=', 'Content-Length': '0'} 2023-09-29T22:49:25.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d691e9c430d08ce2-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.899 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.899 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.899 DEBUG:boto:Method: HEAD 2023-09-29T22:49:25.899 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:49:25.899 DEBUG:boto:Data: 2023-09-29T22:49:25.899 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.900 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.900 DEBUG:boto:Params: {} 2023-09-29T22:49:25.900 DEBUG:boto:Token: None 2023-09-29T22:49:25.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124 2023-09-29T22:49:25.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5qp1vqX8+dpiqSohraw+7nWRqa4= 2023-09-29T22:49:25.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5qp1vqX8+dpiqSohraw+7nWRqa4=', 'Content-Length': '0'} 2023-09-29T22:49:25.902 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030a3007c785e8eac-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:49:25.903 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.903 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.903 DEBUG:boto:Method: GET 2023-09-29T22:49:25.903 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:49:25.903 DEBUG:boto:Data: 2023-09-29T22:49:25.903 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.903 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.903 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.903 DEBUG:boto:Params: {} 2023-09-29T22:49:25.904 DEBUG:boto:Token: None 2023-09-29T22:49:25.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124 2023-09-29T22:49:25.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bAZnhXNxNSJKF/XPyHgqMfxOke0= 2023-09-29T22:49:25.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bAZnhXNxNSJKF/XPyHgqMfxOke0=', 'Content-Length': '0'} 2023-09-29T22:49:25.906 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab594760b757c057-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.906 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:49:25.906 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:49:25.906 DEBUG:boto:Method: GET 2023-09-29T22:49:25.906 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:49:25.906 DEBUG:boto:Data: 2023-09-29T22:49:25.907 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.907 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.907 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.907 DEBUG:boto:Params: {} 2023-09-29T22:49:25.907 DEBUG:boto:Token: None 2023-09-29T22:49:25.907 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj124 2023-09-29T22:49:25.907 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bAZnhXNxNSJKF/XPyHgqMfxOke0= 2023-09-29T22:49:25.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bAZnhXNxNSJKF/XPyHgqMfxOke0=', 'Content-Length': '0'} 2023-09-29T22:49:25.909 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000326aca253850565a-0065175475-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.950 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.950 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.951 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.951 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.951 DEBUG:boto:Method: GET 2023-09-29T22:49:25.951 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.951 DEBUG:boto:Data: 2023-09-29T22:49:25.951 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.951 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.951 DEBUG:boto:Params: {} 2023-09-29T22:49:25.951 DEBUG:boto:Token: None 2023-09-29T22:49:25.952 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002c310698dc8ce3da-0065175475-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.957 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.957 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:25.957 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:25.958 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.958 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.958 DEBUG:boto:Method: GET 2023-09-29T22:49:25.958 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:49:25.958 DEBUG:boto:Data: 2023-09-29T22:49:25.958 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:25.958 DEBUG:boto:Port: 8000 2023-09-29T22:49:25.958 DEBUG:boto:Params: {} 2023-09-29T22:49:25.958 DEBUG:boto:Token: None 2023-09-29T22:49:25.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/ 2023-09-29T22:49:25.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM= 2023-09-29T22:49:25.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DztmkHy0xCPfLWETHLWqa7jcEAM=', 'Content-Length': '0'} 2023-09-29T22:49:25.966 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006b820bd31fa61a82-0065175475-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:25.966 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:25.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:49:25.967 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:25.967 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:25.967 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:25.967 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:25.967 DEBUG:boto:Method: GET 2023-09-29T22:49:25.967 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:25.967 DEBUG:boto:Data: 2023-09-29T22:49:25.968 DEBUG:boto:Headers: {} 2023-09-29T22:49:25.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:25.968 DEBUG:boto:Port: 8001 2023-09-29T22:49:25.968 DEBUG:boto:Params: {} 2023-09-29T22:49:25.968 DEBUG:boto:Token: None 2023-09-29T22:49:25.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:25 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:25.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X9qZbxqYSvI5ousjtGXRlpZso38= 2023-09-29T22:49:25.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X9qZbxqYSvI5ousjtGXRlpZso38=', 'Content-Length': '0'} 2023-09-29T22:49:25.970 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ddf17934b8fb89d-0065175475-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.014 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.015 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.015 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.015 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.015 DEBUG:boto:Method: GET 2023-09-29T22:49:26.015 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.015 DEBUG:boto:Data: 2023-09-29T22:49:26.015 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.016 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.016 DEBUG:boto:Params: {} 2023-09-29T22:49:26.016 DEBUG:boto:Token: None 2023-09-29T22:49:26.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b0sAGtxEmJYY8vykYZS7LHxh/Os= 2023-09-29T22:49:26.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b0sAGtxEmJYY8vykYZS7LHxh/Os=', 'Content-Length': '0'} 2023-09-29T22:49:26.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033eb6ad279b750e6-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.019 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.019 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.019 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.020 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.020 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.020 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.020 DEBUG:boto:Data: 2023-09-29T22:49:26.020 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.020 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.021 DEBUG:boto:Params: {} 2023-09-29T22:49:26.021 DEBUG:boto:Token: None 2023-09-29T22:49:26.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:opNB3WsDmpSCJRL9bwJgzpXNXro= 2023-09-29T22:49:26.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:opNB3WsDmpSCJRL9bwJgzpXNXro=', 'Content-Length': '0'} 2023-09-29T22:49:26.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e07edd775003a1ce-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.024 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.024 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.024 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.025 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.025 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.025 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.025 DEBUG:boto:Data: 2023-09-29T22:49:26.025 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.025 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.025 DEBUG:boto:Params: {} 2023-09-29T22:49:26.025 DEBUG:boto:Token: None 2023-09-29T22:49:26.026 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:opNB3WsDmpSCJRL9bwJgzpXNXro= 2023-09-29T22:49:26.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:opNB3WsDmpSCJRL9bwJgzpXNXro=', 'Content-Length': '0'} 2023-09-29T22:49:26.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f64125388501332-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:49:26.028 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.029 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.029 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.029 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.029 DEBUG:boto:Method: GET 2023-09-29T22:49:26.029 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.029 DEBUG:boto:Data: 2023-09-29T22:49:26.030 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.030 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.030 DEBUG:boto:Params: {} 2023-09-29T22:49:26.030 DEBUG:boto:Token: None 2023-09-29T22:49:26.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.030 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b0sAGtxEmJYY8vykYZS7LHxh/Os= 2023-09-29T22:49:26.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b0sAGtxEmJYY8vykYZS7LHxh/Os=', 'Content-Length': '0'} 2023-09-29T22:49:26.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559ed91409015a33-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.033 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.034 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.034 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.034 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.034 DEBUG:boto:Method: GET 2023-09-29T22:49:26.034 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.034 DEBUG:boto:Data: 2023-09-29T22:49:26.034 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.035 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.035 DEBUG:boto:Params: {} 2023-09-29T22:49:26.035 DEBUG:boto:Token: None 2023-09-29T22:49:26.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:49:26.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b0sAGtxEmJYY8vykYZS7LHxh/Os= 2023-09-29T22:49:26.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b0sAGtxEmJYY8vykYZS7LHxh/Os=', 'Content-Length': '0'} 2023-09-29T22:49:26.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aee4a6e38f8168d8-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.038 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.038 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.038 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.039 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.039 DEBUG:boto:Method: GET 2023-09-29T22:49:26.039 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.039 DEBUG:boto:Data: 2023-09-29T22:49:26.039 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.039 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.039 DEBUG:boto:Params: {} 2023-09-29T22:49:26.040 DEBUG:boto:Token: None 2023-09-29T22:49:26.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.044 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003ec02494f64395bb-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.045 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.045 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.045 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.045 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.045 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.045 DEBUG:boto:Method: GET 2023-09-29T22:49:26.046 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.046 DEBUG:boto:Data: 2023-09-29T22:49:26.046 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.046 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.046 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.046 DEBUG:boto:Params: {} 2023-09-29T22:49:26.046 DEBUG:boto:Token: None 2023-09-29T22:49:26.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.047 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.054 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f0114035b8dbb858-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.055 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.055 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.055 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.055 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.055 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:49:26.055 DEBUG:boto:Data: 2023-09-29T22:49:26.056 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.056 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.056 DEBUG:boto:Params: {} 2023-09-29T22:49:26.056 DEBUG:boto:Token: None 2023-09-29T22:49:26.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145 2023-09-29T22:49:26.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3ySYBG7e817/cT2ADRUrRGHYjgw= 2023-09-29T22:49:26.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3ySYBG7e817/cT2ADRUrRGHYjgw=', 'Content-Length': '0'} 2023-09-29T22:49:26.058 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000523d04d4ff1dbf85-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.059 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.059 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.059 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.059 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:49:26.059 DEBUG:boto:Data: 2023-09-29T22:49:26.059 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.059 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.059 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.060 DEBUG:boto:Params: {} 2023-09-29T22:49:26.060 DEBUG:boto:Token: None 2023-09-29T22:49:26.060 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145 2023-09-29T22:49:26.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3ySYBG7e817/cT2ADRUrRGHYjgw= 2023-09-29T22:49:26.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3ySYBG7e817/cT2ADRUrRGHYjgw=', 'Content-Length': '0'} 2023-09-29T22:49:26.062 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000923cd9e4e9198cbd-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:49:26.062 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.062 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.063 DEBUG:boto:Method: GET 2023-09-29T22:49:26.063 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:49:26.063 DEBUG:boto:Data: 2023-09-29T22:49:26.063 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.063 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.063 DEBUG:boto:Params: {} 2023-09-29T22:49:26.063 DEBUG:boto:Token: None 2023-09-29T22:49:26.063 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145 2023-09-29T22:49:26.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/QCrqrdHwlULkjk+SkbabN16b6Y= 2023-09-29T22:49:26.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/QCrqrdHwlULkjk+SkbabN16b6Y=', 'Content-Length': '0'} 2023-09-29T22:49:26.066 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000344da11157dd3c81-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.106 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:49:26.106 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:49:26.107 DEBUG:boto:Method: GET 2023-09-29T22:49:26.107 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:49:26.107 DEBUG:boto:Data: 2023-09-29T22:49:26.107 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.107 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.107 DEBUG:boto:Params: {} 2023-09-29T22:49:26.107 DEBUG:boto:Token: None 2023-09-29T22:49:26.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj145 2023-09-29T22:49:26.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/QCrqrdHwlULkjk+SkbabN16b6Y= 2023-09-29T22:49:26.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/QCrqrdHwlULkjk+SkbabN16b6Y=', 'Content-Length': '0'} 2023-09-29T22:49:26.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093beff4371d7b71a-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.111 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.111 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.111 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.111 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.112 DEBUG:boto:Method: GET 2023-09-29T22:49:26.112 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.112 DEBUG:boto:Data: 2023-09-29T22:49:26.112 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.112 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.112 DEBUG:boto:Params: {} 2023-09-29T22:49:26.112 DEBUG:boto:Token: None 2023-09-29T22:49:26.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001def87e2b9c6d0f9-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.119 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.119 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.120 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.120 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.120 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.120 DEBUG:boto:Method: GET 2023-09-29T22:49:26.120 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:49:26.120 DEBUG:boto:Data: 2023-09-29T22:49:26.120 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.121 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.121 DEBUG:boto:Params: {} 2023-09-29T22:49:26.121 DEBUG:boto:Token: None 2023-09-29T22:49:26.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4c0a42c33b5b57e-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.129 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:49:26.129 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.130 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.130 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.130 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.130 DEBUG:boto:Method: GET 2023-09-29T22:49:26.130 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.130 DEBUG:boto:Data: 2023-09-29T22:49:26.130 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.130 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.131 DEBUG:boto:Params: {} 2023-09-29T22:49:26.131 DEBUG:boto:Token: None 2023-09-29T22:49:26.131 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.131 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ= 2023-09-29T22:49:26.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ=', 'Content-Length': '0'} 2023-09-29T22:49:26.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013be1c4afd6ee494-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.134 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.134 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.134 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.134 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.134 DEBUG:boto:Method: GET 2023-09-29T22:49:26.134 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.134 DEBUG:boto:Data: 2023-09-29T22:49:26.135 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.135 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.135 DEBUG:boto:Params: {} 2023-09-29T22:49:26.135 DEBUG:boto:Token: None 2023-09-29T22:49:26.135 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.135 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ= 2023-09-29T22:49:26.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ=', 'Content-Length': '0'} 2023-09-29T22:49:26.137 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019a95fa496e72f05-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.138 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.138 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.138 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.138 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.138 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.139 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.139 DEBUG:boto:Data: 2023-09-29T22:49:26.139 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.139 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.139 DEBUG:boto:Params: {} 2023-09-29T22:49:26.139 DEBUG:boto:Token: None 2023-09-29T22:49:26.139 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h1U+Avnp3VjrSjJAN+Gu67WIz6s= 2023-09-29T22:49:26.140 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h1U+Avnp3VjrSjJAN+Gu67WIz6s=', 'Content-Length': '0'} 2023-09-29T22:49:26.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03bda6f7217a939-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.142 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.142 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.142 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.142 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.143 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.143 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.143 DEBUG:boto:Data: 2023-09-29T22:49:26.143 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.143 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.143 DEBUG:boto:Params: {} 2023-09-29T22:49:26.143 DEBUG:boto:Token: None 2023-09-29T22:49:26.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h1U+Avnp3VjrSjJAN+Gu67WIz6s= 2023-09-29T22:49:26.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h1U+Avnp3VjrSjJAN+Gu67WIz6s=', 'Content-Length': '0'} 2023-09-29T22:49:26.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a8ec7747d8eebe-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:49:26.147 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.147 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.147 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.147 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.147 DEBUG:boto:Method: GET 2023-09-29T22:49:26.148 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.148 DEBUG:boto:Data: 2023-09-29T22:49:26.148 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.148 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.148 DEBUG:boto:Params: {} 2023-09-29T22:49:26.148 DEBUG:boto:Token: None 2023-09-29T22:49:26.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ= 2023-09-29T22:49:26.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ=', 'Content-Length': '0'} 2023-09-29T22:49:26.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075466c0eefa8a9df-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.151 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.151 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.152 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.152 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.152 DEBUG:boto:Method: GET 2023-09-29T22:49:26.152 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.152 DEBUG:boto:Data: 2023-09-29T22:49:26.152 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.152 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.152 DEBUG:boto:Params: {} 2023-09-29T22:49:26.152 DEBUG:boto:Token: None 2023-09-29T22:49:26.153 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:49:26.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ= 2023-09-29T22:49:26.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xHGLw+ySf16qt1H067U5aXC16mQ=', 'Content-Length': '0'} 2023-09-29T22:49:26.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eabcf9748e4a7c17-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.198 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.198 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.199 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.199 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.199 DEBUG:boto:Method: GET 2023-09-29T22:49:26.199 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.199 DEBUG:boto:Data: 2023-09-29T22:49:26.199 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.199 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.199 DEBUG:boto:Params: {} 2023-09-29T22:49:26.200 DEBUG:boto:Token: None 2023-09-29T22:49:26.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e3b19881b27b29f-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.204 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.204 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.204 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.204 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.205 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.205 DEBUG:boto:Method: GET 2023-09-29T22:49:26.205 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.205 DEBUG:boto:Data: 2023-09-29T22:49:26.205 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.205 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.205 DEBUG:boto:Params: {} 2023-09-29T22:49:26.205 DEBUG:boto:Token: None 2023-09-29T22:49:26.206 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.206 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.214 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b3108ba53a4428a-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.215 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.215 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.215 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.215 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.215 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:49:26.215 DEBUG:boto:Data: 2023-09-29T22:49:26.215 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.216 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.216 DEBUG:boto:Params: {} 2023-09-29T22:49:26.216 DEBUG:boto:Token: None 2023-09-29T22:49:26.216 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148 2023-09-29T22:49:26.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CjIdIrrdjhEEdTObgppJpX7AmyU= 2023-09-29T22:49:26.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CjIdIrrdjhEEdTObgppJpX7AmyU=', 'Content-Length': '0'} 2023-09-29T22:49:26.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010974fe4399f1692-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.219 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.219 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.219 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.219 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:49:26.219 DEBUG:boto:Data: 2023-09-29T22:49:26.219 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.219 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.219 DEBUG:boto:Params: {} 2023-09-29T22:49:26.220 DEBUG:boto:Token: None 2023-09-29T22:49:26.220 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148 2023-09-29T22:49:26.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CjIdIrrdjhEEdTObgppJpX7AmyU= 2023-09-29T22:49:26.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CjIdIrrdjhEEdTObgppJpX7AmyU=', 'Content-Length': '0'} 2023-09-29T22:49:26.222 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096e95ace08ee0fe2-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:49:26.222 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.222 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.222 DEBUG:boto:Method: GET 2023-09-29T22:49:26.223 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:49:26.223 DEBUG:boto:Data: 2023-09-29T22:49:26.223 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.223 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.223 DEBUG:boto:Params: {} 2023-09-29T22:49:26.223 DEBUG:boto:Token: None 2023-09-29T22:49:26.223 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148 2023-09-29T22:49:26.223 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OaucNQ9TSraYrVRIvnsc+uqFMjM= 2023-09-29T22:49:26.224 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OaucNQ9TSraYrVRIvnsc+uqFMjM=', 'Content-Length': '0'} 2023-09-29T22:49:26.225 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066b46603005ab775-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.266 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:49:26.267 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:49:26.267 DEBUG:boto:Method: GET 2023-09-29T22:49:26.267 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:49:26.267 DEBUG:boto:Data: 2023-09-29T22:49:26.267 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.267 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.267 DEBUG:boto:Params: {} 2023-09-29T22:49:26.268 DEBUG:boto:Token: None 2023-09-29T22:49:26.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj148 2023-09-29T22:49:26.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OaucNQ9TSraYrVRIvnsc+uqFMjM= 2023-09-29T22:49:26.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OaucNQ9TSraYrVRIvnsc+uqFMjM=', 'Content-Length': '0'} 2023-09-29T22:49:26.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007661aedfb4baee6c-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.271 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.271 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.271 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.272 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.272 DEBUG:boto:Method: GET 2023-09-29T22:49:26.272 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.272 DEBUG:boto:Data: 2023-09-29T22:49:26.272 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.272 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.272 DEBUG:boto:Params: {} 2023-09-29T22:49:26.272 DEBUG:boto:Token: None 2023-09-29T22:49:26.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000090e3af50e4898093-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.277 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.277 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.278 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.278 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.278 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.278 DEBUG:boto:Method: GET 2023-09-29T22:49:26.278 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:49:26.278 DEBUG:boto:Data: 2023-09-29T22:49:26.278 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.278 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.279 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.279 DEBUG:boto:Params: {} 2023-09-29T22:49:26.279 DEBUG:boto:Token: None 2023-09-29T22:49:26.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.286 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a681c39b089d93c9-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.330 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:49:26.331 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.331 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.331 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.331 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.331 DEBUG:boto:Method: GET 2023-09-29T22:49:26.331 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.332 DEBUG:boto:Data: 2023-09-29T22:49:26.332 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.332 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.332 DEBUG:boto:Params: {} 2023-09-29T22:49:26.332 DEBUG:boto:Token: None 2023-09-29T22:49:26.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k= 2023-09-29T22:49:26.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k=', 'Content-Length': '0'} 2023-09-29T22:49:26.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008763aaee23498936-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.336 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.336 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.336 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.336 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.336 DEBUG:boto:Method: GET 2023-09-29T22:49:26.336 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.336 DEBUG:boto:Data: 2023-09-29T22:49:26.336 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.337 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.337 DEBUG:boto:Params: {} 2023-09-29T22:49:26.337 DEBUG:boto:Token: None 2023-09-29T22:49:26.337 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k= 2023-09-29T22:49:26.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k=', 'Content-Length': '0'} 2023-09-29T22:49:26.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cf02dbea4d810ac-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.340 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.340 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.340 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.341 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.341 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.341 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.341 DEBUG:boto:Data: 2023-09-29T22:49:26.341 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.341 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.341 DEBUG:boto:Params: {} 2023-09-29T22:49:26.341 DEBUG:boto:Token: None 2023-09-29T22:49:26.342 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bOEXIP1pykoBTkX4Quu+dLNS3Xs= 2023-09-29T22:49:26.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bOEXIP1pykoBTkX4Quu+dLNS3Xs=', 'Content-Length': '0'} 2023-09-29T22:49:26.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001ad6a6439923046-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.345 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.345 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.345 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.345 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.345 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.345 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.345 DEBUG:boto:Data: 2023-09-29T22:49:26.346 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.346 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.346 DEBUG:boto:Params: {} 2023-09-29T22:49:26.346 DEBUG:boto:Token: None 2023-09-29T22:49:26.346 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.346 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bOEXIP1pykoBTkX4Quu+dLNS3Xs= 2023-09-29T22:49:26.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bOEXIP1pykoBTkX4Quu+dLNS3Xs=', 'Content-Length': '0'} 2023-09-29T22:49:26.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000682acdb483a486c4-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:49:26.349 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.350 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.350 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.350 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.350 DEBUG:boto:Method: GET 2023-09-29T22:49:26.350 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.350 DEBUG:boto:Data: 2023-09-29T22:49:26.350 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.350 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.350 DEBUG:boto:Params: {} 2023-09-29T22:49:26.351 DEBUG:boto:Token: None 2023-09-29T22:49:26.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k= 2023-09-29T22:49:26.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k=', 'Content-Length': '0'} 2023-09-29T22:49:26.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec9a1ec1424d6b9d-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.354 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.354 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.354 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.354 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.355 DEBUG:boto:Method: GET 2023-09-29T22:49:26.355 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.355 DEBUG:boto:Data: 2023-09-29T22:49:26.355 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.355 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.355 DEBUG:boto:Params: {} 2023-09-29T22:49:26.355 DEBUG:boto:Token: None 2023-09-29T22:49:26.355 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:49:26.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k= 2023-09-29T22:49:26.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gJzS7fh69JJwimklcOnXgG8QD5k=', 'Content-Length': '0'} 2023-09-29T22:49:26.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1b735a27f6a5b8a-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.359 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.359 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.359 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.359 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.359 DEBUG:boto:Method: GET 2023-09-29T22:49:26.359 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.359 DEBUG:boto:Data: 2023-09-29T22:49:26.359 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.360 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.360 DEBUG:boto:Params: {} 2023-09-29T22:49:26.360 DEBUG:boto:Token: None 2023-09-29T22:49:26.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc5bb0349daf80d7-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.366 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.367 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.367 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.367 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.367 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.367 DEBUG:boto:Method: GET 2023-09-29T22:49:26.367 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.367 DEBUG:boto:Data: 2023-09-29T22:49:26.368 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.368 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.368 DEBUG:boto:Params: {} 2023-09-29T22:49:26.368 DEBUG:boto:Token: None 2023-09-29T22:49:26.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.374 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f9f582cae2ee618-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.422 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.423 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.423 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.423 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.423 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:49:26.423 DEBUG:boto:Data: 2023-09-29T22:49:26.423 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.424 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.424 DEBUG:boto:Params: {} 2023-09-29T22:49:26.424 DEBUG:boto:Token: None 2023-09-29T22:49:26.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155 2023-09-29T22:49:26.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+/7IB94xdTt3ox7dNdSemfqy9Ho= 2023-09-29T22:49:26.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+/7IB94xdTt3ox7dNdSemfqy9Ho=', 'Content-Length': '0'} 2023-09-29T22:49:26.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dab9afb03eaf64a-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.427 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.428 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.428 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.428 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:49:26.428 DEBUG:boto:Data: 2023-09-29T22:49:26.428 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.428 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.428 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.429 DEBUG:boto:Params: {} 2023-09-29T22:49:26.429 DEBUG:boto:Token: None 2023-09-29T22:49:26.429 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155 2023-09-29T22:49:26.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+/7IB94xdTt3ox7dNdSemfqy9Ho= 2023-09-29T22:49:26.429 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+/7IB94xdTt3ox7dNdSemfqy9Ho=', 'Content-Length': '0'} 2023-09-29T22:49:26.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e828d1e87f3a66fa-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:49:26.432 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.432 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.432 DEBUG:boto:Method: GET 2023-09-29T22:49:26.433 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:49:26.433 DEBUG:boto:Data: 2023-09-29T22:49:26.433 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.433 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.433 DEBUG:boto:Params: {} 2023-09-29T22:49:26.433 DEBUG:boto:Token: None 2023-09-29T22:49:26.433 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155 2023-09-29T22:49:26.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CXDpodOKTdG+kWTVSsaHsB7N/NQ= 2023-09-29T22:49:26.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CXDpodOKTdG+kWTVSsaHsB7N/NQ=', 'Content-Length': '0'} 2023-09-29T22:49:26.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003914c62c216acc36-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.482 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:49:26.482 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:49:26.483 DEBUG:boto:Method: GET 2023-09-29T22:49:26.483 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:49:26.483 DEBUG:boto:Data: 2023-09-29T22:49:26.483 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.483 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.483 DEBUG:boto:Params: {} 2023-09-29T22:49:26.483 DEBUG:boto:Token: None 2023-09-29T22:49:26.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj155 2023-09-29T22:49:26.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CXDpodOKTdG+kWTVSsaHsB7N/NQ= 2023-09-29T22:49:26.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CXDpodOKTdG+kWTVSsaHsB7N/NQ=', 'Content-Length': '0'} 2023-09-29T22:49:26.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000397f87c4e158d2d-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.487 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.487 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.487 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.487 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.487 DEBUG:boto:Method: GET 2023-09-29T22:49:26.487 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.487 DEBUG:boto:Data: 2023-09-29T22:49:26.488 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.488 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.488 DEBUG:boto:Params: {} 2023-09-29T22:49:26.488 DEBUG:boto:Token: None 2023-09-29T22:49:26.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa4067a84bad0309-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.494 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.494 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.494 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.494 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.494 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.494 DEBUG:boto:Method: GET 2023-09-29T22:49:26.495 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:49:26.495 DEBUG:boto:Data: 2023-09-29T22:49:26.495 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.495 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.495 DEBUG:boto:Params: {} 2023-09-29T22:49:26.495 DEBUG:boto:Token: None 2023-09-29T22:49:26.495 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.495 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.504 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ffbb623ecf747ba9-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.504 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:49:26.504 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.505 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.505 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.505 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.505 DEBUG:boto:Method: GET 2023-09-29T22:49:26.505 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.505 DEBUG:boto:Data: 2023-09-29T22:49:26.505 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.505 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.505 DEBUG:boto:Params: {} 2023-09-29T22:49:26.506 DEBUG:boto:Token: None 2023-09-29T22:49:26.506 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.506 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8= 2023-09-29T22:49:26.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8=', 'Content-Length': '0'} 2023-09-29T22:49:26.508 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e3abaab6a817162-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.509 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.509 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.509 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.509 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.509 DEBUG:boto:Method: GET 2023-09-29T22:49:26.510 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.510 DEBUG:boto:Data: 2023-09-29T22:49:26.510 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.510 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.510 DEBUG:boto:Params: {} 2023-09-29T22:49:26.511 DEBUG:boto:Token: None 2023-09-29T22:49:26.511 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.511 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8= 2023-09-29T22:49:26.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8=', 'Content-Length': '0'} 2023-09-29T22:49:26.513 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e151b0e16f23fd4a-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.557 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.557 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.557 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.557 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.558 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.558 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.558 DEBUG:boto:Data: 2023-09-29T22:49:26.558 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.558 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.558 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.558 DEBUG:boto:Params: {} 2023-09-29T22:49:26.558 DEBUG:boto:Token: None 2023-09-29T22:49:26.559 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.559 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ArRWsH6hPdjB5qpCxiq62U9sefo= 2023-09-29T22:49:26.559 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ArRWsH6hPdjB5qpCxiq62U9sefo=', 'Content-Length': '0'} 2023-09-29T22:49:26.561 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005266d69bf18d3836-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.562 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.562 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.562 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.562 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.562 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.562 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.562 DEBUG:boto:Data: 2023-09-29T22:49:26.563 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.563 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.563 DEBUG:boto:Params: {} 2023-09-29T22:49:26.563 DEBUG:boto:Token: None 2023-09-29T22:49:26.563 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.563 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ArRWsH6hPdjB5qpCxiq62U9sefo= 2023-09-29T22:49:26.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ArRWsH6hPdjB5qpCxiq62U9sefo=', 'Content-Length': '0'} 2023-09-29T22:49:26.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b21c5e000072d01-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:49:26.567 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.567 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.567 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.567 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.567 DEBUG:boto:Method: GET 2023-09-29T22:49:26.567 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.567 DEBUG:boto:Data: 2023-09-29T22:49:26.567 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.568 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.568 DEBUG:boto:Params: {} 2023-09-29T22:49:26.568 DEBUG:boto:Token: None 2023-09-29T22:49:26.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8= 2023-09-29T22:49:26.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8=', 'Content-Length': '0'} 2023-09-29T22:49:26.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3021966119df269-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.571 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.571 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.571 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.571 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.572 DEBUG:boto:Method: GET 2023-09-29T22:49:26.572 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.572 DEBUG:boto:Data: 2023-09-29T22:49:26.572 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.572 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.572 DEBUG:boto:Params: {} 2023-09-29T22:49:26.572 DEBUG:boto:Token: None 2023-09-29T22:49:26.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:49:26.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8= 2023-09-29T22:49:26.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ozWgS4q3sHdcZ5HRy5gIBASiTI8=', 'Content-Length': '0'} 2023-09-29T22:49:26.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eda85a3cd8db128b-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.576 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.576 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.576 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.576 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.576 DEBUG:boto:Method: GET 2023-09-29T22:49:26.576 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.577 DEBUG:boto:Data: 2023-09-29T22:49:26.577 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.577 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.577 DEBUG:boto:Params: {} 2023-09-29T22:49:26.577 DEBUG:boto:Token: None 2023-09-29T22:49:26.577 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.582 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000009a8d1cc2e285486-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.626 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.627 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.627 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.627 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.627 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.627 DEBUG:boto:Method: GET 2023-09-29T22:49:26.627 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.628 DEBUG:boto:Data: 2023-09-29T22:49:26.628 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.628 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.628 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.628 DEBUG:boto:Params: {} 2023-09-29T22:49:26.628 DEBUG:boto:Token: None 2023-09-29T22:49:26.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e9eae01bce969c85-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.638 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.638 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.638 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.639 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.639 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:49:26.639 DEBUG:boto:Data: 2023-09-29T22:49:26.639 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.639 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.639 DEBUG:boto:Params: {} 2023-09-29T22:49:26.639 DEBUG:boto:Token: None 2023-09-29T22:49:26.639 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17 2023-09-29T22:49:26.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXjRqd2a5PAm1M0dPBBqJw6q1Lk= 2023-09-29T22:49:26.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXjRqd2a5PAm1M0dPBBqJw6q1Lk=', 'Content-Length': '0'} 2023-09-29T22:49:26.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7a60727ebab494f-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.642 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.642 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.642 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.642 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:49:26.642 DEBUG:boto:Data: 2023-09-29T22:49:26.643 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.643 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.643 DEBUG:boto:Params: {} 2023-09-29T22:49:26.643 DEBUG:boto:Token: None 2023-09-29T22:49:26.643 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17 2023-09-29T22:49:26.643 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXjRqd2a5PAm1M0dPBBqJw6q1Lk= 2023-09-29T22:49:26.643 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXjRqd2a5PAm1M0dPBBqJw6q1Lk=', 'Content-Length': '0'} 2023-09-29T22:49:26.645 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c94c9e5c6552894-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:49:26.646 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.646 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.646 DEBUG:boto:Method: GET 2023-09-29T22:49:26.646 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:49:26.646 DEBUG:boto:Data: 2023-09-29T22:49:26.646 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.646 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.646 DEBUG:boto:Params: {} 2023-09-29T22:49:26.647 DEBUG:boto:Token: None 2023-09-29T22:49:26.647 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17 2023-09-29T22:49:26.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K70D2fmSoqX+QWGautwY07UNLyY= 2023-09-29T22:49:26.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K70D2fmSoqX+QWGautwY07UNLyY=', 'Content-Length': '0'} 2023-09-29T22:49:26.649 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ece7cfb79a50b6ae-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.649 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:49:26.649 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:49:26.649 DEBUG:boto:Method: GET 2023-09-29T22:49:26.649 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:49:26.650 DEBUG:boto:Data: 2023-09-29T22:49:26.650 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.650 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.650 DEBUG:boto:Params: {} 2023-09-29T22:49:26.650 DEBUG:boto:Token: None 2023-09-29T22:49:26.650 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj17 2023-09-29T22:49:26.650 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K70D2fmSoqX+QWGautwY07UNLyY= 2023-09-29T22:49:26.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K70D2fmSoqX+QWGautwY07UNLyY=', 'Content-Length': '0'} 2023-09-29T22:49:26.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df913f1df34c0e7b-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.694 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.695 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.695 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.695 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.695 DEBUG:boto:Method: GET 2023-09-29T22:49:26.695 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.695 DEBUG:boto:Data: 2023-09-29T22:49:26.695 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.696 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.696 DEBUG:boto:Params: {} 2023-09-29T22:49:26.696 DEBUG:boto:Token: None 2023-09-29T22:49:26.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.701 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000c7f07f71dcc00dd-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.701 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.702 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.702 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.702 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.702 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.702 DEBUG:boto:Method: GET 2023-09-29T22:49:26.702 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:49:26.702 DEBUG:boto:Data: 2023-09-29T22:49:26.702 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.703 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.703 DEBUG:boto:Params: {} 2023-09-29T22:49:26.703 DEBUG:boto:Token: None 2023-09-29T22:49:26.703 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.703 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.712 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007a0e9d6142c42820-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.712 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:49:26.712 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.713 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.713 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.713 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.713 DEBUG:boto:Method: GET 2023-09-29T22:49:26.713 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.713 DEBUG:boto:Data: 2023-09-29T22:49:26.713 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.713 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.713 DEBUG:boto:Params: {} 2023-09-29T22:49:26.714 DEBUG:boto:Token: None 2023-09-29T22:49:26.714 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.714 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg= 2023-09-29T22:49:26.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg=', 'Content-Length': '0'} 2023-09-29T22:49:26.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc71052eee55eec7-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.717 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.717 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.717 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.717 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.717 DEBUG:boto:Method: GET 2023-09-29T22:49:26.717 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.717 DEBUG:boto:Data: 2023-09-29T22:49:26.717 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.718 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.718 DEBUG:boto:Params: {} 2023-09-29T22:49:26.718 DEBUG:boto:Token: None 2023-09-29T22:49:26.718 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg= 2023-09-29T22:49:26.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg=', 'Content-Length': '0'} 2023-09-29T22:49:26.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001781f9c5ce62deff-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.720 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.721 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.721 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.721 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.721 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.721 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.721 DEBUG:boto:Data: 2023-09-29T22:49:26.721 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.722 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.722 DEBUG:boto:Params: {} 2023-09-29T22:49:26.722 DEBUG:boto:Token: None 2023-09-29T22:49:26.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fOS6kx46wnR2YCoi39T8DU62dyU= 2023-09-29T22:49:26.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fOS6kx46wnR2YCoi39T8DU62dyU=', 'Content-Length': '0'} 2023-09-29T22:49:26.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020ae7e75bd275348-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.724 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.724 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.724 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.725 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.725 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.725 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.725 DEBUG:boto:Data: 2023-09-29T22:49:26.725 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.725 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.725 DEBUG:boto:Params: {} 2023-09-29T22:49:26.725 DEBUG:boto:Token: None 2023-09-29T22:49:26.725 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fOS6kx46wnR2YCoi39T8DU62dyU= 2023-09-29T22:49:26.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fOS6kx46wnR2YCoi39T8DU62dyU=', 'Content-Length': '0'} 2023-09-29T22:49:26.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7dda200affa7324-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:49:26.728 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.729 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.729 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.729 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.729 DEBUG:boto:Method: GET 2023-09-29T22:49:26.729 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.729 DEBUG:boto:Data: 2023-09-29T22:49:26.729 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.729 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.729 DEBUG:boto:Params: {} 2023-09-29T22:49:26.730 DEBUG:boto:Token: None 2023-09-29T22:49:26.730 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg= 2023-09-29T22:49:26.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg=', 'Content-Length': '0'} 2023-09-29T22:49:26.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5281739217c4c9b-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.732 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.732 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.733 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.733 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.733 DEBUG:boto:Method: GET 2023-09-29T22:49:26.733 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.733 DEBUG:boto:Data: 2023-09-29T22:49:26.733 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.733 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.733 DEBUG:boto:Params: {} 2023-09-29T22:49:26.734 DEBUG:boto:Token: None 2023-09-29T22:49:26.734 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:49:26.734 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg= 2023-09-29T22:49:26.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DjYTByT6c97xE42E8KBGvz7QxOg=', 'Content-Length': '0'} 2023-09-29T22:49:26.736 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000988b85b97dc4fc22-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.737 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.737 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.737 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.737 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.737 DEBUG:boto:Method: GET 2023-09-29T22:49:26.737 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.737 DEBUG:boto:Data: 2023-09-29T22:49:26.738 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.738 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.738 DEBUG:boto:Params: {} 2023-09-29T22:49:26.738 DEBUG:boto:Token: None 2023-09-29T22:49:26.738 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.738 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2f219e1fd3ddc40-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.742 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.742 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.742 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.742 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.742 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.743 DEBUG:boto:Method: GET 2023-09-29T22:49:26.743 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.743 DEBUG:boto:Data: 2023-09-29T22:49:26.743 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.743 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.743 DEBUG:boto:Params: {} 2023-09-29T22:49:26.743 DEBUG:boto:Token: None 2023-09-29T22:49:26.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000667c6ea14c960ee7-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.751 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.751 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.751 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.752 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.752 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:49:26.752 DEBUG:boto:Data: 2023-09-29T22:49:26.752 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.752 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.752 DEBUG:boto:Params: {} 2023-09-29T22:49:26.752 DEBUG:boto:Token: None 2023-09-29T22:49:26.752 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185 2023-09-29T22:49:26.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jl6uMurlbe02WCYCsd1tPvYESCc= 2023-09-29T22:49:26.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jl6uMurlbe02WCYCsd1tPvYESCc=', 'Content-Length': '0'} 2023-09-29T22:49:26.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd42c86b01e012f6-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.755 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.755 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.755 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.755 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:49:26.755 DEBUG:boto:Data: 2023-09-29T22:49:26.755 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.755 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.755 DEBUG:boto:Params: {} 2023-09-29T22:49:26.756 DEBUG:boto:Token: None 2023-09-29T22:49:26.756 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185 2023-09-29T22:49:26.756 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jl6uMurlbe02WCYCsd1tPvYESCc= 2023-09-29T22:49:26.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jl6uMurlbe02WCYCsd1tPvYESCc=', 'Content-Length': '0'} 2023-09-29T22:49:26.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006576108d3e9aa012-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:49:26.758 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.758 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.758 DEBUG:boto:Method: GET 2023-09-29T22:49:26.758 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:49:26.758 DEBUG:boto:Data: 2023-09-29T22:49:26.759 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.759 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.759 DEBUG:boto:Params: {} 2023-09-29T22:49:26.759 DEBUG:boto:Token: None 2023-09-29T22:49:26.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185 2023-09-29T22:49:26.759 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z6VZzM/HBh59d4kt6gaVVJnNGpI= 2023-09-29T22:49:26.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z6VZzM/HBh59d4kt6gaVVJnNGpI=', 'Content-Length': '0'} 2023-09-29T22:49:26.761 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7cb433d64312ed-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.802 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:49:26.802 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:49:26.803 DEBUG:boto:Method: GET 2023-09-29T22:49:26.803 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:49:26.803 DEBUG:boto:Data: 2023-09-29T22:49:26.803 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.803 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.803 DEBUG:boto:Params: {} 2023-09-29T22:49:26.803 DEBUG:boto:Token: None 2023-09-29T22:49:26.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj185 2023-09-29T22:49:26.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z6VZzM/HBh59d4kt6gaVVJnNGpI= 2023-09-29T22:49:26.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z6VZzM/HBh59d4kt6gaVVJnNGpI=', 'Content-Length': '0'} 2023-09-29T22:49:26.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d29de2b5ad3510-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.807 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.807 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.807 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.807 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.807 DEBUG:boto:Method: GET 2023-09-29T22:49:26.807 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.807 DEBUG:boto:Data: 2023-09-29T22:49:26.807 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.808 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.808 DEBUG:boto:Params: {} 2023-09-29T22:49:26.808 DEBUG:boto:Token: None 2023-09-29T22:49:26.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.808 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.808 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.812 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000481d08f533fb4662-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.813 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.813 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.813 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.813 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.813 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.814 DEBUG:boto:Method: GET 2023-09-29T22:49:26.814 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:49:26.814 DEBUG:boto:Data: 2023-09-29T22:49:26.814 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.814 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.814 DEBUG:boto:Params: {} 2023-09-29T22:49:26.814 DEBUG:boto:Token: None 2023-09-29T22:49:26.814 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.815 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa75949a910bd733-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.823 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:49:26.823 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.824 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.824 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.824 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.824 DEBUG:boto:Method: GET 2023-09-29T22:49:26.824 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.824 DEBUG:boto:Data: 2023-09-29T22:49:26.824 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.824 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.824 DEBUG:boto:Params: {} 2023-09-29T22:49:26.825 DEBUG:boto:Token: None 2023-09-29T22:49:26.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0= 2023-09-29T22:49:26.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0=', 'Content-Length': '0'} 2023-09-29T22:49:26.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f63bdd204f795b9a-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.827 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.827 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.827 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.828 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.828 DEBUG:boto:Method: GET 2023-09-29T22:49:26.828 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.828 DEBUG:boto:Data: 2023-09-29T22:49:26.828 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.828 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.828 DEBUG:boto:Params: {} 2023-09-29T22:49:26.828 DEBUG:boto:Token: None 2023-09-29T22:49:26.828 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0= 2023-09-29T22:49:26.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0=', 'Content-Length': '0'} 2023-09-29T22:49:26.831 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc3913a85d925a96-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.831 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.831 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.831 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.831 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.832 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.832 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.832 DEBUG:boto:Data: 2023-09-29T22:49:26.832 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.832 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.832 DEBUG:boto:Params: {} 2023-09-29T22:49:26.832 DEBUG:boto:Token: None 2023-09-29T22:49:26.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3cCpRZ2LuT1mjkqVyk4Pj85AT7E= 2023-09-29T22:49:26.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3cCpRZ2LuT1mjkqVyk4Pj85AT7E=', 'Content-Length': '0'} 2023-09-29T22:49:26.835 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b2f1953b59514e1-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.835 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.835 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.835 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.835 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.835 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.835 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.836 DEBUG:boto:Data: 2023-09-29T22:49:26.836 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.836 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.836 DEBUG:boto:Params: {} 2023-09-29T22:49:26.836 DEBUG:boto:Token: None 2023-09-29T22:49:26.836 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3cCpRZ2LuT1mjkqVyk4Pj85AT7E= 2023-09-29T22:49:26.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3cCpRZ2LuT1mjkqVyk4Pj85AT7E=', 'Content-Length': '0'} 2023-09-29T22:49:26.838 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ec1adc9e026699-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:49:26.839 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.839 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.839 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.839 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.839 DEBUG:boto:Method: GET 2023-09-29T22:49:26.839 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.839 DEBUG:boto:Data: 2023-09-29T22:49:26.839 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.840 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.840 DEBUG:boto:Params: {} 2023-09-29T22:49:26.840 DEBUG:boto:Token: None 2023-09-29T22:49:26.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0= 2023-09-29T22:49:26.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0=', 'Content-Length': '0'} 2023-09-29T22:49:26.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000464fce6129ae99c1-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.886 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.887 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.887 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.887 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.887 DEBUG:boto:Method: GET 2023-09-29T22:49:26.887 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.887 DEBUG:boto:Data: 2023-09-29T22:49:26.887 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.888 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.888 DEBUG:boto:Params: {} 2023-09-29T22:49:26.888 DEBUG:boto:Token: None 2023-09-29T22:49:26.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:49:26.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0= 2023-09-29T22:49:26.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:91wJkq30RueNfKUG5bewQ97fZX0=', 'Content-Length': '0'} 2023-09-29T22:49:26.891 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000093a9c22f0d729197-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.892 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.892 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.892 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.892 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.892 DEBUG:boto:Method: GET 2023-09-29T22:49:26.892 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.892 DEBUG:boto:Data: 2023-09-29T22:49:26.893 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.893 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.893 DEBUG:boto:Params: {} 2023-09-29T22:49:26.893 DEBUG:boto:Token: None 2023-09-29T22:49:26.893 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.898 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001508135eb63f8daf-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.898 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.899 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.899 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.899 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.899 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.899 DEBUG:boto:Method: GET 2023-09-29T22:49:26.899 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.899 DEBUG:boto:Data: 2023-09-29T22:49:26.899 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.900 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.900 DEBUG:boto:Params: {} 2023-09-29T22:49:26.900 DEBUG:boto:Token: None 2023-09-29T22:49:26.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.908 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e0053f2870d9717b-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.908 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.908 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.908 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.909 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.909 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:49:26.909 DEBUG:boto:Data: 2023-09-29T22:49:26.909 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.909 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.909 DEBUG:boto:Params: {} 2023-09-29T22:49:26.909 DEBUG:boto:Token: None 2023-09-29T22:49:26.909 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19 2023-09-29T22:49:26.909 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2e8adOzDeWoP7IfLd0lVddlRofA= 2023-09-29T22:49:26.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2e8adOzDeWoP7IfLd0lVddlRofA=', 'Content-Length': '0'} 2023-09-29T22:49:26.911 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa58ad521e55cb9f-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.912 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.912 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.912 DEBUG:boto:Method: HEAD 2023-09-29T22:49:26.912 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:49:26.912 DEBUG:boto:Data: 2023-09-29T22:49:26.912 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.912 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.913 DEBUG:boto:Params: {} 2023-09-29T22:49:26.913 DEBUG:boto:Token: None 2023-09-29T22:49:26.913 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19 2023-09-29T22:49:26.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2e8adOzDeWoP7IfLd0lVddlRofA= 2023-09-29T22:49:26.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2e8adOzDeWoP7IfLd0lVddlRofA=', 'Content-Length': '0'} 2023-09-29T22:49:26.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000354cc16b2f2aa58b-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:49:26.915 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.916 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.916 DEBUG:boto:Method: GET 2023-09-29T22:49:26.916 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:49:26.916 DEBUG:boto:Data: 2023-09-29T22:49:26.916 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.916 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.916 DEBUG:boto:Params: {} 2023-09-29T22:49:26.916 DEBUG:boto:Token: None 2023-09-29T22:49:26.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19 2023-09-29T22:49:26.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zsTtxOENtm72OW2P9nxkdNjDuXM= 2023-09-29T22:49:26.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zsTtxOENtm72OW2P9nxkdNjDuXM=', 'Content-Length': '0'} 2023-09-29T22:49:26.919 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e29f1c366d718432-0065175476-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.962 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:49:26.962 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:49:26.963 DEBUG:boto:Method: GET 2023-09-29T22:49:26.963 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:49:26.963 DEBUG:boto:Data: 2023-09-29T22:49:26.963 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.963 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.963 DEBUG:boto:Params: {} 2023-09-29T22:49:26.963 DEBUG:boto:Token: None 2023-09-29T22:49:26.963 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/obj19 2023-09-29T22:49:26.963 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zsTtxOENtm72OW2P9nxkdNjDuXM= 2023-09-29T22:49:26.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zsTtxOENtm72OW2P9nxkdNjDuXM=', 'Content-Length': '0'} 2023-09-29T22:49:26.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088408a7ae9874fdb-0065175476-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.967 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.967 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.967 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.967 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.967 DEBUG:boto:Method: GET 2023-09-29T22:49:26.967 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.967 DEBUG:boto:Data: 2023-09-29T22:49:26.967 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:26.968 DEBUG:boto:Port: 8001 2023-09-29T22:49:26.968 DEBUG:boto:Params: {} 2023-09-29T22:49:26.968 DEBUG:boto:Token: None 2023-09-29T22:49:26.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292ed2915d4a0019-0065175476-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:26.973 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:26.973 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:26.974 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:26.974 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.974 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.974 DEBUG:boto:Method: GET 2023-09-29T22:49:26.974 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:49:26.974 DEBUG:boto:Data: 2023-09-29T22:49:26.974 DEBUG:boto:Headers: {} 2023-09-29T22:49:26.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:26.974 DEBUG:boto:Port: 8000 2023-09-29T22:49:26.974 DEBUG:boto:Params: {} 2023-09-29T22:49:26.975 DEBUG:boto:Token: None 2023-09-29T22:49:26.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:26 GMT /hcnphz-59/ 2023-09-29T22:49:26.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI= 2023-09-29T22:49:26.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TMajT7329n4SThtKW7hq4UdIzvI=', 'Content-Length': '0'} 2023-09-29T22:49:26.982 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b296b2b2bd04051e-0065175476-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.022 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:49:27.023 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.023 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.023 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.023 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.023 DEBUG:boto:Method: GET 2023-09-29T22:49:27.023 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.023 DEBUG:boto:Data: 2023-09-29T22:49:27.024 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.024 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.024 DEBUG:boto:Params: {} 2023-09-29T22:49:27.024 DEBUG:boto:Token: None 2023-09-29T22:49:27.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274= 2023-09-29T22:49:27.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274=', 'Content-Length': '0'} 2023-09-29T22:49:27.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2fd0990390d80a6-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.027 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.028 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.028 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.028 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.028 DEBUG:boto:Method: GET 2023-09-29T22:49:27.028 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.028 DEBUG:boto:Data: 2023-09-29T22:49:27.028 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.028 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.029 DEBUG:boto:Params: {} 2023-09-29T22:49:27.029 DEBUG:boto:Token: None 2023-09-29T22:49:27.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274= 2023-09-29T22:49:27.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274=', 'Content-Length': '0'} 2023-09-29T22:49:27.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bba535cfc84dcbca-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.032 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.032 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.032 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.032 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.032 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.032 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.033 DEBUG:boto:Data: 2023-09-29T22:49:27.033 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.033 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.033 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.033 DEBUG:boto:Params: {} 2023-09-29T22:49:27.033 DEBUG:boto:Token: None 2023-09-29T22:49:27.033 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.033 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dgGDD0bTFiURbxA4dS1iC+5POCg= 2023-09-29T22:49:27.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dgGDD0bTFiURbxA4dS1iC+5POCg=', 'Content-Length': '0'} 2023-09-29T22:49:27.036 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f560bfdf71f7b6-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.036 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.036 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.036 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.037 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.037 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.037 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.037 DEBUG:boto:Data: 2023-09-29T22:49:27.037 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.037 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.037 DEBUG:boto:Params: {} 2023-09-29T22:49:27.037 DEBUG:boto:Token: None 2023-09-29T22:49:27.038 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.038 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dgGDD0bTFiURbxA4dS1iC+5POCg= 2023-09-29T22:49:27.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dgGDD0bTFiURbxA4dS1iC+5POCg=', 'Content-Length': '0'} 2023-09-29T22:49:27.040 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067bba2568050e326-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:49:27.041 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.041 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.041 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.041 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.041 DEBUG:boto:Method: GET 2023-09-29T22:49:27.041 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.041 DEBUG:boto:Data: 2023-09-29T22:49:27.042 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.042 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.042 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.042 DEBUG:boto:Params: {} 2023-09-29T22:49:27.042 DEBUG:boto:Token: None 2023-09-29T22:49:27.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.042 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274= 2023-09-29T22:49:27.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274=', 'Content-Length': '0'} 2023-09-29T22:49:27.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000748c1c46e2fa4150-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.086 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.086 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.087 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.087 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.087 DEBUG:boto:Method: GET 2023-09-29T22:49:27.087 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.087 DEBUG:boto:Data: 2023-09-29T22:49:27.087 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.087 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.087 DEBUG:boto:Params: {} 2023-09-29T22:49:27.087 DEBUG:boto:Token: None 2023-09-29T22:49:27.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:49:27.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274= 2023-09-29T22:49:27.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BQmCYzklpi8ipmNi14PdLTUi274=', 'Content-Length': '0'} 2023-09-29T22:49:27.090 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000876b731c20f8f08a-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.091 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.091 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.091 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.091 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.091 DEBUG:boto:Method: GET 2023-09-29T22:49:27.091 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.091 DEBUG:boto:Data: 2023-09-29T22:49:27.091 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.092 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.092 DEBUG:boto:Params: {} 2023-09-29T22:49:27.092 DEBUG:boto:Token: None 2023-09-29T22:49:27.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000333697a33d613797-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.097 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.097 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.097 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.097 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.098 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.098 DEBUG:boto:Method: GET 2023-09-29T22:49:27.098 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.098 DEBUG:boto:Data: 2023-09-29T22:49:27.098 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.098 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.098 DEBUG:boto:Params: {} 2023-09-29T22:49:27.098 DEBUG:boto:Token: None 2023-09-29T22:49:27.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.099 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e60b89cc6ac241-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.146 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.147 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.147 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.147 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.147 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:49:27.147 DEBUG:boto:Data: 2023-09-29T22:49:27.147 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.148 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.148 DEBUG:boto:Params: {} 2023-09-29T22:49:27.148 DEBUG:boto:Token: None 2023-09-29T22:49:27.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22 2023-09-29T22:49:27.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KoTNUN2aT7HyWfznYwUDjGutnAE= 2023-09-29T22:49:27.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KoTNUN2aT7HyWfznYwUDjGutnAE=', 'Content-Length': '0'} 2023-09-29T22:49:27.151 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b178ce711e9242e4-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.151 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.151 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.151 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.151 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:49:27.151 DEBUG:boto:Data: 2023-09-29T22:49:27.152 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.152 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.152 DEBUG:boto:Params: {} 2023-09-29T22:49:27.152 DEBUG:boto:Token: None 2023-09-29T22:49:27.152 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22 2023-09-29T22:49:27.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KoTNUN2aT7HyWfznYwUDjGutnAE= 2023-09-29T22:49:27.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KoTNUN2aT7HyWfznYwUDjGutnAE=', 'Content-Length': '0'} 2023-09-29T22:49:27.155 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dffe45ef5d4d3bfa-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:49:27.155 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.156 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.156 DEBUG:boto:Method: GET 2023-09-29T22:49:27.156 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:49:27.156 DEBUG:boto:Data: 2023-09-29T22:49:27.156 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.156 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.156 DEBUG:boto:Params: {} 2023-09-29T22:49:27.156 DEBUG:boto:Token: None 2023-09-29T22:49:27.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22 2023-09-29T22:49:27.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6eFKrjEhRwoXOnFODcDlcKiydqI= 2023-09-29T22:49:27.157 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6eFKrjEhRwoXOnFODcDlcKiydqI=', 'Content-Length': '0'} 2023-09-29T22:49:27.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad4df9b9ff571cb8-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.160 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:49:27.160 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:49:27.160 DEBUG:boto:Method: GET 2023-09-29T22:49:27.160 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:49:27.160 DEBUG:boto:Data: 2023-09-29T22:49:27.160 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.161 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.161 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.161 DEBUG:boto:Params: {} 2023-09-29T22:49:27.161 DEBUG:boto:Token: None 2023-09-29T22:49:27.161 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj22 2023-09-29T22:49:27.161 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6eFKrjEhRwoXOnFODcDlcKiydqI= 2023-09-29T22:49:27.161 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6eFKrjEhRwoXOnFODcDlcKiydqI=', 'Content-Length': '0'} 2023-09-29T22:49:27.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b266c9582ad47ce-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.164 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.164 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.164 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.165 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.165 DEBUG:boto:Method: GET 2023-09-29T22:49:27.165 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.165 DEBUG:boto:Data: 2023-09-29T22:49:27.165 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.165 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.165 DEBUG:boto:Params: {} 2023-09-29T22:49:27.165 DEBUG:boto:Token: None 2023-09-29T22:49:27.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.170 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b99652331186a74b-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.214 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.215 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.215 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.215 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.215 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.215 DEBUG:boto:Method: GET 2023-09-29T22:49:27.216 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:49:27.216 DEBUG:boto:Data: 2023-09-29T22:49:27.216 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.216 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.216 DEBUG:boto:Params: {} 2023-09-29T22:49:27.216 DEBUG:boto:Token: None 2023-09-29T22:49:27.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009f9450e506474212-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.226 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:49:27.227 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.227 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.227 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.227 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.227 DEBUG:boto:Method: GET 2023-09-29T22:49:27.227 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.227 DEBUG:boto:Data: 2023-09-29T22:49:27.227 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.227 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.228 DEBUG:boto:Params: {} 2023-09-29T22:49:27.228 DEBUG:boto:Token: None 2023-09-29T22:49:27.228 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.228 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY= 2023-09-29T22:49:27.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY=', 'Content-Length': '0'} 2023-09-29T22:49:27.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adee89a35765335c-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.231 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.231 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.231 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.231 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.232 DEBUG:boto:Method: GET 2023-09-29T22:49:27.232 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.232 DEBUG:boto:Data: 2023-09-29T22:49:27.232 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.232 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.232 DEBUG:boto:Params: {} 2023-09-29T22:49:27.232 DEBUG:boto:Token: None 2023-09-29T22:49:27.232 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY= 2023-09-29T22:49:27.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY=', 'Content-Length': '0'} 2023-09-29T22:49:27.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5501780c53149fe-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.278 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.279 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.279 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.279 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.279 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.279 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.279 DEBUG:boto:Data: 2023-09-29T22:49:27.279 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.280 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.280 DEBUG:boto:Params: {} 2023-09-29T22:49:27.280 DEBUG:boto:Token: None 2023-09-29T22:49:27.280 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+xC8vSharujkequWv5S8Jy8c9Zw= 2023-09-29T22:49:27.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+xC8vSharujkequWv5S8Jy8c9Zw=', 'Content-Length': '0'} 2023-09-29T22:49:27.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030189433faf227a9-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.283 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.283 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.283 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.283 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.284 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.284 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.284 DEBUG:boto:Data: 2023-09-29T22:49:27.284 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.284 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.284 DEBUG:boto:Params: {} 2023-09-29T22:49:27.284 DEBUG:boto:Token: None 2023-09-29T22:49:27.284 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+xC8vSharujkequWv5S8Jy8c9Zw= 2023-09-29T22:49:27.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+xC8vSharujkequWv5S8Jy8c9Zw=', 'Content-Length': '0'} 2023-09-29T22:49:27.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072f54f8ec2a0ea7f-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:49:27.287 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.288 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.288 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.288 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.288 DEBUG:boto:Method: GET 2023-09-29T22:49:27.288 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.288 DEBUG:boto:Data: 2023-09-29T22:49:27.288 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.288 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.288 DEBUG:boto:Params: {} 2023-09-29T22:49:27.289 DEBUG:boto:Token: None 2023-09-29T22:49:27.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY= 2023-09-29T22:49:27.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY=', 'Content-Length': '0'} 2023-09-29T22:49:27.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f79cde1d59051d26-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.292 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.292 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.292 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.293 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.293 DEBUG:boto:Method: GET 2023-09-29T22:49:27.293 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.293 DEBUG:boto:Data: 2023-09-29T22:49:27.293 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.293 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.293 DEBUG:boto:Params: {} 2023-09-29T22:49:27.293 DEBUG:boto:Token: None 2023-09-29T22:49:27.293 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:49:27.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY= 2023-09-29T22:49:27.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6h/A+6iYnywSK3VzIce2R8qCyTY=', 'Content-Length': '0'} 2023-09-29T22:49:27.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000096fbc6cd7b850650-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.297 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.297 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.297 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.297 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.297 DEBUG:boto:Method: GET 2023-09-29T22:49:27.297 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.297 DEBUG:boto:Data: 2023-09-29T22:49:27.298 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.298 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.298 DEBUG:boto:Params: {} 2023-09-29T22:49:27.298 DEBUG:boto:Token: None 2023-09-29T22:49:27.298 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069ff7bb46028d4a5-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.346 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.347 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.347 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.347 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.347 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.347 DEBUG:boto:Method: GET 2023-09-29T22:49:27.347 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.348 DEBUG:boto:Data: 2023-09-29T22:49:27.348 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.348 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.348 DEBUG:boto:Params: {} 2023-09-29T22:49:27.348 DEBUG:boto:Token: None 2023-09-29T22:49:27.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.358 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000229ccf1bbf4f973f-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.358 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.358 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.358 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.359 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.359 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:49:27.359 DEBUG:boto:Data: 2023-09-29T22:49:27.359 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.359 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.359 DEBUG:boto:Params: {} 2023-09-29T22:49:27.359 DEBUG:boto:Token: None 2023-09-29T22:49:27.359 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23 2023-09-29T22:49:27.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q//a2tppl1AzH0VlWpN3AyK2Hrg= 2023-09-29T22:49:27.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q//a2tppl1AzH0VlWpN3AyK2Hrg=', 'Content-Length': '0'} 2023-09-29T22:49:27.362 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003947df6e561cdf7e-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.362 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.362 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.362 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.362 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:49:27.362 DEBUG:boto:Data: 2023-09-29T22:49:27.363 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.363 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.363 DEBUG:boto:Params: {} 2023-09-29T22:49:27.363 DEBUG:boto:Token: None 2023-09-29T22:49:27.363 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23 2023-09-29T22:49:27.363 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q//a2tppl1AzH0VlWpN3AyK2Hrg= 2023-09-29T22:49:27.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q//a2tppl1AzH0VlWpN3AyK2Hrg=', 'Content-Length': '0'} 2023-09-29T22:49:27.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a51602effd986c-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:49:27.366 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.366 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.366 DEBUG:boto:Method: GET 2023-09-29T22:49:27.366 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:49:27.366 DEBUG:boto:Data: 2023-09-29T22:49:27.366 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.366 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.367 DEBUG:boto:Params: {} 2023-09-29T22:49:27.367 DEBUG:boto:Token: None 2023-09-29T22:49:27.367 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23 2023-09-29T22:49:27.367 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bZOfuzQ/bH8GdDVJGjHHIA0cW3E= 2023-09-29T22:49:27.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bZOfuzQ/bH8GdDVJGjHHIA0cW3E=', 'Content-Length': '0'} 2023-09-29T22:49:27.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000279e8f981750f5df-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.369 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:49:27.369 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:49:27.369 DEBUG:boto:Method: GET 2023-09-29T22:49:27.369 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:49:27.369 DEBUG:boto:Data: 2023-09-29T22:49:27.370 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.370 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.370 DEBUG:boto:Params: {} 2023-09-29T22:49:27.370 DEBUG:boto:Token: None 2023-09-29T22:49:27.370 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj23 2023-09-29T22:49:27.370 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bZOfuzQ/bH8GdDVJGjHHIA0cW3E= 2023-09-29T22:49:27.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bZOfuzQ/bH8GdDVJGjHHIA0cW3E=', 'Content-Length': '0'} 2023-09-29T22:49:27.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000576c27871d7ee24a-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.414 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.414 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.415 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.415 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.415 DEBUG:boto:Method: GET 2023-09-29T22:49:27.415 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.415 DEBUG:boto:Data: 2023-09-29T22:49:27.415 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.415 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.415 DEBUG:boto:Params: {} 2023-09-29T22:49:27.416 DEBUG:boto:Token: None 2023-09-29T22:49:27.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.420 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007fa1f3334445728c-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.420 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.421 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.421 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.421 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.421 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.421 DEBUG:boto:Method: GET 2023-09-29T22:49:27.421 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:49:27.421 DEBUG:boto:Data: 2023-09-29T22:49:27.421 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.421 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.422 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.422 DEBUG:boto:Params: {} 2023-09-29T22:49:27.422 DEBUG:boto:Token: None 2023-09-29T22:49:27.422 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.429 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d8d33b92c48ee06-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.429 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:49:27.430 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.430 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.430 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.430 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.430 DEBUG:boto:Method: GET 2023-09-29T22:49:27.430 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.430 DEBUG:boto:Data: 2023-09-29T22:49:27.431 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.431 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.431 DEBUG:boto:Params: {} 2023-09-29T22:49:27.431 DEBUG:boto:Token: None 2023-09-29T22:49:27.431 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.431 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w= 2023-09-29T22:49:27.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w=', 'Content-Length': '0'} 2023-09-29T22:49:27.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014afc31b6ba5cf51-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.433 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.434 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.434 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.434 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.434 DEBUG:boto:Method: GET 2023-09-29T22:49:27.434 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.434 DEBUG:boto:Data: 2023-09-29T22:49:27.434 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.434 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.435 DEBUG:boto:Params: {} 2023-09-29T22:49:27.435 DEBUG:boto:Token: None 2023-09-29T22:49:27.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.435 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w= 2023-09-29T22:49:27.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w=', 'Content-Length': '0'} 2023-09-29T22:49:27.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b472196e8c01abb8-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.438 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.438 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.438 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.438 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.438 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.438 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.438 DEBUG:boto:Data: 2023-09-29T22:49:27.438 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.438 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.439 DEBUG:boto:Params: {} 2023-09-29T22:49:27.439 DEBUG:boto:Token: None 2023-09-29T22:49:27.439 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.439 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nz0vlAMEwsvhxkqe4fVq4sPratM= 2023-09-29T22:49:27.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nz0vlAMEwsvhxkqe4fVq4sPratM=', 'Content-Length': '0'} 2023-09-29T22:49:27.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000391d4900fed35b2b-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.441 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.441 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.442 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.442 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.442 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.442 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.442 DEBUG:boto:Data: 2023-09-29T22:49:27.442 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.442 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.442 DEBUG:boto:Params: {} 2023-09-29T22:49:27.443 DEBUG:boto:Token: None 2023-09-29T22:49:27.443 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nz0vlAMEwsvhxkqe4fVq4sPratM= 2023-09-29T22:49:27.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nz0vlAMEwsvhxkqe4fVq4sPratM=', 'Content-Length': '0'} 2023-09-29T22:49:27.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed0f8829794a931-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:49:27.445 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.445 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.446 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.446 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.446 DEBUG:boto:Method: GET 2023-09-29T22:49:27.446 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.446 DEBUG:boto:Data: 2023-09-29T22:49:27.446 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.446 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.446 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.446 DEBUG:boto:Params: {} 2023-09-29T22:49:27.447 DEBUG:boto:Token: None 2023-09-29T22:49:27.447 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w= 2023-09-29T22:49:27.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w=', 'Content-Length': '0'} 2023-09-29T22:49:27.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062143c1bb49d1f4b-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.449 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.450 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.450 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.450 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.450 DEBUG:boto:Method: GET 2023-09-29T22:49:27.450 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.450 DEBUG:boto:Data: 2023-09-29T22:49:27.450 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.450 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.450 DEBUG:boto:Params: {} 2023-09-29T22:49:27.451 DEBUG:boto:Token: None 2023-09-29T22:49:27.451 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:49:27.451 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w= 2023-09-29T22:49:27.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NR5fTQRayn+h6kf5U14xyuAf84w=', 'Content-Length': '0'} 2023-09-29T22:49:27.453 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3a75b1bf22e7aa-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.454 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.454 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.454 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.454 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.454 DEBUG:boto:Method: GET 2023-09-29T22:49:27.454 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.455 DEBUG:boto:Data: 2023-09-29T22:49:27.455 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.455 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.455 DEBUG:boto:Params: {} 2023-09-29T22:49:27.455 DEBUG:boto:Token: None 2023-09-29T22:49:27.455 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.455 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.455 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.460 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007747144fd9f291b7-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.460 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.461 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.461 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.461 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.461 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.461 DEBUG:boto:Method: GET 2023-09-29T22:49:27.461 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.461 DEBUG:boto:Data: 2023-09-29T22:49:27.462 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.462 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.462 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.462 DEBUG:boto:Params: {} 2023-09-29T22:49:27.462 DEBUG:boto:Token: None 2023-09-29T22:49:27.462 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.462 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008e29da3282c34776-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.470 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.470 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.471 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.471 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.471 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:49:27.471 DEBUG:boto:Data: 2023-09-29T22:49:27.471 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.471 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.471 DEBUG:boto:Params: {} 2023-09-29T22:49:27.471 DEBUG:boto:Token: None 2023-09-29T22:49:27.471 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29 2023-09-29T22:49:27.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CHQOrIbNCH+lrUnqg51OOMVg61E= 2023-09-29T22:49:27.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CHQOrIbNCH+lrUnqg51OOMVg61E=', 'Content-Length': '0'} 2023-09-29T22:49:27.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086f369559145892e-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.474 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.474 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.474 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.475 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:49:27.475 DEBUG:boto:Data: 2023-09-29T22:49:27.475 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.475 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.475 DEBUG:boto:Params: {} 2023-09-29T22:49:27.475 DEBUG:boto:Token: None 2023-09-29T22:49:27.475 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29 2023-09-29T22:49:27.475 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CHQOrIbNCH+lrUnqg51OOMVg61E= 2023-09-29T22:49:27.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CHQOrIbNCH+lrUnqg51OOMVg61E=', 'Content-Length': '0'} 2023-09-29T22:49:27.477 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce3f2febf3f89a7-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:49:27.478 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.478 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.478 DEBUG:boto:Method: GET 2023-09-29T22:49:27.478 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:49:27.478 DEBUG:boto:Data: 2023-09-29T22:49:27.478 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.478 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.478 DEBUG:boto:Params: {} 2023-09-29T22:49:27.479 DEBUG:boto:Token: None 2023-09-29T22:49:27.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29 2023-09-29T22:49:27.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ty95S2CA37iC2AkyewpfBN6/9HI= 2023-09-29T22:49:27.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ty95S2CA37iC2AkyewpfBN6/9HI=', 'Content-Length': '0'} 2023-09-29T22:49:27.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fb8d82da20e176d-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.522 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:49:27.523 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:49:27.523 DEBUG:boto:Method: GET 2023-09-29T22:49:27.523 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:49:27.523 DEBUG:boto:Data: 2023-09-29T22:49:27.523 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.523 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.523 DEBUG:boto:Params: {} 2023-09-29T22:49:27.523 DEBUG:boto:Token: None 2023-09-29T22:49:27.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj29 2023-09-29T22:49:27.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ty95S2CA37iC2AkyewpfBN6/9HI= 2023-09-29T22:49:27.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ty95S2CA37iC2AkyewpfBN6/9HI=', 'Content-Length': '0'} 2023-09-29T22:49:27.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000606c074026f183a5-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.527 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.527 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.527 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.527 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.528 DEBUG:boto:Method: GET 2023-09-29T22:49:27.528 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.528 DEBUG:boto:Data: 2023-09-29T22:49:27.528 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.528 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.528 DEBUG:boto:Params: {} 2023-09-29T22:49:27.529 DEBUG:boto:Token: None 2023-09-29T22:49:27.529 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.533 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5c49149a12d6361-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.533 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.534 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.534 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.534 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.534 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.534 DEBUG:boto:Method: GET 2023-09-29T22:49:27.535 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:49:27.535 DEBUG:boto:Data: 2023-09-29T22:49:27.535 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.535 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.535 DEBUG:boto:Params: {} 2023-09-29T22:49:27.535 DEBUG:boto:Token: None 2023-09-29T22:49:27.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.543 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000301fe91e3d847154-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.543 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:49:27.544 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.544 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.544 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.544 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.544 DEBUG:boto:Method: GET 2023-09-29T22:49:27.544 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.544 DEBUG:boto:Data: 2023-09-29T22:49:27.544 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.545 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.545 DEBUG:boto:Params: {} 2023-09-29T22:49:27.545 DEBUG:boto:Token: None 2023-09-29T22:49:27.545 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.546 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g= 2023-09-29T22:49:27.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g=', 'Content-Length': '0'} 2023-09-29T22:49:27.548 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000041651fd1a686ff9-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.548 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.548 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.548 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.549 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.549 DEBUG:boto:Method: GET 2023-09-29T22:49:27.549 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.549 DEBUG:boto:Data: 2023-09-29T22:49:27.549 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.549 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.549 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.549 DEBUG:boto:Params: {} 2023-09-29T22:49:27.549 DEBUG:boto:Token: None 2023-09-29T22:49:27.550 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.550 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g= 2023-09-29T22:49:27.550 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g=', 'Content-Length': '0'} 2023-09-29T22:49:27.552 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6fb1f00c57fe9c-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.552 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.552 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.552 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.552 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.553 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.553 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.553 DEBUG:boto:Data: 2023-09-29T22:49:27.553 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.553 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.553 DEBUG:boto:Params: {} 2023-09-29T22:49:27.553 DEBUG:boto:Token: None 2023-09-29T22:49:27.553 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.554 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4+iH/lexx6Wdiq8U6DAlW/d+LtE= 2023-09-29T22:49:27.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4+iH/lexx6Wdiq8U6DAlW/d+LtE=', 'Content-Length': '0'} 2023-09-29T22:49:27.555 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000585b3e653b4f5b11-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.556 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.556 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.556 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.556 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.556 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.556 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.556 DEBUG:boto:Data: 2023-09-29T22:49:27.557 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.557 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.557 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.557 DEBUG:boto:Params: {} 2023-09-29T22:49:27.557 DEBUG:boto:Token: None 2023-09-29T22:49:27.557 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.557 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4+iH/lexx6Wdiq8U6DAlW/d+LtE= 2023-09-29T22:49:27.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4+iH/lexx6Wdiq8U6DAlW/d+LtE=', 'Content-Length': '0'} 2023-09-29T22:49:27.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ddae36deba736d56-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:49:27.560 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.560 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.560 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.560 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.560 DEBUG:boto:Method: GET 2023-09-29T22:49:27.560 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.560 DEBUG:boto:Data: 2023-09-29T22:49:27.560 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.561 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.561 DEBUG:boto:Params: {} 2023-09-29T22:49:27.561 DEBUG:boto:Token: None 2023-09-29T22:49:27.561 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g= 2023-09-29T22:49:27.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g=', 'Content-Length': '0'} 2023-09-29T22:49:27.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ba8b92a95884e84-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.563 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.564 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.564 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.564 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.564 DEBUG:boto:Method: GET 2023-09-29T22:49:27.564 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.564 DEBUG:boto:Data: 2023-09-29T22:49:27.564 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.565 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.565 DEBUG:boto:Params: {} 2023-09-29T22:49:27.565 DEBUG:boto:Token: None 2023-09-29T22:49:27.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:49:27.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g= 2023-09-29T22:49:27.566 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:otZsHmn+rrUZ978cwQWE9KvpR8g=', 'Content-Length': '0'} 2023-09-29T22:49:27.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c8c22f9566a2454-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.610 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.611 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.611 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.611 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.611 DEBUG:boto:Method: GET 2023-09-29T22:49:27.611 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.612 DEBUG:boto:Data: 2023-09-29T22:49:27.612 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.612 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.612 DEBUG:boto:Params: {} 2023-09-29T22:49:27.612 DEBUG:boto:Token: None 2023-09-29T22:49:27.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.613 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001303b5d2651d9b80-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.619 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.619 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.619 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.619 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.620 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.620 DEBUG:boto:Method: GET 2023-09-29T22:49:27.620 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.620 DEBUG:boto:Data: 2023-09-29T22:49:27.620 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.621 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.621 DEBUG:boto:Params: {} 2023-09-29T22:49:27.621 DEBUG:boto:Token: None 2023-09-29T22:49:27.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.621 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e57661ece858115d-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.629 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.629 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.629 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.629 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.629 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:49:27.630 DEBUG:boto:Data: 2023-09-29T22:49:27.630 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.630 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.630 DEBUG:boto:Params: {} 2023-09-29T22:49:27.630 DEBUG:boto:Token: None 2023-09-29T22:49:27.631 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31 2023-09-29T22:49:27.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NgeJKXgYL+BoIvMZH225TONaYWU= 2023-09-29T22:49:27.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NgeJKXgYL+BoIvMZH225TONaYWU=', 'Content-Length': '0'} 2023-09-29T22:49:27.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab9fb7105703df9-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.634 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.634 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.634 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.634 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:49:27.635 DEBUG:boto:Data: 2023-09-29T22:49:27.635 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.635 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.635 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.635 DEBUG:boto:Params: {} 2023-09-29T22:49:27.635 DEBUG:boto:Token: None 2023-09-29T22:49:27.636 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31 2023-09-29T22:49:27.636 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NgeJKXgYL+BoIvMZH225TONaYWU= 2023-09-29T22:49:27.636 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NgeJKXgYL+BoIvMZH225TONaYWU=', 'Content-Length': '0'} 2023-09-29T22:49:27.638 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032f2d6f27e30f127-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:49:27.638 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.639 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.639 DEBUG:boto:Method: GET 2023-09-29T22:49:27.639 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:49:27.639 DEBUG:boto:Data: 2023-09-29T22:49:27.639 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.640 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.640 DEBUG:boto:Params: {} 2023-09-29T22:49:27.640 DEBUG:boto:Token: None 2023-09-29T22:49:27.640 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31 2023-09-29T22:49:27.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DYfOx3rpUHoG5EsFVWcUm2uj5iY= 2023-09-29T22:49:27.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DYfOx3rpUHoG5EsFVWcUm2uj5iY=', 'Content-Length': '0'} 2023-09-29T22:49:27.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2f3c5b458a4f65d-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.686 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:49:27.687 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:49:27.687 DEBUG:boto:Method: GET 2023-09-29T22:49:27.687 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:49:27.687 DEBUG:boto:Data: 2023-09-29T22:49:27.687 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.688 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.688 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.688 DEBUG:boto:Params: {} 2023-09-29T22:49:27.688 DEBUG:boto:Token: None 2023-09-29T22:49:27.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj31 2023-09-29T22:49:27.689 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DYfOx3rpUHoG5EsFVWcUm2uj5iY= 2023-09-29T22:49:27.689 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DYfOx3rpUHoG5EsFVWcUm2uj5iY=', 'Content-Length': '0'} 2023-09-29T22:49:27.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c52c6c6eba69f68-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.692 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.692 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.693 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.693 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.693 DEBUG:boto:Method: GET 2023-09-29T22:49:27.693 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.693 DEBUG:boto:Data: 2023-09-29T22:49:27.693 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.694 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.694 DEBUG:boto:Params: {} 2023-09-29T22:49:27.694 DEBUG:boto:Token: None 2023-09-29T22:49:27.694 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.699 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7b1879f51ba4a28-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.700 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.700 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.700 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.701 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.701 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.701 DEBUG:boto:Method: GET 2023-09-29T22:49:27.701 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:49:27.701 DEBUG:boto:Data: 2023-09-29T22:49:27.701 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.701 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.701 DEBUG:boto:Params: {} 2023-09-29T22:49:27.702 DEBUG:boto:Token: None 2023-09-29T22:49:27.702 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.709 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f8ec6e3770c21402-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.750 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:49:27.751 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.751 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.752 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.752 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.752 DEBUG:boto:Method: GET 2023-09-29T22:49:27.752 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.752 DEBUG:boto:Data: 2023-09-29T22:49:27.752 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.753 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.753 DEBUG:boto:Params: {} 2023-09-29T22:49:27.753 DEBUG:boto:Token: None 2023-09-29T22:49:27.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg= 2023-09-29T22:49:27.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg=', 'Content-Length': '0'} 2023-09-29T22:49:27.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a3babf23e3d3891-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.757 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.757 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.757 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.757 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.757 DEBUG:boto:Method: GET 2023-09-29T22:49:27.758 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.758 DEBUG:boto:Data: 2023-09-29T22:49:27.758 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.758 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.758 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.758 DEBUG:boto:Params: {} 2023-09-29T22:49:27.759 DEBUG:boto:Token: None 2023-09-29T22:49:27.759 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.759 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg= 2023-09-29T22:49:27.759 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg=', 'Content-Length': '0'} 2023-09-29T22:49:27.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000852a912dc69a93c9-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.762 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.762 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.762 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.763 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.763 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.763 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.763 DEBUG:boto:Data: 2023-09-29T22:49:27.763 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.764 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.764 DEBUG:boto:Params: {} 2023-09-29T22:49:27.764 DEBUG:boto:Token: None 2023-09-29T22:49:27.764 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WQpLAT0LYwYIrzqhTjmVC/Op2SU= 2023-09-29T22:49:27.765 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WQpLAT0LYwYIrzqhTjmVC/Op2SU=', 'Content-Length': '0'} 2023-09-29T22:49:27.767 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f47c021220ebddc-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.768 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.768 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.768 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.768 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.768 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.769 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.769 DEBUG:boto:Data: 2023-09-29T22:49:27.769 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.769 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.769 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.769 DEBUG:boto:Params: {} 2023-09-29T22:49:27.770 DEBUG:boto:Token: None 2023-09-29T22:49:27.770 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.770 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WQpLAT0LYwYIrzqhTjmVC/Op2SU= 2023-09-29T22:49:27.770 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WQpLAT0LYwYIrzqhTjmVC/Op2SU=', 'Content-Length': '0'} 2023-09-29T22:49:27.773 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040dce3f3c880c803-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:49:27.774 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.774 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.774 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.775 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.775 DEBUG:boto:Method: GET 2023-09-29T22:49:27.775 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.775 DEBUG:boto:Data: 2023-09-29T22:49:27.775 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.776 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.776 DEBUG:boto:Params: {} 2023-09-29T22:49:27.776 DEBUG:boto:Token: None 2023-09-29T22:49:27.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg= 2023-09-29T22:49:27.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg=', 'Content-Length': '0'} 2023-09-29T22:49:27.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005668fd1933de951a-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.780 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.780 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.780 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.780 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.780 DEBUG:boto:Method: GET 2023-09-29T22:49:27.781 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.781 DEBUG:boto:Data: 2023-09-29T22:49:27.781 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.781 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.781 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.782 DEBUG:boto:Params: {} 2023-09-29T22:49:27.782 DEBUG:boto:Token: None 2023-09-29T22:49:27.782 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:49:27.782 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg= 2023-09-29T22:49:27.782 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j+aGDHSE7rahRk98dg+7Kci2NXg=', 'Content-Length': '0'} 2023-09-29T22:49:27.785 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b975fbaa7be5ca35-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.786 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.786 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.786 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.786 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.787 DEBUG:boto:Method: GET 2023-09-29T22:49:27.787 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.787 DEBUG:boto:Data: 2023-09-29T22:49:27.787 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.787 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.787 DEBUG:boto:Params: {} 2023-09-29T22:49:27.788 DEBUG:boto:Token: None 2023-09-29T22:49:27.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.793 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000072e109895cefdc53-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.834 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.835 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.835 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.835 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.835 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.836 DEBUG:boto:Method: GET 2023-09-29T22:49:27.836 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.836 DEBUG:boto:Data: 2023-09-29T22:49:27.836 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.837 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.837 DEBUG:boto:Params: {} 2023-09-29T22:49:27.837 DEBUG:boto:Token: None 2023-09-29T22:49:27.837 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dedf6e9ec258b2d-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.847 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.847 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.847 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.847 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.848 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:49:27.848 DEBUG:boto:Data: 2023-09-29T22:49:27.848 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.848 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.848 DEBUG:boto:Params: {} 2023-09-29T22:49:27.849 DEBUG:boto:Token: None 2023-09-29T22:49:27.849 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33 2023-09-29T22:49:27.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bh8B5ltqJ/wmXOaGofsJipceOc0= 2023-09-29T22:49:27.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bh8B5ltqJ/wmXOaGofsJipceOc0=', 'Content-Length': '0'} 2023-09-29T22:49:27.851 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000944abdfd9cd6076d-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.852 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.852 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.852 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.852 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:49:27.852 DEBUG:boto:Data: 2023-09-29T22:49:27.853 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.853 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.853 DEBUG:boto:Params: {} 2023-09-29T22:49:27.853 DEBUG:boto:Token: None 2023-09-29T22:49:27.853 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33 2023-09-29T22:49:27.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bh8B5ltqJ/wmXOaGofsJipceOc0= 2023-09-29T22:49:27.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bh8B5ltqJ/wmXOaGofsJipceOc0=', 'Content-Length': '0'} 2023-09-29T22:49:27.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000429ab191e9961378-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:49:27.856 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.856 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.856 DEBUG:boto:Method: GET 2023-09-29T22:49:27.857 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:49:27.857 DEBUG:boto:Data: 2023-09-29T22:49:27.857 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.857 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.857 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.857 DEBUG:boto:Params: {} 2023-09-29T22:49:27.858 DEBUG:boto:Token: None 2023-09-29T22:49:27.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33 2023-09-29T22:49:27.858 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:36vSAi3tBuN2DE26IuTgJ+HEIcE= 2023-09-29T22:49:27.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:36vSAi3tBuN2DE26IuTgJ+HEIcE=', 'Content-Length': '0'} 2023-09-29T22:49:27.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000174a8b0ae0b32bb8-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.861 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:49:27.861 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:49:27.861 DEBUG:boto:Method: GET 2023-09-29T22:49:27.861 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:49:27.861 DEBUG:boto:Data: 2023-09-29T22:49:27.861 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.862 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.862 DEBUG:boto:Params: {} 2023-09-29T22:49:27.862 DEBUG:boto:Token: None 2023-09-29T22:49:27.862 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj33 2023-09-29T22:49:27.863 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:36vSAi3tBuN2DE26IuTgJ+HEIcE= 2023-09-29T22:49:27.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:36vSAi3tBuN2DE26IuTgJ+HEIcE=', 'Content-Length': '0'} 2023-09-29T22:49:27.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc633a5d3ba344e-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.865 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.865 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.866 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.866 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.866 DEBUG:boto:Method: GET 2023-09-29T22:49:27.866 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.866 DEBUG:boto:Data: 2023-09-29T22:49:27.867 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.867 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.867 DEBUG:boto:Params: {} 2023-09-29T22:49:27.867 DEBUG:boto:Token: None 2023-09-29T22:49:27.867 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f933763d5e88db8-0065175477-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.871 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.872 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:27.872 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:27.872 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.872 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.872 DEBUG:boto:Method: GET 2023-09-29T22:49:27.873 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:49:27.873 DEBUG:boto:Data: 2023-09-29T22:49:27.873 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.873 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.873 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.873 DEBUG:boto:Params: {} 2023-09-29T22:49:27.874 DEBUG:boto:Token: None 2023-09-29T22:49:27.874 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/ 2023-09-29T22:49:27.874 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg= 2023-09-29T22:49:27.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qntCVY3LlDGPOGnVAXKQiPOLXZg=', 'Content-Length': '0'} 2023-09-29T22:49:27.881 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e3b219e5f4c2390-0065175477-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.881 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:27.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:49:27.882 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:27.882 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:27.882 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.882 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.882 DEBUG:boto:Method: GET 2023-09-29T22:49:27.883 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.883 DEBUG:boto:Data: 2023-09-29T22:49:27.883 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.883 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.884 DEBUG:boto:Params: {} 2023-09-29T22:49:27.884 DEBUG:boto:Token: None 2023-09-29T22:49:27.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8= 2023-09-29T22:49:27.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8=', 'Content-Length': '0'} 2023-09-29T22:49:27.887 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d1bedf834ed951-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.930 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:27.931 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:27.931 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.931 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.931 DEBUG:boto:Method: GET 2023-09-29T22:49:27.931 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.931 DEBUG:boto:Data: 2023-09-29T22:49:27.932 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.932 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.932 DEBUG:boto:Params: {} 2023-09-29T22:49:27.932 DEBUG:boto:Token: None 2023-09-29T22:49:27.933 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.933 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8= 2023-09-29T22:49:27.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8=', 'Content-Length': '0'} 2023-09-29T22:49:27.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f915e066322d2ff-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.937 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:27.937 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:27.937 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.937 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.937 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.937 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.937 DEBUG:boto:Data: 2023-09-29T22:49:27.938 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.938 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.938 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.938 DEBUG:boto:Params: {} 2023-09-29T22:49:27.938 DEBUG:boto:Token: None 2023-09-29T22:49:27.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.939 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EyVI3r4BdjECBSb0vjEauMH71Xk= 2023-09-29T22:49:27.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EyVI3r4BdjECBSb0vjEauMH71Xk=', 'Content-Length': '0'} 2023-09-29T22:49:27.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc85875892df1d6a-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.942 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:27.942 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:27.942 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.942 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.943 DEBUG:boto:Method: HEAD 2023-09-29T22:49:27.943 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.943 DEBUG:boto:Data: 2023-09-29T22:49:27.943 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.944 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.944 DEBUG:boto:Params: {} 2023-09-29T22:49:27.944 DEBUG:boto:Token: None 2023-09-29T22:49:27.944 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EyVI3r4BdjECBSb0vjEauMH71Xk= 2023-09-29T22:49:27.945 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EyVI3r4BdjECBSb0vjEauMH71Xk=', 'Content-Length': '0'} 2023-09-29T22:49:27.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cffc72524e9c21-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:49:27.948 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:27.948 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:27.948 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.948 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.948 DEBUG:boto:Method: GET 2023-09-29T22:49:27.948 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.948 DEBUG:boto:Data: 2023-09-29T22:49:27.948 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:27.949 DEBUG:boto:Port: 8001 2023-09-29T22:49:27.949 DEBUG:boto:Params: {} 2023-09-29T22:49:27.949 DEBUG:boto:Token: None 2023-09-29T22:49:27.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8= 2023-09-29T22:49:27.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8=', 'Content-Length': '0'} 2023-09-29T22:49:27.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcd1226d99161342-0065175477-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:27.952 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:27.952 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:27.953 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.953 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.953 DEBUG:boto:Method: GET 2023-09-29T22:49:27.953 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.953 DEBUG:boto:Data: 2023-09-29T22:49:27.953 DEBUG:boto:Headers: {} 2023-09-29T22:49:27.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:27.953 DEBUG:boto:Port: 8000 2023-09-29T22:49:27.953 DEBUG:boto:Params: {} 2023-09-29T22:49:27.953 DEBUG:boto:Token: None 2023-09-29T22:49:27.954 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:27 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:49:27.954 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8= 2023-09-29T22:49:27.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uPmJa849dWemQaKO5BbAUtZSZB8=', 'Content-Length': '0'} 2023-09-29T22:49:27.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000729d96c845b7ed44-0065175477-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.002 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.003 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.003 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.003 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.003 DEBUG:boto:Method: GET 2023-09-29T22:49:28.003 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.003 DEBUG:boto:Data: 2023-09-29T22:49:28.003 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.004 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.004 DEBUG:boto:Params: {} 2023-09-29T22:49:28.004 DEBUG:boto:Token: None 2023-09-29T22:49:28.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.009 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084f1e8b3380330a6-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.009 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.010 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.010 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.010 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.010 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.010 DEBUG:boto:Method: GET 2023-09-29T22:49:28.010 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.010 DEBUG:boto:Data: 2023-09-29T22:49:28.010 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.011 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.011 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.011 DEBUG:boto:Params: {} 2023-09-29T22:49:28.011 DEBUG:boto:Token: None 2023-09-29T22:49:28.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.019 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aceb0a2fc229fec9-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.020 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.020 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:28.020 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:28.020 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.020 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:49:28.021 DEBUG:boto:Data: 2023-09-29T22:49:28.021 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.021 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.021 DEBUG:boto:Params: {} 2023-09-29T22:49:28.021 DEBUG:boto:Token: None 2023-09-29T22:49:28.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj34 2023-09-29T22:49:28.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G+445r3ihQ7sEFi5vwHtZhv+8qU= 2023-09-29T22:49:28.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G+445r3ihQ7sEFi5vwHtZhv+8qU=', 'Content-Length': '0'} 2023-09-29T22:49:28.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000427c778be9283b9d-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.024 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:28.024 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:28.024 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.024 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:49:28.024 DEBUG:boto:Data: 2023-09-29T22:49:28.024 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.025 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.025 DEBUG:boto:Params: {} 2023-09-29T22:49:28.025 DEBUG:boto:Token: None 2023-09-29T22:49:28.025 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj34 2023-09-29T22:49:28.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G+445r3ihQ7sEFi5vwHtZhv+8qU= 2023-09-29T22:49:28.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G+445r3ihQ7sEFi5vwHtZhv+8qU=', 'Content-Length': '0'} 2023-09-29T22:49:28.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020946cd7de9a6181-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:49:28.028 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:28.028 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:28.028 DEBUG:boto:Method: GET 2023-09-29T22:49:28.028 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:49:28.028 DEBUG:boto:Data: 2023-09-29T22:49:28.028 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.028 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.028 DEBUG:boto:Params: {} 2023-09-29T22:49:28.029 DEBUG:boto:Token: None 2023-09-29T22:49:28.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj34 2023-09-29T22:49:28.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NWtmX/8CM+uvwRwkwn7o+4v8v3A= 2023-09-29T22:49:28.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NWtmX/8CM+uvwRwkwn7o+4v8v3A=', 'Content-Length': '0'} 2023-09-29T22:49:28.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a59d92c781ad8a1-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.031 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:49:28.031 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:49:28.031 DEBUG:boto:Method: GET 2023-09-29T22:49:28.031 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:49:28.032 DEBUG:boto:Data: 2023-09-29T22:49:28.032 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.032 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.032 DEBUG:boto:Params: {} 2023-09-29T22:49:28.032 DEBUG:boto:Token: None 2023-09-29T22:49:28.032 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj34 2023-09-29T22:49:28.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NWtmX/8CM+uvwRwkwn7o+4v8v3A= 2023-09-29T22:49:28.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NWtmX/8CM+uvwRwkwn7o+4v8v3A=', 'Content-Length': '0'} 2023-09-29T22:49:28.035 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1d889af170f0cc9-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.078 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.079 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.079 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.079 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.079 DEBUG:boto:Method: GET 2023-09-29T22:49:28.079 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.079 DEBUG:boto:Data: 2023-09-29T22:49:28.079 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.079 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.080 DEBUG:boto:Params: {} 2023-09-29T22:49:28.080 DEBUG:boto:Token: None 2023-09-29T22:49:28.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000163635c0292d6394-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.085 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.086 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.086 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.086 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.086 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.086 DEBUG:boto:Method: GET 2023-09-29T22:49:28.086 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:49:28.086 DEBUG:boto:Data: 2023-09-29T22:49:28.086 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.087 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.087 DEBUG:boto:Params: {} 2023-09-29T22:49:28.087 DEBUG:boto:Token: None 2023-09-29T22:49:28.087 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea017ab6e63ce010-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.096 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:49:28.097 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.097 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.097 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.098 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.098 DEBUG:boto:Method: GET 2023-09-29T22:49:28.098 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.098 DEBUG:boto:Data: 2023-09-29T22:49:28.098 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.099 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.099 DEBUG:boto:Params: {} 2023-09-29T22:49:28.099 DEBUG:boto:Token: None 2023-09-29T22:49:28.100 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U= 2023-09-29T22:49:28.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U=', 'Content-Length': '0'} 2023-09-29T22:49:28.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032b615c955c9200f-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.146 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.146 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.147 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.147 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.147 DEBUG:boto:Method: GET 2023-09-29T22:49:28.147 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.147 DEBUG:boto:Data: 2023-09-29T22:49:28.147 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.147 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.147 DEBUG:boto:Params: {} 2023-09-29T22:49:28.148 DEBUG:boto:Token: None 2023-09-29T22:49:28.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U= 2023-09-29T22:49:28.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U=', 'Content-Length': '0'} 2023-09-29T22:49:28.150 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a622c695d1c364d-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.151 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.151 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.151 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.151 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.151 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.151 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.151 DEBUG:boto:Data: 2023-09-29T22:49:28.151 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.152 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.152 DEBUG:boto:Params: {} 2023-09-29T22:49:28.152 DEBUG:boto:Token: None 2023-09-29T22:49:28.152 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.152 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SlfuMc7Ucvef3aVOZ73NC6m+ZBE= 2023-09-29T22:49:28.152 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SlfuMc7Ucvef3aVOZ73NC6m+ZBE=', 'Content-Length': '0'} 2023-09-29T22:49:28.154 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062d1a8b459f08da2-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.155 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.155 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.155 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.155 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.155 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.155 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.155 DEBUG:boto:Data: 2023-09-29T22:49:28.155 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.156 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.156 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.156 DEBUG:boto:Params: {} 2023-09-29T22:49:28.156 DEBUG:boto:Token: None 2023-09-29T22:49:28.156 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.156 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SlfuMc7Ucvef3aVOZ73NC6m+ZBE= 2023-09-29T22:49:28.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SlfuMc7Ucvef3aVOZ73NC6m+ZBE=', 'Content-Length': '0'} 2023-09-29T22:49:28.158 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2b58c254f351bf6-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:49:28.159 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.159 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.159 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.159 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.159 DEBUG:boto:Method: GET 2023-09-29T22:49:28.159 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.160 DEBUG:boto:Data: 2023-09-29T22:49:28.160 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.160 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.160 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.160 DEBUG:boto:Params: {} 2023-09-29T22:49:28.160 DEBUG:boto:Token: None 2023-09-29T22:49:28.160 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.160 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U= 2023-09-29T22:49:28.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U=', 'Content-Length': '0'} 2023-09-29T22:49:28.163 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024b3fdfab6ff42da-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.163 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.163 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.163 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.163 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.163 DEBUG:boto:Method: GET 2023-09-29T22:49:28.164 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.164 DEBUG:boto:Data: 2023-09-29T22:49:28.164 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.164 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.164 DEBUG:boto:Params: {} 2023-09-29T22:49:28.164 DEBUG:boto:Token: None 2023-09-29T22:49:28.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:49:28.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U= 2023-09-29T22:49:28.165 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C1h5nLb93YUSUn9Crjl/ZWhA03U=', 'Content-Length': '0'} 2023-09-29T22:49:28.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba3c0fb191672a06-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.210 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.211 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.211 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.211 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.211 DEBUG:boto:Method: GET 2023-09-29T22:49:28.211 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.212 DEBUG:boto:Data: 2023-09-29T22:49:28.212 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.212 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.212 DEBUG:boto:Params: {} 2023-09-29T22:49:28.212 DEBUG:boto:Token: None 2023-09-29T22:49:28.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc083ccf518334aa-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.218 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.218 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.219 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.219 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.219 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.219 DEBUG:boto:Method: GET 2023-09-29T22:49:28.219 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.219 DEBUG:boto:Data: 2023-09-29T22:49:28.219 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.220 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.220 DEBUG:boto:Params: {} 2023-09-29T22:49:28.220 DEBUG:boto:Token: None 2023-09-29T22:49:28.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.228 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000103b3e7902184c46-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.229 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.229 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.229 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.229 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.230 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:49:28.230 DEBUG:boto:Data: 2023-09-29T22:49:28.230 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.230 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.230 DEBUG:boto:Params: {} 2023-09-29T22:49:28.230 DEBUG:boto:Token: None 2023-09-29T22:49:28.231 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37 2023-09-29T22:49:28.231 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O/UTFaLPr//99oD6jE3k57BXum0= 2023-09-29T22:49:28.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O/UTFaLPr//99oD6jE3k57BXum0=', 'Content-Length': '0'} 2023-09-29T22:49:28.233 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048513081734b9ab5-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.233 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.234 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.234 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.234 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:49:28.235 DEBUG:boto:Data: 2023-09-29T22:49:28.235 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.235 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.235 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.235 DEBUG:boto:Params: {} 2023-09-29T22:49:28.236 DEBUG:boto:Token: None 2023-09-29T22:49:28.236 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37 2023-09-29T22:49:28.236 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O/UTFaLPr//99oD6jE3k57BXum0= 2023-09-29T22:49:28.237 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O/UTFaLPr//99oD6jE3k57BXum0=', 'Content-Length': '0'} 2023-09-29T22:49:28.239 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000908df47aed25fc5f-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:49:28.240 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.240 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.240 DEBUG:boto:Method: GET 2023-09-29T22:49:28.240 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:49:28.241 DEBUG:boto:Data: 2023-09-29T22:49:28.241 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.241 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.241 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.241 DEBUG:boto:Params: {} 2023-09-29T22:49:28.242 DEBUG:boto:Token: None 2023-09-29T22:49:28.242 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37 2023-09-29T22:49:28.242 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3R+MhEkHwD3H87JmlQhEUgjWW+0= 2023-09-29T22:49:28.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3R+MhEkHwD3H87JmlQhEUgjWW+0=', 'Content-Length': '0'} 2023-09-29T22:49:28.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcbe0de65010cd83-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.286 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:49:28.287 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:49:28.287 DEBUG:boto:Method: GET 2023-09-29T22:49:28.287 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:49:28.288 DEBUG:boto:Data: 2023-09-29T22:49:28.288 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.288 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.288 DEBUG:boto:Params: {} 2023-09-29T22:49:28.289 DEBUG:boto:Token: None 2023-09-29T22:49:28.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj37 2023-09-29T22:49:28.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3R+MhEkHwD3H87JmlQhEUgjWW+0= 2023-09-29T22:49:28.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3R+MhEkHwD3H87JmlQhEUgjWW+0=', 'Content-Length': '0'} 2023-09-29T22:49:28.293 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da0a7ad5e3e67400-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.293 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.294 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.294 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.294 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.294 DEBUG:boto:Method: GET 2023-09-29T22:49:28.295 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.295 DEBUG:boto:Data: 2023-09-29T22:49:28.295 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.295 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.296 DEBUG:boto:Params: {} 2023-09-29T22:49:28.296 DEBUG:boto:Token: None 2023-09-29T22:49:28.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.302 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbec16a221d6806c-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.302 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.303 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.303 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.303 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.303 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.304 DEBUG:boto:Method: GET 2023-09-29T22:49:28.304 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:49:28.304 DEBUG:boto:Data: 2023-09-29T22:49:28.304 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.305 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.305 DEBUG:boto:Params: {} 2023-09-29T22:49:28.305 DEBUG:boto:Token: None 2023-09-29T22:49:28.305 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.313 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bc1909202eeb1306-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.313 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:49:28.314 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.314 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.314 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.314 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.314 DEBUG:boto:Method: GET 2023-09-29T22:49:28.315 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.315 DEBUG:boto:Data: 2023-09-29T22:49:28.315 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.315 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.316 DEBUG:boto:Params: {} 2023-09-29T22:49:28.316 DEBUG:boto:Token: None 2023-09-29T22:49:28.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA= 2023-09-29T22:49:28.317 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA=', 'Content-Length': '0'} 2023-09-29T22:49:28.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009c2d3a3049de1c7-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.320 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.320 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.320 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.320 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.320 DEBUG:boto:Method: GET 2023-09-29T22:49:28.320 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.320 DEBUG:boto:Data: 2023-09-29T22:49:28.320 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.321 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.321 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.321 DEBUG:boto:Params: {} 2023-09-29T22:49:28.321 DEBUG:boto:Token: None 2023-09-29T22:49:28.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA= 2023-09-29T22:49:28.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA=', 'Content-Length': '0'} 2023-09-29T22:49:28.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c21558e986ae8790-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.366 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.367 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.367 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.367 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.367 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.367 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.367 DEBUG:boto:Data: 2023-09-29T22:49:28.367 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.367 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.367 DEBUG:boto:Params: {} 2023-09-29T22:49:28.368 DEBUG:boto:Token: None 2023-09-29T22:49:28.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jjRlXks+YsHhDM87+mi1WuQoaIw= 2023-09-29T22:49:28.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jjRlXks+YsHhDM87+mi1WuQoaIw=', 'Content-Length': '0'} 2023-09-29T22:49:28.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd0be400db1fce1a-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.371 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.371 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.371 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.371 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.372 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.372 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.372 DEBUG:boto:Data: 2023-09-29T22:49:28.372 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.372 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.372 DEBUG:boto:Params: {} 2023-09-29T22:49:28.372 DEBUG:boto:Token: None 2023-09-29T22:49:28.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jjRlXks+YsHhDM87+mi1WuQoaIw= 2023-09-29T22:49:28.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jjRlXks+YsHhDM87+mi1WuQoaIw=', 'Content-Length': '0'} 2023-09-29T22:49:28.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a7aaa442caf1f8-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:49:28.377 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.377 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.377 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.377 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.377 DEBUG:boto:Method: GET 2023-09-29T22:49:28.377 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.377 DEBUG:boto:Data: 2023-09-29T22:49:28.377 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.378 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.378 DEBUG:boto:Params: {} 2023-09-29T22:49:28.378 DEBUG:boto:Token: None 2023-09-29T22:49:28.378 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.378 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA= 2023-09-29T22:49:28.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA=', 'Content-Length': '0'} 2023-09-29T22:49:28.381 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b5477b4ffad8d2f-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.422 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.423 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.423 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.423 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.423 DEBUG:boto:Method: GET 2023-09-29T22:49:28.423 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.423 DEBUG:boto:Data: 2023-09-29T22:49:28.423 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.424 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.424 DEBUG:boto:Params: {} 2023-09-29T22:49:28.424 DEBUG:boto:Token: None 2023-09-29T22:49:28.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:49:28.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA= 2023-09-29T22:49:28.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+eBFuD72uFU4/loGfmQZ1eTXdmA=', 'Content-Length': '0'} 2023-09-29T22:49:28.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c80b54cb9fc4526-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.428 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.428 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.428 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.428 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.428 DEBUG:boto:Method: GET 2023-09-29T22:49:28.428 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.429 DEBUG:boto:Data: 2023-09-29T22:49:28.429 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.429 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.429 DEBUG:boto:Params: {} 2023-09-29T22:49:28.429 DEBUG:boto:Token: None 2023-09-29T22:49:28.429 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.429 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.434 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000134b1cf503895b8-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.434 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.434 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.435 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.435 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.435 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.435 DEBUG:boto:Method: GET 2023-09-29T22:49:28.435 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.435 DEBUG:boto:Data: 2023-09-29T22:49:28.435 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.435 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.436 DEBUG:boto:Params: {} 2023-09-29T22:49:28.436 DEBUG:boto:Token: None 2023-09-29T22:49:28.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.443 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6ec8b6c8bbc2f65-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.443 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.444 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.444 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.444 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.444 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:49:28.444 DEBUG:boto:Data: 2023-09-29T22:49:28.444 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.445 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.445 DEBUG:boto:Params: {} 2023-09-29T22:49:28.445 DEBUG:boto:Token: None 2023-09-29T22:49:28.445 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38 2023-09-29T22:49:28.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EdXoIviKEDfF/N6fxza+01f7h1Q= 2023-09-29T22:49:28.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EdXoIviKEDfF/N6fxza+01f7h1Q=', 'Content-Length': '0'} 2023-09-29T22:49:28.448 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e79da940a133b47-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.448 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.448 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.448 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.448 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:49:28.448 DEBUG:boto:Data: 2023-09-29T22:49:28.449 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.449 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.449 DEBUG:boto:Params: {} 2023-09-29T22:49:28.449 DEBUG:boto:Token: None 2023-09-29T22:49:28.449 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38 2023-09-29T22:49:28.449 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EdXoIviKEDfF/N6fxza+01f7h1Q= 2023-09-29T22:49:28.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EdXoIviKEDfF/N6fxza+01f7h1Q=', 'Content-Length': '0'} 2023-09-29T22:49:28.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d261a3e4c73ede4-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:49:28.452 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.452 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.452 DEBUG:boto:Method: GET 2023-09-29T22:49:28.452 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:49:28.452 DEBUG:boto:Data: 2023-09-29T22:49:28.452 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.453 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.453 DEBUG:boto:Params: {} 2023-09-29T22:49:28.453 DEBUG:boto:Token: None 2023-09-29T22:49:28.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38 2023-09-29T22:49:28.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zKTRFi0u5WHQOP877JIukNQawro= 2023-09-29T22:49:28.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zKTRFi0u5WHQOP877JIukNQawro=', 'Content-Length': '0'} 2023-09-29T22:49:28.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005808d293f64ffbad-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.498 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:49:28.499 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:49:28.499 DEBUG:boto:Method: GET 2023-09-29T22:49:28.499 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:49:28.499 DEBUG:boto:Data: 2023-09-29T22:49:28.499 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.499 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.499 DEBUG:boto:Params: {} 2023-09-29T22:49:28.500 DEBUG:boto:Token: None 2023-09-29T22:49:28.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj38 2023-09-29T22:49:28.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zKTRFi0u5WHQOP877JIukNQawro= 2023-09-29T22:49:28.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zKTRFi0u5WHQOP877JIukNQawro=', 'Content-Length': '0'} 2023-09-29T22:49:28.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1dce7b85470d667-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.504 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.504 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.504 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.504 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.504 DEBUG:boto:Method: GET 2023-09-29T22:49:28.504 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.504 DEBUG:boto:Data: 2023-09-29T22:49:28.504 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.505 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.505 DEBUG:boto:Params: {} 2023-09-29T22:49:28.505 DEBUG:boto:Token: None 2023-09-29T22:49:28.505 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003315a43c574d26af-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.510 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.510 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.511 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.511 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.511 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.511 DEBUG:boto:Method: GET 2023-09-29T22:49:28.511 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:49:28.511 DEBUG:boto:Data: 2023-09-29T22:49:28.511 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.511 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.512 DEBUG:boto:Params: {} 2023-09-29T22:49:28.512 DEBUG:boto:Token: None 2023-09-29T22:49:28.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3056554626bfaaf-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.519 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:49:28.520 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.520 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.520 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.520 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.520 DEBUG:boto:Method: GET 2023-09-29T22:49:28.521 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.521 DEBUG:boto:Data: 2023-09-29T22:49:28.521 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.521 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.521 DEBUG:boto:Params: {} 2023-09-29T22:49:28.521 DEBUG:boto:Token: None 2023-09-29T22:49:28.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.521 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I= 2023-09-29T22:49:28.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I=', 'Content-Length': '0'} 2023-09-29T22:49:28.523 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c6440d22227ea20-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.524 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.524 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.524 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.524 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.524 DEBUG:boto:Method: GET 2023-09-29T22:49:28.524 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.524 DEBUG:boto:Data: 2023-09-29T22:49:28.525 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.525 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.525 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.525 DEBUG:boto:Params: {} 2023-09-29T22:49:28.525 DEBUG:boto:Token: None 2023-09-29T22:49:28.525 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.525 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I= 2023-09-29T22:49:28.526 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I=', 'Content-Length': '0'} 2023-09-29T22:49:28.527 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d86ec397be1ec06e-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.528 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.528 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.528 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.528 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.528 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.528 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.528 DEBUG:boto:Data: 2023-09-29T22:49:28.529 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.529 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.529 DEBUG:boto:Params: {} 2023-09-29T22:49:28.529 DEBUG:boto:Token: None 2023-09-29T22:49:28.529 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.529 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FqfrkaiiPvd84w3uq3+/gFd9F5Q= 2023-09-29T22:49:28.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FqfrkaiiPvd84w3uq3+/gFd9F5Q=', 'Content-Length': '0'} 2023-09-29T22:49:28.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b647480c2a7c0146-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.532 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.532 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.532 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.532 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.532 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.532 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.532 DEBUG:boto:Data: 2023-09-29T22:49:28.532 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.533 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.533 DEBUG:boto:Params: {} 2023-09-29T22:49:28.533 DEBUG:boto:Token: None 2023-09-29T22:49:28.533 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.533 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FqfrkaiiPvd84w3uq3+/gFd9F5Q= 2023-09-29T22:49:28.533 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FqfrkaiiPvd84w3uq3+/gFd9F5Q=', 'Content-Length': '0'} 2023-09-29T22:49:28.536 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000070cf8aac4ff61752-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:49:28.536 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.536 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.536 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.536 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.537 DEBUG:boto:Method: GET 2023-09-29T22:49:28.537 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.537 DEBUG:boto:Data: 2023-09-29T22:49:28.537 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.537 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.537 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.537 DEBUG:boto:Params: {} 2023-09-29T22:49:28.537 DEBUG:boto:Token: None 2023-09-29T22:49:28.538 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.538 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I= 2023-09-29T22:49:28.538 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I=', 'Content-Length': '0'} 2023-09-29T22:49:28.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000963f80d1ed49185c-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.582 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.583 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.583 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.583 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.583 DEBUG:boto:Method: GET 2023-09-29T22:49:28.583 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.583 DEBUG:boto:Data: 2023-09-29T22:49:28.583 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.583 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.584 DEBUG:boto:Params: {} 2023-09-29T22:49:28.584 DEBUG:boto:Token: None 2023-09-29T22:49:28.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:49:28.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I= 2023-09-29T22:49:28.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rDPiQdrWE8FhAmyshBx++XOU94I=', 'Content-Length': '0'} 2023-09-29T22:49:28.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047a8fd54307c8661-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.588 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.588 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.588 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.588 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.588 DEBUG:boto:Method: GET 2023-09-29T22:49:28.588 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.588 DEBUG:boto:Data: 2023-09-29T22:49:28.589 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.589 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.589 DEBUG:boto:Params: {} 2023-09-29T22:49:28.589 DEBUG:boto:Token: None 2023-09-29T22:49:28.589 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.594 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096b3a9e3ffe12e07-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.594 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.594 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.595 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.595 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.595 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.595 DEBUG:boto:Method: GET 2023-09-29T22:49:28.595 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.595 DEBUG:boto:Data: 2023-09-29T22:49:28.595 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.595 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.596 DEBUG:boto:Params: {} 2023-09-29T22:49:28.596 DEBUG:boto:Token: None 2023-09-29T22:49:28.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.603 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002a4626a19e0c7dd7-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.604 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.604 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.604 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.604 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.605 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:49:28.605 DEBUG:boto:Data: 2023-09-29T22:49:28.605 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.605 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.605 DEBUG:boto:Params: {} 2023-09-29T22:49:28.605 DEBUG:boto:Token: None 2023-09-29T22:49:28.605 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44 2023-09-29T22:49:28.606 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FaXGA4fec8fLAJwf0nmkLA8SuNE= 2023-09-29T22:49:28.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FaXGA4fec8fLAJwf0nmkLA8SuNE=', 'Content-Length': '0'} 2023-09-29T22:49:28.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044f2f911c3b140e0-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.608 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.608 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.608 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.608 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:49:28.608 DEBUG:boto:Data: 2023-09-29T22:49:28.609 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.609 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.609 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.609 DEBUG:boto:Params: {} 2023-09-29T22:49:28.609 DEBUG:boto:Token: None 2023-09-29T22:49:28.609 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44 2023-09-29T22:49:28.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FaXGA4fec8fLAJwf0nmkLA8SuNE= 2023-09-29T22:49:28.610 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FaXGA4fec8fLAJwf0nmkLA8SuNE=', 'Content-Length': '0'} 2023-09-29T22:49:28.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db11ec9d7f5fd26-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:49:28.613 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.613 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.614 DEBUG:boto:Method: GET 2023-09-29T22:49:28.614 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:49:28.614 DEBUG:boto:Data: 2023-09-29T22:49:28.614 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.614 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.614 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.614 DEBUG:boto:Params: {} 2023-09-29T22:49:28.614 DEBUG:boto:Token: None 2023-09-29T22:49:28.615 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44 2023-09-29T22:49:28.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ewh+F8LtyC02GLK9NZYmJH3jkTI= 2023-09-29T22:49:28.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ewh+F8LtyC02GLK9NZYmJH3jkTI=', 'Content-Length': '0'} 2023-09-29T22:49:28.617 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8de9531b6b335ce-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.617 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:49:28.617 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:49:28.617 DEBUG:boto:Method: GET 2023-09-29T22:49:28.617 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:49:28.618 DEBUG:boto:Data: 2023-09-29T22:49:28.618 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.618 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.618 DEBUG:boto:Params: {} 2023-09-29T22:49:28.618 DEBUG:boto:Token: None 2023-09-29T22:49:28.618 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj44 2023-09-29T22:49:28.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ewh+F8LtyC02GLK9NZYmJH3jkTI= 2023-09-29T22:49:28.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ewh+F8LtyC02GLK9NZYmJH3jkTI=', 'Content-Length': '0'} 2023-09-29T22:49:28.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ff8e5fe9ba45b28-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.662 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.663 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.663 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.663 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.663 DEBUG:boto:Method: GET 2023-09-29T22:49:28.663 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.663 DEBUG:boto:Data: 2023-09-29T22:49:28.663 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.663 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.664 DEBUG:boto:Params: {} 2023-09-29T22:49:28.664 DEBUG:boto:Token: None 2023-09-29T22:49:28.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.669 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000063b204a6e9d83a21-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.669 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.669 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.669 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.670 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.670 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.670 DEBUG:boto:Method: GET 2023-09-29T22:49:28.670 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:49:28.670 DEBUG:boto:Data: 2023-09-29T22:49:28.670 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.670 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.670 DEBUG:boto:Params: {} 2023-09-29T22:49:28.671 DEBUG:boto:Token: None 2023-09-29T22:49:28.671 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.671 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.678 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd5d8f8b9d82f1e6-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.679 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:49:28.679 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.679 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.680 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.680 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.680 DEBUG:boto:Method: GET 2023-09-29T22:49:28.680 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.680 DEBUG:boto:Data: 2023-09-29T22:49:28.680 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.680 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.680 DEBUG:boto:Params: {} 2023-09-29T22:49:28.680 DEBUG:boto:Token: None 2023-09-29T22:49:28.681 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY= 2023-09-29T22:49:28.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY=', 'Content-Length': '0'} 2023-09-29T22:49:28.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008dcf196477de218d-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.726 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.727 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.727 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.727 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.727 DEBUG:boto:Method: GET 2023-09-29T22:49:28.727 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.727 DEBUG:boto:Data: 2023-09-29T22:49:28.727 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.727 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.727 DEBUG:boto:Params: {} 2023-09-29T22:49:28.728 DEBUG:boto:Token: None 2023-09-29T22:49:28.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY= 2023-09-29T22:49:28.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY=', 'Content-Length': '0'} 2023-09-29T22:49:28.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055b074ef10d97893-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.732 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.732 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.732 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.733 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.733 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.733 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.733 DEBUG:boto:Data: 2023-09-29T22:49:28.734 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.734 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.734 DEBUG:boto:Params: {} 2023-09-29T22:49:28.734 DEBUG:boto:Token: None 2023-09-29T22:49:28.735 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E2KlhwRus1NZrd5tFWdxIk7yJH4= 2023-09-29T22:49:28.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E2KlhwRus1NZrd5tFWdxIk7yJH4=', 'Content-Length': '0'} 2023-09-29T22:49:28.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bab05b2735a25c9a-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.738 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.739 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.739 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.739 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.739 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.739 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.739 DEBUG:boto:Data: 2023-09-29T22:49:28.739 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.740 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.740 DEBUG:boto:Params: {} 2023-09-29T22:49:28.740 DEBUG:boto:Token: None 2023-09-29T22:49:28.740 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E2KlhwRus1NZrd5tFWdxIk7yJH4= 2023-09-29T22:49:28.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E2KlhwRus1NZrd5tFWdxIk7yJH4=', 'Content-Length': '0'} 2023-09-29T22:49:28.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f67ab6058aa13d0c-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:49:28.743 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.743 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.743 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.743 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.743 DEBUG:boto:Method: GET 2023-09-29T22:49:28.743 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.744 DEBUG:boto:Data: 2023-09-29T22:49:28.744 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.744 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.744 DEBUG:boto:Params: {} 2023-09-29T22:49:28.744 DEBUG:boto:Token: None 2023-09-29T22:49:28.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY= 2023-09-29T22:49:28.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY=', 'Content-Length': '0'} 2023-09-29T22:49:28.746 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c0645cb2fc6712-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.747 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.747 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.747 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.747 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.747 DEBUG:boto:Method: GET 2023-09-29T22:49:28.747 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.748 DEBUG:boto:Data: 2023-09-29T22:49:28.748 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.748 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.748 DEBUG:boto:Params: {} 2023-09-29T22:49:28.748 DEBUG:boto:Token: None 2023-09-29T22:49:28.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:49:28.749 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY= 2023-09-29T22:49:28.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4OPBOFxCqPlHn42D0WClWYijtY=', 'Content-Length': '0'} 2023-09-29T22:49:28.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f4506fd5c881f80-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.794 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.795 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.795 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.795 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.796 DEBUG:boto:Method: GET 2023-09-29T22:49:28.796 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.796 DEBUG:boto:Data: 2023-09-29T22:49:28.797 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.797 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.797 DEBUG:boto:Params: {} 2023-09-29T22:49:28.798 DEBUG:boto:Token: None 2023-09-29T22:49:28.798 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.804 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a14638130d00a11b-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.804 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.804 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.805 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.805 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.805 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.805 DEBUG:boto:Method: GET 2023-09-29T22:49:28.806 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.806 DEBUG:boto:Data: 2023-09-29T22:49:28.806 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.806 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.806 DEBUG:boto:Params: {} 2023-09-29T22:49:28.806 DEBUG:boto:Token: None 2023-09-29T22:49:28.807 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.814 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091a32d72196ed9e1-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.814 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.814 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.815 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.815 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.815 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:49:28.815 DEBUG:boto:Data: 2023-09-29T22:49:28.815 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.815 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.815 DEBUG:boto:Params: {} 2023-09-29T22:49:28.815 DEBUG:boto:Token: None 2023-09-29T22:49:28.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46 2023-09-29T22:49:28.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oR6yaWVRN6zz5Csbn5Mdqyxao14= 2023-09-29T22:49:28.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oR6yaWVRN6zz5Csbn5Mdqyxao14=', 'Content-Length': '0'} 2023-09-29T22:49:28.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a25ae15d6f9a6f9a-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.818 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.818 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.819 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.819 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:49:28.819 DEBUG:boto:Data: 2023-09-29T22:49:28.819 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.819 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.819 DEBUG:boto:Params: {} 2023-09-29T22:49:28.819 DEBUG:boto:Token: None 2023-09-29T22:49:28.819 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46 2023-09-29T22:49:28.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oR6yaWVRN6zz5Csbn5Mdqyxao14= 2023-09-29T22:49:28.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oR6yaWVRN6zz5Csbn5Mdqyxao14=', 'Content-Length': '0'} 2023-09-29T22:49:28.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068bb398894912ea0-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:49:28.822 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.822 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.823 DEBUG:boto:Method: GET 2023-09-29T22:49:28.823 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:49:28.823 DEBUG:boto:Data: 2023-09-29T22:49:28.823 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.823 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.823 DEBUG:boto:Params: {} 2023-09-29T22:49:28.823 DEBUG:boto:Token: None 2023-09-29T22:49:28.823 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46 2023-09-29T22:49:28.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FWv4ADoohAid2TZ1QWo4X1Kd+rQ= 2023-09-29T22:49:28.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FWv4ADoohAid2TZ1QWo4X1Kd+rQ=', 'Content-Length': '0'} 2023-09-29T22:49:28.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006373cf27827cc62c-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.826 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:49:28.826 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:49:28.826 DEBUG:boto:Method: GET 2023-09-29T22:49:28.826 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:49:28.827 DEBUG:boto:Data: 2023-09-29T22:49:28.827 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.827 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.827 DEBUG:boto:Params: {} 2023-09-29T22:49:28.827 DEBUG:boto:Token: None 2023-09-29T22:49:28.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj46 2023-09-29T22:49:28.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FWv4ADoohAid2TZ1QWo4X1Kd+rQ= 2023-09-29T22:49:28.828 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FWv4ADoohAid2TZ1QWo4X1Kd+rQ=', 'Content-Length': '0'} 2023-09-29T22:49:28.830 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009594cdd9625c3258-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.830 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.831 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.831 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.831 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.831 DEBUG:boto:Method: GET 2023-09-29T22:49:28.831 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.831 DEBUG:boto:Data: 2023-09-29T22:49:28.832 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.832 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.832 DEBUG:boto:Params: {} 2023-09-29T22:49:28.833 DEBUG:boto:Token: None 2023-09-29T22:49:28.833 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.833 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.838 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068f475656ebe93bd-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.838 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.839 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.839 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.839 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.839 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.839 DEBUG:boto:Method: GET 2023-09-29T22:49:28.839 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:49:28.839 DEBUG:boto:Data: 2023-09-29T22:49:28.839 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.840 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.840 DEBUG:boto:Params: {} 2023-09-29T22:49:28.840 DEBUG:boto:Token: None 2023-09-29T22:49:28.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.848 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd28dfe69866bcfe-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.848 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:49:28.848 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.849 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.849 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.849 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.849 DEBUG:boto:Method: GET 2023-09-29T22:49:28.849 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.849 DEBUG:boto:Data: 2023-09-29T22:49:28.849 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.849 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.850 DEBUG:boto:Params: {} 2023-09-29T22:49:28.850 DEBUG:boto:Token: None 2023-09-29T22:49:28.850 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU= 2023-09-29T22:49:28.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU=', 'Content-Length': '0'} 2023-09-29T22:49:28.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb17922a849fc310-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.894 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.895 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.895 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.895 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.895 DEBUG:boto:Method: GET 2023-09-29T22:49:28.895 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.895 DEBUG:boto:Data: 2023-09-29T22:49:28.895 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.895 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.896 DEBUG:boto:Params: {} 2023-09-29T22:49:28.896 DEBUG:boto:Token: None 2023-09-29T22:49:28.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU= 2023-09-29T22:49:28.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU=', 'Content-Length': '0'} 2023-09-29T22:49:28.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c01ba7efaef16f45-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.899 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.899 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.899 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.900 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.900 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.900 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.900 DEBUG:boto:Data: 2023-09-29T22:49:28.900 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.900 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.900 DEBUG:boto:Params: {} 2023-09-29T22:49:28.900 DEBUG:boto:Token: None 2023-09-29T22:49:28.901 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vgRmSPu96mHyAqH7oMArAhP4++U= 2023-09-29T22:49:28.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vgRmSPu96mHyAqH7oMArAhP4++U=', 'Content-Length': '0'} 2023-09-29T22:49:28.903 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f84ee138242244c-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.903 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.903 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.904 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.904 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.904 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.904 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.904 DEBUG:boto:Data: 2023-09-29T22:49:28.904 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.904 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.904 DEBUG:boto:Params: {} 2023-09-29T22:49:28.904 DEBUG:boto:Token: None 2023-09-29T22:49:28.905 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vgRmSPu96mHyAqH7oMArAhP4++U= 2023-09-29T22:49:28.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vgRmSPu96mHyAqH7oMArAhP4++U=', 'Content-Length': '0'} 2023-09-29T22:49:28.907 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b322e7a2a4b9376-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:49:28.908 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.908 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.908 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.908 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.908 DEBUG:boto:Method: GET 2023-09-29T22:49:28.909 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.909 DEBUG:boto:Data: 2023-09-29T22:49:28.909 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.909 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.909 DEBUG:boto:Params: {} 2023-09-29T22:49:28.909 DEBUG:boto:Token: None 2023-09-29T22:49:28.909 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.909 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU= 2023-09-29T22:49:28.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU=', 'Content-Length': '0'} 2023-09-29T22:49:28.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d8b3618a6ec7c1f-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.912 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.913 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.913 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.913 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.913 DEBUG:boto:Method: GET 2023-09-29T22:49:28.913 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.913 DEBUG:boto:Data: 2023-09-29T22:49:28.913 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.913 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.913 DEBUG:boto:Params: {} 2023-09-29T22:49:28.914 DEBUG:boto:Token: None 2023-09-29T22:49:28.914 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:49:28.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU= 2023-09-29T22:49:28.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N30mLlKG5j+vgUNegiIWgUwpiDU=', 'Content-Length': '0'} 2023-09-29T22:49:28.917 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e098538c5ecc82fa-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.958 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.959 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.959 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:28.959 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:28.959 DEBUG:boto:Method: GET 2023-09-29T22:49:28.959 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:49:28.959 DEBUG:boto:Data: 2023-09-29T22:49:28.959 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.959 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.960 DEBUG:boto:Params: {} 2023-09-29T22:49:28.960 DEBUG:boto:Token: None 2023-09-29T22:49:28.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.965 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009757f18740068e73-0065175478-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.965 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.966 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:28.966 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:28.966 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:28.966 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:28.966 DEBUG:boto:Method: GET 2023-09-29T22:49:28.966 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:49:28.966 DEBUG:boto:Data: 2023-09-29T22:49:28.966 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.967 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.967 DEBUG:boto:Params: {} 2023-09-29T22:49:28.967 DEBUG:boto:Token: None 2023-09-29T22:49:28.967 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/ 2023-09-29T22:49:28.967 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk= 2023-09-29T22:49:28.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bcj0gqoyT+Q+Yq6jqkGgw5EWquk=', 'Content-Length': '0'} 2023-09-29T22:49:28.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001034a68fa073e640-0065175478-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.974 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:28.974 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.974 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.974 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.975 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:49:28.975 DEBUG:boto:Data: 2023-09-29T22:49:28.975 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.975 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.975 DEBUG:boto:Params: {} 2023-09-29T22:49:28.975 DEBUG:boto:Token: None 2023-09-29T22:49:28.975 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51 2023-09-29T22:49:28.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2DTFTydCwoGz2T4fxZZ66IviSAs= 2023-09-29T22:49:28.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2DTFTydCwoGz2T4fxZZ66IviSAs=', 'Content-Length': '0'} 2023-09-29T22:49:28.978 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a910ea5a8a50fc-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.978 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.978 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.978 DEBUG:boto:Method: HEAD 2023-09-29T22:49:28.978 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:49:28.978 DEBUG:boto:Data: 2023-09-29T22:49:28.979 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.979 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:28.979 DEBUG:boto:Port: 8000 2023-09-29T22:49:28.979 DEBUG:boto:Params: {} 2023-09-29T22:49:28.979 DEBUG:boto:Token: None 2023-09-29T22:49:28.979 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51 2023-09-29T22:49:28.979 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2DTFTydCwoGz2T4fxZZ66IviSAs= 2023-09-29T22:49:28.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2DTFTydCwoGz2T4fxZZ66IviSAs=', 'Content-Length': '0'} 2023-09-29T22:49:28.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085533e5f22fcf8c2-0065175478-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:28.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:49:28.982 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:28.982 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:28.982 DEBUG:boto:Method: GET 2023-09-29T22:49:28.982 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:49:28.983 DEBUG:boto:Data: 2023-09-29T22:49:28.983 DEBUG:boto:Headers: {} 2023-09-29T22:49:28.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:28.983 DEBUG:boto:Port: 8001 2023-09-29T22:49:28.983 DEBUG:boto:Params: {} 2023-09-29T22:49:28.983 DEBUG:boto:Token: None 2023-09-29T22:49:28.983 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:28 GMT /hcnphz-59/obj51 2023-09-29T22:49:28.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hIIJKrJjHOCAASCo9CHDqT+6p/s= 2023-09-29T22:49:28.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hIIJKrJjHOCAASCo9CHDqT+6p/s=', 'Content-Length': '0'} 2023-09-29T22:49:28.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1007203647c88da-0065175478-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.026 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:49:29.027 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:49:29.027 DEBUG:boto:Method: GET 2023-09-29T22:49:29.027 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:49:29.027 DEBUG:boto:Data: 2023-09-29T22:49:29.027 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.027 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.027 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.027 DEBUG:boto:Params: {} 2023-09-29T22:49:29.028 DEBUG:boto:Token: None 2023-09-29T22:49:29.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj51 2023-09-29T22:49:29.028 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5Fbhm3P8BedubkRliMxkXszJaHI= 2023-09-29T22:49:29.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5Fbhm3P8BedubkRliMxkXszJaHI=', 'Content-Length': '0'} 2023-09-29T22:49:29.031 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad61360ddcc58914-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.032 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.032 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.032 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:29.032 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:29.032 DEBUG:boto:Method: GET 2023-09-29T22:49:29.032 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:49:29.032 DEBUG:boto:Data: 2023-09-29T22:49:29.032 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.033 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.033 DEBUG:boto:Params: {} 2023-09-29T22:49:29.033 DEBUG:boto:Token: None 2023-09-29T22:49:29.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.033 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.038 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038215971ca69edd3-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.039 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.039 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.039 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.039 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:29.040 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:49:29.040 DEBUG:boto:Method: GET 2023-09-29T22:49:29.040 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:49:29.040 DEBUG:boto:Data: 2023-09-29T22:49:29.040 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.040 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.040 DEBUG:boto:Params: {} 2023-09-29T22:49:29.040 DEBUG:boto:Token: None 2023-09-29T22:49:29.041 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.048 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000099ffb64bde35ff79-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.090 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:49:29.091 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.091 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.091 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.091 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.091 DEBUG:boto:Method: GET 2023-09-29T22:49:29.092 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.092 DEBUG:boto:Data: 2023-09-29T22:49:29.092 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.092 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.092 DEBUG:boto:Params: {} 2023-09-29T22:49:29.092 DEBUG:boto:Token: None 2023-09-29T22:49:29.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k= 2023-09-29T22:49:29.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k=', 'Content-Length': '0'} 2023-09-29T22:49:29.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071a196f708f7c532-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.096 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.096 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.096 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.097 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.097 DEBUG:boto:Method: GET 2023-09-29T22:49:29.097 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.097 DEBUG:boto:Data: 2023-09-29T22:49:29.097 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.097 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.097 DEBUG:boto:Params: {} 2023-09-29T22:49:29.097 DEBUG:boto:Token: None 2023-09-29T22:49:29.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k= 2023-09-29T22:49:29.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k=', 'Content-Length': '0'} 2023-09-29T22:49:29.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f06d7e490749a4da-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.101 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.101 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.101 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.101 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.102 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.102 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.102 DEBUG:boto:Data: 2023-09-29T22:49:29.102 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.102 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.102 DEBUG:boto:Params: {} 2023-09-29T22:49:29.102 DEBUG:boto:Token: None 2023-09-29T22:49:29.102 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.103 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rqa9oQdHXVEWV8RcFnEQ6YTCzLY= 2023-09-29T22:49:29.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rqa9oQdHXVEWV8RcFnEQ6YTCzLY=', 'Content-Length': '0'} 2023-09-29T22:49:29.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048f6171804f5465f-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.105 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.106 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.106 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.106 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.106 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.106 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.107 DEBUG:boto:Data: 2023-09-29T22:49:29.107 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.107 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.107 DEBUG:boto:Params: {} 2023-09-29T22:49:29.108 DEBUG:boto:Token: None 2023-09-29T22:49:29.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rqa9oQdHXVEWV8RcFnEQ6YTCzLY= 2023-09-29T22:49:29.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rqa9oQdHXVEWV8RcFnEQ6YTCzLY=', 'Content-Length': '0'} 2023-09-29T22:49:29.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002515ec0bf02108ce-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:49:29.112 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.112 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.112 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.112 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.113 DEBUG:boto:Method: GET 2023-09-29T22:49:29.113 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.113 DEBUG:boto:Data: 2023-09-29T22:49:29.113 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.113 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.113 DEBUG:boto:Params: {} 2023-09-29T22:49:29.113 DEBUG:boto:Token: None 2023-09-29T22:49:29.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k= 2023-09-29T22:49:29.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k=', 'Content-Length': '0'} 2023-09-29T22:49:29.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000dc794759ca95a3f-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.117 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.117 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.117 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.117 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.117 DEBUG:boto:Method: GET 2023-09-29T22:49:29.117 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.117 DEBUG:boto:Data: 2023-09-29T22:49:29.117 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.118 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.118 DEBUG:boto:Params: {} 2023-09-29T22:49:29.118 DEBUG:boto:Token: None 2023-09-29T22:49:29.118 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:49:29.118 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k= 2023-09-29T22:49:29.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGcpJgiQJH0+kPnTaWi6U+4ap/k=', 'Content-Length': '0'} 2023-09-29T22:49:29.121 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088157c43d4cecd23-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.121 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.121 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.121 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.121 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.122 DEBUG:boto:Method: GET 2023-09-29T22:49:29.122 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.122 DEBUG:boto:Data: 2023-09-29T22:49:29.122 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.122 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.122 DEBUG:boto:Params: {} 2023-09-29T22:49:29.122 DEBUG:boto:Token: None 2023-09-29T22:49:29.122 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.127 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e95ecf4ef9811b59-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.170 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.171 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.171 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.171 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.171 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.171 DEBUG:boto:Method: GET 2023-09-29T22:49:29.172 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.172 DEBUG:boto:Data: 2023-09-29T22:49:29.172 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.172 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.172 DEBUG:boto:Params: {} 2023-09-29T22:49:29.172 DEBUG:boto:Token: None 2023-09-29T22:49:29.172 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.172 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.181 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa5e04c0db9edebb-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.181 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.182 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.182 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.182 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.182 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:49:29.182 DEBUG:boto:Data: 2023-09-29T22:49:29.182 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.182 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.183 DEBUG:boto:Params: {} 2023-09-29T22:49:29.183 DEBUG:boto:Token: None 2023-09-29T22:49:29.183 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55 2023-09-29T22:49:29.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4tKiNY2O7r5gvGQ7Lpqo++FKV7I= 2023-09-29T22:49:29.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4tKiNY2O7r5gvGQ7Lpqo++FKV7I=', 'Content-Length': '0'} 2023-09-29T22:49:29.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c9f894aa95fd229-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.186 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.186 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.186 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.186 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:49:29.186 DEBUG:boto:Data: 2023-09-29T22:49:29.186 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.186 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.186 DEBUG:boto:Params: {} 2023-09-29T22:49:29.187 DEBUG:boto:Token: None 2023-09-29T22:49:29.187 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55 2023-09-29T22:49:29.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4tKiNY2O7r5gvGQ7Lpqo++FKV7I= 2023-09-29T22:49:29.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4tKiNY2O7r5gvGQ7Lpqo++FKV7I=', 'Content-Length': '0'} 2023-09-29T22:49:29.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000016a4489b1ca4a6ad-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:49:29.190 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.190 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.190 DEBUG:boto:Method: GET 2023-09-29T22:49:29.191 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:49:29.191 DEBUG:boto:Data: 2023-09-29T22:49:29.191 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.191 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.191 DEBUG:boto:Params: {} 2023-09-29T22:49:29.192 DEBUG:boto:Token: None 2023-09-29T22:49:29.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55 2023-09-29T22:49:29.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ADZb5HNxkQ8PhxYeQrd4Y3gAqXg= 2023-09-29T22:49:29.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ADZb5HNxkQ8PhxYeQrd4Y3gAqXg=', 'Content-Length': '0'} 2023-09-29T22:49:29.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5032cfbf2f2bbb4-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.195 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:49:29.195 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:49:29.195 DEBUG:boto:Method: GET 2023-09-29T22:49:29.196 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:49:29.196 DEBUG:boto:Data: 2023-09-29T22:49:29.196 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.196 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.196 DEBUG:boto:Params: {} 2023-09-29T22:49:29.197 DEBUG:boto:Token: None 2023-09-29T22:49:29.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj55 2023-09-29T22:49:29.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ADZb5HNxkQ8PhxYeQrd4Y3gAqXg= 2023-09-29T22:49:29.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ADZb5HNxkQ8PhxYeQrd4Y3gAqXg=', 'Content-Length': '0'} 2023-09-29T22:49:29.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000db15da76031ef952-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.242 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.243 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.243 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.243 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.243 DEBUG:boto:Method: GET 2023-09-29T22:49:29.243 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.243 DEBUG:boto:Data: 2023-09-29T22:49:29.243 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.243 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.244 DEBUG:boto:Params: {} 2023-09-29T22:49:29.244 DEBUG:boto:Token: None 2023-09-29T22:49:29.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009393b7fe3fdccdfc-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.249 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.249 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.249 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.250 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.250 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.250 DEBUG:boto:Method: GET 2023-09-29T22:49:29.250 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:49:29.250 DEBUG:boto:Data: 2023-09-29T22:49:29.250 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.250 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.250 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.250 DEBUG:boto:Params: {} 2023-09-29T22:49:29.251 DEBUG:boto:Token: None 2023-09-29T22:49:29.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.258 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eda28a5eddd02de-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.258 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:49:29.259 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.259 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.259 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.259 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.259 DEBUG:boto:Method: GET 2023-09-29T22:49:29.259 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.259 DEBUG:boto:Data: 2023-09-29T22:49:29.260 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.260 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.260 DEBUG:boto:Params: {} 2023-09-29T22:49:29.260 DEBUG:boto:Token: None 2023-09-29T22:49:29.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8= 2023-09-29T22:49:29.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8=', 'Content-Length': '0'} 2023-09-29T22:49:29.262 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007b7aa5d57451f71d-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.263 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.263 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.263 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.263 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.263 DEBUG:boto:Method: GET 2023-09-29T22:49:29.263 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.263 DEBUG:boto:Data: 2023-09-29T22:49:29.263 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.264 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.264 DEBUG:boto:Params: {} 2023-09-29T22:49:29.264 DEBUG:boto:Token: None 2023-09-29T22:49:29.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8= 2023-09-29T22:49:29.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8=', 'Content-Length': '0'} 2023-09-29T22:49:29.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1f0bf64218a429d-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.266 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.266 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.267 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.267 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.267 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.267 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.267 DEBUG:boto:Data: 2023-09-29T22:49:29.267 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.267 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.267 DEBUG:boto:Params: {} 2023-09-29T22:49:29.268 DEBUG:boto:Token: None 2023-09-29T22:49:29.268 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KjJRudfy8+d9JvDRqpVN5sNyQNs= 2023-09-29T22:49:29.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KjJRudfy8+d9JvDRqpVN5sNyQNs=', 'Content-Length': '0'} 2023-09-29T22:49:29.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6886e90943fdf4-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.270 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.270 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.270 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.270 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.271 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.271 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.271 DEBUG:boto:Data: 2023-09-29T22:49:29.271 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.271 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.271 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.271 DEBUG:boto:Params: {} 2023-09-29T22:49:29.271 DEBUG:boto:Token: None 2023-09-29T22:49:29.272 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.272 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KjJRudfy8+d9JvDRqpVN5sNyQNs= 2023-09-29T22:49:29.272 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KjJRudfy8+d9JvDRqpVN5sNyQNs=', 'Content-Length': '0'} 2023-09-29T22:49:29.274 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b4ed3d43b3163c6-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:49:29.274 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.274 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.274 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.274 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.274 DEBUG:boto:Method: GET 2023-09-29T22:49:29.275 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.275 DEBUG:boto:Data: 2023-09-29T22:49:29.275 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.275 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.275 DEBUG:boto:Params: {} 2023-09-29T22:49:29.275 DEBUG:boto:Token: None 2023-09-29T22:49:29.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8= 2023-09-29T22:49:29.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8=', 'Content-Length': '0'} 2023-09-29T22:49:29.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000420d50b9e8298c77-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.318 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.318 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.319 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.319 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.319 DEBUG:boto:Method: GET 2023-09-29T22:49:29.319 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.319 DEBUG:boto:Data: 2023-09-29T22:49:29.319 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.319 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.319 DEBUG:boto:Params: {} 2023-09-29T22:49:29.319 DEBUG:boto:Token: None 2023-09-29T22:49:29.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:49:29.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8= 2023-09-29T22:49:29.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:T0/ukj49DeBU6XjyEnGcz+zRxO8=', 'Content-Length': '0'} 2023-09-29T22:49:29.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d76a06ab0612dbe0-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.323 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.323 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.323 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.324 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.324 DEBUG:boto:Method: GET 2023-09-29T22:49:29.324 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.324 DEBUG:boto:Data: 2023-09-29T22:49:29.324 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.324 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.324 DEBUG:boto:Params: {} 2023-09-29T22:49:29.324 DEBUG:boto:Token: None 2023-09-29T22:49:29.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e5bb3a2636e3fec3-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.330 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.330 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.330 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.330 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.330 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.331 DEBUG:boto:Method: GET 2023-09-29T22:49:29.331 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.331 DEBUG:boto:Data: 2023-09-29T22:49:29.331 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.331 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.331 DEBUG:boto:Params: {} 2023-09-29T22:49:29.331 DEBUG:boto:Token: None 2023-09-29T22:49:29.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f97de2f3898a06ef-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.341 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.341 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.342 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.342 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.342 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:49:29.342 DEBUG:boto:Data: 2023-09-29T22:49:29.342 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.342 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.342 DEBUG:boto:Params: {} 2023-09-29T22:49:29.343 DEBUG:boto:Token: None 2023-09-29T22:49:29.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57 2023-09-29T22:49:29.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0DaWiVWX7Pf45uqG23+uCZQbSB4= 2023-09-29T22:49:29.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0DaWiVWX7Pf45uqG23+uCZQbSB4=', 'Content-Length': '0'} 2023-09-29T22:49:29.345 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007402a2a6e2003883-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.345 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.345 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.345 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.346 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:49:29.346 DEBUG:boto:Data: 2023-09-29T22:49:29.346 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.346 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.346 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.346 DEBUG:boto:Params: {} 2023-09-29T22:49:29.346 DEBUG:boto:Token: None 2023-09-29T22:49:29.346 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57 2023-09-29T22:49:29.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0DaWiVWX7Pf45uqG23+uCZQbSB4= 2023-09-29T22:49:29.347 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0DaWiVWX7Pf45uqG23+uCZQbSB4=', 'Content-Length': '0'} 2023-09-29T22:49:29.348 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008eee4d5e427bd552-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:49:29.349 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.349 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.349 DEBUG:boto:Method: GET 2023-09-29T22:49:29.349 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:49:29.349 DEBUG:boto:Data: 2023-09-29T22:49:29.350 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.350 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.350 DEBUG:boto:Params: {} 2023-09-29T22:49:29.350 DEBUG:boto:Token: None 2023-09-29T22:49:29.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57 2023-09-29T22:49:29.350 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m0t+5ETQQgtxG1j5UTuM9/Ji3F4= 2023-09-29T22:49:29.350 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m0t+5ETQQgtxG1j5UTuM9/Ji3F4=', 'Content-Length': '0'} 2023-09-29T22:49:29.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006afe5ea817f90a8d-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.353 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:49:29.353 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:49:29.353 DEBUG:boto:Method: GET 2023-09-29T22:49:29.353 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:49:29.353 DEBUG:boto:Data: 2023-09-29T22:49:29.353 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.354 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.354 DEBUG:boto:Params: {} 2023-09-29T22:49:29.354 DEBUG:boto:Token: None 2023-09-29T22:49:29.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj57 2023-09-29T22:49:29.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m0t+5ETQQgtxG1j5UTuM9/Ji3F4= 2023-09-29T22:49:29.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m0t+5ETQQgtxG1j5UTuM9/Ji3F4=', 'Content-Length': '0'} 2023-09-29T22:49:29.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4586427bcd0e646-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.356 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.357 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.357 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.357 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.357 DEBUG:boto:Method: GET 2023-09-29T22:49:29.357 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.357 DEBUG:boto:Data: 2023-09-29T22:49:29.357 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.357 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.358 DEBUG:boto:Params: {} 2023-09-29T22:49:29.358 DEBUG:boto:Token: None 2023-09-29T22:49:29.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.361 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e03c0a718abc75b-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.362 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.362 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.362 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.362 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.362 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.363 DEBUG:boto:Method: GET 2023-09-29T22:49:29.363 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:49:29.363 DEBUG:boto:Data: 2023-09-29T22:49:29.363 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.363 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.364 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.364 DEBUG:boto:Params: {} 2023-09-29T22:49:29.364 DEBUG:boto:Token: None 2023-09-29T22:49:29.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bf184f40ada9f06-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.372 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:49:29.372 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.372 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.372 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.372 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.373 DEBUG:boto:Method: GET 2023-09-29T22:49:29.373 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.373 DEBUG:boto:Data: 2023-09-29T22:49:29.373 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.373 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.373 DEBUG:boto:Params: {} 2023-09-29T22:49:29.373 DEBUG:boto:Token: None 2023-09-29T22:49:29.373 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.374 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0= 2023-09-29T22:49:29.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0=', 'Content-Length': '0'} 2023-09-29T22:49:29.376 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9343a9f831bffe3-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.376 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.376 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.377 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.377 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.377 DEBUG:boto:Method: GET 2023-09-29T22:49:29.377 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.377 DEBUG:boto:Data: 2023-09-29T22:49:29.377 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.377 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.377 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.377 DEBUG:boto:Params: {} 2023-09-29T22:49:29.377 DEBUG:boto:Token: None 2023-09-29T22:49:29.378 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.378 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0= 2023-09-29T22:49:29.378 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0=', 'Content-Length': '0'} 2023-09-29T22:49:29.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4b75dfa521bc497-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.426 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.427 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.427 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.427 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.427 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.427 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.427 DEBUG:boto:Data: 2023-09-29T22:49:29.427 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.428 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.428 DEBUG:boto:Params: {} 2023-09-29T22:49:29.428 DEBUG:boto:Token: None 2023-09-29T22:49:29.428 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXa/3/KE1BWxkGInpBloiDG0V3Q= 2023-09-29T22:49:29.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXa/3/KE1BWxkGInpBloiDG0V3Q=', 'Content-Length': '0'} 2023-09-29T22:49:29.434 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfec60146f81157a-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.435 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.435 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.435 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.435 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.435 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.436 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.436 DEBUG:boto:Data: 2023-09-29T22:49:29.436 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.436 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.436 DEBUG:boto:Params: {} 2023-09-29T22:49:29.436 DEBUG:boto:Token: None 2023-09-29T22:49:29.436 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.437 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXa/3/KE1BWxkGInpBloiDG0V3Q= 2023-09-29T22:49:29.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXa/3/KE1BWxkGInpBloiDG0V3Q=', 'Content-Length': '0'} 2023-09-29T22:49:29.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539ce14572a10af3-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:49:29.440 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.440 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.440 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.440 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.441 DEBUG:boto:Method: GET 2023-09-29T22:49:29.441 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.441 DEBUG:boto:Data: 2023-09-29T22:49:29.441 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.441 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.441 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.441 DEBUG:boto:Params: {} 2023-09-29T22:49:29.441 DEBUG:boto:Token: None 2023-09-29T22:49:29.442 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.442 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0= 2023-09-29T22:49:29.442 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0=', 'Content-Length': '0'} 2023-09-29T22:49:29.445 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf783f3c61210e93-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.486 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.487 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.487 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.487 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.487 DEBUG:boto:Method: GET 2023-09-29T22:49:29.487 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.487 DEBUG:boto:Data: 2023-09-29T22:49:29.487 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.488 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.488 DEBUG:boto:Params: {} 2023-09-29T22:49:29.488 DEBUG:boto:Token: None 2023-09-29T22:49:29.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:49:29.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0= 2023-09-29T22:49:29.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1mvlTypilavpmT5/wAJSzbz+bE0=', 'Content-Length': '0'} 2023-09-29T22:49:29.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000de57c236adc3bd74-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.492 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.492 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.492 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.492 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.493 DEBUG:boto:Method: GET 2023-09-29T22:49:29.493 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.493 DEBUG:boto:Data: 2023-09-29T22:49:29.493 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.493 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.493 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.493 DEBUG:boto:Params: {} 2023-09-29T22:49:29.493 DEBUG:boto:Token: None 2023-09-29T22:49:29.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.494 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.498 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e04b5a3664f56dcf-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.498 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.499 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.499 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.499 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.499 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.499 DEBUG:boto:Method: GET 2023-09-29T22:49:29.500 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.500 DEBUG:boto:Data: 2023-09-29T22:49:29.500 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.500 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.500 DEBUG:boto:Params: {} 2023-09-29T22:49:29.501 DEBUG:boto:Token: None 2023-09-29T22:49:29.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.510 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7a8633881760587-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.510 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.511 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.511 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.511 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.511 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:49:29.511 DEBUG:boto:Data: 2023-09-29T22:49:29.511 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.512 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.512 DEBUG:boto:Params: {} 2023-09-29T22:49:29.512 DEBUG:boto:Token: None 2023-09-29T22:49:29.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58 2023-09-29T22:49:29.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qHkQae36iLsT6V3ObVN8YScFAng= 2023-09-29T22:49:29.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qHkQae36iLsT6V3ObVN8YScFAng=', 'Content-Length': '0'} 2023-09-29T22:49:29.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037d8c3f790f1089f-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.515 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.515 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.515 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.515 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:49:29.515 DEBUG:boto:Data: 2023-09-29T22:49:29.516 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.516 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.516 DEBUG:boto:Params: {} 2023-09-29T22:49:29.516 DEBUG:boto:Token: None 2023-09-29T22:49:29.516 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58 2023-09-29T22:49:29.516 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qHkQae36iLsT6V3ObVN8YScFAng= 2023-09-29T22:49:29.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qHkQae36iLsT6V3ObVN8YScFAng=', 'Content-Length': '0'} 2023-09-29T22:49:29.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041cd7695a91411ce-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:49:29.519 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.519 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.519 DEBUG:boto:Method: GET 2023-09-29T22:49:29.519 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:49:29.519 DEBUG:boto:Data: 2023-09-29T22:49:29.520 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.520 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.520 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.520 DEBUG:boto:Params: {} 2023-09-29T22:49:29.520 DEBUG:boto:Token: None 2023-09-29T22:49:29.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58 2023-09-29T22:49:29.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o8IeLJ7LHAL9n8MmmpczpniO7Tc= 2023-09-29T22:49:29.521 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o8IeLJ7LHAL9n8MmmpczpniO7Tc=', 'Content-Length': '0'} 2023-09-29T22:49:29.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fd16ba6649f82ade-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.566 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:49:29.566 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:49:29.567 DEBUG:boto:Method: GET 2023-09-29T22:49:29.567 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:49:29.567 DEBUG:boto:Data: 2023-09-29T22:49:29.567 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.567 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.567 DEBUG:boto:Params: {} 2023-09-29T22:49:29.568 DEBUG:boto:Token: None 2023-09-29T22:49:29.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj58 2023-09-29T22:49:29.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:o8IeLJ7LHAL9n8MmmpczpniO7Tc= 2023-09-29T22:49:29.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:o8IeLJ7LHAL9n8MmmpczpniO7Tc=', 'Content-Length': '0'} 2023-09-29T22:49:29.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082e8e88706529dd8-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.571 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.571 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.571 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.572 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.572 DEBUG:boto:Method: GET 2023-09-29T22:49:29.572 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.572 DEBUG:boto:Data: 2023-09-29T22:49:29.572 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.572 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.572 DEBUG:boto:Params: {} 2023-09-29T22:49:29.572 DEBUG:boto:Token: None 2023-09-29T22:49:29.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f77cc2286175d75-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.577 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.577 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.578 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.578 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.578 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.578 DEBUG:boto:Method: GET 2023-09-29T22:49:29.578 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:49:29.578 DEBUG:boto:Data: 2023-09-29T22:49:29.579 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.579 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.579 DEBUG:boto:Params: {} 2023-09-29T22:49:29.579 DEBUG:boto:Token: None 2023-09-29T22:49:29.579 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.587 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c551a807d2b326b-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.587 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:49:29.587 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.588 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.588 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.588 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.588 DEBUG:boto:Method: GET 2023-09-29T22:49:29.588 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.588 DEBUG:boto:Data: 2023-09-29T22:49:29.588 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.589 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.589 DEBUG:boto:Params: {} 2023-09-29T22:49:29.589 DEBUG:boto:Token: None 2023-09-29T22:49:29.589 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc= 2023-09-29T22:49:29.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc=', 'Content-Length': '0'} 2023-09-29T22:49:29.591 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086c1c3fb1b4e6d52-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.592 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.592 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.592 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.592 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.592 DEBUG:boto:Method: GET 2023-09-29T22:49:29.592 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.592 DEBUG:boto:Data: 2023-09-29T22:49:29.592 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.593 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.593 DEBUG:boto:Params: {} 2023-09-29T22:49:29.593 DEBUG:boto:Token: None 2023-09-29T22:49:29.593 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc= 2023-09-29T22:49:29.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc=', 'Content-Length': '0'} 2023-09-29T22:49:29.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c40a2bd1df81d09-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.596 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.596 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.596 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.596 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.596 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.596 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.597 DEBUG:boto:Data: 2023-09-29T22:49:29.597 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.597 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.597 DEBUG:boto:Params: {} 2023-09-29T22:49:29.597 DEBUG:boto:Token: None 2023-09-29T22:49:29.597 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Dz0Hj6lXaLRLa0IBvHOmoaRC9M= 2023-09-29T22:49:29.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Dz0Hj6lXaLRLa0IBvHOmoaRC9M=', 'Content-Length': '0'} 2023-09-29T22:49:29.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d95fe32f66fc69b0-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.600 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.600 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.600 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.600 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.600 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.601 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.601 DEBUG:boto:Data: 2023-09-29T22:49:29.601 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.601 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.601 DEBUG:boto:Params: {} 2023-09-29T22:49:29.601 DEBUG:boto:Token: None 2023-09-29T22:49:29.601 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Dz0Hj6lXaLRLa0IBvHOmoaRC9M= 2023-09-29T22:49:29.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Dz0Hj6lXaLRLa0IBvHOmoaRC9M=', 'Content-Length': '0'} 2023-09-29T22:49:29.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afc14cf9b846ae16-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.604 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:49:29.604 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.604 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.604 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.604 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.604 DEBUG:boto:Method: GET 2023-09-29T22:49:29.604 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.605 DEBUG:boto:Data: 2023-09-29T22:49:29.605 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.605 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.605 DEBUG:boto:Params: {} 2023-09-29T22:49:29.605 DEBUG:boto:Token: None 2023-09-29T22:49:29.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.605 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc= 2023-09-29T22:49:29.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc=', 'Content-Length': '0'} 2023-09-29T22:49:29.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058b63a9264a74770-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.654 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.655 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.655 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.655 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.655 DEBUG:boto:Method: GET 2023-09-29T22:49:29.655 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.655 DEBUG:boto:Data: 2023-09-29T22:49:29.655 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.655 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.655 DEBUG:boto:Params: {} 2023-09-29T22:49:29.656 DEBUG:boto:Token: None 2023-09-29T22:49:29.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:49:29.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc= 2023-09-29T22:49:29.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4/hy73cCVN3HC9yTeOur4VoymSc=', 'Content-Length': '0'} 2023-09-29T22:49:29.659 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc4ecb7c60eae98b-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.659 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.659 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.659 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.659 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.660 DEBUG:boto:Method: GET 2023-09-29T22:49:29.660 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.660 DEBUG:boto:Data: 2023-09-29T22:49:29.660 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.660 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.660 DEBUG:boto:Params: {} 2023-09-29T22:49:29.660 DEBUG:boto:Token: None 2023-09-29T22:49:29.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.661 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006f19228d79f41f9a-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.666 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.666 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.666 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.666 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.666 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.667 DEBUG:boto:Method: GET 2023-09-29T22:49:29.667 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.667 DEBUG:boto:Data: 2023-09-29T22:49:29.667 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.667 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.667 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.667 DEBUG:boto:Params: {} 2023-09-29T22:49:29.667 DEBUG:boto:Token: None 2023-09-29T22:49:29.667 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.675 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b438afc41a56e0b4-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.676 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.676 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.676 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.676 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.676 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:49:29.677 DEBUG:boto:Data: 2023-09-29T22:49:29.677 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.677 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.677 DEBUG:boto:Params: {} 2023-09-29T22:49:29.677 DEBUG:boto:Token: None 2023-09-29T22:49:29.677 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62 2023-09-29T22:49:29.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pJ6LnfbavKibOj746/jA0MPggVE= 2023-09-29T22:49:29.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pJ6LnfbavKibOj746/jA0MPggVE=', 'Content-Length': '0'} 2023-09-29T22:49:29.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005366574a7a41b72d-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.680 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.680 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.680 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.680 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:49:29.680 DEBUG:boto:Data: 2023-09-29T22:49:29.680 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.681 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.681 DEBUG:boto:Params: {} 2023-09-29T22:49:29.681 DEBUG:boto:Token: None 2023-09-29T22:49:29.681 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62 2023-09-29T22:49:29.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pJ6LnfbavKibOj746/jA0MPggVE= 2023-09-29T22:49:29.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pJ6LnfbavKibOj746/jA0MPggVE=', 'Content-Length': '0'} 2023-09-29T22:49:29.683 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff1f82c75f0bbbfe-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:49:29.683 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.684 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.684 DEBUG:boto:Method: GET 2023-09-29T22:49:29.684 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:49:29.684 DEBUG:boto:Data: 2023-09-29T22:49:29.684 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.684 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.684 DEBUG:boto:Params: {} 2023-09-29T22:49:29.684 DEBUG:boto:Token: None 2023-09-29T22:49:29.685 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62 2023-09-29T22:49:29.685 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k4oRcrIFHDUR0BwHPJipCbCM3yo= 2023-09-29T22:49:29.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k4oRcrIFHDUR0BwHPJipCbCM3yo=', 'Content-Length': '0'} 2023-09-29T22:49:29.687 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d36ba38fb1d31f54-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.730 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:49:29.730 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:49:29.731 DEBUG:boto:Method: GET 2023-09-29T22:49:29.731 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:49:29.731 DEBUG:boto:Data: 2023-09-29T22:49:29.731 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.731 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.731 DEBUG:boto:Params: {} 2023-09-29T22:49:29.731 DEBUG:boto:Token: None 2023-09-29T22:49:29.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj62 2023-09-29T22:49:29.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k4oRcrIFHDUR0BwHPJipCbCM3yo= 2023-09-29T22:49:29.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k4oRcrIFHDUR0BwHPJipCbCM3yo=', 'Content-Length': '0'} 2023-09-29T22:49:29.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abd39ae77371fadb-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.735 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.735 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.735 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.735 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.735 DEBUG:boto:Method: GET 2023-09-29T22:49:29.735 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.736 DEBUG:boto:Data: 2023-09-29T22:49:29.736 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.736 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.736 DEBUG:boto:Params: {} 2023-09-29T22:49:29.736 DEBUG:boto:Token: None 2023-09-29T22:49:29.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.737 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.741 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000039534d5632a85c9a-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.741 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.742 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.742 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.742 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.742 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.742 DEBUG:boto:Method: GET 2023-09-29T22:49:29.742 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:49:29.743 DEBUG:boto:Data: 2023-09-29T22:49:29.743 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.743 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.743 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.743 DEBUG:boto:Params: {} 2023-09-29T22:49:29.743 DEBUG:boto:Token: None 2023-09-29T22:49:29.743 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.743 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.751 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027f3d119507ae880-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.798 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:49:29.799 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.799 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.799 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.799 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.800 DEBUG:boto:Method: GET 2023-09-29T22:49:29.800 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.800 DEBUG:boto:Data: 2023-09-29T22:49:29.800 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.800 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.800 DEBUG:boto:Params: {} 2023-09-29T22:49:29.800 DEBUG:boto:Token: None 2023-09-29T22:49:29.800 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY= 2023-09-29T22:49:29.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY=', 'Content-Length': '0'} 2023-09-29T22:49:29.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c863dab705f022b-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.804 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.804 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.804 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.804 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.805 DEBUG:boto:Method: GET 2023-09-29T22:49:29.805 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.805 DEBUG:boto:Data: 2023-09-29T22:49:29.805 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.805 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.805 DEBUG:boto:Params: {} 2023-09-29T22:49:29.805 DEBUG:boto:Token: None 2023-09-29T22:49:29.805 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY= 2023-09-29T22:49:29.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY=', 'Content-Length': '0'} 2023-09-29T22:49:29.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7af176a919c0411-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.809 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.809 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.809 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.810 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.810 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.810 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.810 DEBUG:boto:Data: 2023-09-29T22:49:29.810 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.810 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.810 DEBUG:boto:Params: {} 2023-09-29T22:49:29.811 DEBUG:boto:Token: None 2023-09-29T22:49:29.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kwju6kgBWlvqefr8iWrYhIyfGiE= 2023-09-29T22:49:29.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kwju6kgBWlvqefr8iWrYhIyfGiE=', 'Content-Length': '0'} 2023-09-29T22:49:29.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a483cf5d04b20f-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.814 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.814 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.814 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.814 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.814 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.814 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.815 DEBUG:boto:Data: 2023-09-29T22:49:29.815 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.815 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.815 DEBUG:boto:Params: {} 2023-09-29T22:49:29.815 DEBUG:boto:Token: None 2023-09-29T22:49:29.815 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.815 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kwju6kgBWlvqefr8iWrYhIyfGiE= 2023-09-29T22:49:29.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kwju6kgBWlvqefr8iWrYhIyfGiE=', 'Content-Length': '0'} 2023-09-29T22:49:29.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d3c0b4b2d363ff-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:49:29.819 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.819 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.819 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.819 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.819 DEBUG:boto:Method: GET 2023-09-29T22:49:29.819 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.820 DEBUG:boto:Data: 2023-09-29T22:49:29.820 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.820 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.820 DEBUG:boto:Params: {} 2023-09-29T22:49:29.820 DEBUG:boto:Token: None 2023-09-29T22:49:29.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY= 2023-09-29T22:49:29.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY=', 'Content-Length': '0'} 2023-09-29T22:49:29.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006345b03a5b544cd5-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.824 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.824 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.824 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.824 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.824 DEBUG:boto:Method: GET 2023-09-29T22:49:29.824 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.824 DEBUG:boto:Data: 2023-09-29T22:49:29.824 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.825 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.825 DEBUG:boto:Params: {} 2023-09-29T22:49:29.825 DEBUG:boto:Token: None 2023-09-29T22:49:29.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:49:29.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY= 2023-09-29T22:49:29.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gmdXse8EQr0rLC8TglzA7m48/PY=', 'Content-Length': '0'} 2023-09-29T22:49:29.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052d342258bebaae4-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.828 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.829 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.829 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.829 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.829 DEBUG:boto:Method: GET 2023-09-29T22:49:29.829 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.829 DEBUG:boto:Data: 2023-09-29T22:49:29.829 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.830 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.830 DEBUG:boto:Params: {} 2023-09-29T22:49:29.830 DEBUG:boto:Token: None 2023-09-29T22:49:29.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de1c621d3a6ce958-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.878 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.879 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.879 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.879 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.879 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.879 DEBUG:boto:Method: GET 2023-09-29T22:49:29.880 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.880 DEBUG:boto:Data: 2023-09-29T22:49:29.880 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.880 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.880 DEBUG:boto:Params: {} 2023-09-29T22:49:29.880 DEBUG:boto:Token: None 2023-09-29T22:49:29.880 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.889 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cfc6d256cd312e7-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.889 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.890 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.890 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.890 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.890 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:49:29.890 DEBUG:boto:Data: 2023-09-29T22:49:29.890 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.890 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.890 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.891 DEBUG:boto:Params: {} 2023-09-29T22:49:29.891 DEBUG:boto:Token: None 2023-09-29T22:49:29.891 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68 2023-09-29T22:49:29.891 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G1Mcu8y9QnfVKaS5szJC8c9DOMk= 2023-09-29T22:49:29.891 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G1Mcu8y9QnfVKaS5szJC8c9DOMk=', 'Content-Length': '0'} 2023-09-29T22:49:29.893 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f797d616472eb9-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.893 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.893 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.894 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.894 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:49:29.894 DEBUG:boto:Data: 2023-09-29T22:49:29.894 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.894 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.894 DEBUG:boto:Params: {} 2023-09-29T22:49:29.894 DEBUG:boto:Token: None 2023-09-29T22:49:29.894 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68 2023-09-29T22:49:29.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G1Mcu8y9QnfVKaS5szJC8c9DOMk= 2023-09-29T22:49:29.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G1Mcu8y9QnfVKaS5szJC8c9DOMk=', 'Content-Length': '0'} 2023-09-29T22:49:29.897 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a220aaa214c4ec51-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:49:29.897 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.897 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.897 DEBUG:boto:Method: GET 2023-09-29T22:49:29.898 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:49:29.898 DEBUG:boto:Data: 2023-09-29T22:49:29.898 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.898 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.898 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.898 DEBUG:boto:Params: {} 2023-09-29T22:49:29.898 DEBUG:boto:Token: None 2023-09-29T22:49:29.898 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68 2023-09-29T22:49:29.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lI9gX1LiXbTR+HWolYo2B4cCoFo= 2023-09-29T22:49:29.899 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lI9gX1LiXbTR+HWolYo2B4cCoFo=', 'Content-Length': '0'} 2023-09-29T22:49:29.901 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c44e18d4b56b52d-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.901 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:49:29.901 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:49:29.901 DEBUG:boto:Method: GET 2023-09-29T22:49:29.902 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:49:29.902 DEBUG:boto:Data: 2023-09-29T22:49:29.902 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.902 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.902 DEBUG:boto:Params: {} 2023-09-29T22:49:29.902 DEBUG:boto:Token: None 2023-09-29T22:49:29.902 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj68 2023-09-29T22:49:29.903 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lI9gX1LiXbTR+HWolYo2B4cCoFo= 2023-09-29T22:49:29.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lI9gX1LiXbTR+HWolYo2B4cCoFo=', 'Content-Length': '0'} 2023-09-29T22:49:29.905 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000098f3b4f76c0f5e79-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.905 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.905 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.905 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.905 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.905 DEBUG:boto:Method: GET 2023-09-29T22:49:29.906 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.906 DEBUG:boto:Data: 2023-09-29T22:49:29.906 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.906 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.906 DEBUG:boto:Params: {} 2023-09-29T22:49:29.906 DEBUG:boto:Token: None 2023-09-29T22:49:29.906 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.906 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.910 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef2a9a3fef75a2b-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.910 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.911 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.911 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.911 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.911 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.912 DEBUG:boto:Method: GET 2023-09-29T22:49:29.912 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:49:29.912 DEBUG:boto:Data: 2023-09-29T22:49:29.912 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.912 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.912 DEBUG:boto:Params: {} 2023-09-29T22:49:29.912 DEBUG:boto:Token: None 2023-09-29T22:49:29.913 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.913 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:29.920 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cdbbb33104f742ae-0065175479-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.920 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:29.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:49:29.921 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:29.921 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:29.921 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.921 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.921 DEBUG:boto:Method: GET 2023-09-29T22:49:29.921 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.921 DEBUG:boto:Data: 2023-09-29T22:49:29.921 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.922 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.922 DEBUG:boto:Params: {} 2023-09-29T22:49:29.922 DEBUG:boto:Token: None 2023-09-29T22:49:29.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw= 2023-09-29T22:49:29.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw=', 'Content-Length': '0'} 2023-09-29T22:49:29.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000344795b7be52bf3-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.925 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:29.925 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:29.925 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.925 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.925 DEBUG:boto:Method: GET 2023-09-29T22:49:29.925 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.925 DEBUG:boto:Data: 2023-09-29T22:49:29.925 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.926 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.926 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.926 DEBUG:boto:Params: {} 2023-09-29T22:49:29.926 DEBUG:boto:Token: None 2023-09-29T22:49:29.926 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.926 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw= 2023-09-29T22:49:29.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw=', 'Content-Length': '0'} 2023-09-29T22:49:29.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000256ef2f81867cd61-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.974 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:29.974 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:29.975 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.975 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.975 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.975 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.975 DEBUG:boto:Data: 2023-09-29T22:49:29.975 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.975 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.975 DEBUG:boto:Params: {} 2023-09-29T22:49:29.976 DEBUG:boto:Token: None 2023-09-29T22:49:29.976 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZVektUjzLUlCJamDzUKAvielNUU= 2023-09-29T22:49:29.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZVektUjzLUlCJamDzUKAvielNUU=', 'Content-Length': '0'} 2023-09-29T22:49:29.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080ba32407d8bd7ba-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.979 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:29.979 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:29.980 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.980 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.980 DEBUG:boto:Method: HEAD 2023-09-29T22:49:29.980 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.980 DEBUG:boto:Data: 2023-09-29T22:49:29.980 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.980 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.980 DEBUG:boto:Params: {} 2023-09-29T22:49:29.981 DEBUG:boto:Token: None 2023-09-29T22:49:29.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZVektUjzLUlCJamDzUKAvielNUU= 2023-09-29T22:49:29.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZVektUjzLUlCJamDzUKAvielNUU=', 'Content-Length': '0'} 2023-09-29T22:49:29.984 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5598301f9ed4fe8-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:49:29.984 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:29.985 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:29.985 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.985 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.985 DEBUG:boto:Method: GET 2023-09-29T22:49:29.985 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.985 DEBUG:boto:Data: 2023-09-29T22:49:29.985 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.985 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.985 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.985 DEBUG:boto:Params: {} 2023-09-29T22:49:29.986 DEBUG:boto:Token: None 2023-09-29T22:49:29.986 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.986 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw= 2023-09-29T22:49:29.986 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw=', 'Content-Length': '0'} 2023-09-29T22:49:29.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ae15e4eb8692410-0065175479-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.989 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:29.989 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:29.990 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.990 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.990 DEBUG:boto:Method: GET 2023-09-29T22:49:29.990 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.990 DEBUG:boto:Data: 2023-09-29T22:49:29.990 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:29.990 DEBUG:boto:Port: 8000 2023-09-29T22:49:29.990 DEBUG:boto:Params: {} 2023-09-29T22:49:29.991 DEBUG:boto:Token: None 2023-09-29T22:49:29.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:49:29.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw= 2023-09-29T22:49:29.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BGbga4MjKyhW0qrQmtwqy4+tfIw=', 'Content-Length': '0'} 2023-09-29T22:49:29.994 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006595dad28c4ca8ca-0065175479-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:29.994 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:29.994 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:29.995 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:29.995 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:29.995 DEBUG:boto:Method: GET 2023-09-29T22:49:29.995 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:49:29.996 DEBUG:boto:Data: 2023-09-29T22:49:29.996 DEBUG:boto:Headers: {} 2023-09-29T22:49:29.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:29.996 DEBUG:boto:Port: 8001 2023-09-29T22:49:29.996 DEBUG:boto:Params: {} 2023-09-29T22:49:29.997 DEBUG:boto:Token: None 2023-09-29T22:49:29.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:29 GMT /hcnphz-59/ 2023-09-29T22:49:29.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk= 2023-09-29T22:49:29.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SveiKZAkhBeVT8b9FTjobAgjGUk=', 'Content-Length': '0'} 2023-09-29T22:49:30.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000bdfd449f8558938-0065175479-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.003 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.003 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.003 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.004 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.004 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.004 DEBUG:boto:Method: GET 2023-09-29T22:49:30.004 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.005 DEBUG:boto:Data: 2023-09-29T22:49:30.005 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.005 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.006 DEBUG:boto:Params: {} 2023-09-29T22:49:30.006 DEBUG:boto:Token: None 2023-09-29T22:49:30.006 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.006 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.007 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.014 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000492959ca4ee7a407-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.014 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.015 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:30.015 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:30.015 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.015 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:49:30.016 DEBUG:boto:Data: 2023-09-29T22:49:30.016 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.016 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.016 DEBUG:boto:Params: {} 2023-09-29T22:49:30.017 DEBUG:boto:Token: None 2023-09-29T22:49:30.017 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj69 2023-09-29T22:49:30.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DyOpD6FT002jm6+PA17law2aKOQ= 2023-09-29T22:49:30.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DyOpD6FT002jm6+PA17law2aKOQ=', 'Content-Length': '0'} 2023-09-29T22:49:30.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085f49ba4e45c7ffe-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.020 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:30.020 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:30.020 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.021 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:49:30.021 DEBUG:boto:Data: 2023-09-29T22:49:30.021 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.021 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.021 DEBUG:boto:Params: {} 2023-09-29T22:49:30.022 DEBUG:boto:Token: None 2023-09-29T22:49:30.022 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj69 2023-09-29T22:49:30.022 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DyOpD6FT002jm6+PA17law2aKOQ= 2023-09-29T22:49:30.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DyOpD6FT002jm6+PA17law2aKOQ=', 'Content-Length': '0'} 2023-09-29T22:49:30.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18afd99f1f5dfd1-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:49:30.025 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:30.025 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:30.025 DEBUG:boto:Method: GET 2023-09-29T22:49:30.026 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:49:30.026 DEBUG:boto:Data: 2023-09-29T22:49:30.026 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.026 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.027 DEBUG:boto:Params: {} 2023-09-29T22:49:30.027 DEBUG:boto:Token: None 2023-09-29T22:49:30.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj69 2023-09-29T22:49:30.027 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:avWdYZHO4fbDtIFnPlK/baSJoOY= 2023-09-29T22:49:30.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:avWdYZHO4fbDtIFnPlK/baSJoOY=', 'Content-Length': '0'} 2023-09-29T22:49:30.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008caaab457efc6e6b-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.074 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:49:30.075 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:49:30.075 DEBUG:boto:Method: GET 2023-09-29T22:49:30.075 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:49:30.075 DEBUG:boto:Data: 2023-09-29T22:49:30.075 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.075 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.075 DEBUG:boto:Params: {} 2023-09-29T22:49:30.076 DEBUG:boto:Token: None 2023-09-29T22:49:30.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj69 2023-09-29T22:49:30.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:avWdYZHO4fbDtIFnPlK/baSJoOY= 2023-09-29T22:49:30.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:avWdYZHO4fbDtIFnPlK/baSJoOY=', 'Content-Length': '0'} 2023-09-29T22:49:30.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000656b7d0fd212c540-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.079 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.080 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.080 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.080 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.080 DEBUG:boto:Method: GET 2023-09-29T22:49:30.080 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.080 DEBUG:boto:Data: 2023-09-29T22:49:30.080 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.080 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.081 DEBUG:boto:Params: {} 2023-09-29T22:49:30.081 DEBUG:boto:Token: None 2023-09-29T22:49:30.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.085 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b218e44e380bc03-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.086 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.086 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.086 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.086 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.087 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.087 DEBUG:boto:Method: GET 2023-09-29T22:49:30.087 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:49:30.087 DEBUG:boto:Data: 2023-09-29T22:49:30.087 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.087 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.087 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.087 DEBUG:boto:Params: {} 2023-09-29T22:49:30.088 DEBUG:boto:Token: None 2023-09-29T22:49:30.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001669ffe53896c17b-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.095 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:49:30.096 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.096 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.096 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.096 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.096 DEBUG:boto:Method: GET 2023-09-29T22:49:30.096 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.096 DEBUG:boto:Data: 2023-09-29T22:49:30.096 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.097 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.097 DEBUG:boto:Params: {} 2023-09-29T22:49:30.097 DEBUG:boto:Token: None 2023-09-29T22:49:30.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc= 2023-09-29T22:49:30.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc=', 'Content-Length': '0'} 2023-09-29T22:49:30.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e79d4a3af07765c4-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.100 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.100 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.100 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.100 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.100 DEBUG:boto:Method: GET 2023-09-29T22:49:30.100 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.101 DEBUG:boto:Data: 2023-09-29T22:49:30.101 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.101 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.101 DEBUG:boto:Params: {} 2023-09-29T22:49:30.101 DEBUG:boto:Token: None 2023-09-29T22:49:30.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.101 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc= 2023-09-29T22:49:30.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc=', 'Content-Length': '0'} 2023-09-29T22:49:30.104 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006aecc8b113264c15-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.104 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.104 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.104 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.104 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.104 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.105 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.105 DEBUG:boto:Data: 2023-09-29T22:49:30.105 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.105 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.105 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.105 DEBUG:boto:Params: {} 2023-09-29T22:49:30.105 DEBUG:boto:Token: None 2023-09-29T22:49:30.105 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.105 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7d74gPG/DQHTKEEWdnQ1uYIS+QU= 2023-09-29T22:49:30.106 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7d74gPG/DQHTKEEWdnQ1uYIS+QU=', 'Content-Length': '0'} 2023-09-29T22:49:30.108 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c7c7ba5ef441c0a-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.108 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.108 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.108 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.108 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.108 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.108 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.108 DEBUG:boto:Data: 2023-09-29T22:49:30.109 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.109 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.109 DEBUG:boto:Params: {} 2023-09-29T22:49:30.109 DEBUG:boto:Token: None 2023-09-29T22:49:30.109 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7d74gPG/DQHTKEEWdnQ1uYIS+QU= 2023-09-29T22:49:30.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7d74gPG/DQHTKEEWdnQ1uYIS+QU=', 'Content-Length': '0'} 2023-09-29T22:49:30.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bde974655c0e65d4-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:49:30.112 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.112 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.112 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.112 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.112 DEBUG:boto:Method: GET 2023-09-29T22:49:30.112 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.113 DEBUG:boto:Data: 2023-09-29T22:49:30.113 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.113 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.113 DEBUG:boto:Params: {} 2023-09-29T22:49:30.113 DEBUG:boto:Token: None 2023-09-29T22:49:30.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc= 2023-09-29T22:49:30.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc=', 'Content-Length': '0'} 2023-09-29T22:49:30.116 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d18f98618e7f5820-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.162 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.163 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.163 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.163 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.163 DEBUG:boto:Method: GET 2023-09-29T22:49:30.163 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.163 DEBUG:boto:Data: 2023-09-29T22:49:30.163 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.164 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.164 DEBUG:boto:Params: {} 2023-09-29T22:49:30.164 DEBUG:boto:Token: None 2023-09-29T22:49:30.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:49:30.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc= 2023-09-29T22:49:30.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J0ne6HmIQYE3OBM7Jss42qhI4wc=', 'Content-Length': '0'} 2023-09-29T22:49:30.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e2018bc61f1659a-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.167 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.167 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.167 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.167 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.167 DEBUG:boto:Method: GET 2023-09-29T22:49:30.168 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.168 DEBUG:boto:Data: 2023-09-29T22:49:30.168 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.168 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.168 DEBUG:boto:Params: {} 2023-09-29T22:49:30.168 DEBUG:boto:Token: None 2023-09-29T22:49:30.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.172 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f59f5b298bba6493-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.172 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.173 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.173 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.173 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.173 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.173 DEBUG:boto:Method: GET 2023-09-29T22:49:30.173 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.173 DEBUG:boto:Data: 2023-09-29T22:49:30.174 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.174 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.174 DEBUG:boto:Params: {} 2023-09-29T22:49:30.174 DEBUG:boto:Token: None 2023-09-29T22:49:30.174 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.182 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000336328ca988a220a-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.182 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.182 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.182 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.182 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.182 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:49:30.183 DEBUG:boto:Data: 2023-09-29T22:49:30.183 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.183 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.183 DEBUG:boto:Params: {} 2023-09-29T22:49:30.183 DEBUG:boto:Token: None 2023-09-29T22:49:30.183 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74 2023-09-29T22:49:30.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d58tn+EMjtLGaKUNnpMdi+xxOv4= 2023-09-29T22:49:30.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d58tn+EMjtLGaKUNnpMdi+xxOv4=', 'Content-Length': '0'} 2023-09-29T22:49:30.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000091cb35d5214bae0-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.185 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.186 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.186 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.186 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:49:30.186 DEBUG:boto:Data: 2023-09-29T22:49:30.186 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.186 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.186 DEBUG:boto:Params: {} 2023-09-29T22:49:30.186 DEBUG:boto:Token: None 2023-09-29T22:49:30.186 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74 2023-09-29T22:49:30.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d58tn+EMjtLGaKUNnpMdi+xxOv4= 2023-09-29T22:49:30.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d58tn+EMjtLGaKUNnpMdi+xxOv4=', 'Content-Length': '0'} 2023-09-29T22:49:30.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f736815a59fba45-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:49:30.189 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.189 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.189 DEBUG:boto:Method: GET 2023-09-29T22:49:30.189 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:49:30.190 DEBUG:boto:Data: 2023-09-29T22:49:30.190 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.190 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.190 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.190 DEBUG:boto:Params: {} 2023-09-29T22:49:30.190 DEBUG:boto:Token: None 2023-09-29T22:49:30.190 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74 2023-09-29T22:49:30.190 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L9Y7G2Q0YvMOlITzKha0u/wGoVo= 2023-09-29T22:49:30.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L9Y7G2Q0YvMOlITzKha0u/wGoVo=', 'Content-Length': '0'} 2023-09-29T22:49:30.192 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af84f8bdc7bbe226-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.193 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:49:30.193 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:49:30.193 DEBUG:boto:Method: GET 2023-09-29T22:49:30.193 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:49:30.193 DEBUG:boto:Data: 2023-09-29T22:49:30.194 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.194 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.194 DEBUG:boto:Params: {} 2023-09-29T22:49:30.194 DEBUG:boto:Token: None 2023-09-29T22:49:30.194 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj74 2023-09-29T22:49:30.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L9Y7G2Q0YvMOlITzKha0u/wGoVo= 2023-09-29T22:49:30.194 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L9Y7G2Q0YvMOlITzKha0u/wGoVo=', 'Content-Length': '0'} 2023-09-29T22:49:30.197 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb633401ee4de292-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.197 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.197 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.197 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.197 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.197 DEBUG:boto:Method: GET 2023-09-29T22:49:30.198 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.198 DEBUG:boto:Data: 2023-09-29T22:49:30.198 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.198 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.198 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.198 DEBUG:boto:Params: {} 2023-09-29T22:49:30.198 DEBUG:boto:Token: None 2023-09-29T22:49:30.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.199 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.199 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b90ae6fd115d0720-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.202 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.203 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.203 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.203 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.203 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.203 DEBUG:boto:Method: GET 2023-09-29T22:49:30.203 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:49:30.204 DEBUG:boto:Data: 2023-09-29T22:49:30.204 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.204 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.204 DEBUG:boto:Params: {} 2023-09-29T22:49:30.204 DEBUG:boto:Token: None 2023-09-29T22:49:30.204 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.204 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.211 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cadbf644329d5c39-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.212 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:49:30.212 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.212 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.212 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.212 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.213 DEBUG:boto:Method: GET 2023-09-29T22:49:30.213 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.213 DEBUG:boto:Data: 2023-09-29T22:49:30.213 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.213 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.213 DEBUG:boto:Params: {} 2023-09-29T22:49:30.213 DEBUG:boto:Token: None 2023-09-29T22:49:30.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc= 2023-09-29T22:49:30.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc=', 'Content-Length': '0'} 2023-09-29T22:49:30.216 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000003f02405061812-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.262 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.263 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.263 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.263 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.263 DEBUG:boto:Method: GET 2023-09-29T22:49:30.263 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.263 DEBUG:boto:Data: 2023-09-29T22:49:30.263 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.263 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.264 DEBUG:boto:Params: {} 2023-09-29T22:49:30.264 DEBUG:boto:Token: None 2023-09-29T22:49:30.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc= 2023-09-29T22:49:30.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc=', 'Content-Length': '0'} 2023-09-29T22:49:30.267 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d97d40fdf693548-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.267 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.267 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.267 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.268 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.268 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.268 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.268 DEBUG:boto:Data: 2023-09-29T22:49:30.268 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.268 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.268 DEBUG:boto:Params: {} 2023-09-29T22:49:30.268 DEBUG:boto:Token: None 2023-09-29T22:49:30.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H48r/5A8MoFRt5emigoQETSsqj0= 2023-09-29T22:49:30.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H48r/5A8MoFRt5emigoQETSsqj0=', 'Content-Length': '0'} 2023-09-29T22:49:30.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb6e3254f19fcf32-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.272 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.272 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.272 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.272 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.272 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.272 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.272 DEBUG:boto:Data: 2023-09-29T22:49:30.272 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.273 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.273 DEBUG:boto:Params: {} 2023-09-29T22:49:30.273 DEBUG:boto:Token: None 2023-09-29T22:49:30.273 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:H48r/5A8MoFRt5emigoQETSsqj0= 2023-09-29T22:49:30.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H48r/5A8MoFRt5emigoQETSsqj0=', 'Content-Length': '0'} 2023-09-29T22:49:30.276 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae43dd0799bb7a9b-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:49:30.276 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.276 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.276 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.276 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.276 DEBUG:boto:Method: GET 2023-09-29T22:49:30.277 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.277 DEBUG:boto:Data: 2023-09-29T22:49:30.277 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.277 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.277 DEBUG:boto:Params: {} 2023-09-29T22:49:30.277 DEBUG:boto:Token: None 2023-09-29T22:49:30.277 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc= 2023-09-29T22:49:30.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc=', 'Content-Length': '0'} 2023-09-29T22:49:30.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f8246b0c5d77152-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.280 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.281 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.281 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.281 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.281 DEBUG:boto:Method: GET 2023-09-29T22:49:30.281 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.281 DEBUG:boto:Data: 2023-09-29T22:49:30.281 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.281 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.281 DEBUG:boto:Params: {} 2023-09-29T22:49:30.282 DEBUG:boto:Token: None 2023-09-29T22:49:30.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:49:30.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc= 2023-09-29T22:49:30.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vLVvgFyn/aM/iY48mfZOdTA8Fzc=', 'Content-Length': '0'} 2023-09-29T22:49:30.284 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c544d7d1c059f1e4-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.326 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.326 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.327 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.327 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.327 DEBUG:boto:Method: GET 2023-09-29T22:49:30.327 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.327 DEBUG:boto:Data: 2023-09-29T22:49:30.327 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.327 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.327 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.327 DEBUG:boto:Params: {} 2023-09-29T22:49:30.328 DEBUG:boto:Token: None 2023-09-29T22:49:30.328 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.328 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.328 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.333 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043c06a7d8c1e4cde-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.333 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.333 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.333 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.333 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.334 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.334 DEBUG:boto:Method: GET 2023-09-29T22:49:30.334 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.334 DEBUG:boto:Data: 2023-09-29T22:49:30.334 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.334 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.334 DEBUG:boto:Params: {} 2023-09-29T22:49:30.334 DEBUG:boto:Token: None 2023-09-29T22:49:30.335 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.335 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.335 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.342 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000926116a1fea91ac-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.342 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.343 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.343 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.343 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.343 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:49:30.343 DEBUG:boto:Data: 2023-09-29T22:49:30.343 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.343 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.343 DEBUG:boto:Params: {} 2023-09-29T22:49:30.344 DEBUG:boto:Token: None 2023-09-29T22:49:30.344 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76 2023-09-29T22:49:30.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hIBU2s/dd4B/j1ZoV5mtJw+/laQ= 2023-09-29T22:49:30.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hIBU2s/dd4B/j1ZoV5mtJw+/laQ=', 'Content-Length': '0'} 2023-09-29T22:49:30.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9f5e743f67067c6-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.346 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.346 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.347 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.347 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:49:30.347 DEBUG:boto:Data: 2023-09-29T22:49:30.347 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.347 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.347 DEBUG:boto:Params: {} 2023-09-29T22:49:30.347 DEBUG:boto:Token: None 2023-09-29T22:49:30.347 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76 2023-09-29T22:49:30.347 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hIBU2s/dd4B/j1ZoV5mtJw+/laQ= 2023-09-29T22:49:30.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hIBU2s/dd4B/j1ZoV5mtJw+/laQ=', 'Content-Length': '0'} 2023-09-29T22:49:30.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74815edde11e9fc-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:49:30.350 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.350 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.350 DEBUG:boto:Method: GET 2023-09-29T22:49:30.350 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:49:30.350 DEBUG:boto:Data: 2023-09-29T22:49:30.350 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.351 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.351 DEBUG:boto:Params: {} 2023-09-29T22:49:30.351 DEBUG:boto:Token: None 2023-09-29T22:49:30.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76 2023-09-29T22:49:30.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7SczXAeaqPUTgLu6kIBAiau12n0= 2023-09-29T22:49:30.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7SczXAeaqPUTgLu6kIBAiau12n0=', 'Content-Length': '0'} 2023-09-29T22:49:30.353 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b36c4b56c7de1161-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.398 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:49:30.398 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:49:30.399 DEBUG:boto:Method: GET 2023-09-29T22:49:30.399 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:49:30.399 DEBUG:boto:Data: 2023-09-29T22:49:30.399 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.399 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.399 DEBUG:boto:Params: {} 2023-09-29T22:49:30.399 DEBUG:boto:Token: None 2023-09-29T22:49:30.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj76 2023-09-29T22:49:30.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7SczXAeaqPUTgLu6kIBAiau12n0= 2023-09-29T22:49:30.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7SczXAeaqPUTgLu6kIBAiau12n0=', 'Content-Length': '0'} 2023-09-29T22:49:30.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e66eb7f84428d42f-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.402 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.402 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.402 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.402 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.403 DEBUG:boto:Method: GET 2023-09-29T22:49:30.403 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.403 DEBUG:boto:Data: 2023-09-29T22:49:30.403 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.403 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.403 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.403 DEBUG:boto:Params: {} 2023-09-29T22:49:30.403 DEBUG:boto:Token: None 2023-09-29T22:49:30.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.404 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.404 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.408 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000033b1dc7df53eeebe-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.408 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.408 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.408 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.408 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.408 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.409 DEBUG:boto:Method: GET 2023-09-29T22:49:30.409 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:49:30.409 DEBUG:boto:Data: 2023-09-29T22:49:30.409 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.409 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.409 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.409 DEBUG:boto:Params: {} 2023-09-29T22:49:30.409 DEBUG:boto:Token: None 2023-09-29T22:49:30.409 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000eee0bfd20f23f18-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.416 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:49:30.417 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.417 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.417 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.417 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.417 DEBUG:boto:Method: GET 2023-09-29T22:49:30.417 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.417 DEBUG:boto:Data: 2023-09-29T22:49:30.418 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.418 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.418 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.418 DEBUG:boto:Params: {} 2023-09-29T22:49:30.418 DEBUG:boto:Token: None 2023-09-29T22:49:30.418 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc= 2023-09-29T22:49:30.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc=', 'Content-Length': '0'} 2023-09-29T22:49:30.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd8c1be13cc7e76c-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.421 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.421 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.421 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.421 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.421 DEBUG:boto:Method: GET 2023-09-29T22:49:30.421 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.421 DEBUG:boto:Data: 2023-09-29T22:49:30.422 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.422 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.422 DEBUG:boto:Params: {} 2023-09-29T22:49:30.422 DEBUG:boto:Token: None 2023-09-29T22:49:30.422 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc= 2023-09-29T22:49:30.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc=', 'Content-Length': '0'} 2023-09-29T22:49:30.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fe36a1e970cf982-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.466 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.466 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.466 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.467 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.467 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.467 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.467 DEBUG:boto:Data: 2023-09-29T22:49:30.467 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.467 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.467 DEBUG:boto:Params: {} 2023-09-29T22:49:30.467 DEBUG:boto:Token: None 2023-09-29T22:49:30.468 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zgs70iDV4rJkm0AIk35riWcL+rM= 2023-09-29T22:49:30.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zgs70iDV4rJkm0AIk35riWcL+rM=', 'Content-Length': '0'} 2023-09-29T22:49:30.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043d8c44a091be794-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.471 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.471 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.471 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.471 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.471 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.471 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.471 DEBUG:boto:Data: 2023-09-29T22:49:30.471 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.472 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.472 DEBUG:boto:Params: {} 2023-09-29T22:49:30.472 DEBUG:boto:Token: None 2023-09-29T22:49:30.472 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Zgs70iDV4rJkm0AIk35riWcL+rM= 2023-09-29T22:49:30.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Zgs70iDV4rJkm0AIk35riWcL+rM=', 'Content-Length': '0'} 2023-09-29T22:49:30.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080d0e68f22602202-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:49:30.475 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.475 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.475 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.475 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.476 DEBUG:boto:Method: GET 2023-09-29T22:49:30.476 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.476 DEBUG:boto:Data: 2023-09-29T22:49:30.476 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.476 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.476 DEBUG:boto:Params: {} 2023-09-29T22:49:30.476 DEBUG:boto:Token: None 2023-09-29T22:49:30.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc= 2023-09-29T22:49:30.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc=', 'Content-Length': '0'} 2023-09-29T22:49:30.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f53c35d8b310bf6-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.480 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.480 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.480 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.480 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.480 DEBUG:boto:Method: GET 2023-09-29T22:49:30.480 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.480 DEBUG:boto:Data: 2023-09-29T22:49:30.480 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.481 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.481 DEBUG:boto:Params: {} 2023-09-29T22:49:30.481 DEBUG:boto:Token: None 2023-09-29T22:49:30.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:49:30.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc= 2023-09-29T22:49:30.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VClfYnY4ndgpZ4fAM308tDu3chc=', 'Content-Length': '0'} 2023-09-29T22:49:30.484 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d24f6dc770263a61-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.484 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.485 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.485 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.485 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.485 DEBUG:boto:Method: GET 2023-09-29T22:49:30.485 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.485 DEBUG:boto:Data: 2023-09-29T22:49:30.485 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.485 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.485 DEBUG:boto:Params: {} 2023-09-29T22:49:30.486 DEBUG:boto:Token: None 2023-09-29T22:49:30.486 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.486 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.490 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb573ddacbe1bfa8-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.530 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.531 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.531 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.531 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.531 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.531 DEBUG:boto:Method: GET 2023-09-29T22:49:30.531 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.531 DEBUG:boto:Data: 2023-09-29T22:49:30.531 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.532 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.532 DEBUG:boto:Params: {} 2023-09-29T22:49:30.532 DEBUG:boto:Token: None 2023-09-29T22:49:30.532 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c790d230cb6ef0b5-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.540 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.541 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.541 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.541 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.541 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:49:30.541 DEBUG:boto:Data: 2023-09-29T22:49:30.541 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.541 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.541 DEBUG:boto:Params: {} 2023-09-29T22:49:30.542 DEBUG:boto:Token: None 2023-09-29T22:49:30.542 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85 2023-09-29T22:49:30.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:peRj2zA5BaWi7a/PGNGV1QRuYxM= 2023-09-29T22:49:30.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:peRj2zA5BaWi7a/PGNGV1QRuYxM=', 'Content-Length': '0'} 2023-09-29T22:49:30.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000898515f0dab50344-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.544 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.544 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.544 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.544 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:49:30.545 DEBUG:boto:Data: 2023-09-29T22:49:30.545 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.545 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.545 DEBUG:boto:Params: {} 2023-09-29T22:49:30.545 DEBUG:boto:Token: None 2023-09-29T22:49:30.545 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85 2023-09-29T22:49:30.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:peRj2zA5BaWi7a/PGNGV1QRuYxM= 2023-09-29T22:49:30.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:peRj2zA5BaWi7a/PGNGV1QRuYxM=', 'Content-Length': '0'} 2023-09-29T22:49:30.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cad8ef07a50b1ac-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:49:30.548 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.548 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.548 DEBUG:boto:Method: GET 2023-09-29T22:49:30.548 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:49:30.548 DEBUG:boto:Data: 2023-09-29T22:49:30.548 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.548 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.549 DEBUG:boto:Params: {} 2023-09-29T22:49:30.549 DEBUG:boto:Token: None 2023-09-29T22:49:30.549 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85 2023-09-29T22:49:30.549 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qB2Uyi7itJYVTGsko4VvNfeME6g= 2023-09-29T22:49:30.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qB2Uyi7itJYVTGsko4VvNfeME6g=', 'Content-Length': '0'} 2023-09-29T22:49:30.551 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d33cc1e4a6cb62b6-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.551 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:49:30.551 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:49:30.551 DEBUG:boto:Method: GET 2023-09-29T22:49:30.552 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:49:30.552 DEBUG:boto:Data: 2023-09-29T22:49:30.552 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.552 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.552 DEBUG:boto:Params: {} 2023-09-29T22:49:30.552 DEBUG:boto:Token: None 2023-09-29T22:49:30.552 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj85 2023-09-29T22:49:30.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qB2Uyi7itJYVTGsko4VvNfeME6g= 2023-09-29T22:49:30.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qB2Uyi7itJYVTGsko4VvNfeME6g=', 'Content-Length': '0'} 2023-09-29T22:49:30.554 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20adcf666397b1a-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.602 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.603 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.603 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.603 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.603 DEBUG:boto:Method: GET 2023-09-29T22:49:30.603 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.603 DEBUG:boto:Data: 2023-09-29T22:49:30.603 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.603 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.603 DEBUG:boto:Params: {} 2023-09-29T22:49:30.604 DEBUG:boto:Token: None 2023-09-29T22:49:30.604 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.609 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000595b4b6a0bfc6ce4-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.609 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.609 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.609 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.610 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.610 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.610 DEBUG:boto:Method: GET 2023-09-29T22:49:30.610 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:49:30.610 DEBUG:boto:Data: 2023-09-29T22:49:30.610 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.610 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.611 DEBUG:boto:Params: {} 2023-09-29T22:49:30.611 DEBUG:boto:Token: None 2023-09-29T22:49:30.611 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.620 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000049b132f3fa4ce8b7-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.620 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:49:30.620 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.620 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.621 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.621 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.621 DEBUG:boto:Method: GET 2023-09-29T22:49:30.621 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.621 DEBUG:boto:Data: 2023-09-29T22:49:30.621 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.621 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.621 DEBUG:boto:Params: {} 2023-09-29T22:49:30.621 DEBUG:boto:Token: None 2023-09-29T22:49:30.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY= 2023-09-29T22:49:30.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY=', 'Content-Length': '0'} 2023-09-29T22:49:30.624 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ea4e0225a5c2b35-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.624 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.624 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.625 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.625 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.625 DEBUG:boto:Method: GET 2023-09-29T22:49:30.625 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.625 DEBUG:boto:Data: 2023-09-29T22:49:30.625 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.625 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.625 DEBUG:boto:Params: {} 2023-09-29T22:49:30.626 DEBUG:boto:Token: None 2023-09-29T22:49:30.626 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.626 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY= 2023-09-29T22:49:30.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY=', 'Content-Length': '0'} 2023-09-29T22:49:30.628 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7e3f634cee51422-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.628 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.628 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.628 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.629 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.629 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.629 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.629 DEBUG:boto:Data: 2023-09-29T22:49:30.629 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.629 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.629 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.629 DEBUG:boto:Params: {} 2023-09-29T22:49:30.629 DEBUG:boto:Token: None 2023-09-29T22:49:30.630 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.630 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RVahhEYfbbMezmpeQQDrqLOp42I= 2023-09-29T22:49:30.630 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RVahhEYfbbMezmpeQQDrqLOp42I=', 'Content-Length': '0'} 2023-09-29T22:49:30.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052706a23dad07590-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.632 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.632 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.632 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.632 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.632 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.633 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.633 DEBUG:boto:Data: 2023-09-29T22:49:30.633 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.633 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.633 DEBUG:boto:Params: {} 2023-09-29T22:49:30.633 DEBUG:boto:Token: None 2023-09-29T22:49:30.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RVahhEYfbbMezmpeQQDrqLOp42I= 2023-09-29T22:49:30.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RVahhEYfbbMezmpeQQDrqLOp42I=', 'Content-Length': '0'} 2023-09-29T22:49:30.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000418e343fb7d8e2b1-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:49:30.636 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.636 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.636 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.636 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.636 DEBUG:boto:Method: GET 2023-09-29T22:49:30.636 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.637 DEBUG:boto:Data: 2023-09-29T22:49:30.637 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.637 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.637 DEBUG:boto:Params: {} 2023-09-29T22:49:30.637 DEBUG:boto:Token: None 2023-09-29T22:49:30.637 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY= 2023-09-29T22:49:30.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY=', 'Content-Length': '0'} 2023-09-29T22:49:30.640 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8994009a8151ff7-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.640 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.640 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.640 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.640 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.640 DEBUG:boto:Method: GET 2023-09-29T22:49:30.641 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.641 DEBUG:boto:Data: 2023-09-29T22:49:30.641 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.641 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.641 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.641 DEBUG:boto:Params: {} 2023-09-29T22:49:30.641 DEBUG:boto:Token: None 2023-09-29T22:49:30.641 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:49:30.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY= 2023-09-29T22:49:30.642 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ynC3qY+SPEr2CXChNJbbHAPr5dY=', 'Content-Length': '0'} 2023-09-29T22:49:30.644 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000995c6a989f458414-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.686 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.686 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.687 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.687 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.687 DEBUG:boto:Method: GET 2023-09-29T22:49:30.687 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.687 DEBUG:boto:Data: 2023-09-29T22:49:30.687 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.687 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.687 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.687 DEBUG:boto:Params: {} 2023-09-29T22:49:30.687 DEBUG:boto:Token: None 2023-09-29T22:49:30.688 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.688 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.688 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.693 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000105c710e08932a7a-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.693 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.693 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.693 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.693 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.694 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.694 DEBUG:boto:Method: GET 2023-09-29T22:49:30.694 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.694 DEBUG:boto:Data: 2023-09-29T22:49:30.694 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.694 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.694 DEBUG:boto:Params: {} 2023-09-29T22:49:30.694 DEBUG:boto:Token: None 2023-09-29T22:49:30.695 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.695 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a9576980ae45c71-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.703 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.703 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.703 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.703 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.703 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:49:30.703 DEBUG:boto:Data: 2023-09-29T22:49:30.704 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.704 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.704 DEBUG:boto:Params: {} 2023-09-29T22:49:30.704 DEBUG:boto:Token: None 2023-09-29T22:49:30.704 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87 2023-09-29T22:49:30.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/D4dmPM5yWAPXJlanLxxWRpxWqw= 2023-09-29T22:49:30.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/D4dmPM5yWAPXJlanLxxWRpxWqw=', 'Content-Length': '0'} 2023-09-29T22:49:30.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004544026c52e744dc-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.707 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.707 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.707 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.707 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:49:30.707 DEBUG:boto:Data: 2023-09-29T22:49:30.707 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.707 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.708 DEBUG:boto:Params: {} 2023-09-29T22:49:30.708 DEBUG:boto:Token: None 2023-09-29T22:49:30.708 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87 2023-09-29T22:49:30.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/D4dmPM5yWAPXJlanLxxWRpxWqw= 2023-09-29T22:49:30.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/D4dmPM5yWAPXJlanLxxWRpxWqw=', 'Content-Length': '0'} 2023-09-29T22:49:30.710 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000037847f28d9628ce-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:49:30.710 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.711 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.711 DEBUG:boto:Method: GET 2023-09-29T22:49:30.711 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:49:30.711 DEBUG:boto:Data: 2023-09-29T22:49:30.711 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.711 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.711 DEBUG:boto:Params: {} 2023-09-29T22:49:30.711 DEBUG:boto:Token: None 2023-09-29T22:49:30.711 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87 2023-09-29T22:49:30.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ci2ZOTQ4RO1PWh9/eQ3kHFmAF5k= 2023-09-29T22:49:30.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ci2ZOTQ4RO1PWh9/eQ3kHFmAF5k=', 'Content-Length': '0'} 2023-09-29T22:49:30.714 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f12f91b60f540a23-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.754 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:49:30.754 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:49:30.755 DEBUG:boto:Method: GET 2023-09-29T22:49:30.755 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:49:30.755 DEBUG:boto:Data: 2023-09-29T22:49:30.755 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.755 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.755 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.755 DEBUG:boto:Params: {} 2023-09-29T22:49:30.755 DEBUG:boto:Token: None 2023-09-29T22:49:30.755 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj87 2023-09-29T22:49:30.755 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ci2ZOTQ4RO1PWh9/eQ3kHFmAF5k= 2023-09-29T22:49:30.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ci2ZOTQ4RO1PWh9/eQ3kHFmAF5k=', 'Content-Length': '0'} 2023-09-29T22:49:30.758 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc4cede30bfb5f13-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.758 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.759 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.759 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.759 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.759 DEBUG:boto:Method: GET 2023-09-29T22:49:30.759 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.759 DEBUG:boto:Data: 2023-09-29T22:49:30.759 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.759 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.760 DEBUG:boto:Params: {} 2023-09-29T22:49:30.760 DEBUG:boto:Token: None 2023-09-29T22:49:30.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcb223a3f3beac45-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.765 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.765 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.765 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.765 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.765 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.765 DEBUG:boto:Method: GET 2023-09-29T22:49:30.765 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:49:30.766 DEBUG:boto:Data: 2023-09-29T22:49:30.766 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.766 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.766 DEBUG:boto:Params: {} 2023-09-29T22:49:30.766 DEBUG:boto:Token: None 2023-09-29T22:49:30.766 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.766 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.773 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd9e10c2b247710e-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.814 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:49:30.815 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.815 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.815 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.815 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.815 DEBUG:boto:Method: GET 2023-09-29T22:49:30.815 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.815 DEBUG:boto:Data: 2023-09-29T22:49:30.816 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.816 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.816 DEBUG:boto:Params: {} 2023-09-29T22:49:30.816 DEBUG:boto:Token: None 2023-09-29T22:49:30.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM= 2023-09-29T22:49:30.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM=', 'Content-Length': '0'} 2023-09-29T22:49:30.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bba0d0f78787c22-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.819 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.819 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.820 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.820 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.820 DEBUG:boto:Method: GET 2023-09-29T22:49:30.820 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.820 DEBUG:boto:Data: 2023-09-29T22:49:30.820 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.820 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.820 DEBUG:boto:Params: {} 2023-09-29T22:49:30.821 DEBUG:boto:Token: None 2023-09-29T22:49:30.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM= 2023-09-29T22:49:30.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM=', 'Content-Length': '0'} 2023-09-29T22:49:30.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000805ffe76baf880f9-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.824 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.824 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.825 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.825 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.825 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.825 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.825 DEBUG:boto:Data: 2023-09-29T22:49:30.825 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.825 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.825 DEBUG:boto:Params: {} 2023-09-29T22:49:30.825 DEBUG:boto:Token: None 2023-09-29T22:49:30.826 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.826 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AImaybxCbb5RsB7GAyb34q4zU3s= 2023-09-29T22:49:30.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AImaybxCbb5RsB7GAyb34q4zU3s=', 'Content-Length': '0'} 2023-09-29T22:49:30.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b814196f7ed632bc-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.829 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.829 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.829 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.829 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.829 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.829 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.829 DEBUG:boto:Data: 2023-09-29T22:49:30.829 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.830 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.830 DEBUG:boto:Params: {} 2023-09-29T22:49:30.830 DEBUG:boto:Token: None 2023-09-29T22:49:30.830 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AImaybxCbb5RsB7GAyb34q4zU3s= 2023-09-29T22:49:30.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AImaybxCbb5RsB7GAyb34q4zU3s=', 'Content-Length': '0'} 2023-09-29T22:49:30.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e18e4b2d05d26b-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:49:30.833 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.833 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.833 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.833 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.833 DEBUG:boto:Method: GET 2023-09-29T22:49:30.834 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.834 DEBUG:boto:Data: 2023-09-29T22:49:30.834 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.834 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.834 DEBUG:boto:Params: {} 2023-09-29T22:49:30.834 DEBUG:boto:Token: None 2023-09-29T22:49:30.834 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.834 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM= 2023-09-29T22:49:30.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM=', 'Content-Length': '0'} 2023-09-29T22:49:30.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9a107ee4fd3581-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.837 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.837 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.838 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.838 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.838 DEBUG:boto:Method: GET 2023-09-29T22:49:30.838 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.838 DEBUG:boto:Data: 2023-09-29T22:49:30.838 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.838 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.838 DEBUG:boto:Params: {} 2023-09-29T22:49:30.839 DEBUG:boto:Token: None 2023-09-29T22:49:30.839 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:49:30.839 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM= 2023-09-29T22:49:30.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:e1oS79T8Bc6b3VE9ztyemSioYPM=', 'Content-Length': '0'} 2023-09-29T22:49:30.842 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b72bec5c6ca87150-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.842 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.842 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.842 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.842 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.842 DEBUG:boto:Method: GET 2023-09-29T22:49:30.843 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.843 DEBUG:boto:Data: 2023-09-29T22:49:30.843 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.843 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.843 DEBUG:boto:Params: {} 2023-09-29T22:49:30.843 DEBUG:boto:Token: None 2023-09-29T22:49:30.843 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.843 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.843 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.847 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c257e6cba8863128-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.848 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.848 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.848 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.848 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.848 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.848 DEBUG:boto:Method: GET 2023-09-29T22:49:30.849 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.849 DEBUG:boto:Data: 2023-09-29T22:49:30.849 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.849 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.849 DEBUG:boto:Params: {} 2023-09-29T22:49:30.849 DEBUG:boto:Token: None 2023-09-29T22:49:30.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.857 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005d12b419f992c394-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.858 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.858 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.858 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.858 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.858 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:49:30.858 DEBUG:boto:Data: 2023-09-29T22:49:30.859 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.859 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.859 DEBUG:boto:Params: {} 2023-09-29T22:49:30.859 DEBUG:boto:Token: None 2023-09-29T22:49:30.859 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92 2023-09-29T22:49:30.859 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PgX3nMktvlPExeJVGW27Z3NXdqo= 2023-09-29T22:49:30.859 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PgX3nMktvlPExeJVGW27Z3NXdqo=', 'Content-Length': '0'} 2023-09-29T22:49:30.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fcdc7f156cc89238-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.862 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.862 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.862 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.862 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:49:30.862 DEBUG:boto:Data: 2023-09-29T22:49:30.862 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.862 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.862 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.862 DEBUG:boto:Params: {} 2023-09-29T22:49:30.863 DEBUG:boto:Token: None 2023-09-29T22:49:30.863 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92 2023-09-29T22:49:30.863 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PgX3nMktvlPExeJVGW27Z3NXdqo= 2023-09-29T22:49:30.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PgX3nMktvlPExeJVGW27Z3NXdqo=', 'Content-Length': '0'} 2023-09-29T22:49:30.865 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031216ce2ce62584f-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:49:30.866 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.866 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.866 DEBUG:boto:Method: GET 2023-09-29T22:49:30.866 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:49:30.866 DEBUG:boto:Data: 2023-09-29T22:49:30.866 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.866 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.867 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.867 DEBUG:boto:Params: {} 2023-09-29T22:49:30.867 DEBUG:boto:Token: None 2023-09-29T22:49:30.867 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92 2023-09-29T22:49:30.867 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4eBu/x0UEIIl3N0Urtoi4HUykw4= 2023-09-29T22:49:30.867 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4eBu/x0UEIIl3N0Urtoi4HUykw4=', 'Content-Length': '0'} 2023-09-29T22:49:30.869 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbd91afbe5938eb7-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.910 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:49:30.911 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:49:30.911 DEBUG:boto:Method: GET 2023-09-29T22:49:30.911 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:49:30.911 DEBUG:boto:Data: 2023-09-29T22:49:30.911 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.911 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.911 DEBUG:boto:Params: {} 2023-09-29T22:49:30.911 DEBUG:boto:Token: None 2023-09-29T22:49:30.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj92 2023-09-29T22:49:30.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4eBu/x0UEIIl3N0Urtoi4HUykw4= 2023-09-29T22:49:30.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4eBu/x0UEIIl3N0Urtoi4HUykw4=', 'Content-Length': '0'} 2023-09-29T22:49:30.914 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a60f81d46fedfc73-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.915 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.915 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.915 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.915 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.915 DEBUG:boto:Method: GET 2023-09-29T22:49:30.915 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.915 DEBUG:boto:Data: 2023-09-29T22:49:30.916 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.916 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.916 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.916 DEBUG:boto:Params: {} 2023-09-29T22:49:30.916 DEBUG:boto:Token: None 2023-09-29T22:49:30.916 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.916 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.916 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.921 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004dabb7e952c0fc43-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.921 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.921 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.922 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.922 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.922 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.922 DEBUG:boto:Method: GET 2023-09-29T22:49:30.922 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:49:30.922 DEBUG:boto:Data: 2023-09-29T22:49:30.922 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.922 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.923 DEBUG:boto:Params: {} 2023-09-29T22:49:30.923 DEBUG:boto:Token: None 2023-09-29T22:49:30.923 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.923 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.923 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.930 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcabcba372e0567a-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.930 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:49:30.930 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.931 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.931 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.931 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.931 DEBUG:boto:Method: GET 2023-09-29T22:49:30.931 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.931 DEBUG:boto:Data: 2023-09-29T22:49:30.931 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.931 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.931 DEBUG:boto:Params: {} 2023-09-29T22:49:30.932 DEBUG:boto:Token: None 2023-09-29T22:49:30.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q= 2023-09-29T22:49:30.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q=', 'Content-Length': '0'} 2023-09-29T22:49:30.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ed2f7ab1dba2468-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.934 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.934 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.935 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.935 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.935 DEBUG:boto:Method: GET 2023-09-29T22:49:30.935 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.935 DEBUG:boto:Data: 2023-09-29T22:49:30.935 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.935 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.935 DEBUG:boto:Params: {} 2023-09-29T22:49:30.936 DEBUG:boto:Token: None 2023-09-29T22:49:30.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q= 2023-09-29T22:49:30.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q=', 'Content-Length': '0'} 2023-09-29T22:49:30.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003caba2cbddec347f-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.938 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.938 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.938 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.938 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.939 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.939 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.939 DEBUG:boto:Data: 2023-09-29T22:49:30.939 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.939 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.939 DEBUG:boto:Params: {} 2023-09-29T22:49:30.939 DEBUG:boto:Token: None 2023-09-29T22:49:30.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fD90UvuG17sBi/53ZB5GYdZEghw= 2023-09-29T22:49:30.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fD90UvuG17sBi/53ZB5GYdZEghw=', 'Content-Length': '0'} 2023-09-29T22:49:30.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b955dea783c4de36-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.942 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.942 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.942 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.942 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.942 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.942 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.943 DEBUG:boto:Data: 2023-09-29T22:49:30.943 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.943 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.943 DEBUG:boto:Params: {} 2023-09-29T22:49:30.943 DEBUG:boto:Token: None 2023-09-29T22:49:30.943 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fD90UvuG17sBi/53ZB5GYdZEghw= 2023-09-29T22:49:30.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fD90UvuG17sBi/53ZB5GYdZEghw=', 'Content-Length': '0'} 2023-09-29T22:49:30.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000005efe8657b456594-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:49:30.946 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.946 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.946 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.946 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.946 DEBUG:boto:Method: GET 2023-09-29T22:49:30.947 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.947 DEBUG:boto:Data: 2023-09-29T22:49:30.947 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.947 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.947 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.947 DEBUG:boto:Params: {} 2023-09-29T22:49:30.947 DEBUG:boto:Token: None 2023-09-29T22:49:30.947 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.947 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q= 2023-09-29T22:49:30.948 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q=', 'Content-Length': '0'} 2023-09-29T22:49:30.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018044f00a5acb393-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.950 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.950 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.950 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.950 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.950 DEBUG:boto:Method: GET 2023-09-29T22:49:30.950 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.951 DEBUG:boto:Data: 2023-09-29T22:49:30.951 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.951 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.951 DEBUG:boto:Params: {} 2023-09-29T22:49:30.951 DEBUG:boto:Token: None 2023-09-29T22:49:30.951 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:49:30.951 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q= 2023-09-29T22:49:30.951 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CyTMMtACFSb8LgCAGCuWKt3Nm5Q=', 'Content-Length': '0'} 2023-09-29T22:49:30.954 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d073e1c58e2523a-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.954 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.954 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.954 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:30.954 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:30.955 DEBUG:boto:Method: GET 2023-09-29T22:49:30.955 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:49:30.955 DEBUG:boto:Data: 2023-09-29T22:49:30.955 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.955 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.955 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.955 DEBUG:boto:Params: {} 2023-09-29T22:49:30.955 DEBUG:boto:Token: None 2023-09-29T22:49:30.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.960 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e602255d15c121f2-006517547a-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.960 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.960 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:30.961 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:30.961 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:30.961 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:30.961 DEBUG:boto:Method: GET 2023-09-29T22:49:30.961 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:49:30.961 DEBUG:boto:Data: 2023-09-29T22:49:30.961 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.961 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.962 DEBUG:boto:Params: {} 2023-09-29T22:49:30.962 DEBUG:boto:Token: None 2023-09-29T22:49:30.962 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/ 2023-09-29T22:49:30.962 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk= 2023-09-29T22:49:30.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0JuShJPLJUzUsLCDwUMmJOGbTJk=', 'Content-Length': '0'} 2023-09-29T22:49:30.970 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091ed8f5b3f09a42d-006517547a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.970 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:30.970 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.970 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.971 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.971 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:49:30.971 DEBUG:boto:Data: 2023-09-29T22:49:30.971 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.971 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.971 DEBUG:boto:Params: {} 2023-09-29T22:49:30.971 DEBUG:boto:Token: None 2023-09-29T22:49:30.971 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93 2023-09-29T22:49:30.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fklqowAFc3tQVt/PpUJkDIG8MWk= 2023-09-29T22:49:30.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fklqowAFc3tQVt/PpUJkDIG8MWk=', 'Content-Length': '0'} 2023-09-29T22:49:30.974 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bd8dbaf851aad71-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.974 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.974 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.974 DEBUG:boto:Method: HEAD 2023-09-29T22:49:30.974 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:49:30.974 DEBUG:boto:Data: 2023-09-29T22:49:30.974 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:30.975 DEBUG:boto:Port: 8000 2023-09-29T22:49:30.975 DEBUG:boto:Params: {} 2023-09-29T22:49:30.975 DEBUG:boto:Token: None 2023-09-29T22:49:30.975 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93 2023-09-29T22:49:30.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fklqowAFc3tQVt/PpUJkDIG8MWk= 2023-09-29T22:49:30.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fklqowAFc3tQVt/PpUJkDIG8MWk=', 'Content-Length': '0'} 2023-09-29T22:49:30.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001093aec05d20eee-006517547a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:30.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:49:30.977 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:30.978 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:30.978 DEBUG:boto:Method: GET 2023-09-29T22:49:30.978 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:49:30.978 DEBUG:boto:Data: 2023-09-29T22:49:30.978 DEBUG:boto:Headers: {} 2023-09-29T22:49:30.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:30.978 DEBUG:boto:Port: 8001 2023-09-29T22:49:30.978 DEBUG:boto:Params: {} 2023-09-29T22:49:30.978 DEBUG:boto:Token: None 2023-09-29T22:49:30.979 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:30 GMT /hcnphz-59/obj93 2023-09-29T22:49:30.979 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nx7efKPCT9Hi4JNIv19ABw1OZjU= 2023-09-29T22:49:30.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nx7efKPCT9Hi4JNIv19ABw1OZjU=', 'Content-Length': '0'} 2023-09-29T22:49:30.981 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c013a47292d7761b-006517547a-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.022 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:49:31.023 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:49:31.023 DEBUG:boto:Method: GET 2023-09-29T22:49:31.023 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:49:31.023 DEBUG:boto:Data: 2023-09-29T22:49:31.023 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.023 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.023 DEBUG:boto:Params: {} 2023-09-29T22:49:31.024 DEBUG:boto:Token: None 2023-09-29T22:49:31.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj93 2023-09-29T22:49:31.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ET7TCrPQT/AVEKPrpAmB3J06m8Y= 2023-09-29T22:49:31.024 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ET7TCrPQT/AVEKPrpAmB3J06m8Y=', 'Content-Length': '0'} 2023-09-29T22:49:31.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000940a79cf275e863a-006517547b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.027 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:31.027 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:31.027 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:31.027 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:31.027 DEBUG:boto:Method: GET 2023-09-29T22:49:31.028 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:49:31.028 DEBUG:boto:Data: 2023-09-29T22:49:31.028 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.028 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.028 DEBUG:boto:Params: {} 2023-09-29T22:49:31.028 DEBUG:boto:Token: None 2023-09-29T22:49:31.028 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/ 2023-09-29T22:49:31.028 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk= 2023-09-29T22:49:31.029 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk=', 'Content-Length': '0'} 2023-09-29T22:49:31.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000073e862176b60206d-006517547b-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.033 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:31.033 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:31.033 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:31.033 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:31.034 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:49:31.034 DEBUG:boto:Method: GET 2023-09-29T22:49:31.034 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:49:31.034 DEBUG:boto:Data: 2023-09-29T22:49:31.034 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.034 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.034 DEBUG:boto:Params: {} 2023-09-29T22:49:31.034 DEBUG:boto:Token: None 2023-09-29T22:49:31.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/ 2023-09-29T22:49:31.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk= 2023-09-29T22:49:31.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk=', 'Content-Length': '0'} 2023-09-29T22:49:31.043 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000afadc5f0b4880a68-006517547b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.043 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:31.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:49:31.044 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.044 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.044 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.044 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.044 DEBUG:boto:Method: GET 2023-09-29T22:49:31.044 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.044 DEBUG:boto:Data: 2023-09-29T22:49:31.045 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.045 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.045 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.045 DEBUG:boto:Params: {} 2023-09-29T22:49:31.045 DEBUG:boto:Token: None 2023-09-29T22:49:31.045 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.045 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M= 2023-09-29T22:49:31.046 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M=', 'Content-Length': '0'} 2023-09-29T22:49:31.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061b0ccd2b5ec98cb-006517547b-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.048 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.048 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.048 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.048 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.048 DEBUG:boto:Method: GET 2023-09-29T22:49:31.048 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.048 DEBUG:boto:Data: 2023-09-29T22:49:31.048 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.049 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.049 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.049 DEBUG:boto:Params: {} 2023-09-29T22:49:31.049 DEBUG:boto:Token: None 2023-09-29T22:49:31.049 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M= 2023-09-29T22:49:31.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M=', 'Content-Length': '0'} 2023-09-29T22:49:31.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002009493c8329c525-006517547b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.052 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.052 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.052 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.052 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.052 DEBUG:boto:Method: HEAD 2023-09-29T22:49:31.052 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.052 DEBUG:boto:Data: 2023-09-29T22:49:31.053 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.053 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.053 DEBUG:boto:Params: {} 2023-09-29T22:49:31.053 DEBUG:boto:Token: None 2023-09-29T22:49:31.053 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.053 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R5gBGbLh6Gv9Bc7vVcHwGjHyAAc= 2023-09-29T22:49:31.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R5gBGbLh6Gv9Bc7vVcHwGjHyAAc=', 'Content-Length': '0'} 2023-09-29T22:49:31.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f591163c3a5e6c5-006517547b-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.056 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.056 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.056 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.056 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.056 DEBUG:boto:Method: HEAD 2023-09-29T22:49:31.056 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.056 DEBUG:boto:Data: 2023-09-29T22:49:31.056 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.057 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.057 DEBUG:boto:Params: {} 2023-09-29T22:49:31.057 DEBUG:boto:Token: None 2023-09-29T22:49:31.057 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R5gBGbLh6Gv9Bc7vVcHwGjHyAAc= 2023-09-29T22:49:31.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R5gBGbLh6Gv9Bc7vVcHwGjHyAAc=', 'Content-Length': '0'} 2023-09-29T22:49:31.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055abb9dba3c06a6b-006517547b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:49:31.060 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.060 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.060 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.060 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.060 DEBUG:boto:Method: GET 2023-09-29T22:49:31.060 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.060 DEBUG:boto:Data: 2023-09-29T22:49:31.060 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.061 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.061 DEBUG:boto:Params: {} 2023-09-29T22:49:31.061 DEBUG:boto:Token: None 2023-09-29T22:49:31.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M= 2023-09-29T22:49:31.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M=', 'Content-Length': '0'} 2023-09-29T22:49:31.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c669bb0b7570022-006517547b-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.106 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.107 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.107 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.107 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.107 DEBUG:boto:Method: GET 2023-09-29T22:49:31.107 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.107 DEBUG:boto:Data: 2023-09-29T22:49:31.107 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.107 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.108 DEBUG:boto:Params: {} 2023-09-29T22:49:31.108 DEBUG:boto:Token: None 2023-09-29T22:49:31.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:49:31.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M= 2023-09-29T22:49:31.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bd9CF0DJnwqD5olEyFy2vhtWQ+M=', 'Content-Length': '0'} 2023-09-29T22:49:31.111 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5355b41ef229243-006517547b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.111 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:31.111 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:31.112 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.112 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.112 DEBUG:boto:Method: GET 2023-09-29T22:49:31.112 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.112 DEBUG:boto:Data: 2023-09-29T22:49:31.112 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.112 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.112 DEBUG:boto:Params: {} 2023-09-29T22:49:31.112 DEBUG:boto:Token: None 2023-09-29T22:49:31.113 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/ 2023-09-29T22:49:31.113 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk= 2023-09-29T22:49:31.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk=', 'Content-Length': '0'} 2023-09-29T22:49:31.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008d4c60e0eda98084-006517547b-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.118 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:31.118 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:31.118 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:31.118 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.118 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.119 DEBUG:boto:Method: GET 2023-09-29T22:49:31.119 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.119 DEBUG:boto:Data: 2023-09-29T22:49:31.119 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.119 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.119 DEBUG:boto:Params: {} 2023-09-29T22:49:31.119 DEBUG:boto:Token: None 2023-09-29T22:49:31.119 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/ 2023-09-29T22:49:31.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk= 2023-09-29T22:49:31.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk=', 'Content-Length': '0'} 2023-09-29T22:49:31.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bbc1b83ce7494669-006517547b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.128 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:31.128 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.128 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.129 DEBUG:boto:Method: HEAD 2023-09-29T22:49:31.129 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:49:31.129 DEBUG:boto:Data: 2023-09-29T22:49:31.129 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.129 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.129 DEBUG:boto:Params: {} 2023-09-29T22:49:31.129 DEBUG:boto:Token: None 2023-09-29T22:49:31.129 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95 2023-09-29T22:49:31.130 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xBOGoj1c/5KRzIvzqUkdoa5JvGI= 2023-09-29T22:49:31.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xBOGoj1c/5KRzIvzqUkdoa5JvGI=', 'Content-Length': '0'} 2023-09-29T22:49:31.132 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b220731a184b5e0-006517547b-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.132 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.132 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.132 DEBUG:boto:Method: HEAD 2023-09-29T22:49:31.132 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:49:31.132 DEBUG:boto:Data: 2023-09-29T22:49:31.133 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.133 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.133 DEBUG:boto:Params: {} 2023-09-29T22:49:31.133 DEBUG:boto:Token: None 2023-09-29T22:49:31.133 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95 2023-09-29T22:49:31.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xBOGoj1c/5KRzIvzqUkdoa5JvGI= 2023-09-29T22:49:31.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xBOGoj1c/5KRzIvzqUkdoa5JvGI=', 'Content-Length': '0'} 2023-09-29T22:49:31.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2079d873bd25169-006517547b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:49:31.136 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.136 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.136 DEBUG:boto:Method: GET 2023-09-29T22:49:31.136 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:49:31.136 DEBUG:boto:Data: 2023-09-29T22:49:31.136 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.136 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.136 DEBUG:boto:Params: {} 2023-09-29T22:49:31.137 DEBUG:boto:Token: None 2023-09-29T22:49:31.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95 2023-09-29T22:49:31.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PDJEYDq4B2K1rfX9K9D0Xv9btZA= 2023-09-29T22:49:31.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PDJEYDq4B2K1rfX9K9D0Xv9btZA=', 'Content-Length': '0'} 2023-09-29T22:49:31.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027ddbae37cbb1c73-006517547b-1b41-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.139 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:49:31.139 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:49:31.139 DEBUG:boto:Method: GET 2023-09-29T22:49:31.139 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:49:31.140 DEBUG:boto:Data: 2023-09-29T22:49:31.140 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.140 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.140 DEBUG:boto:Params: {} 2023-09-29T22:49:31.140 DEBUG:boto:Token: None 2023-09-29T22:49:31.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/obj95 2023-09-29T22:49:31.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PDJEYDq4B2K1rfX9K9D0Xv9btZA= 2023-09-29T22:49:31.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PDJEYDq4B2K1rfX9K9D0Xv9btZA=', 'Content-Length': '0'} 2023-09-29T22:49:31.142 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ff82637594b2da3-006517547b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.186 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:31.187 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:31.187 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.187 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.187 DEBUG:boto:Method: GET 2023-09-29T22:49:31.187 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.187 DEBUG:boto:Data: 2023-09-29T22:49:31.187 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:31.187 DEBUG:boto:Port: 8001 2023-09-29T22:49:31.188 DEBUG:boto:Params: {} 2023-09-29T22:49:31.188 DEBUG:boto:Token: None 2023-09-29T22:49:31.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/ 2023-09-29T22:49:31.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk= 2023-09-29T22:49:31.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk=', 'Content-Length': '0'} 2023-09-29T22:49:31.193 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d7bf2acf302dfc45-006517547b-1b41-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.193 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:31.194 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:49:31.194 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:49:31.194 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.194 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.194 DEBUG:boto:Method: GET 2023-09-29T22:49:31.194 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:49:31.194 DEBUG:boto:Data: 2023-09-29T22:49:31.195 DEBUG:boto:Headers: {} 2023-09-29T22:49:31.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:49:31.195 DEBUG:boto:Port: 8000 2023-09-29T22:49:31.195 DEBUG:boto:Params: {} 2023-09-29T22:49:31.195 DEBUG:boto:Token: None 2023-09-29T22:49:31.195 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:49:31 GMT /hcnphz-59/ 2023-09-29T22:49:31.195 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk= 2023-09-29T22:49:31.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:49:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jXOENgPehCsLP0Wg7Hg2OIBQOUk=', 'Content-Length': '0'} 2023-09-29T22:49:31.203 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbb979f8cd0accc9-006517547b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:49:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:49:31.203 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:49:31.204 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:49:31.204 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T22:49:31.204 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:49:31.205 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:49:31.204+0000 7f5056b93700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 66548) UID: 0 2023-09-29T22:49:31.206 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:49:31.204+0000 7f505a3f2f80 -1 shutting down 2023-09-29T22:49:34.476 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T22:49:37.305 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T22:49:37.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:37.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:37.305 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:37.353 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:37.353 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:37.417 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:49:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj11", 2023-09-29T22:49:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.343021Z", 2023-09-29T22:49:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_nLxq9CRAuEXGXZHqLpTC1ko3H-Smp8G", 2023-09-29T22:49:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj111", 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 18 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.413973Z", 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_DKNyfqiBvobnTunrAfkklhzFTjryjwa", 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj115", 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.872201Z", 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_6MH-ot_iCAxnWm7upctnq6_xqwkQB0R", 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj116", 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.297302Z", 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_Ts7PALUtmsbuGmWadgeNIyzTplDhkMa", 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj124", 2023-09-29T22:49:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.084101Z", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_zYMq_ruyt504u5HjUZm8_WGmOb1vk1m", 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj145", 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.431 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.314031Z", 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.432 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_2dyrd3IEXpG_0PmOyrA9_0YCNBnd2LD", 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.433 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj148", 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:37.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.432471Z", 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.435 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_j2S9UXRgJVCqevVjFxMpucuxkC9pRYH", 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj155", 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.437 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.102172Z", 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.438 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_uWh6gEGLgQ_KV7jL98mNkOCem8yQ3SO", 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.439 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj17", 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.440 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.883322Z", 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.441 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_rwMuAawrqCOlMXer_llNDNu_qieTHbH", 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj185", 2023-09-29T22:49:37.442 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:37.443 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.490207Z", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.444 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_5ggt_GGqVhRpdmq2K_YRiNEFJkSWImH", 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj19", 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.445 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.804834Z", 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.446 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_r5NCx6N4aawSz1CGDnKDNhFHkevQm8i", 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.447 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj22", 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.448 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.359768Z", 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.449 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_61Bp2Mg6bFAoHZ_h8hDIvEGj2oYhOBf", 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.450 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj23", 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.451 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.719274Z", 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.452 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_VxtFpNLsjSALsypRRwst9W_Mmv_nnVg", 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj29", 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.454 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.172122Z", 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.455 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "__toXtOzddVwVJi4CpTR5n8A-fSGrCa2", 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.456 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj31", 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.457 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:47.954760Z", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.458 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_dODnQ8BTDWbSjYQ8TpEy7nfinrcxrXm", 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj33", 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.459 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.460 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.410124Z", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.461 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_fhZKlWLFu6K7r5PRsrLRxUXcdMi_6F7", 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj34", 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.462 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.935683Z", 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.463 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_gK90WyGzBB6gBvMyZzzPYYvL43t0JYB", 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.464 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj37", 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.465 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.340977Z", 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.466 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_rpzQWCj0Wq1Q3Y4xdrYHH-whdJhSmOX", 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj38", 2023-09-29T22:49:37.467 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.468 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.859697Z", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_j453QpWqrq4nxPHVNcCJT28CiW1we5T", 2023-09-29T22:49:37.469 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj44", 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.470 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.427697Z", 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.471 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_demxZzdi9SGL6LFiZHmLb1ovqLLquzS", 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.472 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj46", 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.473 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.736604Z", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.474 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_CjjPLLf6x81ukeVuusivZnvetrL_o3K", 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj51", 2023-09-29T22:49:37.475 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.953139Z", 2023-09-29T22:49:37.476 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_sXM08vQ63a_hBVJmsTE3EdnuS_S7tSZ", 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj55", 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.478 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.478252Z", 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.479 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_61qPOyxHxPRb-GFO0vL10UTPQAIo621", 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj57", 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.481 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.874896Z", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.482 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_-1msGt4jW0UjUKBtQByyszzKqEJEtLj", 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj58", 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.224394Z", 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.484 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_VxkdxXbHdWPxJghF9zhR2qeCIkyC4D2", 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.485 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj62", 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.486 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.012224Z", 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.487 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_IrS2hHnz59XFClXA4MhaPteHySW-7Y3", 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj68", 2023-09-29T22:49:37.488 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:37.489 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:06.006280Z", 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.490 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_4hTWGdItXLPO2s-_NosZbxStPb70p-Y", 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.491 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj69", 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.492 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.793247Z", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.493 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_SbnJkS2Zhgt_aCBbAcw6t9GXcXMYho0", 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj74", 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.494 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.358603Z", 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.495 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_SYSdA9K9ymn-2JlNurLJLPR-rK2vl_6", 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.496 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj76", 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.497 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.927079Z", 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.498 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_dUvhJmhq_vUMtQ4SwKXMjG6qyZ__qVl", 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj85", 2023-09-29T22:49:37.499 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.500 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:06.023186Z", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_JXBXQEK33cs6UOfjvzSry1bST47J3K6", 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj87", 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.503 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.242721Z", 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.504 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_kqOom02c-Xlu2RLFAoUsKVuDxAcvMCb", 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj92", 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.506 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.813048Z", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.507 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_aNGAqTBDZp6Edp44SimayJRS0-ya3wc", 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj93", 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.508 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.509 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.031005Z", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.510 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_JgHJAkl3soWguyku5VyKfZLL0wdcbq8", 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj95", 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:37.511 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.945391Z", 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:37.512 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_HeciEHsNoQPxriwIdk_4w-W0cGKmm-e", 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:37.513 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:37.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:49:37.514 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:49:37.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:37.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:37.514 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj11 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:37.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:37.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:37.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:37.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj111', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:37.730 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj111 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:37.775 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:37.958 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:37.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj115', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:37.959 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj115 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:38.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj116', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.185 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj116 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:38.232 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:38.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj124', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.416 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj124 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:38.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:38.641 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj145', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.642 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj145 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:38.689 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:38.690 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:38.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj148', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:38.872 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj148 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:38.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:38.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:39.094 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj155', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.094 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj155 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:39.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:39.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:39.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.320 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj17 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:39.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:39.364 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:39.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj185', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.544 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj185 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:39.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:39.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:39.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.772 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj19 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:39.819 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:39.819 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:39.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:39.997 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj22 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:40.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:40.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:40.080 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T22:49:40.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.221 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj23 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:40.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:40.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj29', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.446 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj29 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:40.492 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:40.492 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:40.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj31', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.671 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj31 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:40.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:40.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:40.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj33', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:40.944 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj33 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:40.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:40.991 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:41.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj34', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.172 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj34 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:41.219 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:41.397 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj37', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.397 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj37 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:41.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:41.443 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:41.626 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj38', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.627 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj38 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:41.672 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:41.673 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:41.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:41.847 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj44 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:41.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:41.895 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:42.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.069 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj46 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:42.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:42.115 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:42.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj51', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.296 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj51 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:42.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:42.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.523 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:42.569 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:42.569 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:42.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj57', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.753 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj57 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:42.797 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:42.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj58', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:42.981 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj58 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:43.028 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:43.028 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:43.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.203 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj62 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:43.250 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:43.251 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:43.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.428 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:43.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:43.473 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:43.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.660 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj69 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:43.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:43.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:43.893 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:43.893 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:43.939 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:43.939 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:44.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj76', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.113 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj76 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:44.159 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:44.160 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:44.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj85', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.337 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj85 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:44.385 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:44.386 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:44.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj87', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.575 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj87 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:44.618 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:44.618 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:44.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj92', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:44.794 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj92 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:44.880 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:44.881 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:45.064 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.065 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj93', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.065 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj93 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:45.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:45.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:45.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'object', 'rm', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--object', 'obj95', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 object rm --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --object obj95 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:45.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:45.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:45.511 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'trim', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.512 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog trim --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:45.556 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:45.556 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:45.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.650 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:45.737 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:45.738 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:45.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'run', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:49:45.827 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync run --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:49:45.872 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:45.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:46.018 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.016+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: init. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.016+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: waiting on bucket lease 2023-09-29T22:49:46.027 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.028+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: status after acquiring the lock is: init 2023-09-29T22:49:46.047 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.048+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bucket for full sync 2023-09-29T22:49:46.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj11 2023-09-29T22:49:46.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj11 op=0 op_state=1 2023-09-29T22:49:46.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj11[0] 2023-09-29T22:49:46.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj111 op=0 op_state=1 2023-09-29T22:49:46.056 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj115 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj115 op=0 op_state=1 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj115[0] 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj116 op=0 op_state=1 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj116[0] 2023-09-29T22:49:46.057 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj124 op=0 op_state=1 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj124[0] 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj145 op=0 op_state=1 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj145[0] 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj148 op=0 op_state=1 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj148[0] 2023-09-29T22:49:46.058 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj155 op=0 op_state=1 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj17 op=0 op_state=1 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj17[0] 2023-09-29T22:49:46.059 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj185 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj185 op=0 op_state=1 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj185[0] 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj19 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj19 op=0 op_state=1 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj19[0] 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj22 op=0 op_state=1 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: : injecting data sync error on key=obj22 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.060 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.052+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj23 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj23 op=0 op_state=1 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj23[0] 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj29 op=0 op_state=1 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.061 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj31 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj31 op=0 op_state=1 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj31[0] 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj33 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj33 op=0 op_state=1 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj33[0] 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj34 op=0 op_state=1 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: : injecting data sync error on key=obj34 2023-09-29T22:49:46.062 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj37 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj37 op=0 op_state=1 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj37[0] 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj38 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj38 op=0 op_state=1 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj38[0] 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj44 2023-09-29T22:49:46.063 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj44 op=0 op_state=1 2023-09-29T22:49:46.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj44[0] 2023-09-29T22:49:46.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj46 2023-09-29T22:49:46.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj46 op=0 op_state=1 2023-09-29T22:49:46.064 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.056+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj46[0] 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj51 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj51 op=0 op_state=1 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj51[0] 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj55 op=0 op_state=1 2023-09-29T22:49:46.065 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.060+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj55[0] 2023-09-29T22:49:46.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj57 op=0 op_state=1 2023-09-29T22:49:46.066 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj57[0] 2023-09-29T22:49:46.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj58 2023-09-29T22:49:46.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj58 op=0 op_state=1 2023-09-29T22:49:46.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj58[0] 2023-09-29T22:49:46.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.067 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj62 op=0 op_state=1 2023-09-29T22:49:46.068 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.064+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj62[0] 2023-09-29T22:49:46.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: success 2023-09-29T22:49:46.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj68 op=0 op_state=1 2023-09-29T22:49:46.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj68[0] 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: success 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: success 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj69 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj69 op=0 op_state=1 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj69[0] 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj74 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj74 op=0 op_state=1 2023-09-29T22:49:46.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj74[0] 2023-09-29T22:49:46.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: success 2023-09-29T22:49:46.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj76 op=0 op_state=1 2023-09-29T22:49:46.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj76[0] 2023-09-29T22:49:46.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: success 2023-09-29T22:49:46.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj85 op=0 op_state=1 2023-09-29T22:49:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.088+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj85[0] 2023-09-29T22:49:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: success 2023-09-29T22:49:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: success 2023-09-29T22:49:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj87 op=0 op_state=1 2023-09-29T22:49:46.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj87[0] 2023-09-29T22:49:46.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj92 2023-09-29T22:49:46.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj92 op=0 op_state=1 2023-09-29T22:49:46.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.092+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj92[0] 2023-09-29T22:49:46.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: success 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: success 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: success 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj93 op=0 op_state=1 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj93[0] 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj95 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj95 op=0 op_state=1 2023-09-29T22:49:46.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj95[0] 2023-09-29T22:49:46.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: success 2023-09-29T22:49:46.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: success 2023-09-29T22:49:46.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: success 2023-09-29T22:49:46.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: success 2023-09-29T22:49:46.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: success 2023-09-29T22:49:46.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: success 2023-09-29T22:49:46.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: success 2023-09-29T22:49:46.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: success 2023-09-29T22:49:46.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.096+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: success 2023-09-29T22:49:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.104+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: success 2023-09-29T22:49:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.104+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: success 2023-09-29T22:49:46.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.104+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: success 2023-09-29T22:49:46.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.104+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: success 2023-09-29T22:49:46.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.104+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: success 2023-09-29T22:49:46.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.108+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: success 2023-09-29T22:49:46.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.108+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: success 2023-09-29T22:49:46.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.108+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: success 2023-09-29T22:49:46.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.152+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: success 2023-09-29T22:49:46.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.152+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: success 2023-09-29T22:49:46.152 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.152+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: success 2023-09-29T22:49:46.153 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.152+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj11 2023-09-29T22:49:46.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.152+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: backing out with sync_status=-5 2023-09-29T22:49:46.155 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.152+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: ERROR: full sync failed. error: -5 2023-09-29T22:49:46.156 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.156+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.157 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.156+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: waiting on bucket lease 2023-09-29T22:49:46.161 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.160+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: status after acquiring the lock is: full 2023-09-29T22:49:46.176 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.176+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bucket for full sync 2023-09-29T22:49:46.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj11 2023-09-29T22:49:46.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj11 op=0 op_state=1 2023-09-29T22:49:46.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj11[0] 2023-09-29T22:49:46.183 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj111 op=0 op_state=1 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj111[0] 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj115 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj115 op=0 op_state=1 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj115[0] 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj116 op=0 op_state=1 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj116[0] 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj124 op=0 op_state=1 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj124[0] 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj145 op=0 op_state=1 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj145[0] 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj148 op=0 op_state=1 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj148[0] 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.185 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj155 op=0 op_state=1 2023-09-29T22:49:46.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj155[0] 2023-09-29T22:49:46.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj17 op=0 op_state=1 2023-09-29T22:49:46.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj17[0] 2023-09-29T22:49:46.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj185 2023-09-29T22:49:46.186 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj185 op=0 op_state=1 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj185[0] 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.180+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj19 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj19 op=0 op_state=1 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj19[0] 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj22 op=0 op_state=1 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj22[0] 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj23 2023-09-29T22:49:46.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj23 op=0 op_state=1 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj23[0] 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj29 op=0 op_state=1 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj29[0] 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj31 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj31 op=0 op_state=1 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj31[0] 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj33 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj33 op=0 op_state=1 2023-09-29T22:49:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj33[0] 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj34 op=0 op_state=1 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj34[0] 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj37 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj37 op=0 op_state=1 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj37[0] 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj38 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj38 op=0 op_state=1 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj38[0] 2023-09-29T22:49:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj44 2023-09-29T22:49:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj44 op=0 op_state=1 2023-09-29T22:49:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj44[0] 2023-09-29T22:49:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj46 2023-09-29T22:49:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj46 op=0 op_state=1 2023-09-29T22:49:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.184+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj46[0] 2023-09-29T22:49:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.188+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: success 2023-09-29T22:49:46.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.188+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj51 2023-09-29T22:49:46.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.188+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj51 op=0 op_state=1 2023-09-29T22:49:46.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.188+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj51[0] 2023-09-29T22:49:46.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: success 2023-09-29T22:49:46.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj55 op=0 op_state=1 2023-09-29T22:49:46.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj55[0] 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: success 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: success 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj57 op=0 op_state=1 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: success 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj57[0] 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj58 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj58 op=0 op_state=1 2023-09-29T22:49:46.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj58[0] 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: success 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: success 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj62 op=0 op_state=1 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj62[0] 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: success 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj68 op=0 op_state=1 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: : injecting data sync error on key=obj68 2023-09-29T22:49:46.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj69 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj69 op=0 op_state=1 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj69[0] 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj74 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj74 op=0 op_state=1 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj74[0] 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: success 2023-09-29T22:49:46.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: success 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj76 op=0 op_state=1 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: : injecting data sync error on key=obj76 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj85 op=0 op_state=1 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj85[0] 2023-09-29T22:49:46.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: success 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: success 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj87 op=0 op_state=1 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: success 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj92 2023-09-29T22:49:46.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj92 op=0 op_state=1 2023-09-29T22:49:46.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj92[0] 2023-09-29T22:49:46.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj93 op=0 op_state=1 2023-09-29T22:49:46.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj93[0] 2023-09-29T22:49:46.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: success 2023-09-29T22:49:46.199 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj95 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj95 op=0 op_state=1 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj95[0] 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: success 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.192+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: success 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: success 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: success 2023-09-29T22:49:46.201 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.196+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: success 2023-09-29T22:49:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: success 2023-09-29T22:49:46.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: success 2023-09-29T22:49:46.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: success 2023-09-29T22:49:46.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: success 2023-09-29T22:49:46.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.200+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: success 2023-09-29T22:49:46.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.236+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: success 2023-09-29T22:49:46.236 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.236+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj62 2023-09-29T22:49:46.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.236+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: backing out with sync_status=-5 2023-09-29T22:49:46.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.236+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: ERROR: full sync failed. error: -5 2023-09-29T22:49:46.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.240+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.240+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: waiting on bucket lease 2023-09-29T22:49:46.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.248+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: status after acquiring the lock is: full 2023-09-29T22:49:46.267 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.268+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bucket for full sync 2023-09-29T22:49:46.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj11 2023-09-29T22:49:46.319 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj11 op=0 op_state=1 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj11[0] 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj111 op=0 op_state=1 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj111[0] 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj115 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj115 op=0 op_state=1 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj115[0] 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj116 op=0 op_state=1 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: : injecting data sync error on key=obj116 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.316+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj124 op=0 op_state=1 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj124[0] 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj145 op=0 op_state=1 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj145[0] 2023-09-29T22:49:46.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj148 op=0 op_state=1 2023-09-29T22:49:46.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj148[0] 2023-09-29T22:49:46.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj155 op=0 op_state=1 2023-09-29T22:49:46.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: : injecting data sync error on key=obj155 2023-09-29T22:49:46.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj17 op=0 op_state=1 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj17[0] 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj185 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj185 op=0 op_state=1 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj185[0] 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj19 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj19 op=0 op_state=1 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj19[0] 2023-09-29T22:49:46.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj22 op=0 op_state=1 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj22[0] 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj23 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj23 op=0 op_state=1 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.320+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj23[0] 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj29 op=0 op_state=1 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj29[0] 2023-09-29T22:49:46.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj31 2023-09-29T22:49:46.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj31 op=0 op_state=1 2023-09-29T22:49:46.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj31[0] 2023-09-29T22:49:46.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj33 2023-09-29T22:49:46.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj33 op=0 op_state=1 2023-09-29T22:49:46.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj33[0] 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj34 op=0 op_state=1 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj34[0] 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj37 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj37 op=0 op_state=1 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj37[0] 2023-09-29T22:49:46.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj38 2023-09-29T22:49:46.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj38 op=0 op_state=1 2023-09-29T22:49:46.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.324+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj38[0] 2023-09-29T22:49:46.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj44 2023-09-29T22:49:46.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj44 op=0 op_state=1 2023-09-29T22:49:46.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj44[0] 2023-09-29T22:49:46.329 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj46 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj46 op=0 op_state=1 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: success 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: success 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj46[0] 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: success 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj51 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj51 op=0 op_state=1 2023-09-29T22:49:46.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: success 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj51[0] 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj55 op=0 op_state=1 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: success 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: success 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: success 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: success 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.328+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj55[0] 2023-09-29T22:49:46.333 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj57 op=0 op_state=1 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj57[0] 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj58 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj58 op=0 op_state=1 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj58[0] 2023-09-29T22:49:46.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: success 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj62 op=0 op_state=1 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: : injecting data sync error on key=obj62 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.337 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj68 op=0 op_state=1 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: success 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj68[0] 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj69 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj69 op=0 op_state=1 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj69[0] 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj74 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj74 op=0 op_state=1 2023-09-29T22:49:46.338 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj74[0] 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj76 op=0 op_state=1 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.332+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj76[0] 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj85 op=0 op_state=1 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: success 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: success 2023-09-29T22:49:46.339 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: success 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: success 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj87 op=0 op_state=1 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: success 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: : injecting data sync error on key=obj87 2023-09-29T22:49:46.340 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: success 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: success 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj92 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj92 op=0 op_state=1 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj92[0] 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: success 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: success 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: success 2023-09-29T22:49:46.341 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: success 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj93 op=0 op_state=1 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: success 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: : injecting data sync error on key=obj93 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj95 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj95 op=0 op_state=1 2023-09-29T22:49:46.342 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj95[0] 2023-09-29T22:49:46.343 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: success 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: success 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: success 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: success 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: success 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.336+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: success 2023-09-29T22:49:46.344 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: success 2023-09-29T22:49:46.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj115 2023-09-29T22:49:46.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: backing out with sync_status=-5 2023-09-29T22:49:46.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: ERROR: full sync failed. error: -5 2023-09-29T22:49:46.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.345 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.340+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: waiting on bucket lease 2023-09-29T22:49:46.348 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.348+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: status after acquiring the lock is: full 2023-09-29T22:49:46.362 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.360+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bucket for full sync 2023-09-29T22:49:46.369 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj11 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj11 op=0 op_state=1 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj11[0] 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj111 op=0 op_state=1 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: : injecting data sync error on key=obj111 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.370 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj115 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj115 op=0 op_state=1 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj115[0] 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj116 op=0 op_state=1 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj116[0] 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj124 op=0 op_state=1 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: : injecting data sync error on key=obj124 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.371 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj145 op=0 op_state=1 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj145[0] 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj148 op=0 op_state=1 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: : injecting data sync error on key=obj148 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.372 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj155 op=0 op_state=1 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj155[0] 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj17 op=0 op_state=1 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: : injecting data sync error on key=obj17 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj185 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj185 op=0 op_state=1 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj185[0] 2023-09-29T22:49:46.373 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj19 2023-09-29T22:49:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj19 op=0 op_state=1 2023-09-29T22:49:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj19[0] 2023-09-29T22:49:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj22 op=0 op_state=1 2023-09-29T22:49:46.374 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj22[0] 2023-09-29T22:49:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj23 2023-09-29T22:49:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj23 op=0 op_state=1 2023-09-29T22:49:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj23[0] 2023-09-29T22:49:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.376 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj29 op=0 op_state=1 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj29[0] 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj31 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj31 op=0 op_state=1 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj31[0] 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj33 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj33 op=0 op_state=1 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.368+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj33[0] 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj34 op=0 op_state=1 2023-09-29T22:49:46.377 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj34[0] 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj37 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj37 op=0 op_state=1 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj37[0] 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj38 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj38 op=0 op_state=1 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj38[0] 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj44 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj44 op=0 op_state=1 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj44[0] 2023-09-29T22:49:46.378 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj46 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj46 op=0 op_state=1 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj46[0] 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: success 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj51 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj51 op=0 op_state=1 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: success 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj51[0] 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.379 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj55 op=0 op_state=1 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: : injecting data sync error on key=obj55 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj57 op=0 op_state=1 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.372+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: : injecting data sync error on key=obj57 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.380 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.383 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: success 2023-09-29T22:49:46.384 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: success 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: success 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj58 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj58 op=0 op_state=1 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj58[0] 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj62 op=0 op_state=1 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj62[0] 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj68 op=0 op_state=1 2023-09-29T22:49:46.385 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj68[0] 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: success 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: success 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj69 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj69 op=0 op_state=1 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj69[0] 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj74 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj74 op=0 op_state=1 2023-09-29T22:49:46.386 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj74[0] 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj76 op=0 op_state=1 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj76[0] 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj85 op=0 op_state=1 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj85[0] 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: success 2023-09-29T22:49:46.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: success 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.376+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj87 op=0 op_state=1 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj87[0] 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj92 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj92 op=0 op_state=1 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj92[0] 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj93 op=0 op_state=1 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj93[0] 2023-09-29T22:49:46.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj95 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj95 op=0 op_state=1 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj95[0] 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: success 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: success 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: success 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: success 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: success 2023-09-29T22:49:46.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: success 2023-09-29T22:49:46.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: success 2023-09-29T22:49:46.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: success 2023-09-29T22:49:46.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: success 2023-09-29T22:49:46.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: success 2023-09-29T22:49:46.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.380+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: success 2023-09-29T22:49:46.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.384+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj11 2023-09-29T22:49:46.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.384+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: backing out with sync_status=-5 2023-09-29T22:49:46.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.384+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: ERROR: full sync failed. error: -5 2023-09-29T22:49:46.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.384+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.384+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: waiting on bucket lease 2023-09-29T22:49:46.432 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.432+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: status after acquiring the lock is: full 2023-09-29T22:49:46.450 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.448+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bucket for full sync 2023-09-29T22:49:46.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj11 2023-09-29T22:49:46.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj11 op=0 op_state=1 2023-09-29T22:49:46.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj11[0] 2023-09-29T22:49:46.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.458 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj111 op=0 op_state=1 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj111[0] 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj115 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj115 op=0 op_state=1 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj115[0] 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj116 op=0 op_state=1 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj116[0] 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.459 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj124 op=0 op_state=1 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj124[0] 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj145 op=0 op_state=1 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: : injecting data sync error on key=obj145 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj148 op=0 op_state=1 2023-09-29T22:49:46.460 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj148[0] 2023-09-29T22:49:46.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.461 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj155 op=0 op_state=1 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj155[0] 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj17 op=0 op_state=1 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj17[0] 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj185 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj185 op=0 op_state=1 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj185[0] 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj19 2023-09-29T22:49:46.462 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj19 op=0 op_state=1 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj19[0] 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj22 op=0 op_state=1 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj22[0] 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj23 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj23 op=0 op_state=1 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj23[0] 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.463 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj29 op=0 op_state=1 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: : injecting data sync error on key=obj29 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj31 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj31 op=0 op_state=1 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj31[0] 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj33 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj33 op=0 op_state=1 2023-09-29T22:49:46.464 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj33[0] 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj34 op=0 op_state=1 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj34[0] 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj37 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj37 op=0 op_state=1 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.456+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj37[0] 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj38 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj38 op=0 op_state=1 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj38[0] 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj44 2023-09-29T22:49:46.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj44 op=0 op_state=1 2023-09-29T22:49:46.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj44[0] 2023-09-29T22:49:46.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj46 2023-09-29T22:49:46.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj46 op=0 op_state=1 2023-09-29T22:49:46.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj46[0] 2023-09-29T22:49:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: success 2023-09-29T22:49:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: success 2023-09-29T22:49:46.469 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj51 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj51 op=0 op_state=1 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj51[0] 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.460+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj55 op=0 op_state=1 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj55[0] 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: success 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.470 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj57 op=0 op_state=1 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj57[0] 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: success 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: success 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj58 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj58 op=0 op_state=1 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj58[0] 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: success 2023-09-29T22:49:46.471 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: success 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: success 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: success 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: success 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj62 op=0 op_state=1 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj62[0] 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: success 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj124 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj68 op=0 op_state=1 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj68[0] 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: success 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: success 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj69 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj69 op=0 op_state=1 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: success 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: success 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: success 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj69[0] 2023-09-29T22:49:46.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: success 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: success 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: success 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj74 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj74 op=0 op_state=1 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.464+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj74[0] 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj76 op=0 op_state=1 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj76[0] 2023-09-29T22:49:46.474 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj85 op=0 op_state=1 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: : injecting data sync error on key=obj85 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: success 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: failed, retcode=-5 ((5) Input/output error) 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 0 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: ERROR: failed to sync object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj87 op=0 op_state=1 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: success 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj87[0] 2023-09-29T22:49:46.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj92 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj92 op=0 op_state=1 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj92[0] 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: success 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj93 op=0 op_state=1 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj93[0] 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj95 2023-09-29T22:49:46.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj95 op=0 op_state=1 2023-09-29T22:49:46.477 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj95[0] 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: success 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.468+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: success 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: success 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: success 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: a sync operation returned error 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: success 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: success 2023-09-29T22:49:46.478 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: success 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: success 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: success 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: success 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: backing out with sync_status=-5 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: ERROR: full sync failed. error: -5 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: full. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.472+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: waiting on bucket lease 2023-09-29T22:49:46.479 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.476+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: status after acquiring the lock is: full 2023-09-29T22:49:46.492 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.492+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bucket for full sync 2023-09-29T22:49:46.499 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj11 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj11 op=0 op_state=1 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj11[0] 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj111 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj111 op=0 op_state=1 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj111[0] 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj115 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj115 op=0 op_state=1 2023-09-29T22:49:46.500 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj115[0] 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj116 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj116 op=0 op_state=1 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj116[0] 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj124 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj124 op=0 op_state=1 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj124[0] 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj145 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj145 op=0 op_state=1 2023-09-29T22:49:46.501 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj145[0] 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj148 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj148 op=0 op_state=1 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj148[0] 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj155 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj155 op=0 op_state=1 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj155[0] 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj17 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj17 op=0 op_state=1 2023-09-29T22:49:46.502 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj17[0] 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj185 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj185 op=0 op_state=1 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj185[0] 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj19 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj19 op=0 op_state=1 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj19[0] 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj22 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj22 op=0 op_state=1 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj22[0] 2023-09-29T22:49:46.503 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj23 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj23 op=0 op_state=1 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj23[0] 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj29 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj29 op=0 op_state=1 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj29[0] 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj31 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj31 op=0 op_state=1 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj31[0] 2023-09-29T22:49:46.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj33 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj33 op=0 op_state=1 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.496+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj33[0] 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj34 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj34 op=0 op_state=1 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj34[0] 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj37 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj37 op=0 op_state=1 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj37[0] 2023-09-29T22:49:46.505 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj38 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj38 op=0 op_state=1 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj38[0] 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj44 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj44 op=0 op_state=1 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj44[0] 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj46 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj46 op=0 op_state=1 2023-09-29T22:49:46.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj46[0] 2023-09-29T22:49:46.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj124]: success 2023-09-29T22:49:46.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj51 2023-09-29T22:49:46.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj51 op=0 op_state=1 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj51[0] 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj116]: success 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj55 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj55 op=0 op_state=1 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.500+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj55[0] 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj11]: success 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj111]: success 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj155]: success 2023-09-29T22:49:46.508 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj115]: success 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj148]: success 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj22]: success 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj57 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj57 op=0 op_state=1 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj57[0] 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj58 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj58 op=0 op_state=1 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj58[0] 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj145]: success 2023-09-29T22:49:46.509 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj62 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj62 op=0 op_state=1 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj62[0] 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj34]: success 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj68 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj68 op=0 op_state=1 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj68[0] 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj69 2023-09-29T22:49:46.510 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj69 op=0 op_state=1 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj69[0] 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj74 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj74 op=0 op_state=1 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj74[0] 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj76 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj76 op=0 op_state=1 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj76[0] 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj85 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj85 op=0 op_state=1 2023-09-29T22:49:46.511 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj85[0] 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj31]: success 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj37]: success 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj87 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj87 op=0 op_state=1 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj87[0] 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj92 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj92 op=0 op_state=1 2023-09-29T22:49:46.512 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.504+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj92[0] 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj33]: success 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj19]: success 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj93 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj93 op=0 op_state=1 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj93[0] 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: [full sync] syncing object: hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0/obj95 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b=bucket sync single entry (source_zone=a75fe7d4-5635-4f57-8059-26c1aacd3a0f) b= log_entry=obj95 op=0 op_state=1 2023-09-29T22:49:46.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: bucket sync: sync obj: a75fe7d4-5635-4f57-8059-26c1aacd3a0f/:hcnphz-59[a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24])/obj95[0] 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj23]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj29]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj17]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj17 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj185]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj38]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj46]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj58]: success 2023-09-29T22:49:46.515 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj76]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj51]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.508+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj44]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj74]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj68]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj62]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj69]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj95]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj57]: success 2023-09-29T22:49:46.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj87]: success 2023-09-29T22:49:46.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj85]: success 2023-09-29T22:49:46.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj55]: success 2023-09-29T22:49:46.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj87 2023-09-29T22:49:46.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj93]: success 2023-09-29T22:49:46.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:entry[obj92]: success 2023-09-29T22:49:46.517 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: updating marker oid=bucket.full-sync-status.a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24 marker=obj95 2023-09-29T22:49:46.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.512+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: set bucket state=incremental 2023-09-29T22:49:46.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.516+0000 7f74104d1f40 5 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:full_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: bucket status objv={r=_BNrp0CE4gPm8RzxLmhlVwL8:149,w=:0} 2023-09-29T22:49:46.518 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.516+0000 7f74104d1f40 10 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: WARNING: requested sync of past generation 0 < 6, returning success 2023-09-29T22:49:46.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.519 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:1]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:3]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:2]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.520 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:4]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: sync status for source bucket shard: 2 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:0]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:5]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:1]: sync status for source bucket shard: 2 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:2]: sync status for source bucket shard: 2 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:4]: sync status for source bucket shard: 2 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:3]: sync status for source bucket shard: 2 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:1]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:1]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:2]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:2]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.521 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:4]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:4]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:3]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:3]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:7]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:6]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:5]: sync status for source bucket shard: 2 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:5]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:5]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:8]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:9]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:10]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:11]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:12]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:13]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:7]: sync status for source bucket shard: 2 2023-09-29T22:49:46.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:6]: sync status for source bucket shard: 2 2023-09-29T22:49:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:7]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:7]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:6]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:6]: listing bilog for incremental sync; position=00000000002.105.6 2023-09-29T22:49:46.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:14]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:8]: sync status for source bucket shard: 2 2023-09-29T22:49:46.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:8]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:8]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:15]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.520+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:16]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:9]: sync status for source bucket shard: 2 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:17]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:10]: sync status for source bucket shard: 2 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:13]: sync status for source bucket shard: 2 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:12]: sync status for source bucket shard: 2 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:11]: sync status for source bucket shard: 2 2023-09-29T22:49:46.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:9]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:9]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:10]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:10]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:13]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:13]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:12]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:12]: listing bilog for incremental sync; position=00000000002.84.6 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:11]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:11]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:18]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:20]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:19]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:14]: sync status for source bucket shard: 2 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:16]: sync status for source bucket shard: 2 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:15]: sync status for source bucket shard: 2 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:14]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:14]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:16]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:16]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:15]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:15]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:21]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:17]: sync status for source bucket shard: 2 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:17]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:17]: listing bilog for incremental sync; position=00000000002.99.6 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:22]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:18]: sync status for source bucket shard: 2 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:18]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:18]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:23]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:20]: sync status for source bucket shard: 2 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:20]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:20]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:24]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:19]: sync status for source bucket shard: 2 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:19]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:19]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:21]: sync status for source bucket shard: 2 2023-09-29T22:49:46.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:21]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:21]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:25]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:26]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:22]: sync status for source bucket shard: 2 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:23]: sync status for source bucket shard: 2 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:24]: sync status for source bucket shard: 2 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:22]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:22]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:23]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:23]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:24]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:24]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:27]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:28]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:26]: sync status for source bucket shard: 2 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:26]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:26]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:29]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:25]: sync status for source bucket shard: 2 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:25]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:25]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:30]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:31]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:27]: sync status for source bucket shard: 2 2023-09-29T22:49:46.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:28]: sync status for source bucket shard: 2 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:27]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:27]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:28]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:28]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.524+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:32]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:29]: sync status for source bucket shard: 2 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:33]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:34]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:29]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:29]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:30]: sync status for source bucket shard: 2 2023-09-29T22:49:46.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:30]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:30]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:35]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:36]: sync status for source bucket: incremental. lease is: not taken. stop indications is: 0 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:31]: sync status for source bucket shard: 2 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:32]: sync status for source bucket shard: 2 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:31]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:31]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:32]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:32]: listing bilog for incremental sync; position=00000000001.113.6 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:34]: sync status for source bucket shard: 2 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:33]: sync status for source bucket shard: 2 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:34]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:34]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:33]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:33]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:35]: sync status for source bucket shard: 2 2023-09-29T22:49:46.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:36]: sync status for source bucket shard: 2 2023-09-29T22:49:46.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:35]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:35]: listing bilog for incremental sync; position= 2023-09-29T22:49:46.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:46.528+0000 7f74104d1f40 20 RGW-SYNC:bucket[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24<-hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:36]:inc_sync[hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24:36]: listing bilog for incremental sync; position=00000000001.95.6 2023-09-29T22:49:46.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:46.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:46.622 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:46.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:46.666 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:46.729 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:49:46.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj11", 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.730 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.343021Z", 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.731 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.18201579254359014931", 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj111", 2023-09-29T22:49:46.732 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.733 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.413973Z", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.578916349546261479", 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj115", 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.872201Z", 2023-09-29T22:49:46.736 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15195203411226195419", 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.737 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj116", 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:46.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.297302Z", 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.739 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.3507860784608234968", 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.740 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj124", 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.741 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.084101Z", 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.742 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11689542820289710159", 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.743 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj145", 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.744 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.314031Z", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.745 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16387013039995494822", 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj148", 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.746 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.432471Z", 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.747 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.17209412161982628365", 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj155", 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:46.749 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.102172Z", 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.750 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.4755937345482590133", 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.751 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj17", 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.752 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.883322Z", 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.753 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16485214782536472946", 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.754 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj185", 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.755 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.490207Z", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.756 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.12685125432679294942", 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj19", 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.757 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.804834Z", 2023-09-29T22:49:46.758 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11421695641587843545", 2023-09-29T22:49:46.759 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj22", 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.760 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.359768Z", 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.761 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.2372691188230227799", 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj23", 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.763 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.719274Z", 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.764 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.18394685256790188913", 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj29", 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.766 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.172122Z", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.13174965069759993315", 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj31", 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.768 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:47.954760Z", 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.769 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.3996482307891896839", 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.770 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj33", 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.771 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.410124Z", 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.772 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.1015282961201443769", 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj34", 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.774 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.935683Z", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10353882933797428902", 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj37", 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.776 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.340977Z", 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.777 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.14548865906993801742", 2023-09-29T22:49:46.778 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj38", 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:46.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.859697Z", 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.780 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10189194630914998245", 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj44", 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 1 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.782 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.427697Z", 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.783 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.297198074572545119", 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj46", 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.785 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.736604Z", 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.786 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.1317944465906780066", 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj51", 2023-09-29T22:49:46.787 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.788 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.953139Z", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.789 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.3477674747286355681", 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj55", 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.478252Z", 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.791 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.9469431131229407232", 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.792 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj57", 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.793 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.874896Z", 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.794 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10946243682012997680", 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj58", 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.796 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.224394Z", 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.797 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.14102178547748409075", 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.798 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj62", 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.799 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.012224Z", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.800 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5263285671870198005", 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj68", 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.801 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.802 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:06.006280Z", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.803 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.8139114442762038235", 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj69", 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 2 2023-09-29T22:49:46.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.793247Z", 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.805 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.11307711544551731657", 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.806 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj74", 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.807 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.358603Z", 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.808 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10711570540550992624", 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj76", 2023-09-29T22:49:46.809 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 3 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.810 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.927079Z", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16015696285108618599", 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj85", 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.812 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.813 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 24 2023-09-29T22:49:46.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:06.023186Z", 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.814 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.2056696409414008335", 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.815 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj87", 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.816 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.242721Z", 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.817 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.5719498196360444980", 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj92", 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 18 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.819 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.813048Z", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.820 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.9519047108401087845", 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj93", 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.821 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.031005Z", 2023-09-29T22:49:46.822 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.3369156137688468945", 2023-09-29T22:49:46.823 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj95", 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.825 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 9, 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.945391Z", 2023-09-29T22:49:46.826 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.827 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.15424734167430648244", 2023-09-29T22:49:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:49:46.828 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:49:46.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:46.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'list', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-59', '--max-entries', '10000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:49:46.829 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket list --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-59 --max-entries 10000 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:49:46.902 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:46.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:46.965 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:49:46.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj11", 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.966 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.343021Z", 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.967 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_VvnCXnp-yGceZn9IT9jrEn3sv4PMMit", 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj111", 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.968 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.413973Z", 2023-09-29T22:49:46.969 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.970 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_4natiqxEPHgBFphVJAU2bz1iwB7d-3R", 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj115", 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.971 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.872201Z", 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.972 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_sNpOOGgxn1frNSlzIFbNB-bgkTnvDc6", 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.973 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj116", 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.974 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.297302Z", 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.975 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_wqC3d3pdGzXZfZKL0yECRL1TwldmVBm", 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.976 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj124", 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.977 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.084101Z", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_il0TFUz5nhvlmT898r7yvHuP6g_4hkM", 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj145", 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.979 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.314031Z", 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.980 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_-kPMA7ls3EYg9wy8sJN2WKyhXvXMmlX", 2023-09-29T22:49:46.981 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj148", 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:46.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.432471Z", 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.983 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_XVKyQOQ7q1QqU6QPLYsgsDYFwYRj9e_", 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj155", 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.985 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.102172Z", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.986 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_sEqGwLHuRI_j0YNdu8m7LEbWYgIYTI3", 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj17", 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.987 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 23 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.883322Z", 2023-09-29T22:49:46.988 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_W-W1G6zGSmUjuK8Ets_zxdADnYkjIkc", 2023-09-29T22:49:46.989 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj185", 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:46.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.490207Z", 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.991 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_SmaCcmWR-I9yBTVBbD3UKTHp-h_4jbV", 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.992 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj19", 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.993 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.804834Z", 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.994 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_2dk4DGvF0DqPR3yoEZ07qUKy9zwi7Bg", 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj22", 2023-09-29T22:49:46.995 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 12 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:46.996 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.359768Z", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.997 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_eN50Jrvwb8eoioAsYGoaDDO9kwmDmcJ", 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj23", 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:46.998 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.719274Z", 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:46.999 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_PhY2e3AAuC7F-TB8pr5yPqv8wUCq1Ph", 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.000 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj29", 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.001 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.172122Z", 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.002 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_kEfz7YjBvwXjimt9t8Fu-MrDGRO1LGY", 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj31", 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.004 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:47.954760Z", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_nGLJqgAOK0Qd6n8Zjel64gre9qHpipk", 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj33", 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.006 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 9 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.410124Z", 2023-09-29T22:49:47.007 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_lvxDeSS3TW4iQfgkkehQRgFTNprwR1w", 2023-09-29T22:49:47.008 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj34", 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.009 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.935683Z", 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.010 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_2Y4frTOniA533l80uOizVQUAIdNj5lQ", 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.011 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj37", 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.012 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.340977Z", 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.013 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_FUvjAq38cr0J9LY505JKHF3WwZkGvfe", 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj38", 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.015 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.859697Z", 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.016 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_pgVqr5EV-y8qReMrr5E9umHYNDAvt6t", 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj44", 2023-09-29T22:49:47.017 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.018 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.427697Z", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_wFq7H1lyY4ZSo2ECnkJfyyA5AO0FNUh", 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj46", 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.736604Z", 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.021 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_dq3w6PhF1Z1W-IX9o-jk3mdIQT9E_-v", 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj51", 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.023 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:05.953139Z", 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.024 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_WMg84ZUlI0OB3FaMuQtvg_gnCt8pW0_", 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj55", 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.026 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:35:44.478252Z", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.027 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_rq9mI5X5db1JzEGpXFvYbwrYZbZM7SY", 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj57", 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.028 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.874896Z", 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.029 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_UhFf3oWzw4ckG5LDMwm3urfv3-yLVPB", 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.030 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj58", 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:47.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.224394Z", 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.032 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_WYhEfPwafoCZrK9sbB9x2F13JzxAwQc", 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.033 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj62", 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 5 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.034 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.012224Z", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_NJkckZBtg_uBVRfypno85u9KIxfczs3", 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj68", 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.036 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:06.006280Z", 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.037 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_WpEtwnXmZ-rrzSwFzQPKa-FAzEKlp11", 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.038 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj69", 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 4 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.039 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.793247Z", 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.040 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_4TQW9SOuY_ZyO6iT1UdLVC5dtUS-Doj", 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj74", 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.042 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:48:09.358603Z", 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.043 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_EKITAxN2Q_s72tW_QwqjyEeLoGaeHDC", 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.044 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj76", 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 6 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:47.045 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.927079Z", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_EvcyTFtNkLCQ4WiMXrBPo3_sn6M1cUa", 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj85", 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.047 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 11 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 3, 2023-09-29T22:49:47.048 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:36:06.023186Z", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.049 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_VLILXcGy-C03JTWf0Bk9rR_MIQ0BHcM", 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj87", 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.050 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:49:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.051 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:43:47.242721Z", 2023-09-29T22:49:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.052 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_OLbQCZESTtjwj3KM50MA6r_h8J6s9e0", 2023-09-29T22:49:47.053 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj92", 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.054 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 10 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 5, 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:42:07.813048Z", 2023-09-29T22:49:47.055 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_xuHutwGn326t40TnfY1XtPtvHpzvvpp", 2023-09-29T22:49:47.056 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj93", 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.057 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 8 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 16, 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:46:48.031005Z", 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.058 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_h-mFqdfn6M8ge4ZejMc67nK6Oa-keqe", 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.059 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "obj95", 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: "instance": "", 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: "ver": { 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: "pool": 13, 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 7 2023-09-29T22:49:47.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "locator": "", 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "exists": true, 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "meta": { 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "category": 1, 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "size": 6, 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "mtime": "2023-09-29T22:38:48.945391Z", 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "etag": "acbd18db4cc2f85cedef654fccc4a4d8", 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_class": "", 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "owner": "rgw-multisite-test-user", 2023-09-29T22:49:47.061 INFO:teuthology.orchestra.run.smithi112.stdout: "owner_display_name": "Multisite Test User", 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "content_type": "application/octet-stream", 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "accounted_size": 3, 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "user_data": "", 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "appendable": false 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "tag": "_77z-i5Px2mwCLCgfIJiAlXKrczlSUJE", 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "flags": 0, 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pending_map": [], 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: "versioned_epoch": 0 2023-09-29T22:49:47.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:49:47.063 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:49:47.063 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:49:47.063 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:49:47.066 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:49:47.066 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:49:47.087 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:49:47.088 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:49:47.088 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:49:47.088 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T22:49:48.089 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:49:48.103 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:49:48.103 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:49:48.104 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:49:48.104 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T22:49:48.105 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-59 2023-09-29T22:49:48.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:49:48.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:49:48.105 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:49:48.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:48.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:48.263 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:48.264+0000 7fd37fbb1f40 1 bucket sync caught up with source: 2023-09-29T22:49:48.264 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:48.264 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.105.6, , , , , , 00000000002.84.6, , , , , 00000000002.99.6, , , , , , , , , , , , , , , 00000000001.113.6, , , , 00000000001.95.6] 2023-09-29T22:49:48.264 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:48.264+0000 7fd37fbb1f40 0 bucket checkpoint complete 2023-09-29T22:49:48.283 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-59 2023-09-29T22:49:48.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:49:48.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-59', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:49:48.284 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-59 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:49:48.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:49:48.333 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:49:48.396 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:48.396+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:48.396 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:49:48.396 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:49:53.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:53.396+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:53.399 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:49:53.399 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:49:58.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:49:58.400+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:49:58.402 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:49:58.402 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:03.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:03.404+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:03.404 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:03.404 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:08.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:08.403+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:08.407 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:08.407 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:13.409 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:13.407+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:13.409 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:13.409 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:18.411 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:18.411+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:18.412 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:18.412 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:23.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:23.411+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:23.414 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:23.414 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:28.416 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:28.415+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:28.416 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:28.416 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:33.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:33.415+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:33.419 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:33.419 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:38.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:38.419+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:38.421 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:38.421 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:43.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:43.423+0000 7f1fce019f40 1 waiting for incremental sync to catch up: 2023-09-29T22:50:43.424 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000005.221.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000004.300.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:43.424 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:48.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:48.422+0000 7f1fce019f40 1 bucket sync caught up with source: 2023-09-29T22:50:48.427 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:48.427 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000009.289.6, 00000000006.606.6, 00000000006.283.6, 00000000015.268.6, 00000000012.288.6, 00000000003.195.6, 00000000003.324.6, 00000000012.322.6, 00000000021.273.6, 00000000018.367.6] 2023-09-29T22:50:48.427 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:48.422+0000 7f1fce019f40 0 bucket checkpoint complete 2023-09-29T22:50:48.455 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:50:48.455 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.455 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.455 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.455 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:50:48.456 DEBUG:boto:Data: 2023-09-29T22:50:48.456 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.456 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.456 DEBUG:boto:Params: {} 2023-09-29T22:50:48.456 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:50:48.457 DEBUG:boto:Token: None 2023-09-29T22:50:48.457 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MY0VTmDFm3efm+Dxcmvw7gkwpFw= 2023-09-29T22:50:48.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MY0VTmDFm3efm+Dxcmvw7gkwpFw=', 'Content-Length': '0'} 2023-09-29T22:50:48.466 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005a620590492ff093-00651754c8-1da5-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.467 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.467 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.467 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.467 DEBUG:boto:Path: /hcnphz-59/ 2023-09-29T22:50:48.467 DEBUG:boto:Data: 2023-09-29T22:50:48.467 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.467 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.467 DEBUG:boto:Params: {} 2023-09-29T22:50:48.468 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:50:48.468 DEBUG:boto:Token: None 2023-09-29T22:50:48.468 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MY0VTmDFm3efm+Dxcmvw7gkwpFw= 2023-09-29T22:50:48.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MY0VTmDFm3efm+Dxcmvw7gkwpFw=', 'Content-Length': '0'} 2023-09-29T22:50:48.476 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008b9c71b3e96a54c7-00651754c8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.476 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:50:48.476 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.477 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.477 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.477 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.477 DEBUG:boto:Method: GET 2023-09-29T22:50:48.477 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.477 DEBUG:boto:Data: 2023-09-29T22:50:48.477 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.477 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.477 DEBUG:boto:Params: {} 2023-09-29T22:50:48.478 DEBUG:boto:Token: None 2023-09-29T22:50:48.478 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/?versions 2023-09-29T22:50:48.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE= 2023-09-29T22:50:48.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE=', 'Content-Length': '0'} 2023-09-29T22:50:48.482 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd1fea588027d643-00651754c8-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.483 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-29T22:50:48.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:50:48.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:50:48.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:50:48.488 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:50:48.488 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.488 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.489 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.489 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.489 DEBUG:boto:Method: GET 2023-09-29T22:50:48.489 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.489 DEBUG:boto:Data: 2023-09-29T22:50:48.489 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.489 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.489 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.489 DEBUG:boto:Params: {} 2023-09-29T22:50:48.490 DEBUG:boto:Token: None 2023-09-29T22:50:48.490 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/?versions 2023-09-29T22:50:48.490 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE= 2023-09-29T22:50:48.490 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE=', 'Content-Length': '0'} 2023-09-29T22:50:48.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000006591d316fb30cf8-00651754c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.497 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:50:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-29T22:50:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:50:48.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:50:48.502 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:50:48.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:50:48.503 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.503 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.503 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.503 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.503 DEBUG:boto:Method: GET 2023-09-29T22:50:48.503 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.503 DEBUG:boto:Data: 2023-09-29T22:50:48.503 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.504 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.504 DEBUG:boto:Params: {} 2023-09-29T22:50:48.504 DEBUG:boto:Token: None 2023-09-29T22:50:48.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/?versions 2023-09-29T22:50:48.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE= 2023-09-29T22:50:48.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE=', 'Content-Length': '0'} 2023-09-29T22:50:48.508 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f68abc9242a269d-00651754c8-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.508 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.510 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.510 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.511 DEBUG:boto:path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.511 DEBUG:boto:auth_path=/hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.511 DEBUG:boto:Method: GET 2023-09-29T22:50:48.511 DEBUG:boto:Path: /hcnphz-59/?versions&max-keys=999 2023-09-29T22:50:48.511 DEBUG:boto:Data: 2023-09-29T22:50:48.511 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.511 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.511 DEBUG:boto:Params: {} 2023-09-29T22:50:48.511 DEBUG:boto:Token: None 2023-09-29T22:50:48.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/?versions 2023-09-29T22:50:48.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE= 2023-09-29T22:50:48.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+e2d8zr3fDAlBP0CSa3eb8dWmJE=', 'Content-Length': '0'} 2023-09-29T22:50:48.519 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083e651c8fc6fd1f0-00651754c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.520 DEBUG:boto:b'hcnphz-59999falseobj11nulltrue2023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:50:48.522 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.522 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.522 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.522 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.522 DEBUG:boto:Method: GET 2023-09-29T22:50:48.523 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.523 DEBUG:boto:Data: 2023-09-29T22:50:48.523 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.523 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.523 DEBUG:boto:Params: {} 2023-09-29T22:50:48.523 DEBUG:boto:Token: None 2023-09-29T22:50:48.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4= 2023-09-29T22:50:48.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4=', 'Content-Length': '0'} 2023-09-29T22:50:48.529 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3865912046db0b8-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.574 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.574 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.575 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.575 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.575 DEBUG:boto:Method: GET 2023-09-29T22:50:48.575 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.575 DEBUG:boto:Data: 2023-09-29T22:50:48.575 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.575 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.575 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.576 DEBUG:boto:Params: {} 2023-09-29T22:50:48.576 DEBUG:boto:Token: None 2023-09-29T22:50:48.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4= 2023-09-29T22:50:48.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4=', 'Content-Length': '0'} 2023-09-29T22:50:48.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605ccdead834305e-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.579 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.580 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.580 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.580 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.580 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.580 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.580 DEBUG:boto:Data: 2023-09-29T22:50:48.580 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.581 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.581 DEBUG:boto:Params: {} 2023-09-29T22:50:48.581 DEBUG:boto:Token: None 2023-09-29T22:50:48.581 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/+18LsfK9pAyAjsf/FhqsB+tPfo= 2023-09-29T22:50:48.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/+18LsfK9pAyAjsf/FhqsB+tPfo=', 'Content-Length': '0'} 2023-09-29T22:50:48.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5d1c0ad243da4d1-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.584 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.584 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.585 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.585 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.585 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.585 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.585 DEBUG:boto:Data: 2023-09-29T22:50:48.585 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.585 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.585 DEBUG:boto:Params: {} 2023-09-29T22:50:48.586 DEBUG:boto:Token: None 2023-09-29T22:50:48.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/+18LsfK9pAyAjsf/FhqsB+tPfo= 2023-09-29T22:50:48.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/+18LsfK9pAyAjsf/FhqsB+tPfo=', 'Content-Length': '0'} 2023-09-29T22:50:48.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023cb7f157adb64ba-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:50:48.589 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.589 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.590 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.590 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.590 DEBUG:boto:Method: GET 2023-09-29T22:50:48.590 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.590 DEBUG:boto:Data: 2023-09-29T22:50:48.590 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.590 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.590 DEBUG:boto:Params: {} 2023-09-29T22:50:48.591 DEBUG:boto:Token: None 2023-09-29T22:50:48.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4= 2023-09-29T22:50:48.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4=', 'Content-Length': '0'} 2023-09-29T22:50:48.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000828659b67ac0a3ae-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.594 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.594 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.594 DEBUG:boto:path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.594 DEBUG:boto:auth_path=/hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.595 DEBUG:boto:Method: GET 2023-09-29T22:50:48.595 DEBUG:boto:Path: /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.595 DEBUG:boto:Data: 2023-09-29T22:50:48.595 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.595 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.595 DEBUG:boto:Params: {} 2023-09-29T22:50:48.595 DEBUG:boto:Token: None 2023-09-29T22:50:48.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11?versionId=null 2023-09-29T22:50:48.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4= 2023-09-29T22:50:48.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k3sIglAYTfvJeRtLVMv1AfqUuu4=', 'Content-Length': '0'} 2023-09-29T22:50:48.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2095ed1623db2ee-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.649 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.649 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.650 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.650 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.650 DEBUG:boto:Method: GET 2023-09-29T22:50:48.650 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.650 DEBUG:boto:Data: 2023-09-29T22:50:48.650 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.650 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.650 DEBUG:boto:Params: {} 2023-09-29T22:50:48.651 DEBUG:boto:Token: None 2023-09-29T22:50:48.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.657 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009bf6926bc23602bd-00651754c8-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.657 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.658 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.658 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.658 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.658 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.658 DEBUG:boto:Method: GET 2023-09-29T22:50:48.658 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.658 DEBUG:boto:Data: 2023-09-29T22:50:48.658 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.659 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.659 DEBUG:boto:Params: {} 2023-09-29T22:50:48.659 DEBUG:boto:Token: None 2023-09-29T22:50:48.659 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.666 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000784d0cc6b3d14a2a-00651754c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.667 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.667 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.667 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.667 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.667 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:50:48.668 DEBUG:boto:Data: 2023-09-29T22:50:48.668 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.668 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.668 DEBUG:boto:Params: {} 2023-09-29T22:50:48.668 DEBUG:boto:Token: None 2023-09-29T22:50:48.668 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11 2023-09-29T22:50:48.668 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gApjNKReKEaUvARRBEXV6A3AWFI= 2023-09-29T22:50:48.668 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gApjNKReKEaUvARRBEXV6A3AWFI=', 'Content-Length': '0'} 2023-09-29T22:50:48.670 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2f8346dfe9f92b-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.670 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.671 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.671 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.671 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:50:48.671 DEBUG:boto:Data: 2023-09-29T22:50:48.671 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.671 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.671 DEBUG:boto:Params: {} 2023-09-29T22:50:48.671 DEBUG:boto:Token: None 2023-09-29T22:50:48.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11 2023-09-29T22:50:48.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gApjNKReKEaUvARRBEXV6A3AWFI= 2023-09-29T22:50:48.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gApjNKReKEaUvARRBEXV6A3AWFI=', 'Content-Length': '0'} 2023-09-29T22:50:48.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f44ca1f9d8de53-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:50:48.675 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.675 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.675 DEBUG:boto:Method: GET 2023-09-29T22:50:48.675 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:50:48.675 DEBUG:boto:Data: 2023-09-29T22:50:48.675 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.675 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.676 DEBUG:boto:Params: {} 2023-09-29T22:50:48.676 DEBUG:boto:Token: None 2023-09-29T22:50:48.676 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11 2023-09-29T22:50:48.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Grkk5pBURnfJJDrV1CBYNmo0/Tg= 2023-09-29T22:50:48.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Grkk5pBURnfJJDrV1CBYNmo0/Tg=', 'Content-Length': '0'} 2023-09-29T22:50:48.679 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e624ceb75c70efa-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.723 DEBUG:boto:path=/hcnphz-59/obj11 2023-09-29T22:50:48.723 DEBUG:boto:auth_path=/hcnphz-59/obj11 2023-09-29T22:50:48.723 DEBUG:boto:Method: GET 2023-09-29T22:50:48.724 DEBUG:boto:Path: /hcnphz-59/obj11 2023-09-29T22:50:48.724 DEBUG:boto:Data: 2023-09-29T22:50:48.724 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.724 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.724 DEBUG:boto:Params: {} 2023-09-29T22:50:48.724 DEBUG:boto:Token: None 2023-09-29T22:50:48.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj11 2023-09-29T22:50:48.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Grkk5pBURnfJJDrV1CBYNmo0/Tg= 2023-09-29T22:50:48.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Grkk5pBURnfJJDrV1CBYNmo0/Tg=', 'Content-Length': '0'} 2023-09-29T22:50:48.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045b3451a622f959a-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.729 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.729 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.729 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.730 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.730 DEBUG:boto:Method: GET 2023-09-29T22:50:48.730 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.730 DEBUG:boto:Data: 2023-09-29T22:50:48.730 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.730 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.730 DEBUG:boto:Params: {} 2023-09-29T22:50:48.730 DEBUG:boto:Token: None 2023-09-29T22:50:48.731 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.731 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.731 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.736 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000253469a49337a8c-00651754c8-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.736 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.737 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.737 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.737 DEBUG:boto:path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.737 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.737 DEBUG:boto:Method: GET 2023-09-29T22:50:48.737 DEBUG:boto:Path: /hcnphz-59/?prefix=obj11 2023-09-29T22:50:48.738 DEBUG:boto:Data: 2023-09-29T22:50:48.738 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.738 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.738 DEBUG:boto:Params: {} 2023-09-29T22:50:48.738 DEBUG:boto:Token: None 2023-09-29T22:50:48.738 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.738 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.744 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e70864fe144312f-00651754c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.745 DEBUG:boto:b'hcnphz-59obj111000falseobj112023-09-29T22:35:44.343Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:50:48.745 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.745 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.746 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.746 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.746 DEBUG:boto:Method: GET 2023-09-29T22:50:48.746 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.746 DEBUG:boto:Data: 2023-09-29T22:50:48.746 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.746 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.746 DEBUG:boto:Params: {} 2023-09-29T22:50:48.747 DEBUG:boto:Token: None 2023-09-29T22:50:48.747 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.747 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8= 2023-09-29T22:50:48.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8=', 'Content-Length': '0'} 2023-09-29T22:50:48.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000057675170a28bbdea-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.750 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.750 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.750 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.750 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.750 DEBUG:boto:Method: GET 2023-09-29T22:50:48.750 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.750 DEBUG:boto:Data: 2023-09-29T22:50:48.751 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.751 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.751 DEBUG:boto:Params: {} 2023-09-29T22:50:48.751 DEBUG:boto:Token: None 2023-09-29T22:50:48.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.751 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8= 2023-09-29T22:50:48.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8=', 'Content-Length': '0'} 2023-09-29T22:50:48.753 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4e822bc34e398ce-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.798 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.798 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.799 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.799 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.799 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.799 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.799 DEBUG:boto:Data: 2023-09-29T22:50:48.799 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.799 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.799 DEBUG:boto:Params: {} 2023-09-29T22:50:48.800 DEBUG:boto:Token: None 2023-09-29T22:50:48.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oSS8NwhbVXzJSW4eWFQb/W1AwsU= 2023-09-29T22:50:48.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oSS8NwhbVXzJSW4eWFQb/W1AwsU=', 'Content-Length': '0'} 2023-09-29T22:50:48.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0aee596b20db722-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.803 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.803 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.804 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.804 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.804 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.804 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.804 DEBUG:boto:Data: 2023-09-29T22:50:48.804 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.804 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.804 DEBUG:boto:Params: {} 2023-09-29T22:50:48.805 DEBUG:boto:Token: None 2023-09-29T22:50:48.805 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oSS8NwhbVXzJSW4eWFQb/W1AwsU= 2023-09-29T22:50:48.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oSS8NwhbVXzJSW4eWFQb/W1AwsU=', 'Content-Length': '0'} 2023-09-29T22:50:48.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000019305c39072c2ef6-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:50:48.808 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.808 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.808 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.808 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.809 DEBUG:boto:Method: GET 2023-09-29T22:50:48.809 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.809 DEBUG:boto:Data: 2023-09-29T22:50:48.809 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.809 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.809 DEBUG:boto:Params: {} 2023-09-29T22:50:48.809 DEBUG:boto:Token: None 2023-09-29T22:50:48.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8= 2023-09-29T22:50:48.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8=', 'Content-Length': '0'} 2023-09-29T22:50:48.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000066e93273943b61b0-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.854 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.854 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.855 DEBUG:boto:path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.855 DEBUG:boto:auth_path=/hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.855 DEBUG:boto:Method: GET 2023-09-29T22:50:48.855 DEBUG:boto:Path: /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.855 DEBUG:boto:Data: 2023-09-29T22:50:48.855 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.855 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.855 DEBUG:boto:Params: {} 2023-09-29T22:50:48.856 DEBUG:boto:Token: None 2023-09-29T22:50:48.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111?versionId=null 2023-09-29T22:50:48.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8= 2023-09-29T22:50:48.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QORFteoizjbrAjSuuaXAXp67fd8=', 'Content-Length': '0'} 2023-09-29T22:50:48.859 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d54c12b395a2d58-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.859 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.859 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.859 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.860 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.860 DEBUG:boto:Method: GET 2023-09-29T22:50:48.860 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.860 DEBUG:boto:Data: 2023-09-29T22:50:48.860 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.860 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.860 DEBUG:boto:Params: {} 2023-09-29T22:50:48.860 DEBUG:boto:Token: None 2023-09-29T22:50:48.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.866 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009fe8156c7c0f266d-00651754c8-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.866 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.866 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.867 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.867 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.867 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.867 DEBUG:boto:Method: GET 2023-09-29T22:50:48.867 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.867 DEBUG:boto:Data: 2023-09-29T22:50:48.867 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.867 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.868 DEBUG:boto:Params: {} 2023-09-29T22:50:48.868 DEBUG:boto:Token: None 2023-09-29T22:50:48.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.876 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a700089b4a78521-00651754c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.876 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.877 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.877 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.877 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.877 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:50:48.877 DEBUG:boto:Data: 2023-09-29T22:50:48.877 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.878 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.878 DEBUG:boto:Params: {} 2023-09-29T22:50:48.878 DEBUG:boto:Token: None 2023-09-29T22:50:48.878 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111 2023-09-29T22:50:48.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CeDk390IfLT9MvH3PlcX6kAg9vE= 2023-09-29T22:50:48.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CeDk390IfLT9MvH3PlcX6kAg9vE=', 'Content-Length': '0'} 2023-09-29T22:50:48.880 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b32cfbf84ff3c44f-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.880 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.881 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.881 DEBUG:boto:Method: HEAD 2023-09-29T22:50:48.881 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:50:48.881 DEBUG:boto:Data: 2023-09-29T22:50:48.881 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.881 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.881 DEBUG:boto:Params: {} 2023-09-29T22:50:48.881 DEBUG:boto:Token: None 2023-09-29T22:50:48.882 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111 2023-09-29T22:50:48.882 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CeDk390IfLT9MvH3PlcX6kAg9vE= 2023-09-29T22:50:48.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CeDk390IfLT9MvH3PlcX6kAg9vE=', 'Content-Length': '0'} 2023-09-29T22:50:48.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008113df62fb1049e9-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:50:48.884 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.884 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.885 DEBUG:boto:Method: GET 2023-09-29T22:50:48.885 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:50:48.885 DEBUG:boto:Data: 2023-09-29T22:50:48.885 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.885 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.885 DEBUG:boto:Params: {} 2023-09-29T22:50:48.885 DEBUG:boto:Token: None 2023-09-29T22:50:48.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111 2023-09-29T22:50:48.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9GVEJSToDGbifWs/SAUZs3UzHSo= 2023-09-29T22:50:48.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9GVEJSToDGbifWs/SAUZs3UzHSo=', 'Content-Length': '0'} 2023-09-29T22:50:48.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f79af7addf90c7d-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.888 DEBUG:boto:path=/hcnphz-59/obj111 2023-09-29T22:50:48.888 DEBUG:boto:auth_path=/hcnphz-59/obj111 2023-09-29T22:50:48.888 DEBUG:boto:Method: GET 2023-09-29T22:50:48.888 DEBUG:boto:Path: /hcnphz-59/obj111 2023-09-29T22:50:48.889 DEBUG:boto:Data: 2023-09-29T22:50:48.889 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.889 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.889 DEBUG:boto:Params: {} 2023-09-29T22:50:48.889 DEBUG:boto:Token: None 2023-09-29T22:50:48.889 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj111 2023-09-29T22:50:48.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9GVEJSToDGbifWs/SAUZs3UzHSo= 2023-09-29T22:50:48.890 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9GVEJSToDGbifWs/SAUZs3UzHSo=', 'Content-Length': '0'} 2023-09-29T22:50:48.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080c49bf9d4a8123e-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.934 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.934 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.935 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.935 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.935 DEBUG:boto:Method: GET 2023-09-29T22:50:48.935 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.935 DEBUG:boto:Data: 2023-09-29T22:50:48.935 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.935 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.935 DEBUG:boto:Params: {} 2023-09-29T22:50:48.936 DEBUG:boto:Token: None 2023-09-29T22:50:48.936 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.936 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.940 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000debf52d8cab672b2-00651754c8-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.941 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.941 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:48.941 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:48.941 DEBUG:boto:path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.941 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.942 DEBUG:boto:Method: GET 2023-09-29T22:50:48.942 DEBUG:boto:Path: /hcnphz-59/?prefix=obj111 2023-09-29T22:50:48.942 DEBUG:boto:Data: 2023-09-29T22:50:48.942 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.942 DEBUG:boto:Port: 8000 2023-09-29T22:50:48.942 DEBUG:boto:Params: {} 2023-09-29T22:50:48.942 DEBUG:boto:Token: None 2023-09-29T22:50:48.943 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/ 2023-09-29T22:50:48.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg= 2023-09-29T22:50:48.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lti2D9TjMMPA0nFvFkucESkdEgg=', 'Content-Length': '0'} 2023-09-29T22:50:48.951 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000010f30d67c61e6b19-00651754c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.951 DEBUG:boto:b'hcnphz-59obj1111000falseobj1112023-09-29T22:48:09.413Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:48.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:50:48.952 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:48.952 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:48.952 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.952 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.952 DEBUG:boto:Method: GET 2023-09-29T22:50:48.952 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.952 DEBUG:boto:Data: 2023-09-29T22:50:48.952 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:48.953 DEBUG:boto:Port: 8001 2023-09-29T22:50:48.953 DEBUG:boto:Params: {} 2023-09-29T22:50:48.953 DEBUG:boto:Token: None 2023-09-29T22:50:48.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:48 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9bkFyxqDizvNFq9EX2p+Erm3g5I= 2023-09-29T22:50:48.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9bkFyxqDizvNFq9EX2p+Erm3g5I=', 'Content-Length': '0'} 2023-09-29T22:50:48.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9b01cdb4baf759b-00651754c8-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:48.998 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:48.999 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:48.999 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.999 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.999 DEBUG:boto:Method: GET 2023-09-29T22:50:48.999 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:48.999 DEBUG:boto:Data: 2023-09-29T22:50:48.999 DEBUG:boto:Headers: {} 2023-09-29T22:50:48.999 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:48.999 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.000 DEBUG:boto:Params: {} 2023-09-29T22:50:49.000 DEBUG:boto:Token: None 2023-09-29T22:50:49.000 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ohayAfvD6VeR27gNr5Jx/PF8sQ= 2023-09-29T22:50:49.000 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ohayAfvD6VeR27gNr5Jx/PF8sQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.003 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008aae9ac5efae6907-00651754c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.004 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.004 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.004 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.004 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.004 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.004 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.004 DEBUG:boto:Data: 2023-09-29T22:50:49.004 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.005 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.005 DEBUG:boto:Params: {} 2023-09-29T22:50:49.005 DEBUG:boto:Token: None 2023-09-29T22:50:49.005 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qzcrbqxTCmywPvXYCPve8TUSNYo= 2023-09-29T22:50:49.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qzcrbqxTCmywPvXYCPve8TUSNYo=', 'Content-Length': '0'} 2023-09-29T22:50:49.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041dafa4f176411fb-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.008 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.008 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.008 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.009 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.009 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.009 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.009 DEBUG:boto:Data: 2023-09-29T22:50:49.009 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.009 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.009 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.009 DEBUG:boto:Params: {} 2023-09-29T22:50:49.009 DEBUG:boto:Token: None 2023-09-29T22:50:49.010 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.010 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qzcrbqxTCmywPvXYCPve8TUSNYo= 2023-09-29T22:50:49.010 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qzcrbqxTCmywPvXYCPve8TUSNYo=', 'Content-Length': '0'} 2023-09-29T22:50:49.013 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b739ac4179fb54c-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:50:49.013 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.013 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.014 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.014 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.014 DEBUG:boto:Method: GET 2023-09-29T22:50:49.014 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.014 DEBUG:boto:Data: 2023-09-29T22:50:49.015 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.015 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.015 DEBUG:boto:Params: {} 2023-09-29T22:50:49.015 DEBUG:boto:Token: None 2023-09-29T22:50:49.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ohayAfvD6VeR27gNr5Jx/PF8sQ= 2023-09-29T22:50:49.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ohayAfvD6VeR27gNr5Jx/PF8sQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e68f73610d85bbab-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.019 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.020 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.020 DEBUG:boto:path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.020 DEBUG:boto:auth_path=/hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.020 DEBUG:boto:Method: GET 2023-09-29T22:50:49.020 DEBUG:boto:Path: /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.020 DEBUG:boto:Data: 2023-09-29T22:50:49.020 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.020 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.021 DEBUG:boto:Params: {} 2023-09-29T22:50:49.021 DEBUG:boto:Token: None 2023-09-29T22:50:49.021 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115?versionId=null 2023-09-29T22:50:49.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ohayAfvD6VeR27gNr5Jx/PF8sQ= 2023-09-29T22:50:49.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ohayAfvD6VeR27gNr5Jx/PF8sQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058eef606fcfa14d7-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.024 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.025 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.025 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.025 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.025 DEBUG:boto:Method: GET 2023-09-29T22:50:49.026 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.026 DEBUG:boto:Data: 2023-09-29T22:50:49.026 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.026 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.026 DEBUG:boto:Params: {} 2023-09-29T22:50:49.027 DEBUG:boto:Token: None 2023-09-29T22:50:49.027 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.027 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.028 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.033 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c8a9c7e8e274d2c-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.074 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.075 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.075 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.075 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.075 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.075 DEBUG:boto:Method: GET 2023-09-29T22:50:49.075 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.076 DEBUG:boto:Data: 2023-09-29T22:50:49.076 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.076 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.076 DEBUG:boto:Params: {} 2023-09-29T22:50:49.076 DEBUG:boto:Token: None 2023-09-29T22:50:49.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000829289d83ae7b1e9-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.086 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.087 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.087 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.087 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.087 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:50:49.087 DEBUG:boto:Data: 2023-09-29T22:50:49.088 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.088 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.088 DEBUG:boto:Params: {} 2023-09-29T22:50:49.088 DEBUG:boto:Token: None 2023-09-29T22:50:49.089 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115 2023-09-29T22:50:49.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tsn0G/omQAZYC/eRsNk+I6WbzxY= 2023-09-29T22:50:49.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tsn0G/omQAZYC/eRsNk+I6WbzxY=', 'Content-Length': '0'} 2023-09-29T22:50:49.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e25e147013aaf2b4-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.092 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.092 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.092 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.092 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:50:49.092 DEBUG:boto:Data: 2023-09-29T22:50:49.092 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.093 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.093 DEBUG:boto:Params: {} 2023-09-29T22:50:49.093 DEBUG:boto:Token: None 2023-09-29T22:50:49.094 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115 2023-09-29T22:50:49.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tsn0G/omQAZYC/eRsNk+I6WbzxY= 2023-09-29T22:50:49.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tsn0G/omQAZYC/eRsNk+I6WbzxY=', 'Content-Length': '0'} 2023-09-29T22:50:49.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca9fdade5f40f68-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:50:49.096 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.096 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.097 DEBUG:boto:Method: GET 2023-09-29T22:50:49.097 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:50:49.097 DEBUG:boto:Data: 2023-09-29T22:50:49.097 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.097 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.097 DEBUG:boto:Params: {} 2023-09-29T22:50:49.097 DEBUG:boto:Token: None 2023-09-29T22:50:49.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115 2023-09-29T22:50:49.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Du8kOulrAyTXfzXd1EK02pTdPJs= 2023-09-29T22:50:49.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Du8kOulrAyTXfzXd1EK02pTdPJs=', 'Content-Length': '0'} 2023-09-29T22:50:49.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002741c43c006e18e8-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.100 DEBUG:boto:path=/hcnphz-59/obj115 2023-09-29T22:50:49.100 DEBUG:boto:auth_path=/hcnphz-59/obj115 2023-09-29T22:50:49.100 DEBUG:boto:Method: GET 2023-09-29T22:50:49.100 DEBUG:boto:Path: /hcnphz-59/obj115 2023-09-29T22:50:49.100 DEBUG:boto:Data: 2023-09-29T22:50:49.100 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.101 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.101 DEBUG:boto:Params: {} 2023-09-29T22:50:49.101 DEBUG:boto:Token: None 2023-09-29T22:50:49.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj115 2023-09-29T22:50:49.101 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Du8kOulrAyTXfzXd1EK02pTdPJs= 2023-09-29T22:50:49.101 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Du8kOulrAyTXfzXd1EK02pTdPJs=', 'Content-Length': '0'} 2023-09-29T22:50:49.103 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a46d0830d2b7e55d-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.147 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.147 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.147 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.147 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.147 DEBUG:boto:Method: GET 2023-09-29T22:50:49.147 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.148 DEBUG:boto:Data: 2023-09-29T22:50:49.148 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.148 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.148 DEBUG:boto:Params: {} 2023-09-29T22:50:49.149 DEBUG:boto:Token: None 2023-09-29T22:50:49.149 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.154 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d360c3d59e3c45c-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.155 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.155 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.156 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.156 DEBUG:boto:path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.156 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.156 DEBUG:boto:Method: GET 2023-09-29T22:50:49.156 DEBUG:boto:Path: /hcnphz-59/?prefix=obj115 2023-09-29T22:50:49.156 DEBUG:boto:Data: 2023-09-29T22:50:49.156 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.157 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.157 DEBUG:boto:Params: {} 2023-09-29T22:50:49.157 DEBUG:boto:Token: None 2023-09-29T22:50:49.157 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.157 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.165 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000af9797a5546e47a4-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.165 DEBUG:boto:b'hcnphz-59obj1151000falseobj1152023-09-29T22:42:07.872Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:50:49.166 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.166 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.166 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.166 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.167 DEBUG:boto:Method: GET 2023-09-29T22:50:49.167 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.167 DEBUG:boto:Data: 2023-09-29T22:50:49.167 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.167 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.167 DEBUG:boto:Params: {} 2023-09-29T22:50:49.168 DEBUG:boto:Token: None 2023-09-29T22:50:49.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY= 2023-09-29T22:50:49.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY=', 'Content-Length': '0'} 2023-09-29T22:50:49.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030dc1830400084ef-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.214 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.215 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.215 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.215 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.215 DEBUG:boto:Method: GET 2023-09-29T22:50:49.215 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.215 DEBUG:boto:Data: 2023-09-29T22:50:49.215 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.216 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.216 DEBUG:boto:Params: {} 2023-09-29T22:50:49.216 DEBUG:boto:Token: None 2023-09-29T22:50:49.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY= 2023-09-29T22:50:49.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY=', 'Content-Length': '0'} 2023-09-29T22:50:49.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7ed8398150291f8-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.219 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.220 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.220 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.220 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.220 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.220 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.220 DEBUG:boto:Data: 2023-09-29T22:50:49.220 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.221 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.221 DEBUG:boto:Params: {} 2023-09-29T22:50:49.221 DEBUG:boto:Token: None 2023-09-29T22:50:49.221 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIT9WbRuU6ru1MPPTImBgVGRIvQ= 2023-09-29T22:50:49.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIT9WbRuU6ru1MPPTImBgVGRIvQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cda34ffe04e52d-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.224 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.224 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.224 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.224 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.224 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.224 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.224 DEBUG:boto:Data: 2023-09-29T22:50:49.224 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.225 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.225 DEBUG:boto:Params: {} 2023-09-29T22:50:49.225 DEBUG:boto:Token: None 2023-09-29T22:50:49.225 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tIT9WbRuU6ru1MPPTImBgVGRIvQ= 2023-09-29T22:50:49.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tIT9WbRuU6ru1MPPTImBgVGRIvQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.228 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d93d9f54545bbb3f-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:50:49.228 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.228 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.228 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.228 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.228 DEBUG:boto:Method: GET 2023-09-29T22:50:49.229 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.229 DEBUG:boto:Data: 2023-09-29T22:50:49.229 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.229 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.229 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.229 DEBUG:boto:Params: {} 2023-09-29T22:50:49.229 DEBUG:boto:Token: None 2023-09-29T22:50:49.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.230 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY= 2023-09-29T22:50:49.230 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY=', 'Content-Length': '0'} 2023-09-29T22:50:49.232 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eac1fd32a0a7cb08-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.232 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.233 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.233 DEBUG:boto:path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.233 DEBUG:boto:auth_path=/hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.233 DEBUG:boto:Method: GET 2023-09-29T22:50:49.233 DEBUG:boto:Path: /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.233 DEBUG:boto:Data: 2023-09-29T22:50:49.233 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.233 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.234 DEBUG:boto:Params: {} 2023-09-29T22:50:49.234 DEBUG:boto:Token: None 2023-09-29T22:50:49.234 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116?versionId=null 2023-09-29T22:50:49.234 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY= 2023-09-29T22:50:49.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vh+xx/PTeicOolaNoieYMv0xKBY=', 'Content-Length': '0'} 2023-09-29T22:50:49.237 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004010f5af32a0d9e5-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.278 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.279 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.279 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.279 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.279 DEBUG:boto:Method: GET 2023-09-29T22:50:49.279 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.279 DEBUG:boto:Data: 2023-09-29T22:50:49.279 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.280 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.280 DEBUG:boto:Params: {} 2023-09-29T22:50:49.280 DEBUG:boto:Token: None 2023-09-29T22:50:49.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.285 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000379ccac7cf17b227-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.286 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.286 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.286 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.286 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.287 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.287 DEBUG:boto:Method: GET 2023-09-29T22:50:49.287 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.287 DEBUG:boto:Data: 2023-09-29T22:50:49.287 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.287 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.287 DEBUG:boto:Params: {} 2023-09-29T22:50:49.287 DEBUG:boto:Token: None 2023-09-29T22:50:49.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.295 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000083c0a0e1a804385e-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.295 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.296 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.296 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.296 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.296 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:50:49.296 DEBUG:boto:Data: 2023-09-29T22:50:49.296 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.297 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.297 DEBUG:boto:Params: {} 2023-09-29T22:50:49.297 DEBUG:boto:Token: None 2023-09-29T22:50:49.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116 2023-09-29T22:50:49.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zbyNsiTMZynWSFBW9rJqqLvnmG8= 2023-09-29T22:50:49.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zbyNsiTMZynWSFBW9rJqqLvnmG8=', 'Content-Length': '0'} 2023-09-29T22:50:49.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fce938e8738d4243-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.299 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.300 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.300 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.300 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:50:49.300 DEBUG:boto:Data: 2023-09-29T22:50:49.300 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.300 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.300 DEBUG:boto:Params: {} 2023-09-29T22:50:49.300 DEBUG:boto:Token: None 2023-09-29T22:50:49.301 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116 2023-09-29T22:50:49.301 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zbyNsiTMZynWSFBW9rJqqLvnmG8= 2023-09-29T22:50:49.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zbyNsiTMZynWSFBW9rJqqLvnmG8=', 'Content-Length': '0'} 2023-09-29T22:50:49.303 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005594530c236083d8-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:50:49.303 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.303 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.303 DEBUG:boto:Method: GET 2023-09-29T22:50:49.304 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:50:49.304 DEBUG:boto:Data: 2023-09-29T22:50:49.304 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.304 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.304 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.304 DEBUG:boto:Params: {} 2023-09-29T22:50:49.304 DEBUG:boto:Token: None 2023-09-29T22:50:49.304 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116 2023-09-29T22:50:49.304 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJ5sgaHIyBUGoyNf4T8YXrLZodo= 2023-09-29T22:50:49.305 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJ5sgaHIyBUGoyNf4T8YXrLZodo=', 'Content-Length': '0'} 2023-09-29T22:50:49.307 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f01de85eb694630-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.350 DEBUG:boto:path=/hcnphz-59/obj116 2023-09-29T22:50:49.351 DEBUG:boto:auth_path=/hcnphz-59/obj116 2023-09-29T22:50:49.351 DEBUG:boto:Method: GET 2023-09-29T22:50:49.351 DEBUG:boto:Path: /hcnphz-59/obj116 2023-09-29T22:50:49.351 DEBUG:boto:Data: 2023-09-29T22:50:49.351 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.351 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.351 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.351 DEBUG:boto:Params: {} 2023-09-29T22:50:49.351 DEBUG:boto:Token: None 2023-09-29T22:50:49.352 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj116 2023-09-29T22:50:49.352 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tJ5sgaHIyBUGoyNf4T8YXrLZodo= 2023-09-29T22:50:49.352 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tJ5sgaHIyBUGoyNf4T8YXrLZodo=', 'Content-Length': '0'} 2023-09-29T22:50:49.355 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4adae751a7e19ee-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.355 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.355 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.355 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.356 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.356 DEBUG:boto:Method: GET 2023-09-29T22:50:49.356 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.356 DEBUG:boto:Data: 2023-09-29T22:50:49.356 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.357 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.357 DEBUG:boto:Params: {} 2023-09-29T22:50:49.357 DEBUG:boto:Token: None 2023-09-29T22:50:49.357 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.362 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095bef36a7fc4416d-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.363 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.363 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.364 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.364 DEBUG:boto:path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.364 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.364 DEBUG:boto:Method: GET 2023-09-29T22:50:49.364 DEBUG:boto:Path: /hcnphz-59/?prefix=obj116 2023-09-29T22:50:49.364 DEBUG:boto:Data: 2023-09-29T22:50:49.365 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.365 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.365 DEBUG:boto:Params: {} 2023-09-29T22:50:49.365 DEBUG:boto:Token: None 2023-09-29T22:50:49.365 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.372 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e4854a9f43baab7f-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.414 DEBUG:boto:b'hcnphz-59obj1161000falseobj1162023-09-29T22:43:47.297Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:50:49.415 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.415 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.415 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.415 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.416 DEBUG:boto:Method: GET 2023-09-29T22:50:49.416 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.416 DEBUG:boto:Data: 2023-09-29T22:50:49.416 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.416 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.416 DEBUG:boto:Params: {} 2023-09-29T22:50:49.416 DEBUG:boto:Token: None 2023-09-29T22:50:49.417 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo= 2023-09-29T22:50:49.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo=', 'Content-Length': '0'} 2023-09-29T22:50:49.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067509140179f6eb7-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.421 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.421 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.421 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.421 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.421 DEBUG:boto:Method: GET 2023-09-29T22:50:49.421 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.421 DEBUG:boto:Data: 2023-09-29T22:50:49.421 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.422 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.422 DEBUG:boto:Params: {} 2023-09-29T22:50:49.422 DEBUG:boto:Token: None 2023-09-29T22:50:49.422 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo= 2023-09-29T22:50:49.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo=', 'Content-Length': '0'} 2023-09-29T22:50:49.425 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000121a59d9b798db3c-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.425 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.425 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.426 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.426 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.426 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.426 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.426 DEBUG:boto:Data: 2023-09-29T22:50:49.426 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.427 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.427 DEBUG:boto:Params: {} 2023-09-29T22:50:49.427 DEBUG:boto:Token: None 2023-09-29T22:50:49.427 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:psmY/cb32lLNeYiEeBuErV94WZk= 2023-09-29T22:50:49.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:psmY/cb32lLNeYiEeBuErV94WZk=', 'Content-Length': '0'} 2023-09-29T22:50:49.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008e8b89ec9bec419e-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.430 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.431 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.431 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.431 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.431 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.431 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.431 DEBUG:boto:Data: 2023-09-29T22:50:49.431 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.431 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.431 DEBUG:boto:Params: {} 2023-09-29T22:50:49.432 DEBUG:boto:Token: None 2023-09-29T22:50:49.432 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:psmY/cb32lLNeYiEeBuErV94WZk= 2023-09-29T22:50:49.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:psmY/cb32lLNeYiEeBuErV94WZk=', 'Content-Length': '0'} 2023-09-29T22:50:49.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4c267dfbb5d73f2-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:50:49.435 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.435 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.436 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.436 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.436 DEBUG:boto:Method: GET 2023-09-29T22:50:49.436 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.436 DEBUG:boto:Data: 2023-09-29T22:50:49.436 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.437 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.437 DEBUG:boto:Params: {} 2023-09-29T22:50:49.437 DEBUG:boto:Token: None 2023-09-29T22:50:49.437 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo= 2023-09-29T22:50:49.438 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo=', 'Content-Length': '0'} 2023-09-29T22:50:49.441 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c8e293a203149c3-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.441 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.441 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.441 DEBUG:boto:path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.442 DEBUG:boto:auth_path=/hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.442 DEBUG:boto:Method: GET 2023-09-29T22:50:49.442 DEBUG:boto:Path: /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.442 DEBUG:boto:Data: 2023-09-29T22:50:49.442 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.442 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.442 DEBUG:boto:Params: {} 2023-09-29T22:50:49.443 DEBUG:boto:Token: None 2023-09-29T22:50:49.443 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124?versionId=null 2023-09-29T22:50:49.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo= 2023-09-29T22:50:49.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Rf3H/XMNQ9uplSAQfEeIRYMydDo=', 'Content-Length': '0'} 2023-09-29T22:50:49.446 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f977bb8f619a7801-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.447 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.447 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.447 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.447 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.448 DEBUG:boto:Method: GET 2023-09-29T22:50:49.448 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.448 DEBUG:boto:Data: 2023-09-29T22:50:49.448 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.448 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.449 DEBUG:boto:Params: {} 2023-09-29T22:50:49.449 DEBUG:boto:Token: None 2023-09-29T22:50:49.449 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.449 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.449 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.454 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b8efab6c9a02ddbf-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.454 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.455 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.455 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.455 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.456 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.456 DEBUG:boto:Method: GET 2023-09-29T22:50:49.456 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.456 DEBUG:boto:Data: 2023-09-29T22:50:49.456 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.457 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.457 DEBUG:boto:Params: {} 2023-09-29T22:50:49.457 DEBUG:boto:Token: None 2023-09-29T22:50:49.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.465 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de2fd02b315be378-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.510 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.511 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.511 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.511 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.511 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:50:49.512 DEBUG:boto:Data: 2023-09-29T22:50:49.512 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.512 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.512 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.512 DEBUG:boto:Params: {} 2023-09-29T22:50:49.512 DEBUG:boto:Token: None 2023-09-29T22:50:49.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124 2023-09-29T22:50:49.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oWJkRLECtwT0ulaLNDCekCNwyjA= 2023-09-29T22:50:49.513 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oWJkRLECtwT0ulaLNDCekCNwyjA=', 'Content-Length': '0'} 2023-09-29T22:50:49.515 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d09442f0292119b-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.516 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.516 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.516 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.516 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:50:49.516 DEBUG:boto:Data: 2023-09-29T22:50:49.516 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.516 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.517 DEBUG:boto:Params: {} 2023-09-29T22:50:49.517 DEBUG:boto:Token: None 2023-09-29T22:50:49.517 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124 2023-09-29T22:50:49.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oWJkRLECtwT0ulaLNDCekCNwyjA= 2023-09-29T22:50:49.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oWJkRLECtwT0ulaLNDCekCNwyjA=', 'Content-Length': '0'} 2023-09-29T22:50:49.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bec36312202febb6-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:50:49.520 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.521 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.521 DEBUG:boto:Method: GET 2023-09-29T22:50:49.521 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:50:49.521 DEBUG:boto:Data: 2023-09-29T22:50:49.521 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.522 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.522 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.522 DEBUG:boto:Params: {} 2023-09-29T22:50:49.522 DEBUG:boto:Token: None 2023-09-29T22:50:49.523 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124 2023-09-29T22:50:49.523 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GlhNUtzgKQ5h0ZIvCCFRk7Lev3A= 2023-09-29T22:50:49.523 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GlhNUtzgKQ5h0ZIvCCFRk7Lev3A=', 'Content-Length': '0'} 2023-09-29T22:50:49.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b899a7157ced1741-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.566 DEBUG:boto:path=/hcnphz-59/obj124 2023-09-29T22:50:49.566 DEBUG:boto:auth_path=/hcnphz-59/obj124 2023-09-29T22:50:49.567 DEBUG:boto:Method: GET 2023-09-29T22:50:49.567 DEBUG:boto:Path: /hcnphz-59/obj124 2023-09-29T22:50:49.567 DEBUG:boto:Data: 2023-09-29T22:50:49.567 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.567 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.567 DEBUG:boto:Params: {} 2023-09-29T22:50:49.567 DEBUG:boto:Token: None 2023-09-29T22:50:49.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj124 2023-09-29T22:50:49.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GlhNUtzgKQ5h0ZIvCCFRk7Lev3A= 2023-09-29T22:50:49.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GlhNUtzgKQ5h0ZIvCCFRk7Lev3A=', 'Content-Length': '0'} 2023-09-29T22:50:49.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aea0b06db4bdc876-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.571 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.571 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.572 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.572 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.572 DEBUG:boto:Method: GET 2023-09-29T22:50:49.572 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.572 DEBUG:boto:Data: 2023-09-29T22:50:49.572 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.572 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.573 DEBUG:boto:Params: {} 2023-09-29T22:50:49.573 DEBUG:boto:Token: None 2023-09-29T22:50:49.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.578 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005063233ef7b2c643-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.578 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.579 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.579 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.579 DEBUG:boto:path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.579 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.579 DEBUG:boto:Method: GET 2023-09-29T22:50:49.579 DEBUG:boto:Path: /hcnphz-59/?prefix=obj124 2023-09-29T22:50:49.579 DEBUG:boto:Data: 2023-09-29T22:50:49.580 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.580 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.580 DEBUG:boto:Params: {} 2023-09-29T22:50:49.580 DEBUG:boto:Token: None 2023-09-29T22:50:49.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.588 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6a34e4c3c483e15-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.588 DEBUG:boto:b'hcnphz-59obj1241000falseobj1242023-09-29T22:46:48.084Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:50:49.589 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.589 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.589 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.589 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.589 DEBUG:boto:Method: GET 2023-09-29T22:50:49.589 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.590 DEBUG:boto:Data: 2023-09-29T22:50:49.590 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.590 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.590 DEBUG:boto:Params: {} 2023-09-29T22:50:49.590 DEBUG:boto:Token: None 2023-09-29T22:50:49.590 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.590 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0= 2023-09-29T22:50:49.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0=', 'Content-Length': '0'} 2023-09-29T22:50:49.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004acd419f27eaa848-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.594 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.594 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.594 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.594 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.594 DEBUG:boto:Method: GET 2023-09-29T22:50:49.594 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.594 DEBUG:boto:Data: 2023-09-29T22:50:49.594 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.595 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.595 DEBUG:boto:Params: {} 2023-09-29T22:50:49.595 DEBUG:boto:Token: None 2023-09-29T22:50:49.595 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.595 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0= 2023-09-29T22:50:49.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0=', 'Content-Length': '0'} 2023-09-29T22:50:49.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023cb59c4e7b8284f-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.638 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.638 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.639 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.639 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.639 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.639 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.639 DEBUG:boto:Data: 2023-09-29T22:50:49.639 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.639 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.639 DEBUG:boto:Params: {} 2023-09-29T22:50:49.640 DEBUG:boto:Token: None 2023-09-29T22:50:49.640 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CioLCKv5nTwI5nuqQxEZax8xZBg= 2023-09-29T22:50:49.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CioLCKv5nTwI5nuqQxEZax8xZBg=', 'Content-Length': '0'} 2023-09-29T22:50:49.642 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd841191fd6614d-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.643 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.643 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.643 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.643 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.643 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.643 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.643 DEBUG:boto:Data: 2023-09-29T22:50:49.644 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.644 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.644 DEBUG:boto:Params: {} 2023-09-29T22:50:49.644 DEBUG:boto:Token: None 2023-09-29T22:50:49.644 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CioLCKv5nTwI5nuqQxEZax8xZBg= 2023-09-29T22:50:49.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CioLCKv5nTwI5nuqQxEZax8xZBg=', 'Content-Length': '0'} 2023-09-29T22:50:49.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000670e7eaf3505dd89-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:50:49.647 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.647 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.648 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.648 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.648 DEBUG:boto:Method: GET 2023-09-29T22:50:49.648 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.648 DEBUG:boto:Data: 2023-09-29T22:50:49.648 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.648 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.648 DEBUG:boto:Params: {} 2023-09-29T22:50:49.649 DEBUG:boto:Token: None 2023-09-29T22:50:49.649 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0= 2023-09-29T22:50:49.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0=', 'Content-Length': '0'} 2023-09-29T22:50:49.652 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae246a53f8f27818-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.694 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.695 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.695 DEBUG:boto:path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.695 DEBUG:boto:auth_path=/hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.695 DEBUG:boto:Method: GET 2023-09-29T22:50:49.695 DEBUG:boto:Path: /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.695 DEBUG:boto:Data: 2023-09-29T22:50:49.695 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.695 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.696 DEBUG:boto:Params: {} 2023-09-29T22:50:49.696 DEBUG:boto:Token: None 2023-09-29T22:50:49.696 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145?versionId=null 2023-09-29T22:50:49.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0= 2023-09-29T22:50:49.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AqRJaM0LVwwxL1aDCsPKCrgP5t0=', 'Content-Length': '0'} 2023-09-29T22:50:49.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023cfeec1c148b076-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.700 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.700 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.701 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.701 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.701 DEBUG:boto:Method: GET 2023-09-29T22:50:49.701 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.701 DEBUG:boto:Data: 2023-09-29T22:50:49.701 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.702 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.702 DEBUG:boto:Params: {} 2023-09-29T22:50:49.702 DEBUG:boto:Token: None 2023-09-29T22:50:49.702 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a2a941a1a191e1f-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.708 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.708 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.709 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.709 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.709 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.709 DEBUG:boto:Method: GET 2023-09-29T22:50:49.709 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.709 DEBUG:boto:Data: 2023-09-29T22:50:49.709 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.709 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.710 DEBUG:boto:Params: {} 2023-09-29T22:50:49.710 DEBUG:boto:Token: None 2023-09-29T22:50:49.710 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.716 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000748e9aad2a60fcce-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.716 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.716 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.716 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.717 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.717 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:50:49.717 DEBUG:boto:Data: 2023-09-29T22:50:49.717 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.717 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.717 DEBUG:boto:Params: {} 2023-09-29T22:50:49.717 DEBUG:boto:Token: None 2023-09-29T22:50:49.718 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145 2023-09-29T22:50:49.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LTHeD6hbrfvYZcDVrlXNh6Xc0+o= 2023-09-29T22:50:49.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LTHeD6hbrfvYZcDVrlXNh6Xc0+o=', 'Content-Length': '0'} 2023-09-29T22:50:49.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ad21f78a6d80f14-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.720 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.720 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.720 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.720 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:50:49.721 DEBUG:boto:Data: 2023-09-29T22:50:49.721 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.721 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.721 DEBUG:boto:Params: {} 2023-09-29T22:50:49.721 DEBUG:boto:Token: None 2023-09-29T22:50:49.721 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145 2023-09-29T22:50:49.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LTHeD6hbrfvYZcDVrlXNh6Xc0+o= 2023-09-29T22:50:49.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LTHeD6hbrfvYZcDVrlXNh6Xc0+o=', 'Content-Length': '0'} 2023-09-29T22:50:49.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b515f56814efeeff-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:50:49.724 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.724 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.724 DEBUG:boto:Method: GET 2023-09-29T22:50:49.724 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:50:49.724 DEBUG:boto:Data: 2023-09-29T22:50:49.724 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.725 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.725 DEBUG:boto:Params: {} 2023-09-29T22:50:49.725 DEBUG:boto:Token: None 2023-09-29T22:50:49.725 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145 2023-09-29T22:50:49.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ECUzREQsG4z0frDWeFaf2cSNf0= 2023-09-29T22:50:49.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ECUzREQsG4z0frDWeFaf2cSNf0=', 'Content-Length': '0'} 2023-09-29T22:50:49.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081420d962a0dff7c-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.728 DEBUG:boto:path=/hcnphz-59/obj145 2023-09-29T22:50:49.728 DEBUG:boto:auth_path=/hcnphz-59/obj145 2023-09-29T22:50:49.728 DEBUG:boto:Method: GET 2023-09-29T22:50:49.728 DEBUG:boto:Path: /hcnphz-59/obj145 2023-09-29T22:50:49.728 DEBUG:boto:Data: 2023-09-29T22:50:49.729 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.729 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.729 DEBUG:boto:Params: {} 2023-09-29T22:50:49.729 DEBUG:boto:Token: None 2023-09-29T22:50:49.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj145 2023-09-29T22:50:49.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9ECUzREQsG4z0frDWeFaf2cSNf0= 2023-09-29T22:50:49.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9ECUzREQsG4z0frDWeFaf2cSNf0=', 'Content-Length': '0'} 2023-09-29T22:50:49.732 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d8ab001c228a900-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.774 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.775 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.775 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.775 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.775 DEBUG:boto:Method: GET 2023-09-29T22:50:49.775 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.775 DEBUG:boto:Data: 2023-09-29T22:50:49.775 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.776 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.776 DEBUG:boto:Params: {} 2023-09-29T22:50:49.776 DEBUG:boto:Token: None 2023-09-29T22:50:49.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.781 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001554e1f29a28bf20-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.781 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.782 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.782 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.782 DEBUG:boto:path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.783 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.783 DEBUG:boto:Method: GET 2023-09-29T22:50:49.783 DEBUG:boto:Path: /hcnphz-59/?prefix=obj145 2023-09-29T22:50:49.783 DEBUG:boto:Data: 2023-09-29T22:50:49.783 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.783 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.784 DEBUG:boto:Params: {} 2023-09-29T22:50:49.784 DEBUG:boto:Token: None 2023-09-29T22:50:49.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.794 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001f6bfb6356daf724-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.794 DEBUG:boto:b'hcnphz-59obj1451000falseobj1452023-09-29T22:43:47.314Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:50:49.795 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.795 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.795 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.795 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.796 DEBUG:boto:Method: GET 2023-09-29T22:50:49.796 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.796 DEBUG:boto:Data: 2023-09-29T22:50:49.796 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.796 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.796 DEBUG:boto:Params: {} 2023-09-29T22:50:49.797 DEBUG:boto:Token: None 2023-09-29T22:50:49.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU= 2023-09-29T22:50:49.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU=', 'Content-Length': '0'} 2023-09-29T22:50:49.801 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009170392877b29fee-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.842 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.843 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.843 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.843 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.843 DEBUG:boto:Method: GET 2023-09-29T22:50:49.843 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.843 DEBUG:boto:Data: 2023-09-29T22:50:49.843 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.843 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.843 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.844 DEBUG:boto:Params: {} 2023-09-29T22:50:49.844 DEBUG:boto:Token: None 2023-09-29T22:50:49.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.844 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU= 2023-09-29T22:50:49.844 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU=', 'Content-Length': '0'} 2023-09-29T22:50:49.847 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004b3136cb6b454d0c-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.847 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.847 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.848 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.848 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.848 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.848 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.848 DEBUG:boto:Data: 2023-09-29T22:50:49.848 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.848 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.848 DEBUG:boto:Params: {} 2023-09-29T22:50:49.849 DEBUG:boto:Token: None 2023-09-29T22:50:49.849 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YriWhtoO5ywcfdjMCx5MC+s4RTQ= 2023-09-29T22:50:49.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YriWhtoO5ywcfdjMCx5MC+s4RTQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.852 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053ecee04004a35d6-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.852 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.852 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.852 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.852 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.852 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.852 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.853 DEBUG:boto:Data: 2023-09-29T22:50:49.853 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.853 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.853 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.853 DEBUG:boto:Params: {} 2023-09-29T22:50:49.853 DEBUG:boto:Token: None 2023-09-29T22:50:49.853 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.854 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YriWhtoO5ywcfdjMCx5MC+s4RTQ= 2023-09-29T22:50:49.854 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YriWhtoO5ywcfdjMCx5MC+s4RTQ=', 'Content-Length': '0'} 2023-09-29T22:50:49.856 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005bc79d7dad4c6529-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:50:49.857 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.857 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.857 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.857 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.857 DEBUG:boto:Method: GET 2023-09-29T22:50:49.857 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.857 DEBUG:boto:Data: 2023-09-29T22:50:49.857 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.858 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.858 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.858 DEBUG:boto:Params: {} 2023-09-29T22:50:49.858 DEBUG:boto:Token: None 2023-09-29T22:50:49.858 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.858 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU= 2023-09-29T22:50:49.858 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU=', 'Content-Length': '0'} 2023-09-29T22:50:49.861 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8d9dd86cb599d5b-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.862 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.862 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.862 DEBUG:boto:path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.862 DEBUG:boto:auth_path=/hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.862 DEBUG:boto:Method: GET 2023-09-29T22:50:49.862 DEBUG:boto:Path: /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.862 DEBUG:boto:Data: 2023-09-29T22:50:49.862 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.863 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.863 DEBUG:boto:Params: {} 2023-09-29T22:50:49.863 DEBUG:boto:Token: None 2023-09-29T22:50:49.863 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148?versionId=null 2023-09-29T22:50:49.863 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU= 2023-09-29T22:50:49.863 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VOd+v70qfi4Sp1PiYHaqgBiPJZU=', 'Content-Length': '0'} 2023-09-29T22:50:49.866 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004048b916cbbd4c89-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.867 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.867 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.867 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.867 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.867 DEBUG:boto:Method: GET 2023-09-29T22:50:49.867 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.867 DEBUG:boto:Data: 2023-09-29T22:50:49.867 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.868 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.868 DEBUG:boto:Params: {} 2023-09-29T22:50:49.868 DEBUG:boto:Token: None 2023-09-29T22:50:49.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.873 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a6aabe570cc4424b-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.873 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.874 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.874 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.874 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.874 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.874 DEBUG:boto:Method: GET 2023-09-29T22:50:49.874 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.875 DEBUG:boto:Data: 2023-09-29T22:50:49.875 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.875 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.875 DEBUG:boto:Params: {} 2023-09-29T22:50:49.875 DEBUG:boto:Token: None 2023-09-29T22:50:49.875 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.875 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7f4080fecad3e17-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.926 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.927 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.927 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.927 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.927 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:50:49.928 DEBUG:boto:Data: 2023-09-29T22:50:49.928 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.928 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.928 DEBUG:boto:Params: {} 2023-09-29T22:50:49.928 DEBUG:boto:Token: None 2023-09-29T22:50:49.928 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148 2023-09-29T22:50:49.928 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uL9YuijENzog1puxcwiUPxOKfzU= 2023-09-29T22:50:49.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uL9YuijENzog1puxcwiUPxOKfzU=', 'Content-Length': '0'} 2023-09-29T22:50:49.931 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c144c1eb8d1ee74e-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.932 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.932 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.932 DEBUG:boto:Method: HEAD 2023-09-29T22:50:49.932 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:50:49.932 DEBUG:boto:Data: 2023-09-29T22:50:49.932 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.932 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.933 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.933 DEBUG:boto:Params: {} 2023-09-29T22:50:49.933 DEBUG:boto:Token: None 2023-09-29T22:50:49.933 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148 2023-09-29T22:50:49.933 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uL9YuijENzog1puxcwiUPxOKfzU= 2023-09-29T22:50:49.933 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uL9YuijENzog1puxcwiUPxOKfzU=', 'Content-Length': '0'} 2023-09-29T22:50:49.936 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2562acf24d4178b-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:50:49.936 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.936 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.937 DEBUG:boto:Method: GET 2023-09-29T22:50:49.937 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:50:49.937 DEBUG:boto:Data: 2023-09-29T22:50:49.937 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.937 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.938 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.938 DEBUG:boto:Params: {} 2023-09-29T22:50:49.938 DEBUG:boto:Token: None 2023-09-29T22:50:49.938 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148 2023-09-29T22:50:49.939 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OQoyJ6lX5SFnYcjY864WwncrEc0= 2023-09-29T22:50:49.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OQoyJ6lX5SFnYcjY864WwncrEc0=', 'Content-Length': '0'} 2023-09-29T22:50:49.942 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cef2816bea5711fb-00651754c9-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.982 DEBUG:boto:path=/hcnphz-59/obj148 2023-09-29T22:50:49.983 DEBUG:boto:auth_path=/hcnphz-59/obj148 2023-09-29T22:50:49.983 DEBUG:boto:Method: GET 2023-09-29T22:50:49.983 DEBUG:boto:Path: /hcnphz-59/obj148 2023-09-29T22:50:49.983 DEBUG:boto:Data: 2023-09-29T22:50:49.983 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.983 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.984 DEBUG:boto:Params: {} 2023-09-29T22:50:49.984 DEBUG:boto:Token: None 2023-09-29T22:50:49.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/obj148 2023-09-29T22:50:49.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OQoyJ6lX5SFnYcjY864WwncrEc0= 2023-09-29T22:50:49.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OQoyJ6lX5SFnYcjY864WwncrEc0=', 'Content-Length': '0'} 2023-09-29T22:50:49.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64ef6fbdb39c528-00651754c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.987 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.987 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.987 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.987 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.988 DEBUG:boto:Method: GET 2023-09-29T22:50:49.988 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.988 DEBUG:boto:Data: 2023-09-29T22:50:49.988 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:49.988 DEBUG:boto:Port: 8001 2023-09-29T22:50:49.988 DEBUG:boto:Params: {} 2023-09-29T22:50:49.988 DEBUG:boto:Token: None 2023-09-29T22:50:49.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:49.994 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d831cb69797f373e-00651754c9-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:49 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:49.994 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:49.995 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:49.995 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:49.995 DEBUG:boto:path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.995 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.995 DEBUG:boto:Method: GET 2023-09-29T22:50:49.995 DEBUG:boto:Path: /hcnphz-59/?prefix=obj148 2023-09-29T22:50:49.995 DEBUG:boto:Data: 2023-09-29T22:50:49.995 DEBUG:boto:Headers: {} 2023-09-29T22:50:49.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:49.996 DEBUG:boto:Port: 8000 2023-09-29T22:50:49.996 DEBUG:boto:Params: {} 2023-09-29T22:50:49.996 DEBUG:boto:Token: None 2023-09-29T22:50:49.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:49 GMT /hcnphz-59/ 2023-09-29T22:50:49.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M= 2023-09-29T22:50:49.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:49 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/gM0WxYFvHOUpcNo2G+ZFqFvU4M=', 'Content-Length': '0'} 2023-09-29T22:50:50.005 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004cd676f474ded596-00651754c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.005 DEBUG:boto:b'hcnphz-59obj1481000falseobj1482023-09-29T22:48:09.432Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:50:50.006 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.006 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.006 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.006 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.006 DEBUG:boto:Method: GET 2023-09-29T22:50:50.007 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.007 DEBUG:boto:Data: 2023-09-29T22:50:50.007 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.007 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.007 DEBUG:boto:Params: {} 2023-09-29T22:50:50.008 DEBUG:boto:Token: None 2023-09-29T22:50:50.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc= 2023-09-29T22:50:50.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc=', 'Content-Length': '0'} 2023-09-29T22:50:50.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acb08861c0d40f01-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.011 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.011 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.011 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.011 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.011 DEBUG:boto:Method: GET 2023-09-29T22:50:50.011 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.012 DEBUG:boto:Data: 2023-09-29T22:50:50.012 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.012 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.012 DEBUG:boto:Params: {} 2023-09-29T22:50:50.012 DEBUG:boto:Token: None 2023-09-29T22:50:50.013 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.013 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc= 2023-09-29T22:50:50.013 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc=', 'Content-Length': '0'} 2023-09-29T22:50:50.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7142de5df551187-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.058 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.059 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.059 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.059 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.059 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.059 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.059 DEBUG:boto:Data: 2023-09-29T22:50:50.060 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.060 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.060 DEBUG:boto:Params: {} 2023-09-29T22:50:50.060 DEBUG:boto:Token: None 2023-09-29T22:50:50.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XMSeChy3IpFh03mUWpZg4ncEuFw= 2023-09-29T22:50:50.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XMSeChy3IpFh03mUWpZg4ncEuFw=', 'Content-Length': '0'} 2023-09-29T22:50:50.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009221e2d114dee4d9-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.064 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.064 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.064 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.064 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.065 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.065 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.065 DEBUG:boto:Data: 2023-09-29T22:50:50.065 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.065 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.065 DEBUG:boto:Params: {} 2023-09-29T22:50:50.066 DEBUG:boto:Token: None 2023-09-29T22:50:50.066 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XMSeChy3IpFh03mUWpZg4ncEuFw= 2023-09-29T22:50:50.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XMSeChy3IpFh03mUWpZg4ncEuFw=', 'Content-Length': '0'} 2023-09-29T22:50:50.069 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a96d3d25dfe4f45-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:50:50.069 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.069 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.070 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.070 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.070 DEBUG:boto:Method: GET 2023-09-29T22:50:50.070 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.070 DEBUG:boto:Data: 2023-09-29T22:50:50.070 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.071 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.071 DEBUG:boto:Params: {} 2023-09-29T22:50:50.071 DEBUG:boto:Token: None 2023-09-29T22:50:50.071 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc= 2023-09-29T22:50:50.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc=', 'Content-Length': '0'} 2023-09-29T22:50:50.074 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d122749495a1176e-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.118 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.119 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.119 DEBUG:boto:path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.119 DEBUG:boto:auth_path=/hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.119 DEBUG:boto:Method: GET 2023-09-29T22:50:50.120 DEBUG:boto:Path: /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.120 DEBUG:boto:Data: 2023-09-29T22:50:50.120 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.120 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.121 DEBUG:boto:Params: {} 2023-09-29T22:50:50.121 DEBUG:boto:Token: None 2023-09-29T22:50:50.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155?versionId=null 2023-09-29T22:50:50.122 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc= 2023-09-29T22:50:50.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i72YCRgODnZtVaw3LB+pcwWSecc=', 'Content-Length': '0'} 2023-09-29T22:50:50.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000703b839ce58f1dbf-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.125 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.126 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.126 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.126 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.126 DEBUG:boto:Method: GET 2023-09-29T22:50:50.126 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.127 DEBUG:boto:Data: 2023-09-29T22:50:50.127 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.127 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.127 DEBUG:boto:Params: {} 2023-09-29T22:50:50.128 DEBUG:boto:Token: None 2023-09-29T22:50:50.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.133 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000239145b79eabb552-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.133 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.134 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.134 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.134 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.134 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.134 DEBUG:boto:Method: GET 2023-09-29T22:50:50.134 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.134 DEBUG:boto:Data: 2023-09-29T22:50:50.135 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.135 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.135 DEBUG:boto:Params: {} 2023-09-29T22:50:50.135 DEBUG:boto:Token: None 2023-09-29T22:50:50.135 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.135 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.142 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d1217f5917dcc1ba-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.143 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.143 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.143 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.143 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.143 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:50:50.143 DEBUG:boto:Data: 2023-09-29T22:50:50.144 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.144 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.144 DEBUG:boto:Params: {} 2023-09-29T22:50:50.144 DEBUG:boto:Token: None 2023-09-29T22:50:50.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155 2023-09-29T22:50:50.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/PRVl5rkfdAC6rNldUhbEyiimXA= 2023-09-29T22:50:50.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/PRVl5rkfdAC6rNldUhbEyiimXA=', 'Content-Length': '0'} 2023-09-29T22:50:50.146 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009f61f6f433b2ef34-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.148 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.148 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.148 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.148 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:50:50.149 DEBUG:boto:Data: 2023-09-29T22:50:50.149 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.149 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.149 DEBUG:boto:Params: {} 2023-09-29T22:50:50.149 DEBUG:boto:Token: None 2023-09-29T22:50:50.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155 2023-09-29T22:50:50.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/PRVl5rkfdAC6rNldUhbEyiimXA= 2023-09-29T22:50:50.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/PRVl5rkfdAC6rNldUhbEyiimXA=', 'Content-Length': '0'} 2023-09-29T22:50:50.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091acd5e70d6d6abd-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:50:50.152 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.152 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.153 DEBUG:boto:Method: GET 2023-09-29T22:50:50.153 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:50:50.153 DEBUG:boto:Data: 2023-09-29T22:50:50.153 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.154 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.154 DEBUG:boto:Params: {} 2023-09-29T22:50:50.154 DEBUG:boto:Token: None 2023-09-29T22:50:50.154 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155 2023-09-29T22:50:50.155 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zR5yicnmpYYrdJQ5wqE7s8seJXs= 2023-09-29T22:50:50.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zR5yicnmpYYrdJQ5wqE7s8seJXs=', 'Content-Length': '0'} 2023-09-29T22:50:50.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000202e0f4a5f23edb1-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.198 DEBUG:boto:path=/hcnphz-59/obj155 2023-09-29T22:50:50.199 DEBUG:boto:auth_path=/hcnphz-59/obj155 2023-09-29T22:50:50.199 DEBUG:boto:Method: GET 2023-09-29T22:50:50.199 DEBUG:boto:Path: /hcnphz-59/obj155 2023-09-29T22:50:50.199 DEBUG:boto:Data: 2023-09-29T22:50:50.199 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.199 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.199 DEBUG:boto:Params: {} 2023-09-29T22:50:50.199 DEBUG:boto:Token: None 2023-09-29T22:50:50.200 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj155 2023-09-29T22:50:50.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zR5yicnmpYYrdJQ5wqE7s8seJXs= 2023-09-29T22:50:50.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zR5yicnmpYYrdJQ5wqE7s8seJXs=', 'Content-Length': '0'} 2023-09-29T22:50:50.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001976d6bd2c3d670c-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.203 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.203 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.203 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.203 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.204 DEBUG:boto:Method: GET 2023-09-29T22:50:50.204 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.204 DEBUG:boto:Data: 2023-09-29T22:50:50.204 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.204 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.204 DEBUG:boto:Params: {} 2023-09-29T22:50:50.205 DEBUG:boto:Token: None 2023-09-29T22:50:50.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f65f75181086efc9-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.210 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.210 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.210 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.211 DEBUG:boto:path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.211 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.211 DEBUG:boto:Method: GET 2023-09-29T22:50:50.211 DEBUG:boto:Path: /hcnphz-59/?prefix=obj155 2023-09-29T22:50:50.212 DEBUG:boto:Data: 2023-09-29T22:50:50.212 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.212 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.212 DEBUG:boto:Params: {} 2023-09-29T22:50:50.213 DEBUG:boto:Token: None 2023-09-29T22:50:50.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.213 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.222 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000084d68fdbb1a53bbf-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.266 DEBUG:boto:b'hcnphz-59obj1551000falseobj1552023-09-29T22:46:48.102Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:50:50.267 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.267 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.268 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.268 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.268 DEBUG:boto:Method: GET 2023-09-29T22:50:50.268 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.268 DEBUG:boto:Data: 2023-09-29T22:50:50.268 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.268 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.268 DEBUG:boto:Params: {} 2023-09-29T22:50:50.269 DEBUG:boto:Token: None 2023-09-29T22:50:50.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0= 2023-09-29T22:50:50.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0=', 'Content-Length': '0'} 2023-09-29T22:50:50.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045f3e05b044eb9a7-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.273 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.273 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.273 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.273 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.273 DEBUG:boto:Method: GET 2023-09-29T22:50:50.274 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.274 DEBUG:boto:Data: 2023-09-29T22:50:50.274 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.274 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.274 DEBUG:boto:Params: {} 2023-09-29T22:50:50.274 DEBUG:boto:Token: None 2023-09-29T22:50:50.274 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0= 2023-09-29T22:50:50.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0=', 'Content-Length': '0'} 2023-09-29T22:50:50.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009855be09690a330b-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.278 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.278 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.278 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.278 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.278 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.278 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.279 DEBUG:boto:Data: 2023-09-29T22:50:50.279 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.279 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.279 DEBUG:boto:Params: {} 2023-09-29T22:50:50.279 DEBUG:boto:Token: None 2023-09-29T22:50:50.279 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Sq6Wao1vUPpS6/1mDmDSZRIRmM= 2023-09-29T22:50:50.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Sq6Wao1vUPpS6/1mDmDSZRIRmM=', 'Content-Length': '0'} 2023-09-29T22:50:50.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b63391290e5515d-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.283 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.283 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.283 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.283 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.283 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.283 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.283 DEBUG:boto:Data: 2023-09-29T22:50:50.283 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.283 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.284 DEBUG:boto:Params: {} 2023-09-29T22:50:50.284 DEBUG:boto:Token: None 2023-09-29T22:50:50.284 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Sq6Wao1vUPpS6/1mDmDSZRIRmM= 2023-09-29T22:50:50.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Sq6Wao1vUPpS6/1mDmDSZRIRmM=', 'Content-Length': '0'} 2023-09-29T22:50:50.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c2a011282b114f9-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:50:50.288 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.288 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.288 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.288 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.288 DEBUG:boto:Method: GET 2023-09-29T22:50:50.288 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.288 DEBUG:boto:Data: 2023-09-29T22:50:50.288 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.288 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.289 DEBUG:boto:Params: {} 2023-09-29T22:50:50.289 DEBUG:boto:Token: None 2023-09-29T22:50:50.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0= 2023-09-29T22:50:50.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0=', 'Content-Length': '0'} 2023-09-29T22:50:50.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059350b8dee406325-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.292 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.292 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.292 DEBUG:boto:path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.292 DEBUG:boto:auth_path=/hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.292 DEBUG:boto:Method: GET 2023-09-29T22:50:50.293 DEBUG:boto:Path: /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.293 DEBUG:boto:Data: 2023-09-29T22:50:50.293 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.293 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.293 DEBUG:boto:Params: {} 2023-09-29T22:50:50.293 DEBUG:boto:Token: None 2023-09-29T22:50:50.293 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17?versionId=null 2023-09-29T22:50:50.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0= 2023-09-29T22:50:50.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xKxcOBBBVIB2M61CiUxzTpWGoD0=', 'Content-Length': '0'} 2023-09-29T22:50:50.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1cc59eb77194abb-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.298 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.298 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.298 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.298 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.298 DEBUG:boto:Method: GET 2023-09-29T22:50:50.298 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.298 DEBUG:boto:Data: 2023-09-29T22:50:50.298 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.299 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.299 DEBUG:boto:Params: {} 2023-09-29T22:50:50.299 DEBUG:boto:Token: None 2023-09-29T22:50:50.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc898787601f58d7-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.304 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.305 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.305 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.305 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.305 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.306 DEBUG:boto:Method: GET 2023-09-29T22:50:50.306 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.306 DEBUG:boto:Data: 2023-09-29T22:50:50.306 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.306 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.306 DEBUG:boto:Params: {} 2023-09-29T22:50:50.306 DEBUG:boto:Token: None 2023-09-29T22:50:50.306 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.315 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000076f1bfb296fb8358-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.358 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.359 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.359 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.359 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.359 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:50:50.359 DEBUG:boto:Data: 2023-09-29T22:50:50.359 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.360 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.360 DEBUG:boto:Params: {} 2023-09-29T22:50:50.360 DEBUG:boto:Token: None 2023-09-29T22:50:50.360 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17 2023-09-29T22:50:50.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RaE3AJW3A8g355gvPNlEmVit5FU= 2023-09-29T22:50:50.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RaE3AJW3A8g355gvPNlEmVit5FU=', 'Content-Length': '0'} 2023-09-29T22:50:50.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e5ad85bc32d491-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.364 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.364 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.364 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.364 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:50:50.364 DEBUG:boto:Data: 2023-09-29T22:50:50.364 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.364 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.364 DEBUG:boto:Params: {} 2023-09-29T22:50:50.365 DEBUG:boto:Token: None 2023-09-29T22:50:50.365 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17 2023-09-29T22:50:50.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RaE3AJW3A8g355gvPNlEmVit5FU= 2023-09-29T22:50:50.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RaE3AJW3A8g355gvPNlEmVit5FU=', 'Content-Length': '0'} 2023-09-29T22:50:50.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dc0f708a8310d9bf-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:50:50.368 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.368 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.368 DEBUG:boto:Method: GET 2023-09-29T22:50:50.368 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:50:50.368 DEBUG:boto:Data: 2023-09-29T22:50:50.368 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.369 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.369 DEBUG:boto:Params: {} 2023-09-29T22:50:50.369 DEBUG:boto:Token: None 2023-09-29T22:50:50.369 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17 2023-09-29T22:50:50.369 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9MbtP3vOfg3Tueipvppk7WLdRMk= 2023-09-29T22:50:50.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9MbtP3vOfg3Tueipvppk7WLdRMk=', 'Content-Length': '0'} 2023-09-29T22:50:50.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000255cdeb052cd187e-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.418 DEBUG:boto:path=/hcnphz-59/obj17 2023-09-29T22:50:50.419 DEBUG:boto:auth_path=/hcnphz-59/obj17 2023-09-29T22:50:50.419 DEBUG:boto:Method: GET 2023-09-29T22:50:50.419 DEBUG:boto:Path: /hcnphz-59/obj17 2023-09-29T22:50:50.419 DEBUG:boto:Data: 2023-09-29T22:50:50.419 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.419 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.420 DEBUG:boto:Params: {} 2023-09-29T22:50:50.420 DEBUG:boto:Token: None 2023-09-29T22:50:50.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj17 2023-09-29T22:50:50.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9MbtP3vOfg3Tueipvppk7WLdRMk= 2023-09-29T22:50:50.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9MbtP3vOfg3Tueipvppk7WLdRMk=', 'Content-Length': '0'} 2023-09-29T22:50:50.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000881e813674f2946b-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.423 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.423 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.424 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.424 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.424 DEBUG:boto:Method: GET 2023-09-29T22:50:50.424 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.424 DEBUG:boto:Data: 2023-09-29T22:50:50.424 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.424 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.424 DEBUG:boto:Params: {} 2023-09-29T22:50:50.425 DEBUG:boto:Token: None 2023-09-29T22:50:50.425 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.431 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009e4a27dbb3d78780-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.431 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.432 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.432 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.432 DEBUG:boto:path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.432 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.432 DEBUG:boto:Method: GET 2023-09-29T22:50:50.433 DEBUG:boto:Path: /hcnphz-59/?prefix=obj17 2023-09-29T22:50:50.433 DEBUG:boto:Data: 2023-09-29T22:50:50.433 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.434 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.434 DEBUG:boto:Params: {} 2023-09-29T22:50:50.434 DEBUG:boto:Token: None 2023-09-29T22:50:50.434 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.435 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.435 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.442 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f621b6f4e662106c-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.442 DEBUG:boto:b'hcnphz-59obj171000falseobj172023-09-29T22:36:05.883Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:50:50.443 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.443 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.443 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.443 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.443 DEBUG:boto:Method: GET 2023-09-29T22:50:50.444 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.444 DEBUG:boto:Data: 2023-09-29T22:50:50.444 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.444 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.444 DEBUG:boto:Params: {} 2023-09-29T22:50:50.444 DEBUG:boto:Token: None 2023-09-29T22:50:50.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg= 2023-09-29T22:50:50.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg=', 'Content-Length': '0'} 2023-09-29T22:50:50.447 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5649d0b8b96342-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.447 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.447 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.447 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.447 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.447 DEBUG:boto:Method: GET 2023-09-29T22:50:50.448 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.448 DEBUG:boto:Data: 2023-09-29T22:50:50.448 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.448 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.448 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.448 DEBUG:boto:Params: {} 2023-09-29T22:50:50.448 DEBUG:boto:Token: None 2023-09-29T22:50:50.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg= 2023-09-29T22:50:50.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg=', 'Content-Length': '0'} 2023-09-29T22:50:50.451 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b7a1ab0ecf46e8e2-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.494 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.495 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.495 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.495 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.495 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.495 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.495 DEBUG:boto:Data: 2023-09-29T22:50:50.495 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.495 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.495 DEBUG:boto:Params: {} 2023-09-29T22:50:50.496 DEBUG:boto:Token: None 2023-09-29T22:50:50.496 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WuAwL52BE1ivpV2WAo0Sp/Fq/fY= 2023-09-29T22:50:50.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WuAwL52BE1ivpV2WAo0Sp/Fq/fY=', 'Content-Length': '0'} 2023-09-29T22:50:50.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ff56bea532b9a4f-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.499 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.499 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.499 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.499 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.499 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.499 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.499 DEBUG:boto:Data: 2023-09-29T22:50:50.499 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.500 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.500 DEBUG:boto:Params: {} 2023-09-29T22:50:50.500 DEBUG:boto:Token: None 2023-09-29T22:50:50.500 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WuAwL52BE1ivpV2WAo0Sp/Fq/fY= 2023-09-29T22:50:50.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WuAwL52BE1ivpV2WAo0Sp/Fq/fY=', 'Content-Length': '0'} 2023-09-29T22:50:50.503 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000878ab30327b49e70-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:50:50.503 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.503 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.503 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.503 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.504 DEBUG:boto:Method: GET 2023-09-29T22:50:50.504 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.504 DEBUG:boto:Data: 2023-09-29T22:50:50.504 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.504 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.504 DEBUG:boto:Params: {} 2023-09-29T22:50:50.504 DEBUG:boto:Token: None 2023-09-29T22:50:50.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg= 2023-09-29T22:50:50.505 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg=', 'Content-Length': '0'} 2023-09-29T22:50:50.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000832083228a82e184-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.554 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.554 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.555 DEBUG:boto:path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.555 DEBUG:boto:auth_path=/hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.555 DEBUG:boto:Method: GET 2023-09-29T22:50:50.555 DEBUG:boto:Path: /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.555 DEBUG:boto:Data: 2023-09-29T22:50:50.555 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.555 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.555 DEBUG:boto:Params: {} 2023-09-29T22:50:50.555 DEBUG:boto:Token: None 2023-09-29T22:50:50.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185?versionId=null 2023-09-29T22:50:50.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg= 2023-09-29T22:50:50.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZuytcV4qYvnFDass/h51B17z5Wg=', 'Content-Length': '0'} 2023-09-29T22:50:50.558 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e6cab46e70f3861-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.559 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.559 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.559 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.559 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.559 DEBUG:boto:Method: GET 2023-09-29T22:50:50.559 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.559 DEBUG:boto:Data: 2023-09-29T22:50:50.559 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.560 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.560 DEBUG:boto:Params: {} 2023-09-29T22:50:50.560 DEBUG:boto:Token: None 2023-09-29T22:50:50.560 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.560 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.566 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000095d2792702d85f7e-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.566 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.566 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.567 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.567 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.567 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.567 DEBUG:boto:Method: GET 2023-09-29T22:50:50.567 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.567 DEBUG:boto:Data: 2023-09-29T22:50:50.567 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.567 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.567 DEBUG:boto:Params: {} 2023-09-29T22:50:50.568 DEBUG:boto:Token: None 2023-09-29T22:50:50.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.576 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000061a2c616846c7265-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.576 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.576 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.576 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.576 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.576 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:50:50.576 DEBUG:boto:Data: 2023-09-29T22:50:50.577 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.577 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.577 DEBUG:boto:Params: {} 2023-09-29T22:50:50.577 DEBUG:boto:Token: None 2023-09-29T22:50:50.577 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185 2023-09-29T22:50:50.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aaFTu6GQn7ra6W/4DIzjecatybM= 2023-09-29T22:50:50.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aaFTu6GQn7ra6W/4DIzjecatybM=', 'Content-Length': '0'} 2023-09-29T22:50:50.579 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000004d76c83a2ccef20-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.580 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.580 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.580 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.580 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:50:50.580 DEBUG:boto:Data: 2023-09-29T22:50:50.580 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.580 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.581 DEBUG:boto:Params: {} 2023-09-29T22:50:50.581 DEBUG:boto:Token: None 2023-09-29T22:50:50.581 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185 2023-09-29T22:50:50.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aaFTu6GQn7ra6W/4DIzjecatybM= 2023-09-29T22:50:50.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aaFTu6GQn7ra6W/4DIzjecatybM=', 'Content-Length': '0'} 2023-09-29T22:50:50.583 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c70145432cd2aafb-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:50:50.583 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.584 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.584 DEBUG:boto:Method: GET 2023-09-29T22:50:50.584 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:50:50.584 DEBUG:boto:Data: 2023-09-29T22:50:50.584 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.584 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.584 DEBUG:boto:Params: {} 2023-09-29T22:50:50.584 DEBUG:boto:Token: None 2023-09-29T22:50:50.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185 2023-09-29T22:50:50.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hVerzo95Ab+rDbwQ0NMn+zaM0Qg= 2023-09-29T22:50:50.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hVerzo95Ab+rDbwQ0NMn+zaM0Qg=', 'Content-Length': '0'} 2023-09-29T22:50:50.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7fe48554e8879ba-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.630 DEBUG:boto:path=/hcnphz-59/obj185 2023-09-29T22:50:50.630 DEBUG:boto:auth_path=/hcnphz-59/obj185 2023-09-29T22:50:50.631 DEBUG:boto:Method: GET 2023-09-29T22:50:50.631 DEBUG:boto:Path: /hcnphz-59/obj185 2023-09-29T22:50:50.631 DEBUG:boto:Data: 2023-09-29T22:50:50.631 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.631 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.631 DEBUG:boto:Params: {} 2023-09-29T22:50:50.631 DEBUG:boto:Token: None 2023-09-29T22:50:50.632 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj185 2023-09-29T22:50:50.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hVerzo95Ab+rDbwQ0NMn+zaM0Qg= 2023-09-29T22:50:50.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hVerzo95Ab+rDbwQ0NMn+zaM0Qg=', 'Content-Length': '0'} 2023-09-29T22:50:50.635 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000083f37dfbbddcf36-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.635 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.635 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.635 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.636 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.636 DEBUG:boto:Method: GET 2023-09-29T22:50:50.636 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.636 DEBUG:boto:Data: 2023-09-29T22:50:50.636 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.636 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.636 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.636 DEBUG:boto:Params: {} 2023-09-29T22:50:50.636 DEBUG:boto:Token: None 2023-09-29T22:50:50.636 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.637 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.637 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.642 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028ab13a84c835392-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.642 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.642 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.643 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.643 DEBUG:boto:path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.643 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.643 DEBUG:boto:Method: GET 2023-09-29T22:50:50.643 DEBUG:boto:Path: /hcnphz-59/?prefix=obj185 2023-09-29T22:50:50.643 DEBUG:boto:Data: 2023-09-29T22:50:50.643 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.643 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.643 DEBUG:boto:Params: {} 2023-09-29T22:50:50.644 DEBUG:boto:Token: None 2023-09-29T22:50:50.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.652 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e2f73fbe212d1a3-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.652 DEBUG:boto:b'hcnphz-59obj1851000falseobj1852023-09-29T22:48:09.490Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:50:50.653 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.653 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.653 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.653 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.653 DEBUG:boto:Method: GET 2023-09-29T22:50:50.653 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.653 DEBUG:boto:Data: 2023-09-29T22:50:50.653 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.653 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.654 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.654 DEBUG:boto:Params: {} 2023-09-29T22:50:50.654 DEBUG:boto:Token: None 2023-09-29T22:50:50.654 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.654 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww= 2023-09-29T22:50:50.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww=', 'Content-Length': '0'} 2023-09-29T22:50:50.657 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074f5014aae7d80e4-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.657 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.657 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.657 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.657 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.658 DEBUG:boto:Method: GET 2023-09-29T22:50:50.658 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.658 DEBUG:boto:Data: 2023-09-29T22:50:50.658 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.658 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.658 DEBUG:boto:Params: {} 2023-09-29T22:50:50.658 DEBUG:boto:Token: None 2023-09-29T22:50:50.658 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww= 2023-09-29T22:50:50.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww=', 'Content-Length': '0'} 2023-09-29T22:50:50.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e93ef4bba3727215-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.661 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.661 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.661 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.661 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.662 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.662 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.662 DEBUG:boto:Data: 2023-09-29T22:50:50.662 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.662 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.662 DEBUG:boto:Params: {} 2023-09-29T22:50:50.662 DEBUG:boto:Token: None 2023-09-29T22:50:50.662 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/sWi4cpfiLyIqnoW5ncM+Iibrx0= 2023-09-29T22:50:50.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/sWi4cpfiLyIqnoW5ncM+Iibrx0=', 'Content-Length': '0'} 2023-09-29T22:50:50.665 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c14fb9a8ad103afb-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.665 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.665 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.665 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.665 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.665 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.665 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.666 DEBUG:boto:Data: 2023-09-29T22:50:50.666 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.666 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.666 DEBUG:boto:Params: {} 2023-09-29T22:50:50.666 DEBUG:boto:Token: None 2023-09-29T22:50:50.666 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/sWi4cpfiLyIqnoW5ncM+Iibrx0= 2023-09-29T22:50:50.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/sWi4cpfiLyIqnoW5ncM+Iibrx0=', 'Content-Length': '0'} 2023-09-29T22:50:50.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000c8aabd29602b0dc-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:50:50.669 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.669 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.669 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.669 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.669 DEBUG:boto:Method: GET 2023-09-29T22:50:50.669 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.669 DEBUG:boto:Data: 2023-09-29T22:50:50.670 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.670 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.670 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.670 DEBUG:boto:Params: {} 2023-09-29T22:50:50.670 DEBUG:boto:Token: None 2023-09-29T22:50:50.670 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww= 2023-09-29T22:50:50.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww=', 'Content-Length': '0'} 2023-09-29T22:50:50.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c60363b3f90349a1-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.673 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.673 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.673 DEBUG:boto:path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.673 DEBUG:boto:auth_path=/hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.673 DEBUG:boto:Method: GET 2023-09-29T22:50:50.673 DEBUG:boto:Path: /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.673 DEBUG:boto:Data: 2023-09-29T22:50:50.673 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.674 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.674 DEBUG:boto:Params: {} 2023-09-29T22:50:50.674 DEBUG:boto:Token: None 2023-09-29T22:50:50.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19?versionId=null 2023-09-29T22:50:50.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww= 2023-09-29T22:50:50.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3YSLSAcmrcbY81NrLGXNm2g4jww=', 'Content-Length': '0'} 2023-09-29T22:50:50.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002497c9eac034061d-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.677 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.677 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.677 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.677 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.677 DEBUG:boto:Method: GET 2023-09-29T22:50:50.677 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.677 DEBUG:boto:Data: 2023-09-29T22:50:50.678 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.678 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.678 DEBUG:boto:Params: {} 2023-09-29T22:50:50.678 DEBUG:boto:Token: None 2023-09-29T22:50:50.678 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.678 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a3282950b6c2d750-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.683 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.683 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.683 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.683 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.683 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.684 DEBUG:boto:Method: GET 2023-09-29T22:50:50.684 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.684 DEBUG:boto:Data: 2023-09-29T22:50:50.684 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.684 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.684 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.684 DEBUG:boto:Params: {} 2023-09-29T22:50:50.684 DEBUG:boto:Token: None 2023-09-29T22:50:50.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.684 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.685 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.692 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018f36f314844d40d-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.692 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.693 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.693 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.693 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.693 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:50:50.693 DEBUG:boto:Data: 2023-09-29T22:50:50.693 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.693 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.694 DEBUG:boto:Params: {} 2023-09-29T22:50:50.694 DEBUG:boto:Token: None 2023-09-29T22:50:50.694 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19 2023-09-29T22:50:50.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/m7S2ruu6skNZv3ACVV/70z28PI= 2023-09-29T22:50:50.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/m7S2ruu6skNZv3ACVV/70z28PI=', 'Content-Length': '0'} 2023-09-29T22:50:50.696 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab5e4a36c9470de-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.696 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.696 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.696 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.696 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:50:50.696 DEBUG:boto:Data: 2023-09-29T22:50:50.697 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.697 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.697 DEBUG:boto:Params: {} 2023-09-29T22:50:50.697 DEBUG:boto:Token: None 2023-09-29T22:50:50.697 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19 2023-09-29T22:50:50.697 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/m7S2ruu6skNZv3ACVV/70z28PI= 2023-09-29T22:50:50.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/m7S2ruu6skNZv3ACVV/70z28PI=', 'Content-Length': '0'} 2023-09-29T22:50:50.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009254c4782d0c9453-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:50:50.699 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.699 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.700 DEBUG:boto:Method: GET 2023-09-29T22:50:50.700 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:50:50.700 DEBUG:boto:Data: 2023-09-29T22:50:50.700 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.700 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.700 DEBUG:boto:Params: {} 2023-09-29T22:50:50.700 DEBUG:boto:Token: None 2023-09-29T22:50:50.700 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19 2023-09-29T22:50:50.700 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/ksyq/i71zI1iiHFUWWBWZMKpvA= 2023-09-29T22:50:50.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/ksyq/i71zI1iiHFUWWBWZMKpvA=', 'Content-Length': '0'} 2023-09-29T22:50:50.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cfe4592ff1cc211-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.750 DEBUG:boto:path=/hcnphz-59/obj19 2023-09-29T22:50:50.750 DEBUG:boto:auth_path=/hcnphz-59/obj19 2023-09-29T22:50:50.751 DEBUG:boto:Method: GET 2023-09-29T22:50:50.751 DEBUG:boto:Path: /hcnphz-59/obj19 2023-09-29T22:50:50.751 DEBUG:boto:Data: 2023-09-29T22:50:50.751 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.751 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.751 DEBUG:boto:Params: {} 2023-09-29T22:50:50.751 DEBUG:boto:Token: None 2023-09-29T22:50:50.752 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj19 2023-09-29T22:50:50.752 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/ksyq/i71zI1iiHFUWWBWZMKpvA= 2023-09-29T22:50:50.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/ksyq/i71zI1iiHFUWWBWZMKpvA=', 'Content-Length': '0'} 2023-09-29T22:50:50.754 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000405e531d5fcd2600-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.755 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.755 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.755 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.755 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.755 DEBUG:boto:Method: GET 2023-09-29T22:50:50.755 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.755 DEBUG:boto:Data: 2023-09-29T22:50:50.756 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.756 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.756 DEBUG:boto:Params: {} 2023-09-29T22:50:50.756 DEBUG:boto:Token: None 2023-09-29T22:50:50.756 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.756 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.756 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.761 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b6dc0faa77a38f8a-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.761 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.762 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.762 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.762 DEBUG:boto:path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.762 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.762 DEBUG:boto:Method: GET 2023-09-29T22:50:50.762 DEBUG:boto:Path: /hcnphz-59/?prefix=obj19 2023-09-29T22:50:50.762 DEBUG:boto:Data: 2023-09-29T22:50:50.763 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.763 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.763 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.763 DEBUG:boto:Params: {} 2023-09-29T22:50:50.763 DEBUG:boto:Token: None 2023-09-29T22:50:50.763 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.763 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.763 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.770 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc2a1c5f30b4846d-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.771 DEBUG:boto:b'hcnphz-59obj191000falseobj192023-09-29T22:38:48.804Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:50:50.771 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.771 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.771 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.771 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.772 DEBUG:boto:Method: GET 2023-09-29T22:50:50.772 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.772 DEBUG:boto:Data: 2023-09-29T22:50:50.772 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.772 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.772 DEBUG:boto:Params: {} 2023-09-29T22:50:50.772 DEBUG:boto:Token: None 2023-09-29T22:50:50.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag= 2023-09-29T22:50:50.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag=', 'Content-Length': '0'} 2023-09-29T22:50:50.775 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f05a26019b91556-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.775 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.775 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.775 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.776 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.776 DEBUG:boto:Method: GET 2023-09-29T22:50:50.776 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.776 DEBUG:boto:Data: 2023-09-29T22:50:50.776 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.776 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.776 DEBUG:boto:Params: {} 2023-09-29T22:50:50.776 DEBUG:boto:Token: None 2023-09-29T22:50:50.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag= 2023-09-29T22:50:50.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag=', 'Content-Length': '0'} 2023-09-29T22:50:50.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af2ff38d2d804ea4-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.779 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.779 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.779 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.779 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.779 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.780 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.780 DEBUG:boto:Data: 2023-09-29T22:50:50.780 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.780 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.780 DEBUG:boto:Params: {} 2023-09-29T22:50:50.780 DEBUG:boto:Token: None 2023-09-29T22:50:50.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aYimO4YgSm0SJjWbH4VDeNXwhgA= 2023-09-29T22:50:50.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aYimO4YgSm0SJjWbH4VDeNXwhgA=', 'Content-Length': '0'} 2023-09-29T22:50:50.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bafe6fbf40be18da-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.783 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.783 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.783 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.783 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.783 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.783 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.784 DEBUG:boto:Data: 2023-09-29T22:50:50.784 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.784 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.784 DEBUG:boto:Params: {} 2023-09-29T22:50:50.784 DEBUG:boto:Token: None 2023-09-29T22:50:50.784 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aYimO4YgSm0SJjWbH4VDeNXwhgA= 2023-09-29T22:50:50.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aYimO4YgSm0SJjWbH4VDeNXwhgA=', 'Content-Length': '0'} 2023-09-29T22:50:50.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fddccf6f6006934-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:50:50.787 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.787 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.787 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.787 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.787 DEBUG:boto:Method: GET 2023-09-29T22:50:50.787 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.787 DEBUG:boto:Data: 2023-09-29T22:50:50.787 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.788 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.788 DEBUG:boto:Params: {} 2023-09-29T22:50:50.788 DEBUG:boto:Token: None 2023-09-29T22:50:50.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag= 2023-09-29T22:50:50.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag=', 'Content-Length': '0'} 2023-09-29T22:50:50.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049c20a211d4722ca-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.838 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.838 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.839 DEBUG:boto:path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.839 DEBUG:boto:auth_path=/hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.839 DEBUG:boto:Method: GET 2023-09-29T22:50:50.839 DEBUG:boto:Path: /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.839 DEBUG:boto:Data: 2023-09-29T22:50:50.839 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.839 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.839 DEBUG:boto:Params: {} 2023-09-29T22:50:50.840 DEBUG:boto:Token: None 2023-09-29T22:50:50.840 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22?versionId=null 2023-09-29T22:50:50.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag= 2023-09-29T22:50:50.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GF4r9HqqIyyuWCp3mb7C5y2YOag=', 'Content-Length': '0'} 2023-09-29T22:50:50.843 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020517500866bb817-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.843 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.843 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.843 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.843 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.844 DEBUG:boto:Method: GET 2023-09-29T22:50:50.844 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.844 DEBUG:boto:Data: 2023-09-29T22:50:50.844 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.844 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.844 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.844 DEBUG:boto:Params: {} 2023-09-29T22:50:50.844 DEBUG:boto:Token: None 2023-09-29T22:50:50.844 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.849 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f7714c9c00f7c90a-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.849 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.850 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.850 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.850 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.850 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.850 DEBUG:boto:Method: GET 2023-09-29T22:50:50.850 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.850 DEBUG:boto:Data: 2023-09-29T22:50:50.850 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.851 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.851 DEBUG:boto:Params: {} 2023-09-29T22:50:50.851 DEBUG:boto:Token: None 2023-09-29T22:50:50.851 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.859 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d0094cede5116cad-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.859 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.860 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.860 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.860 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.860 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:50:50.860 DEBUG:boto:Data: 2023-09-29T22:50:50.860 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.860 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.861 DEBUG:boto:Params: {} 2023-09-29T22:50:50.861 DEBUG:boto:Token: None 2023-09-29T22:50:50.861 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22 2023-09-29T22:50:50.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/A7hPtqyy9/AFz0KkY9Px0kK4eo= 2023-09-29T22:50:50.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/A7hPtqyy9/AFz0KkY9Px0kK4eo=', 'Content-Length': '0'} 2023-09-29T22:50:50.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092b115240503ea6a-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.864 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.864 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.864 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.864 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:50:50.864 DEBUG:boto:Data: 2023-09-29T22:50:50.864 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.864 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.864 DEBUG:boto:Params: {} 2023-09-29T22:50:50.865 DEBUG:boto:Token: None 2023-09-29T22:50:50.865 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22 2023-09-29T22:50:50.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/A7hPtqyy9/AFz0KkY9Px0kK4eo= 2023-09-29T22:50:50.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/A7hPtqyy9/AFz0KkY9Px0kK4eo=', 'Content-Length': '0'} 2023-09-29T22:50:50.867 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089ca057c353662b8-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:50:50.867 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.868 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.868 DEBUG:boto:Method: GET 2023-09-29T22:50:50.868 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:50:50.868 DEBUG:boto:Data: 2023-09-29T22:50:50.868 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.868 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.868 DEBUG:boto:Params: {} 2023-09-29T22:50:50.868 DEBUG:boto:Token: None 2023-09-29T22:50:50.869 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22 2023-09-29T22:50:50.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nd4zdwXLU9X9MtEwaYCwsiDV3fo= 2023-09-29T22:50:50.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nd4zdwXLU9X9MtEwaYCwsiDV3fo=', 'Content-Length': '0'} 2023-09-29T22:50:50.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000341c29c5f9851dfb-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.917 DEBUG:boto:path=/hcnphz-59/obj22 2023-09-29T22:50:50.917 DEBUG:boto:auth_path=/hcnphz-59/obj22 2023-09-29T22:50:50.918 DEBUG:boto:Method: GET 2023-09-29T22:50:50.918 DEBUG:boto:Path: /hcnphz-59/obj22 2023-09-29T22:50:50.918 DEBUG:boto:Data: 2023-09-29T22:50:50.918 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.918 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.918 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.918 DEBUG:boto:Params: {} 2023-09-29T22:50:50.919 DEBUG:boto:Token: None 2023-09-29T22:50:50.919 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj22 2023-09-29T22:50:50.919 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nd4zdwXLU9X9MtEwaYCwsiDV3fo= 2023-09-29T22:50:50.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nd4zdwXLU9X9MtEwaYCwsiDV3fo=', 'Content-Length': '0'} 2023-09-29T22:50:50.922 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4b1493ad3bbd71f-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.922 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.922 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.923 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.923 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.923 DEBUG:boto:Method: GET 2023-09-29T22:50:50.923 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.923 DEBUG:boto:Data: 2023-09-29T22:50:50.923 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.923 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.923 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.923 DEBUG:boto:Params: {} 2023-09-29T22:50:50.924 DEBUG:boto:Token: None 2023-09-29T22:50:50.924 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.924 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.924 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fa036c34855ff742-00651754ca-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.929 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.929 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:50.929 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:50.930 DEBUG:boto:path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.930 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.930 DEBUG:boto:Method: GET 2023-09-29T22:50:50.930 DEBUG:boto:Path: /hcnphz-59/?prefix=obj22 2023-09-29T22:50:50.930 DEBUG:boto:Data: 2023-09-29T22:50:50.930 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.930 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.930 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.930 DEBUG:boto:Params: {} 2023-09-29T22:50:50.931 DEBUG:boto:Token: None 2023-09-29T22:50:50.931 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/ 2023-09-29T22:50:50.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4= 2023-09-29T22:50:50.931 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tq0O/NRbEd4Progs9BdoZyJLUN4=', 'Content-Length': '0'} 2023-09-29T22:50:50.939 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5aaeab31984cbfc-00651754ca-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.939 DEBUG:boto:b'hcnphz-59obj221000falseobj222023-09-29T22:35:44.359Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:50.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:50:50.940 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:50.940 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:50.940 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.940 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.940 DEBUG:boto:Method: GET 2023-09-29T22:50:50.940 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.941 DEBUG:boto:Data: 2023-09-29T22:50:50.941 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.941 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.941 DEBUG:boto:Params: {} 2023-09-29T22:50:50.941 DEBUG:boto:Token: None 2023-09-29T22:50:50.941 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BFMIb/JMDpY548zIBQOjv1iAVLI= 2023-09-29T22:50:50.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BFMIb/JMDpY548zIBQOjv1iAVLI=', 'Content-Length': '0'} 2023-09-29T22:50:50.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000567da4b2cb9a940e-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.944 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:50.944 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:50.944 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.944 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.944 DEBUG:boto:Method: GET 2023-09-29T22:50:50.944 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.945 DEBUG:boto:Data: 2023-09-29T22:50:50.945 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:50.945 DEBUG:boto:Port: 8000 2023-09-29T22:50:50.945 DEBUG:boto:Params: {} 2023-09-29T22:50:50.945 DEBUG:boto:Token: None 2023-09-29T22:50:50.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BFMIb/JMDpY548zIBQOjv1iAVLI= 2023-09-29T22:50:50.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BFMIb/JMDpY548zIBQOjv1iAVLI=', 'Content-Length': '0'} 2023-09-29T22:50:50.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbb5c7b487c08987-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.994 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:50.995 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:50.995 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.995 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.995 DEBUG:boto:Method: HEAD 2023-09-29T22:50:50.995 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.995 DEBUG:boto:Data: 2023-09-29T22:50:50.995 DEBUG:boto:Headers: {} 2023-09-29T22:50:50.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:50.995 DEBUG:boto:Port: 8001 2023-09-29T22:50:50.996 DEBUG:boto:Params: {} 2023-09-29T22:50:50.996 DEBUG:boto:Token: None 2023-09-29T22:50:50.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:50 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:50.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c0spYUp9bDggff+p6bc6fEZ2HMA= 2023-09-29T22:50:50.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:50 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c0spYUp9bDggff+p6bc6fEZ2HMA=', 'Content-Length': '0'} 2023-09-29T22:50:50.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007caceba386928fe3-00651754ca-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:50 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:50.999 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:50.999 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.000 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.000 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.000 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.000 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.000 DEBUG:boto:Data: 2023-09-29T22:50:51.000 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.000 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.000 DEBUG:boto:Params: {} 2023-09-29T22:50:51.001 DEBUG:boto:Token: None 2023-09-29T22:50:51.001 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.001 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Ie8z++davC3oq/0JogrHSo5EWU= 2023-09-29T22:50:51.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Ie8z++davC3oq/0JogrHSo5EWU=', 'Content-Length': '0'} 2023-09-29T22:50:51.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bf1831509e4f790-00651754ca-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:50:51.004 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:51.004 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.004 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.004 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.004 DEBUG:boto:Method: GET 2023-09-29T22:50:51.005 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.005 DEBUG:boto:Data: 2023-09-29T22:50:51.005 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.005 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.005 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.005 DEBUG:boto:Params: {} 2023-09-29T22:50:51.005 DEBUG:boto:Token: None 2023-09-29T22:50:51.005 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ViyDKAdZCzZh0Pt12SEuMQ+kTA0= 2023-09-29T22:50:51.006 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ViyDKAdZCzZh0Pt12SEuMQ+kTA0=', 'Content-Length': '0'} 2023-09-29T22:50:51.008 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e58e9f4bd6c16bec-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.050 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:51.051 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.051 DEBUG:boto:path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.051 DEBUG:boto:auth_path=/hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.051 DEBUG:boto:Method: GET 2023-09-29T22:50:51.051 DEBUG:boto:Path: /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.051 DEBUG:boto:Data: 2023-09-29T22:50:51.051 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.051 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.052 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.052 DEBUG:boto:Params: {} 2023-09-29T22:50:51.052 DEBUG:boto:Token: None 2023-09-29T22:50:51.052 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23?versionId=null 2023-09-29T22:50:51.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ViyDKAdZCzZh0Pt12SEuMQ+kTA0= 2023-09-29T22:50:51.052 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ViyDKAdZCzZh0Pt12SEuMQ+kTA0=', 'Content-Length': '0'} 2023-09-29T22:50:51.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ebcfb2d172a4b6d-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.055 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.056 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.056 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.056 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.056 DEBUG:boto:Method: GET 2023-09-29T22:50:51.056 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.056 DEBUG:boto:Data: 2023-09-29T22:50:51.056 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.056 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.057 DEBUG:boto:Params: {} 2023-09-29T22:50:51.057 DEBUG:boto:Token: None 2023-09-29T22:50:51.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002edd5660cf32623-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.063 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.063 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.064 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.064 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.064 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.064 DEBUG:boto:Method: GET 2023-09-29T22:50:51.064 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.064 DEBUG:boto:Data: 2023-09-29T22:50:51.064 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.064 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.064 DEBUG:boto:Params: {} 2023-09-29T22:50:51.065 DEBUG:boto:Token: None 2023-09-29T22:50:51.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.072 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037be13fd612d3c40-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.072 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.072 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:51.073 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.073 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.073 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:50:51.073 DEBUG:boto:Data: 2023-09-29T22:50:51.073 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.073 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.073 DEBUG:boto:Params: {} 2023-09-29T22:50:51.073 DEBUG:boto:Token: None 2023-09-29T22:50:51.074 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23 2023-09-29T22:50:51.074 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DMn4xip4yVhcd/xQDbDeK0aSYCQ= 2023-09-29T22:50:51.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DMn4xip4yVhcd/xQDbDeK0aSYCQ=', 'Content-Length': '0'} 2023-09-29T22:50:51.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000163ac34bfeecb06b-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.076 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:51.076 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.076 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.076 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:50:51.076 DEBUG:boto:Data: 2023-09-29T22:50:51.076 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.076 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.077 DEBUG:boto:Params: {} 2023-09-29T22:50:51.077 DEBUG:boto:Token: None 2023-09-29T22:50:51.077 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23 2023-09-29T22:50:51.077 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DMn4xip4yVhcd/xQDbDeK0aSYCQ= 2023-09-29T22:50:51.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DMn4xip4yVhcd/xQDbDeK0aSYCQ=', 'Content-Length': '0'} 2023-09-29T22:50:51.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a89cb650ee9c6a7-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:50:51.079 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:51.079 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.080 DEBUG:boto:Method: GET 2023-09-29T22:50:51.080 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:50:51.080 DEBUG:boto:Data: 2023-09-29T22:50:51.080 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.080 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.080 DEBUG:boto:Params: {} 2023-09-29T22:50:51.080 DEBUG:boto:Token: None 2023-09-29T22:50:51.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23 2023-09-29T22:50:51.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7XojxvX9HYRPYS2AJm0cQU0Ckz0= 2023-09-29T22:50:51.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7XojxvX9HYRPYS2AJm0cQU0Ckz0=', 'Content-Length': '0'} 2023-09-29T22:50:51.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089be3482a5dc4e0d-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.083 DEBUG:boto:path=/hcnphz-59/obj23 2023-09-29T22:50:51.083 DEBUG:boto:auth_path=/hcnphz-59/obj23 2023-09-29T22:50:51.083 DEBUG:boto:Method: GET 2023-09-29T22:50:51.083 DEBUG:boto:Path: /hcnphz-59/obj23 2023-09-29T22:50:51.083 DEBUG:boto:Data: 2023-09-29T22:50:51.083 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.084 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.084 DEBUG:boto:Params: {} 2023-09-29T22:50:51.084 DEBUG:boto:Token: None 2023-09-29T22:50:51.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj23 2023-09-29T22:50:51.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7XojxvX9HYRPYS2AJm0cQU0Ckz0= 2023-09-29T22:50:51.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7XojxvX9HYRPYS2AJm0cQU0Ckz0=', 'Content-Length': '0'} 2023-09-29T22:50:51.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c15465379233ff56-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.130 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.131 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.131 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.131 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.131 DEBUG:boto:Method: GET 2023-09-29T22:50:51.131 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.131 DEBUG:boto:Data: 2023-09-29T22:50:51.131 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.132 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.132 DEBUG:boto:Params: {} 2023-09-29T22:50:51.132 DEBUG:boto:Token: None 2023-09-29T22:50:51.132 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.136 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000908c63bc1fa06827-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.136 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.137 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.137 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.137 DEBUG:boto:path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.137 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.137 DEBUG:boto:Method: GET 2023-09-29T22:50:51.138 DEBUG:boto:Path: /hcnphz-59/?prefix=obj23 2023-09-29T22:50:51.138 DEBUG:boto:Data: 2023-09-29T22:50:51.138 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.138 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.138 DEBUG:boto:Params: {} 2023-09-29T22:50:51.138 DEBUG:boto:Token: None 2023-09-29T22:50:51.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.139 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.146 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000453b148abe981d24-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.146 DEBUG:boto:b'hcnphz-59obj231000falseobj232023-09-29T22:42:07.719Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:50:51.147 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.147 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.147 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.147 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.147 DEBUG:boto:Method: GET 2023-09-29T22:50:51.147 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.147 DEBUG:boto:Data: 2023-09-29T22:50:51.147 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.147 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.148 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.148 DEBUG:boto:Params: {} 2023-09-29T22:50:51.148 DEBUG:boto:Token: None 2023-09-29T22:50:51.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw= 2023-09-29T22:50:51.148 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw=', 'Content-Length': '0'} 2023-09-29T22:50:51.153 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cfcd5580242754a-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.198 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.199 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.199 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.199 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.199 DEBUG:boto:Method: GET 2023-09-29T22:50:51.199 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.200 DEBUG:boto:Data: 2023-09-29T22:50:51.200 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.200 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.200 DEBUG:boto:Params: {} 2023-09-29T22:50:51.201 DEBUG:boto:Token: None 2023-09-29T22:50:51.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw= 2023-09-29T22:50:51.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw=', 'Content-Length': '0'} 2023-09-29T22:50:51.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b2df26fde58c87f-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.205 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.205 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.205 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.205 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.205 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.206 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.206 DEBUG:boto:Data: 2023-09-29T22:50:51.206 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.206 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.206 DEBUG:boto:Params: {} 2023-09-29T22:50:51.206 DEBUG:boto:Token: None 2023-09-29T22:50:51.207 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vgic+YZB85d6eJ28C5z6+m2JrVc= 2023-09-29T22:50:51.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vgic+YZB85d6eJ28C5z6+m2JrVc=', 'Content-Length': '0'} 2023-09-29T22:50:51.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000274f9f0c9b680abb-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.210 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.210 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.210 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.210 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.211 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.211 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.211 DEBUG:boto:Data: 2023-09-29T22:50:51.211 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.211 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.211 DEBUG:boto:Params: {} 2023-09-29T22:50:51.211 DEBUG:boto:Token: None 2023-09-29T22:50:51.211 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vgic+YZB85d6eJ28C5z6+m2JrVc= 2023-09-29T22:50:51.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vgic+YZB85d6eJ28C5z6+m2JrVc=', 'Content-Length': '0'} 2023-09-29T22:50:51.214 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3346b9b203a30cf-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:50:51.215 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.215 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.215 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.215 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.215 DEBUG:boto:Method: GET 2023-09-29T22:50:51.215 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.215 DEBUG:boto:Data: 2023-09-29T22:50:51.216 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.216 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.216 DEBUG:boto:Params: {} 2023-09-29T22:50:51.216 DEBUG:boto:Token: None 2023-09-29T22:50:51.216 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw= 2023-09-29T22:50:51.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw=', 'Content-Length': '0'} 2023-09-29T22:50:51.219 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d1bc357fefedbb8-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.220 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.220 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.220 DEBUG:boto:path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.220 DEBUG:boto:auth_path=/hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.220 DEBUG:boto:Method: GET 2023-09-29T22:50:51.220 DEBUG:boto:Path: /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.220 DEBUG:boto:Data: 2023-09-29T22:50:51.221 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.221 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.221 DEBUG:boto:Params: {} 2023-09-29T22:50:51.221 DEBUG:boto:Token: None 2023-09-29T22:50:51.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29?versionId=null 2023-09-29T22:50:51.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw= 2023-09-29T22:50:51.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yre5Hlv9HSQ/XfzDLShG07DRRxw=', 'Content-Length': '0'} 2023-09-29T22:50:51.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008871ed2dbd4f2f26-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.270 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.271 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.271 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.271 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.271 DEBUG:boto:Method: GET 2023-09-29T22:50:51.272 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.272 DEBUG:boto:Data: 2023-09-29T22:50:51.272 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.272 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.273 DEBUG:boto:Params: {} 2023-09-29T22:50:51.273 DEBUG:boto:Token: None 2023-09-29T22:50:51.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.274 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037dd46e1613c9e48-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.279 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.279 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.279 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.280 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.280 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.280 DEBUG:boto:Method: GET 2023-09-29T22:50:51.280 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.280 DEBUG:boto:Data: 2023-09-29T22:50:51.281 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.281 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.281 DEBUG:boto:Params: {} 2023-09-29T22:50:51.281 DEBUG:boto:Token: None 2023-09-29T22:50:51.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.289 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009da900d38e49268b-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.289 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.290 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.290 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.290 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.290 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:50:51.290 DEBUG:boto:Data: 2023-09-29T22:50:51.291 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.291 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.291 DEBUG:boto:Params: {} 2023-09-29T22:50:51.291 DEBUG:boto:Token: None 2023-09-29T22:50:51.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29 2023-09-29T22:50:51.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qjHGeBpW3AzKfG6mEJyhpsBZhOI= 2023-09-29T22:50:51.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qjHGeBpW3AzKfG6mEJyhpsBZhOI=', 'Content-Length': '0'} 2023-09-29T22:50:51.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008465268507f584ca-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.294 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.294 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.294 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.294 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:50:51.294 DEBUG:boto:Data: 2023-09-29T22:50:51.294 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.295 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.295 DEBUG:boto:Params: {} 2023-09-29T22:50:51.295 DEBUG:boto:Token: None 2023-09-29T22:50:51.295 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29 2023-09-29T22:50:51.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qjHGeBpW3AzKfG6mEJyhpsBZhOI= 2023-09-29T22:50:51.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qjHGeBpW3AzKfG6mEJyhpsBZhOI=', 'Content-Length': '0'} 2023-09-29T22:50:51.297 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007703e071e2faf946-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:50:51.298 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.298 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.298 DEBUG:boto:Method: GET 2023-09-29T22:50:51.298 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:50:51.298 DEBUG:boto:Data: 2023-09-29T22:50:51.298 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.298 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.299 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.299 DEBUG:boto:Params: {} 2023-09-29T22:50:51.299 DEBUG:boto:Token: None 2023-09-29T22:50:51.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29 2023-09-29T22:50:51.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MU97Z2SAo4RvfPCuZldVgud8sBE= 2023-09-29T22:50:51.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MU97Z2SAo4RvfPCuZldVgud8sBE=', 'Content-Length': '0'} 2023-09-29T22:50:51.302 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd963673eb4c8d4f-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.302 DEBUG:boto:path=/hcnphz-59/obj29 2023-09-29T22:50:51.302 DEBUG:boto:auth_path=/hcnphz-59/obj29 2023-09-29T22:50:51.302 DEBUG:boto:Method: GET 2023-09-29T22:50:51.302 DEBUG:boto:Path: /hcnphz-59/obj29 2023-09-29T22:50:51.302 DEBUG:boto:Data: 2023-09-29T22:50:51.303 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.303 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.303 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.303 DEBUG:boto:Params: {} 2023-09-29T22:50:51.303 DEBUG:boto:Token: None 2023-09-29T22:50:51.303 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj29 2023-09-29T22:50:51.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MU97Z2SAo4RvfPCuZldVgud8sBE= 2023-09-29T22:50:51.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MU97Z2SAo4RvfPCuZldVgud8sBE=', 'Content-Length': '0'} 2023-09-29T22:50:51.305 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001737dbb84b98dd6-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.306 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.306 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.306 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.306 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.306 DEBUG:boto:Method: GET 2023-09-29T22:50:51.306 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.306 DEBUG:boto:Data: 2023-09-29T22:50:51.307 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.307 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.307 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.307 DEBUG:boto:Params: {} 2023-09-29T22:50:51.307 DEBUG:boto:Token: None 2023-09-29T22:50:51.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be58c7f573ff3de6-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.312 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.312 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.312 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.312 DEBUG:boto:path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.312 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.312 DEBUG:boto:Method: GET 2023-09-29T22:50:51.312 DEBUG:boto:Path: /hcnphz-59/?prefix=obj29 2023-09-29T22:50:51.313 DEBUG:boto:Data: 2023-09-29T22:50:51.313 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.313 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.313 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.313 DEBUG:boto:Params: {} 2023-09-29T22:50:51.313 DEBUG:boto:Token: None 2023-09-29T22:50:51.313 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.321 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cef3c38b0f8a0bf2-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.322 DEBUG:boto:b'hcnphz-59obj291000falseobj292023-09-29T22:43:47.172Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:50:51.322 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.322 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.322 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.322 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.323 DEBUG:boto:Method: GET 2023-09-29T22:50:51.323 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.323 DEBUG:boto:Data: 2023-09-29T22:50:51.323 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.323 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.323 DEBUG:boto:Params: {} 2023-09-29T22:50:51.323 DEBUG:boto:Token: None 2023-09-29T22:50:51.323 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk= 2023-09-29T22:50:51.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk=', 'Content-Length': '0'} 2023-09-29T22:50:51.326 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000849421af20109d1e-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.366 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.367 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.367 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.367 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.367 DEBUG:boto:Method: GET 2023-09-29T22:50:51.367 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.367 DEBUG:boto:Data: 2023-09-29T22:50:51.367 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.367 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.368 DEBUG:boto:Params: {} 2023-09-29T22:50:51.368 DEBUG:boto:Token: None 2023-09-29T22:50:51.368 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk= 2023-09-29T22:50:51.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk=', 'Content-Length': '0'} 2023-09-29T22:50:51.371 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000139b952f4850c3fe-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.371 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.372 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.372 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.372 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.372 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.372 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.372 DEBUG:boto:Data: 2023-09-29T22:50:51.372 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.372 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.372 DEBUG:boto:Params: {} 2023-09-29T22:50:51.373 DEBUG:boto:Token: None 2023-09-29T22:50:51.373 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:63ZoHELerrF9YUgD6bv9tFHLmRc= 2023-09-29T22:50:51.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:63ZoHELerrF9YUgD6bv9tFHLmRc=', 'Content-Length': '0'} 2023-09-29T22:50:51.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4481fc2051c3a30-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.376 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.376 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.376 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.376 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.376 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.376 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.376 DEBUG:boto:Data: 2023-09-29T22:50:51.376 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.377 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.377 DEBUG:boto:Params: {} 2023-09-29T22:50:51.377 DEBUG:boto:Token: None 2023-09-29T22:50:51.377 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:63ZoHELerrF9YUgD6bv9tFHLmRc= 2023-09-29T22:50:51.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:63ZoHELerrF9YUgD6bv9tFHLmRc=', 'Content-Length': '0'} 2023-09-29T22:50:51.380 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069fa7c8450213f1a-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:50:51.380 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.380 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.381 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.381 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.381 DEBUG:boto:Method: GET 2023-09-29T22:50:51.381 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.381 DEBUG:boto:Data: 2023-09-29T22:50:51.381 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.381 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.381 DEBUG:boto:Params: {} 2023-09-29T22:50:51.381 DEBUG:boto:Token: None 2023-09-29T22:50:51.382 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.382 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk= 2023-09-29T22:50:51.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk=', 'Content-Length': '0'} 2023-09-29T22:50:51.384 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a06e59c557d873b1-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.384 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.385 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.385 DEBUG:boto:path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.385 DEBUG:boto:auth_path=/hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.385 DEBUG:boto:Method: GET 2023-09-29T22:50:51.385 DEBUG:boto:Path: /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.385 DEBUG:boto:Data: 2023-09-29T22:50:51.385 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.385 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.385 DEBUG:boto:Params: {} 2023-09-29T22:50:51.386 DEBUG:boto:Token: None 2023-09-29T22:50:51.386 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31?versionId=null 2023-09-29T22:50:51.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk= 2023-09-29T22:50:51.386 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mFNb0k3SZ4H08acAxwURgngOfTk=', 'Content-Length': '0'} 2023-09-29T22:50:51.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d5662aebf7829799-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.434 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.434 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.435 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.435 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.435 DEBUG:boto:Method: GET 2023-09-29T22:50:51.435 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.435 DEBUG:boto:Data: 2023-09-29T22:50:51.435 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.435 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.435 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.435 DEBUG:boto:Params: {} 2023-09-29T22:50:51.436 DEBUG:boto:Token: None 2023-09-29T22:50:51.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000203ff7918e2da8ef-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.441 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.441 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.442 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.442 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.442 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.442 DEBUG:boto:Method: GET 2023-09-29T22:50:51.442 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.442 DEBUG:boto:Data: 2023-09-29T22:50:51.442 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.442 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.442 DEBUG:boto:Params: {} 2023-09-29T22:50:51.443 DEBUG:boto:Token: None 2023-09-29T22:50:51.443 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.443 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019e7b4f30e930376-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.451 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.451 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.451 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.452 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.452 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:50:51.452 DEBUG:boto:Data: 2023-09-29T22:50:51.452 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.452 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.452 DEBUG:boto:Params: {} 2023-09-29T22:50:51.452 DEBUG:boto:Token: None 2023-09-29T22:50:51.452 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31 2023-09-29T22:50:51.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cTQWVuoT41yoYQgZsvse3gyzzTc= 2023-09-29T22:50:51.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cTQWVuoT41yoYQgZsvse3gyzzTc=', 'Content-Length': '0'} 2023-09-29T22:50:51.454 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009f196ec0010b011-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.455 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.455 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.455 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.455 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:50:51.455 DEBUG:boto:Data: 2023-09-29T22:50:51.455 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.455 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.456 DEBUG:boto:Params: {} 2023-09-29T22:50:51.456 DEBUG:boto:Token: None 2023-09-29T22:50:51.456 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31 2023-09-29T22:50:51.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cTQWVuoT41yoYQgZsvse3gyzzTc= 2023-09-29T22:50:51.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cTQWVuoT41yoYQgZsvse3gyzzTc=', 'Content-Length': '0'} 2023-09-29T22:50:51.458 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000808fef85d12edc80-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:50:51.458 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.459 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.459 DEBUG:boto:Method: GET 2023-09-29T22:50:51.459 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:50:51.459 DEBUG:boto:Data: 2023-09-29T22:50:51.459 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.459 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.459 DEBUG:boto:Params: {} 2023-09-29T22:50:51.459 DEBUG:boto:Token: None 2023-09-29T22:50:51.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31 2023-09-29T22:50:51.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D6LxXMsDAqsaeDdffuqN/uXpanE= 2023-09-29T22:50:51.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D6LxXMsDAqsaeDdffuqN/uXpanE=', 'Content-Length': '0'} 2023-09-29T22:50:51.462 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a347cd45f0ce9b42-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.502 DEBUG:boto:path=/hcnphz-59/obj31 2023-09-29T22:50:51.503 DEBUG:boto:auth_path=/hcnphz-59/obj31 2023-09-29T22:50:51.503 DEBUG:boto:Method: GET 2023-09-29T22:50:51.503 DEBUG:boto:Path: /hcnphz-59/obj31 2023-09-29T22:50:51.503 DEBUG:boto:Data: 2023-09-29T22:50:51.503 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.503 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.503 DEBUG:boto:Params: {} 2023-09-29T22:50:51.504 DEBUG:boto:Token: None 2023-09-29T22:50:51.504 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj31 2023-09-29T22:50:51.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D6LxXMsDAqsaeDdffuqN/uXpanE= 2023-09-29T22:50:51.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D6LxXMsDAqsaeDdffuqN/uXpanE=', 'Content-Length': '0'} 2023-09-29T22:50:51.507 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f77e0d370b4a028-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.507 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.508 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.508 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.508 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.508 DEBUG:boto:Method: GET 2023-09-29T22:50:51.508 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.508 DEBUG:boto:Data: 2023-09-29T22:50:51.508 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.508 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.508 DEBUG:boto:Params: {} 2023-09-29T22:50:51.509 DEBUG:boto:Token: None 2023-09-29T22:50:51.509 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.514 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000891da7c1cf6cda9e-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.514 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.515 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.515 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.515 DEBUG:boto:path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.515 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.515 DEBUG:boto:Method: GET 2023-09-29T22:50:51.515 DEBUG:boto:Path: /hcnphz-59/?prefix=obj31 2023-09-29T22:50:51.515 DEBUG:boto:Data: 2023-09-29T22:50:51.515 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.516 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.516 DEBUG:boto:Params: {} 2023-09-29T22:50:51.516 DEBUG:boto:Token: None 2023-09-29T22:50:51.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.516 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.524 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000003135a664ada9849-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.566 DEBUG:boto:b'hcnphz-59obj311000falseobj312023-09-29T22:46:47.954Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:50:51.567 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.567 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.567 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.567 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.567 DEBUG:boto:Method: GET 2023-09-29T22:50:51.568 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.568 DEBUG:boto:Data: 2023-09-29T22:50:51.568 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.568 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.568 DEBUG:boto:Params: {} 2023-09-29T22:50:51.568 DEBUG:boto:Token: None 2023-09-29T22:50:51.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc= 2023-09-29T22:50:51.569 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc=', 'Content-Length': '0'} 2023-09-29T22:50:51.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f97254349c274853-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.572 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.572 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.572 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.572 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.572 DEBUG:boto:Method: GET 2023-09-29T22:50:51.572 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.572 DEBUG:boto:Data: 2023-09-29T22:50:51.573 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.573 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.573 DEBUG:boto:Params: {} 2023-09-29T22:50:51.573 DEBUG:boto:Token: None 2023-09-29T22:50:51.573 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc= 2023-09-29T22:50:51.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc=', 'Content-Length': '0'} 2023-09-29T22:50:51.576 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14f672be2c219c8-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.576 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.576 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.577 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.577 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.577 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.577 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.577 DEBUG:boto:Data: 2023-09-29T22:50:51.577 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.577 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.577 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.577 DEBUG:boto:Params: {} 2023-09-29T22:50:51.577 DEBUG:boto:Token: None 2023-09-29T22:50:51.578 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.578 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kvtAtSHdpebeeX+eV1RZqrYaSBo= 2023-09-29T22:50:51.578 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kvtAtSHdpebeeX+eV1RZqrYaSBo=', 'Content-Length': '0'} 2023-09-29T22:50:51.580 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000941115038ab340ee-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.581 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.581 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.581 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.581 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.581 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.581 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.581 DEBUG:boto:Data: 2023-09-29T22:50:51.581 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.582 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.582 DEBUG:boto:Params: {} 2023-09-29T22:50:51.582 DEBUG:boto:Token: None 2023-09-29T22:50:51.582 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kvtAtSHdpebeeX+eV1RZqrYaSBo= 2023-09-29T22:50:51.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kvtAtSHdpebeeX+eV1RZqrYaSBo=', 'Content-Length': '0'} 2023-09-29T22:50:51.585 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007619a260418212c7-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:50:51.585 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.585 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.585 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.585 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.586 DEBUG:boto:Method: GET 2023-09-29T22:50:51.586 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.586 DEBUG:boto:Data: 2023-09-29T22:50:51.586 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.586 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.586 DEBUG:boto:Params: {} 2023-09-29T22:50:51.586 DEBUG:boto:Token: None 2023-09-29T22:50:51.586 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc= 2023-09-29T22:50:51.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc=', 'Content-Length': '0'} 2023-09-29T22:50:51.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080e8021a7c263947-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.590 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.590 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.590 DEBUG:boto:path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.590 DEBUG:boto:auth_path=/hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.590 DEBUG:boto:Method: GET 2023-09-29T22:50:51.590 DEBUG:boto:Path: /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.590 DEBUG:boto:Data: 2023-09-29T22:50:51.590 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.590 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.591 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.591 DEBUG:boto:Params: {} 2023-09-29T22:50:51.591 DEBUG:boto:Token: None 2023-09-29T22:50:51.591 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33?versionId=null 2023-09-29T22:50:51.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc= 2023-09-29T22:50:51.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ipk9gCSgjnqkjYN2qetblK44Kwc=', 'Content-Length': '0'} 2023-09-29T22:50:51.594 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f1bc109f1ceaeeb-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.594 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.595 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.595 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.595 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.595 DEBUG:boto:Method: GET 2023-09-29T22:50:51.595 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.595 DEBUG:boto:Data: 2023-09-29T22:50:51.595 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.595 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.595 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.595 DEBUG:boto:Params: {} 2023-09-29T22:50:51.596 DEBUG:boto:Token: None 2023-09-29T22:50:51.596 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.596 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.596 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.600 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f81ef72591899a1-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.601 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.601 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.601 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.601 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.602 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.602 DEBUG:boto:Method: GET 2023-09-29T22:50:51.602 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.602 DEBUG:boto:Data: 2023-09-29T22:50:51.602 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.602 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.602 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.602 DEBUG:boto:Params: {} 2023-09-29T22:50:51.602 DEBUG:boto:Token: None 2023-09-29T22:50:51.603 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.603 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.610 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000be34fd0d18d1deb2-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.610 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.611 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.611 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.611 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.611 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:50:51.611 DEBUG:boto:Data: 2023-09-29T22:50:51.611 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.611 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.611 DEBUG:boto:Params: {} 2023-09-29T22:50:51.612 DEBUG:boto:Token: None 2023-09-29T22:50:51.612 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33 2023-09-29T22:50:51.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MzcMuRxA7Z+1TNwYGXRrlg8Bx+U= 2023-09-29T22:50:51.612 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MzcMuRxA7Z+1TNwYGXRrlg8Bx+U=', 'Content-Length': '0'} 2023-09-29T22:50:51.614 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d72fffc5c3ea188-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.614 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.615 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.615 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.615 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:50:51.615 DEBUG:boto:Data: 2023-09-29T22:50:51.615 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.615 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.615 DEBUG:boto:Params: {} 2023-09-29T22:50:51.615 DEBUG:boto:Token: None 2023-09-29T22:50:51.616 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33 2023-09-29T22:50:51.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MzcMuRxA7Z+1TNwYGXRrlg8Bx+U= 2023-09-29T22:50:51.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MzcMuRxA7Z+1TNwYGXRrlg8Bx+U=', 'Content-Length': '0'} 2023-09-29T22:50:51.618 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b615816623716171-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:50:51.618 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.618 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.618 DEBUG:boto:Method: GET 2023-09-29T22:50:51.619 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:50:51.619 DEBUG:boto:Data: 2023-09-29T22:50:51.619 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.619 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.619 DEBUG:boto:Params: {} 2023-09-29T22:50:51.619 DEBUG:boto:Token: None 2023-09-29T22:50:51.619 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33 2023-09-29T22:50:51.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U4mNelcJBQy/3twDxcumOYGT2yY= 2023-09-29T22:50:51.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U4mNelcJBQy/3twDxcumOYGT2yY=', 'Content-Length': '0'} 2023-09-29T22:50:51.622 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b18ad35fdc4dfd0d-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.662 DEBUG:boto:path=/hcnphz-59/obj33 2023-09-29T22:50:51.663 DEBUG:boto:auth_path=/hcnphz-59/obj33 2023-09-29T22:50:51.663 DEBUG:boto:Method: GET 2023-09-29T22:50:51.663 DEBUG:boto:Path: /hcnphz-59/obj33 2023-09-29T22:50:51.663 DEBUG:boto:Data: 2023-09-29T22:50:51.663 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.663 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.663 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.663 DEBUG:boto:Params: {} 2023-09-29T22:50:51.663 DEBUG:boto:Token: None 2023-09-29T22:50:51.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj33 2023-09-29T22:50:51.664 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U4mNelcJBQy/3twDxcumOYGT2yY= 2023-09-29T22:50:51.664 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U4mNelcJBQy/3twDxcumOYGT2yY=', 'Content-Length': '0'} 2023-09-29T22:50:51.667 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d04a6aa74a19f10f-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.667 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.667 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.668 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.668 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.668 DEBUG:boto:Method: GET 2023-09-29T22:50:51.668 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.668 DEBUG:boto:Data: 2023-09-29T22:50:51.668 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.668 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.668 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.668 DEBUG:boto:Params: {} 2023-09-29T22:50:51.669 DEBUG:boto:Token: None 2023-09-29T22:50:51.669 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.669 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.669 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.673 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000582b0471ffb6fffc-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.673 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.674 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.674 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.674 DEBUG:boto:path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.674 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.674 DEBUG:boto:Method: GET 2023-09-29T22:50:51.674 DEBUG:boto:Path: /hcnphz-59/?prefix=obj33 2023-09-29T22:50:51.674 DEBUG:boto:Data: 2023-09-29T22:50:51.675 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.675 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.675 DEBUG:boto:Params: {} 2023-09-29T22:50:51.675 DEBUG:boto:Token: None 2023-09-29T22:50:51.675 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.683 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044d2840d1e6e1ff2-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.684 DEBUG:boto:b'hcnphz-59obj331000falseobj332023-09-29T22:35:44.410Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:50:51.684 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.684 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.684 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.685 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.685 DEBUG:boto:Method: GET 2023-09-29T22:50:51.685 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.685 DEBUG:boto:Data: 2023-09-29T22:50:51.685 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.685 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.685 DEBUG:boto:Params: {} 2023-09-29T22:50:51.685 DEBUG:boto:Token: None 2023-09-29T22:50:51.685 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw= 2023-09-29T22:50:51.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw=', 'Content-Length': '0'} 2023-09-29T22:50:51.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e3b552399b96fa9-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.688 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.688 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.688 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.689 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.689 DEBUG:boto:Method: GET 2023-09-29T22:50:51.689 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.689 DEBUG:boto:Data: 2023-09-29T22:50:51.689 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.689 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.689 DEBUG:boto:Params: {} 2023-09-29T22:50:51.689 DEBUG:boto:Token: None 2023-09-29T22:50:51.690 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.690 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw= 2023-09-29T22:50:51.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw=', 'Content-Length': '0'} 2023-09-29T22:50:51.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed21be67d034076f-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.692 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.692 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.692 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.692 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.693 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.693 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.693 DEBUG:boto:Data: 2023-09-29T22:50:51.693 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.693 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.693 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.693 DEBUG:boto:Params: {} 2023-09-29T22:50:51.693 DEBUG:boto:Token: None 2023-09-29T22:50:51.693 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.694 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZPXK6sPFcnU3icFRb9oTx1kSJUc= 2023-09-29T22:50:51.694 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZPXK6sPFcnU3icFRb9oTx1kSJUc=', 'Content-Length': '0'} 2023-09-29T22:50:51.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3980bd118021268-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.696 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.696 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.696 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.696 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.696 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.696 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.696 DEBUG:boto:Data: 2023-09-29T22:50:51.697 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.697 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.697 DEBUG:boto:Params: {} 2023-09-29T22:50:51.697 DEBUG:boto:Token: None 2023-09-29T22:50:51.697 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.697 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZPXK6sPFcnU3icFRb9oTx1kSJUc= 2023-09-29T22:50:51.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZPXK6sPFcnU3icFRb9oTx1kSJUc=', 'Content-Length': '0'} 2023-09-29T22:50:51.699 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007942b442af8828fe-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:50:51.700 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.700 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.700 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.700 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.700 DEBUG:boto:Method: GET 2023-09-29T22:50:51.700 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.701 DEBUG:boto:Data: 2023-09-29T22:50:51.701 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.701 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.701 DEBUG:boto:Params: {} 2023-09-29T22:50:51.701 DEBUG:boto:Token: None 2023-09-29T22:50:51.701 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw= 2023-09-29T22:50:51.701 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw=', 'Content-Length': '0'} 2023-09-29T22:50:51.704 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d63f5c46da8c6dc6-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.746 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.747 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.747 DEBUG:boto:path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.747 DEBUG:boto:auth_path=/hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.747 DEBUG:boto:Method: GET 2023-09-29T22:50:51.747 DEBUG:boto:Path: /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.747 DEBUG:boto:Data: 2023-09-29T22:50:51.747 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.747 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.747 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.748 DEBUG:boto:Params: {} 2023-09-29T22:50:51.748 DEBUG:boto:Token: None 2023-09-29T22:50:51.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34?versionId=null 2023-09-29T22:50:51.748 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw= 2023-09-29T22:50:51.748 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCb74uz6qO2FfZVwHMKpHiiV9Zw=', 'Content-Length': '0'} 2023-09-29T22:50:51.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a70c44fa90dad872-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.751 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.751 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.752 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.752 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.752 DEBUG:boto:Method: GET 2023-09-29T22:50:51.752 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.752 DEBUG:boto:Data: 2023-09-29T22:50:51.752 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.752 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.752 DEBUG:boto:Params: {} 2023-09-29T22:50:51.753 DEBUG:boto:Token: None 2023-09-29T22:50:51.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f617028868f66384-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.758 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.758 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.759 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.759 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.759 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.759 DEBUG:boto:Method: GET 2023-09-29T22:50:51.759 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.759 DEBUG:boto:Data: 2023-09-29T22:50:51.759 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.759 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.759 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.760 DEBUG:boto:Params: {} 2023-09-29T22:50:51.760 DEBUG:boto:Token: None 2023-09-29T22:50:51.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.760 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.768 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000025b7e65282057555-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.810 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.811 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.811 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.811 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.811 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:50:51.811 DEBUG:boto:Data: 2023-09-29T22:50:51.811 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.812 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.812 DEBUG:boto:Params: {} 2023-09-29T22:50:51.812 DEBUG:boto:Token: None 2023-09-29T22:50:51.812 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34 2023-09-29T22:50:51.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Oz/YO186QAl+kV9buR4eSvpd8FA= 2023-09-29T22:50:51.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Oz/YO186QAl+kV9buR4eSvpd8FA=', 'Content-Length': '0'} 2023-09-29T22:50:51.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d79d255e3d8f9678-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.815 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.815 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.816 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.816 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:50:51.816 DEBUG:boto:Data: 2023-09-29T22:50:51.816 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.816 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.816 DEBUG:boto:Params: {} 2023-09-29T22:50:51.816 DEBUG:boto:Token: None 2023-09-29T22:50:51.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34 2023-09-29T22:50:51.817 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Oz/YO186QAl+kV9buR4eSvpd8FA= 2023-09-29T22:50:51.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Oz/YO186QAl+kV9buR4eSvpd8FA=', 'Content-Length': '0'} 2023-09-29T22:50:51.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b96da008b29932e-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:50:51.820 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.820 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.820 DEBUG:boto:Method: GET 2023-09-29T22:50:51.820 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:50:51.820 DEBUG:boto:Data: 2023-09-29T22:50:51.820 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.820 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.821 DEBUG:boto:Params: {} 2023-09-29T22:50:51.821 DEBUG:boto:Token: None 2023-09-29T22:50:51.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34 2023-09-29T22:50:51.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yV+ewnh+t+QMYSKkkxkK34Cg7HU= 2023-09-29T22:50:51.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yV+ewnh+t+QMYSKkkxkK34Cg7HU=', 'Content-Length': '0'} 2023-09-29T22:50:51.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009197c21b3e348f55-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.824 DEBUG:boto:path=/hcnphz-59/obj34 2023-09-29T22:50:51.824 DEBUG:boto:auth_path=/hcnphz-59/obj34 2023-09-29T22:50:51.824 DEBUG:boto:Method: GET 2023-09-29T22:50:51.824 DEBUG:boto:Path: /hcnphz-59/obj34 2023-09-29T22:50:51.825 DEBUG:boto:Data: 2023-09-29T22:50:51.825 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.825 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.825 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.825 DEBUG:boto:Params: {} 2023-09-29T22:50:51.825 DEBUG:boto:Token: None 2023-09-29T22:50:51.825 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj34 2023-09-29T22:50:51.825 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yV+ewnh+t+QMYSKkkxkK34Cg7HU= 2023-09-29T22:50:51.826 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yV+ewnh+t+QMYSKkkxkK34Cg7HU=', 'Content-Length': '0'} 2023-09-29T22:50:51.828 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000046ed4527eafac4e9-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.828 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.829 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.829 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.829 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.829 DEBUG:boto:Method: GET 2023-09-29T22:50:51.829 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.829 DEBUG:boto:Data: 2023-09-29T22:50:51.829 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.829 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.829 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.830 DEBUG:boto:Params: {} 2023-09-29T22:50:51.830 DEBUG:boto:Token: None 2023-09-29T22:50:51.830 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.830 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.830 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.835 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000530b2b30b855eaf1-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.878 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.879 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.879 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.879 DEBUG:boto:path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.879 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.879 DEBUG:boto:Method: GET 2023-09-29T22:50:51.880 DEBUG:boto:Path: /hcnphz-59/?prefix=obj34 2023-09-29T22:50:51.880 DEBUG:boto:Data: 2023-09-29T22:50:51.880 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.880 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.881 DEBUG:boto:Params: {} 2023-09-29T22:50:51.881 DEBUG:boto:Token: None 2023-09-29T22:50:51.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd777d8129c6ff97-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.891 DEBUG:boto:b'hcnphz-59obj341000falseobj342023-09-29T22:36:05.935Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:50:51.891 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.891 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.892 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.892 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.892 DEBUG:boto:Method: GET 2023-09-29T22:50:51.892 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.892 DEBUG:boto:Data: 2023-09-29T22:50:51.892 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.892 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.892 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.892 DEBUG:boto:Params: {} 2023-09-29T22:50:51.892 DEBUG:boto:Token: None 2023-09-29T22:50:51.893 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s= 2023-09-29T22:50:51.893 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s=', 'Content-Length': '0'} 2023-09-29T22:50:51.896 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068d3522a7023338d-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.896 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.896 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.896 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.896 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.897 DEBUG:boto:Method: GET 2023-09-29T22:50:51.897 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.897 DEBUG:boto:Data: 2023-09-29T22:50:51.897 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.897 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.897 DEBUG:boto:Params: {} 2023-09-29T22:50:51.897 DEBUG:boto:Token: None 2023-09-29T22:50:51.897 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s= 2023-09-29T22:50:51.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s=', 'Content-Length': '0'} 2023-09-29T22:50:51.899 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094bfb08216bdefb2-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.942 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.945 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.945 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.945 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.945 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.946 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.946 DEBUG:boto:Data: 2023-09-29T22:50:51.946 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.946 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.946 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.946 DEBUG:boto:Params: {} 2023-09-29T22:50:51.946 DEBUG:boto:Token: None 2023-09-29T22:50:51.947 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.947 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GxYaUOx5r4lvR4p+dTANBp/ojw4= 2023-09-29T22:50:51.947 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GxYaUOx5r4lvR4p+dTANBp/ojw4=', 'Content-Length': '0'} 2023-09-29T22:50:51.950 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c97dfcfdb6e438f1-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.951 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.951 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.951 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.951 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.951 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.951 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.951 DEBUG:boto:Data: 2023-09-29T22:50:51.951 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.952 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.952 DEBUG:boto:Params: {} 2023-09-29T22:50:51.952 DEBUG:boto:Token: None 2023-09-29T22:50:51.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GxYaUOx5r4lvR4p+dTANBp/ojw4= 2023-09-29T22:50:51.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GxYaUOx5r4lvR4p+dTANBp/ojw4=', 'Content-Length': '0'} 2023-09-29T22:50:51.955 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083533c788e037694-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:50:51.956 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.956 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.956 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.956 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.956 DEBUG:boto:Method: GET 2023-09-29T22:50:51.956 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.956 DEBUG:boto:Data: 2023-09-29T22:50:51.956 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.957 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.957 DEBUG:boto:Params: {} 2023-09-29T22:50:51.957 DEBUG:boto:Token: None 2023-09-29T22:50:51.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.957 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s= 2023-09-29T22:50:51.957 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s=', 'Content-Length': '0'} 2023-09-29T22:50:51.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008afb7a35dc3d4fb6-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.960 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.960 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.960 DEBUG:boto:path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.961 DEBUG:boto:auth_path=/hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.961 DEBUG:boto:Method: GET 2023-09-29T22:50:51.961 DEBUG:boto:Path: /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.961 DEBUG:boto:Data: 2023-09-29T22:50:51.961 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.961 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.961 DEBUG:boto:Params: {} 2023-09-29T22:50:51.961 DEBUG:boto:Token: None 2023-09-29T22:50:51.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37?versionId=null 2023-09-29T22:50:51.962 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s= 2023-09-29T22:50:51.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aT+4G8KnTRQR2iy08L7dPm2zp0s=', 'Content-Length': '0'} 2023-09-29T22:50:51.965 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf70aad09dfd7709-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.965 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.965 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.965 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:51.965 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:51.965 DEBUG:boto:Method: GET 2023-09-29T22:50:51.966 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:50:51.966 DEBUG:boto:Data: 2023-09-29T22:50:51.966 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.966 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.966 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.966 DEBUG:boto:Params: {} 2023-09-29T22:50:51.966 DEBUG:boto:Token: None 2023-09-29T22:50:51.966 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.966 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.967 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.971 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c75c6d4becc407a-00651754cb-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.971 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.972 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:51.972 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:51.972 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:51.972 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:51.972 DEBUG:boto:Method: GET 2023-09-29T22:50:51.972 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:50:51.973 DEBUG:boto:Data: 2023-09-29T22:50:51.973 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.973 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.973 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.973 DEBUG:boto:Params: {} 2023-09-29T22:50:51.973 DEBUG:boto:Token: None 2023-09-29T22:50:51.973 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/ 2023-09-29T22:50:51.973 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0= 2023-09-29T22:50:51.974 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hnIrT+13S49yOs0kau4U6aLicu0=', 'Content-Length': '0'} 2023-09-29T22:50:51.981 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000865245c9ed80dfb6-00651754cb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.981 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:51.982 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.982 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.982 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.982 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:50:51.982 DEBUG:boto:Data: 2023-09-29T22:50:51.982 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.982 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.983 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.983 DEBUG:boto:Params: {} 2023-09-29T22:50:51.983 DEBUG:boto:Token: None 2023-09-29T22:50:51.983 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37 2023-09-29T22:50:51.983 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X78HyIqlLdRpUelDu/BHtTTBEPo= 2023-09-29T22:50:51.983 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X78HyIqlLdRpUelDu/BHtTTBEPo=', 'Content-Length': '0'} 2023-09-29T22:50:51.985 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009db7b07616681313-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.986 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.986 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.986 DEBUG:boto:Method: HEAD 2023-09-29T22:50:51.986 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:50:51.986 DEBUG:boto:Data: 2023-09-29T22:50:51.986 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.986 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:51.986 DEBUG:boto:Port: 8000 2023-09-29T22:50:51.986 DEBUG:boto:Params: {} 2023-09-29T22:50:51.987 DEBUG:boto:Token: None 2023-09-29T22:50:51.987 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37 2023-09-29T22:50:51.987 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X78HyIqlLdRpUelDu/BHtTTBEPo= 2023-09-29T22:50:51.987 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X78HyIqlLdRpUelDu/BHtTTBEPo=', 'Content-Length': '0'} 2023-09-29T22:50:51.989 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000112beb166ffecb44-00651754cb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:51.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:50:51.989 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:51.990 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:51.990 DEBUG:boto:Method: GET 2023-09-29T22:50:51.990 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:50:51.990 DEBUG:boto:Data: 2023-09-29T22:50:51.990 DEBUG:boto:Headers: {} 2023-09-29T22:50:51.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:51.990 DEBUG:boto:Port: 8001 2023-09-29T22:50:51.990 DEBUG:boto:Params: {} 2023-09-29T22:50:51.991 DEBUG:boto:Token: None 2023-09-29T22:50:51.991 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:51 GMT /hcnphz-59/obj37 2023-09-29T22:50:51.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZZQ5TxY7EpGkFAv7nCnd4+vkMPg= 2023-09-29T22:50:51.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:51 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZZQ5TxY7EpGkFAv7nCnd4+vkMPg=', 'Content-Length': '0'} 2023-09-29T22:50:51.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c780c2e28efafcb-00651754cb-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:51 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.034 DEBUG:boto:path=/hcnphz-59/obj37 2023-09-29T22:50:52.034 DEBUG:boto:auth_path=/hcnphz-59/obj37 2023-09-29T22:50:52.035 DEBUG:boto:Method: GET 2023-09-29T22:50:52.035 DEBUG:boto:Path: /hcnphz-59/obj37 2023-09-29T22:50:52.035 DEBUG:boto:Data: 2023-09-29T22:50:52.035 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.035 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.035 DEBUG:boto:Params: {} 2023-09-29T22:50:52.035 DEBUG:boto:Token: None 2023-09-29T22:50:52.035 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj37 2023-09-29T22:50:52.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kfomCtlW0K4BFKnci/Faer2idfU= 2023-09-29T22:50:52.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kfomCtlW0K4BFKnci/Faer2idfU=', 'Content-Length': '0'} 2023-09-29T22:50:52.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ce50aab6ead3423-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.039 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.039 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.039 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:52.039 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:52.039 DEBUG:boto:Method: GET 2023-09-29T22:50:52.039 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:50:52.040 DEBUG:boto:Data: 2023-09-29T22:50:52.040 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.040 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.040 DEBUG:boto:Params: {} 2023-09-29T22:50:52.040 DEBUG:boto:Token: None 2023-09-29T22:50:52.040 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.045 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c25ca759186e4294-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.046 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.046 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.046 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.046 DEBUG:boto:path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:52.046 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj37 2023-09-29T22:50:52.047 DEBUG:boto:Method: GET 2023-09-29T22:50:52.047 DEBUG:boto:Path: /hcnphz-59/?prefix=obj37 2023-09-29T22:50:52.047 DEBUG:boto:Data: 2023-09-29T22:50:52.047 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.047 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.047 DEBUG:boto:Params: {} 2023-09-29T22:50:52.047 DEBUG:boto:Token: None 2023-09-29T22:50:52.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.055 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ccf899f8f6aa4fb6-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.056 DEBUG:boto:b'hcnphz-59obj371000falseobj372023-09-29T22:48:09.340Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:50:52.056 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.056 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.056 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.056 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.057 DEBUG:boto:Method: GET 2023-09-29T22:50:52.057 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.057 DEBUG:boto:Data: 2023-09-29T22:50:52.057 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.057 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.057 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.057 DEBUG:boto:Params: {} 2023-09-29T22:50:52.057 DEBUG:boto:Token: None 2023-09-29T22:50:52.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.058 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g= 2023-09-29T22:50:52.058 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g=', 'Content-Length': '0'} 2023-09-29T22:50:52.060 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e299a8da21dc80c-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.060 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.060 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.061 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.061 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.061 DEBUG:boto:Method: GET 2023-09-29T22:50:52.061 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.061 DEBUG:boto:Data: 2023-09-29T22:50:52.061 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.061 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.061 DEBUG:boto:Params: {} 2023-09-29T22:50:52.062 DEBUG:boto:Token: None 2023-09-29T22:50:52.062 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.062 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g= 2023-09-29T22:50:52.062 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g=', 'Content-Length': '0'} 2023-09-29T22:50:52.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a24200c7ad7d03ca-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.064 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.064 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.064 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.065 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.065 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.065 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.065 DEBUG:boto:Data: 2023-09-29T22:50:52.065 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.065 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.065 DEBUG:boto:Params: {} 2023-09-29T22:50:52.065 DEBUG:boto:Token: None 2023-09-29T22:50:52.065 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b/6EBprB9shKjDVRPQngiU5XexA= 2023-09-29T22:50:52.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b/6EBprB9shKjDVRPQngiU5XexA=', 'Content-Length': '0'} 2023-09-29T22:50:52.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000070a557c52bd3e4-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.068 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.068 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.068 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.068 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.068 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.068 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.069 DEBUG:boto:Data: 2023-09-29T22:50:52.069 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.069 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.069 DEBUG:boto:Params: {} 2023-09-29T22:50:52.069 DEBUG:boto:Token: None 2023-09-29T22:50:52.069 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:b/6EBprB9shKjDVRPQngiU5XexA= 2023-09-29T22:50:52.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:b/6EBprB9shKjDVRPQngiU5XexA=', 'Content-Length': '0'} 2023-09-29T22:50:52.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f10fea50d76ac5-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:50:52.072 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.072 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.072 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.072 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.073 DEBUG:boto:Method: GET 2023-09-29T22:50:52.073 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.073 DEBUG:boto:Data: 2023-09-29T22:50:52.073 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.073 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.073 DEBUG:boto:Params: {} 2023-09-29T22:50:52.073 DEBUG:boto:Token: None 2023-09-29T22:50:52.073 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.074 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g= 2023-09-29T22:50:52.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g=', 'Content-Length': '0'} 2023-09-29T22:50:52.076 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e1548bd5660c241-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.076 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.076 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.076 DEBUG:boto:path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.077 DEBUG:boto:auth_path=/hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.077 DEBUG:boto:Method: GET 2023-09-29T22:50:52.077 DEBUG:boto:Path: /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.077 DEBUG:boto:Data: 2023-09-29T22:50:52.077 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.077 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.077 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.077 DEBUG:boto:Params: {} 2023-09-29T22:50:52.077 DEBUG:boto:Token: None 2023-09-29T22:50:52.078 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38?versionId=null 2023-09-29T22:50:52.078 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g= 2023-09-29T22:50:52.078 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xok7ZuHNDaa7hm35kbqCdOTvp6g=', 'Content-Length': '0'} 2023-09-29T22:50:52.080 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000164c074359b1fc88-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.122 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.123 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.123 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.123 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.123 DEBUG:boto:Method: GET 2023-09-29T22:50:52.123 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.123 DEBUG:boto:Data: 2023-09-29T22:50:52.123 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.123 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.123 DEBUG:boto:Params: {} 2023-09-29T22:50:52.124 DEBUG:boto:Token: None 2023-09-29T22:50:52.124 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.129 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093a9f57827bfbfb1-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.129 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.129 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.129 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.130 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.130 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.130 DEBUG:boto:Method: GET 2023-09-29T22:50:52.130 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.130 DEBUG:boto:Data: 2023-09-29T22:50:52.130 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.130 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.130 DEBUG:boto:Params: {} 2023-09-29T22:50:52.131 DEBUG:boto:Token: None 2023-09-29T22:50:52.131 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.131 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.139 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002874df918030c15a-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.139 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.139 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.139 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.140 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.140 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:50:52.140 DEBUG:boto:Data: 2023-09-29T22:50:52.140 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.140 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.140 DEBUG:boto:Params: {} 2023-09-29T22:50:52.140 DEBUG:boto:Token: None 2023-09-29T22:50:52.140 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38 2023-09-29T22:50:52.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SPy77QbJ1AZIVPgW/96BKuDUaXE= 2023-09-29T22:50:52.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SPy77QbJ1AZIVPgW/96BKuDUaXE=', 'Content-Length': '0'} 2023-09-29T22:50:52.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bce5dae1397578a2-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.143 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.143 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.143 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.143 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:50:52.144 DEBUG:boto:Data: 2023-09-29T22:50:52.144 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.144 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.144 DEBUG:boto:Params: {} 2023-09-29T22:50:52.144 DEBUG:boto:Token: None 2023-09-29T22:50:52.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38 2023-09-29T22:50:52.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SPy77QbJ1AZIVPgW/96BKuDUaXE= 2023-09-29T22:50:52.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SPy77QbJ1AZIVPgW/96BKuDUaXE=', 'Content-Length': '0'} 2023-09-29T22:50:52.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000173d355609fcc658-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:50:52.147 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.147 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.147 DEBUG:boto:Method: GET 2023-09-29T22:50:52.147 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:50:52.149 DEBUG:boto:Data: 2023-09-29T22:50:52.149 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.149 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.149 DEBUG:boto:Params: {} 2023-09-29T22:50:52.149 DEBUG:boto:Token: None 2023-09-29T22:50:52.149 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38 2023-09-29T22:50:52.150 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fz93OTwxa3W5rSF/J8PPSlBZD74= 2023-09-29T22:50:52.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fz93OTwxa3W5rSF/J8PPSlBZD74=', 'Content-Length': '0'} 2023-09-29T22:50:52.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021759e6bdc117330-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.194 DEBUG:boto:path=/hcnphz-59/obj38 2023-09-29T22:50:52.194 DEBUG:boto:auth_path=/hcnphz-59/obj38 2023-09-29T22:50:52.195 DEBUG:boto:Method: GET 2023-09-29T22:50:52.195 DEBUG:boto:Path: /hcnphz-59/obj38 2023-09-29T22:50:52.195 DEBUG:boto:Data: 2023-09-29T22:50:52.195 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.195 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.195 DEBUG:boto:Params: {} 2023-09-29T22:50:52.195 DEBUG:boto:Token: None 2023-09-29T22:50:52.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj38 2023-09-29T22:50:52.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fz93OTwxa3W5rSF/J8PPSlBZD74= 2023-09-29T22:50:52.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fz93OTwxa3W5rSF/J8PPSlBZD74=', 'Content-Length': '0'} 2023-09-29T22:50:52.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410b38778b24c25b-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.199 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.199 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.200 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.200 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.200 DEBUG:boto:Method: GET 2023-09-29T22:50:52.200 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.200 DEBUG:boto:Data: 2023-09-29T22:50:52.200 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.200 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.200 DEBUG:boto:Params: {} 2023-09-29T22:50:52.200 DEBUG:boto:Token: None 2023-09-29T22:50:52.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.205 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000194ad70a9bad25c6-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.206 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.206 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.206 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.206 DEBUG:boto:path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.207 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.207 DEBUG:boto:Method: GET 2023-09-29T22:50:52.207 DEBUG:boto:Path: /hcnphz-59/?prefix=obj38 2023-09-29T22:50:52.207 DEBUG:boto:Data: 2023-09-29T22:50:52.207 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.207 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.207 DEBUG:boto:Params: {} 2023-09-29T22:50:52.207 DEBUG:boto:Token: None 2023-09-29T22:50:52.208 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.215 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008725fc0580f40042-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.258 DEBUG:boto:b'hcnphz-59obj381000falseobj382023-09-29T22:38:48.859Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:50:52.259 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.259 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.259 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.259 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.259 DEBUG:boto:Method: GET 2023-09-29T22:50:52.259 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.259 DEBUG:boto:Data: 2023-09-29T22:50:52.260 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.260 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.260 DEBUG:boto:Params: {} 2023-09-29T22:50:52.260 DEBUG:boto:Token: None 2023-09-29T22:50:52.260 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk= 2023-09-29T22:50:52.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk=', 'Content-Length': '0'} 2023-09-29T22:50:52.263 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091961095dfdfee3c-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.264 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.264 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.264 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.264 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.264 DEBUG:boto:Method: GET 2023-09-29T22:50:52.264 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.264 DEBUG:boto:Data: 2023-09-29T22:50:52.264 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.264 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.265 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.265 DEBUG:boto:Params: {} 2023-09-29T22:50:52.265 DEBUG:boto:Token: None 2023-09-29T22:50:52.265 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk= 2023-09-29T22:50:52.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk=', 'Content-Length': '0'} 2023-09-29T22:50:52.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007a2ddb10c85f1a5-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.268 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.268 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.269 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.269 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.269 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.269 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.269 DEBUG:boto:Data: 2023-09-29T22:50:52.269 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.269 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.269 DEBUG:boto:Params: {} 2023-09-29T22:50:52.270 DEBUG:boto:Token: None 2023-09-29T22:50:52.270 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8ia2onzd9knc1YRIITUj/grZWSc= 2023-09-29T22:50:52.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8ia2onzd9knc1YRIITUj/grZWSc=', 'Content-Length': '0'} 2023-09-29T22:50:52.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7ecbad7ae0d023a-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.273 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.273 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.273 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.273 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.274 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.274 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.274 DEBUG:boto:Data: 2023-09-29T22:50:52.274 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.274 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.274 DEBUG:boto:Params: {} 2023-09-29T22:50:52.274 DEBUG:boto:Token: None 2023-09-29T22:50:52.274 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8ia2onzd9knc1YRIITUj/grZWSc= 2023-09-29T22:50:52.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8ia2onzd9knc1YRIITUj/grZWSc=', 'Content-Length': '0'} 2023-09-29T22:50:52.277 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000707c4c5d9f03a726-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:50:52.278 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.278 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.278 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.278 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.278 DEBUG:boto:Method: GET 2023-09-29T22:50:52.278 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.278 DEBUG:boto:Data: 2023-09-29T22:50:52.278 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.279 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.279 DEBUG:boto:Params: {} 2023-09-29T22:50:52.279 DEBUG:boto:Token: None 2023-09-29T22:50:52.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk= 2023-09-29T22:50:52.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk=', 'Content-Length': '0'} 2023-09-29T22:50:52.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000488aa130e88ffc2b-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.282 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.282 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.282 DEBUG:boto:path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.283 DEBUG:boto:auth_path=/hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.283 DEBUG:boto:Method: GET 2023-09-29T22:50:52.283 DEBUG:boto:Path: /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.283 DEBUG:boto:Data: 2023-09-29T22:50:52.283 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.283 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.283 DEBUG:boto:Params: {} 2023-09-29T22:50:52.283 DEBUG:boto:Token: None 2023-09-29T22:50:52.283 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44?versionId=null 2023-09-29T22:50:52.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk= 2023-09-29T22:50:52.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uVxGJe+LX1lvHpPAdlvRqAczhSk=', 'Content-Length': '0'} 2023-09-29T22:50:52.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000fc2361d6fd50ed7-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.287 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.287 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.287 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.287 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.287 DEBUG:boto:Method: GET 2023-09-29T22:50:52.287 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.287 DEBUG:boto:Data: 2023-09-29T22:50:52.287 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.288 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.288 DEBUG:boto:Params: {} 2023-09-29T22:50:52.288 DEBUG:boto:Token: None 2023-09-29T22:50:52.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.293 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e662c36fa2924b2e-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.293 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.294 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.294 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.294 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.294 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.294 DEBUG:boto:Method: GET 2023-09-29T22:50:52.294 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.294 DEBUG:boto:Data: 2023-09-29T22:50:52.294 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.295 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.295 DEBUG:boto:Params: {} 2023-09-29T22:50:52.295 DEBUG:boto:Token: None 2023-09-29T22:50:52.295 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.295 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.303 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d9cfff4e65c86819-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.350 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.354 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.354 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.354 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.355 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:50:52.355 DEBUG:boto:Data: 2023-09-29T22:50:52.355 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.355 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.355 DEBUG:boto:Params: {} 2023-09-29T22:50:52.355 DEBUG:boto:Token: None 2023-09-29T22:50:52.355 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44 2023-09-29T22:50:52.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yBzcB5TErckCWNJoN5uiS3PrGww= 2023-09-29T22:50:52.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yBzcB5TErckCWNJoN5uiS3PrGww=', 'Content-Length': '0'} 2023-09-29T22:50:52.358 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f66889dc38c77e20-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.359 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.359 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.359 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.359 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:50:52.359 DEBUG:boto:Data: 2023-09-29T22:50:52.359 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.359 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.359 DEBUG:boto:Params: {} 2023-09-29T22:50:52.360 DEBUG:boto:Token: None 2023-09-29T22:50:52.360 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44 2023-09-29T22:50:52.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yBzcB5TErckCWNJoN5uiS3PrGww= 2023-09-29T22:50:52.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yBzcB5TErckCWNJoN5uiS3PrGww=', 'Content-Length': '0'} 2023-09-29T22:50:52.363 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071662ec8117b9a5f-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:50:52.363 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.363 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.363 DEBUG:boto:Method: GET 2023-09-29T22:50:52.363 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:50:52.363 DEBUG:boto:Data: 2023-09-29T22:50:52.364 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.364 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.364 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.364 DEBUG:boto:Params: {} 2023-09-29T22:50:52.364 DEBUG:boto:Token: None 2023-09-29T22:50:52.364 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44 2023-09-29T22:50:52.364 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wW00lQBYDxyaN2O/ZB3n8dsM7Bs= 2023-09-29T22:50:52.364 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wW00lQBYDxyaN2O/ZB3n8dsM7Bs=', 'Content-Length': '0'} 2023-09-29T22:50:52.367 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076526e34623a7a49-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.410 DEBUG:boto:path=/hcnphz-59/obj44 2023-09-29T22:50:52.411 DEBUG:boto:auth_path=/hcnphz-59/obj44 2023-09-29T22:50:52.411 DEBUG:boto:Method: GET 2023-09-29T22:50:52.411 DEBUG:boto:Path: /hcnphz-59/obj44 2023-09-29T22:50:52.411 DEBUG:boto:Data: 2023-09-29T22:50:52.411 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.411 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.411 DEBUG:boto:Params: {} 2023-09-29T22:50:52.411 DEBUG:boto:Token: None 2023-09-29T22:50:52.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj44 2023-09-29T22:50:52.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wW00lQBYDxyaN2O/ZB3n8dsM7Bs= 2023-09-29T22:50:52.412 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wW00lQBYDxyaN2O/ZB3n8dsM7Bs=', 'Content-Length': '0'} 2023-09-29T22:50:52.415 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007279c5cf676e1a25-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.415 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.415 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.415 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.415 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.415 DEBUG:boto:Method: GET 2023-09-29T22:50:52.416 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.416 DEBUG:boto:Data: 2023-09-29T22:50:52.416 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.416 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.416 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.416 DEBUG:boto:Params: {} 2023-09-29T22:50:52.416 DEBUG:boto:Token: None 2023-09-29T22:50:52.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.417 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.422 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cc50e2ed73848337-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.422 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.423 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.423 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.423 DEBUG:boto:path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.423 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.423 DEBUG:boto:Method: GET 2023-09-29T22:50:52.423 DEBUG:boto:Path: /hcnphz-59/?prefix=obj44 2023-09-29T22:50:52.423 DEBUG:boto:Data: 2023-09-29T22:50:52.424 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.424 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.424 DEBUG:boto:Params: {} 2023-09-29T22:50:52.424 DEBUG:boto:Token: None 2023-09-29T22:50:52.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.432 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000db3e8876dfebc8c6-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.432 DEBUG:boto:b'hcnphz-59obj441000falseobj442023-09-29T22:35:44.427Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:50:52.433 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.433 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.433 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.433 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.433 DEBUG:boto:Method: GET 2023-09-29T22:50:52.433 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.433 DEBUG:boto:Data: 2023-09-29T22:50:52.434 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.434 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.434 DEBUG:boto:Params: {} 2023-09-29T22:50:52.434 DEBUG:boto:Token: None 2023-09-29T22:50:52.434 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38= 2023-09-29T22:50:52.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38=', 'Content-Length': '0'} 2023-09-29T22:50:52.437 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bfdab2225e7e6e6-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.437 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.437 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.437 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.437 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.437 DEBUG:boto:Method: GET 2023-09-29T22:50:52.438 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.438 DEBUG:boto:Data: 2023-09-29T22:50:52.438 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.438 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.438 DEBUG:boto:Params: {} 2023-09-29T22:50:52.438 DEBUG:boto:Token: None 2023-09-29T22:50:52.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.438 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38= 2023-09-29T22:50:52.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38=', 'Content-Length': '0'} 2023-09-29T22:50:52.440 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003024a6f02c05c60d-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.486 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.486 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.487 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.487 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.487 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.487 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.487 DEBUG:boto:Data: 2023-09-29T22:50:52.487 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.487 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.487 DEBUG:boto:Params: {} 2023-09-29T22:50:52.488 DEBUG:boto:Token: None 2023-09-29T22:50:52.488 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+KeKuTnuXyqkyRlq7vO/UTvxd8I= 2023-09-29T22:50:52.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+KeKuTnuXyqkyRlq7vO/UTvxd8I=', 'Content-Length': '0'} 2023-09-29T22:50:52.490 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c2afad27ac669c1a-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.491 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.491 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.491 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.491 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.491 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.491 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.491 DEBUG:boto:Data: 2023-09-29T22:50:52.491 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.492 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.492 DEBUG:boto:Params: {} 2023-09-29T22:50:52.492 DEBUG:boto:Token: None 2023-09-29T22:50:52.492 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.492 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+KeKuTnuXyqkyRlq7vO/UTvxd8I= 2023-09-29T22:50:52.492 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+KeKuTnuXyqkyRlq7vO/UTvxd8I=', 'Content-Length': '0'} 2023-09-29T22:50:52.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e75343fc79f41fc-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:50:52.495 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.495 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.496 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.496 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.496 DEBUG:boto:Method: GET 2023-09-29T22:50:52.496 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.496 DEBUG:boto:Data: 2023-09-29T22:50:52.496 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.496 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.496 DEBUG:boto:Params: {} 2023-09-29T22:50:52.496 DEBUG:boto:Token: None 2023-09-29T22:50:52.497 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38= 2023-09-29T22:50:52.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38=', 'Content-Length': '0'} 2023-09-29T22:50:52.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e25a6185af6673-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.542 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.542 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.543 DEBUG:boto:path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.543 DEBUG:boto:auth_path=/hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.543 DEBUG:boto:Method: GET 2023-09-29T22:50:52.543 DEBUG:boto:Path: /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.543 DEBUG:boto:Data: 2023-09-29T22:50:52.543 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.543 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.543 DEBUG:boto:Params: {} 2023-09-29T22:50:52.543 DEBUG:boto:Token: None 2023-09-29T22:50:52.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46?versionId=null 2023-09-29T22:50:52.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38= 2023-09-29T22:50:52.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFCQDGHu6gKo8oZEeDRFx+hGv38=', 'Content-Length': '0'} 2023-09-29T22:50:52.547 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000029d716cd28d9331-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.547 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.547 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.547 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.547 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.547 DEBUG:boto:Method: GET 2023-09-29T22:50:52.548 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.548 DEBUG:boto:Data: 2023-09-29T22:50:52.548 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.548 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.548 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.548 DEBUG:boto:Params: {} 2023-09-29T22:50:52.548 DEBUG:boto:Token: None 2023-09-29T22:50:52.548 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.549 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.553 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000987620c704edb673-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.553 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.553 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.554 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.554 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.554 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.554 DEBUG:boto:Method: GET 2023-09-29T22:50:52.554 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.554 DEBUG:boto:Data: 2023-09-29T22:50:52.554 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.555 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.555 DEBUG:boto:Params: {} 2023-09-29T22:50:52.555 DEBUG:boto:Token: None 2023-09-29T22:50:52.555 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.562 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002828c953fad0dac5-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.563 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.563 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.563 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.563 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.563 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:50:52.563 DEBUG:boto:Data: 2023-09-29T22:50:52.564 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.564 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.564 DEBUG:boto:Params: {} 2023-09-29T22:50:52.564 DEBUG:boto:Token: None 2023-09-29T22:50:52.564 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46 2023-09-29T22:50:52.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ch1mL5JaAgowcjoa7hRwikiEmYE= 2023-09-29T22:50:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ch1mL5JaAgowcjoa7hRwikiEmYE=', 'Content-Length': '0'} 2023-09-29T22:50:52.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af5ac16a2137e7b2-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.567 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.567 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.567 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.567 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:50:52.567 DEBUG:boto:Data: 2023-09-29T22:50:52.568 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.568 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.568 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.568 DEBUG:boto:Params: {} 2023-09-29T22:50:52.568 DEBUG:boto:Token: None 2023-09-29T22:50:52.568 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46 2023-09-29T22:50:52.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ch1mL5JaAgowcjoa7hRwikiEmYE= 2023-09-29T22:50:52.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ch1mL5JaAgowcjoa7hRwikiEmYE=', 'Content-Length': '0'} 2023-09-29T22:50:52.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000097b9fb300750e1e-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:50:52.571 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.571 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.571 DEBUG:boto:Method: GET 2023-09-29T22:50:52.571 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:50:52.571 DEBUG:boto:Data: 2023-09-29T22:50:52.571 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.571 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.572 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.572 DEBUG:boto:Params: {} 2023-09-29T22:50:52.572 DEBUG:boto:Token: None 2023-09-29T22:50:52.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46 2023-09-29T22:50:52.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m+l6M1hSZR8dnWGF3qwHAWDumsE= 2023-09-29T22:50:52.572 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m+l6M1hSZR8dnWGF3qwHAWDumsE=', 'Content-Length': '0'} 2023-09-29T22:50:52.574 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c422b351f3ce2560-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.618 DEBUG:boto:path=/hcnphz-59/obj46 2023-09-29T22:50:52.619 DEBUG:boto:auth_path=/hcnphz-59/obj46 2023-09-29T22:50:52.619 DEBUG:boto:Method: GET 2023-09-29T22:50:52.619 DEBUG:boto:Path: /hcnphz-59/obj46 2023-09-29T22:50:52.619 DEBUG:boto:Data: 2023-09-29T22:50:52.619 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.619 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.619 DEBUG:boto:Params: {} 2023-09-29T22:50:52.620 DEBUG:boto:Token: None 2023-09-29T22:50:52.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj46 2023-09-29T22:50:52.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:m+l6M1hSZR8dnWGF3qwHAWDumsE= 2023-09-29T22:50:52.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:m+l6M1hSZR8dnWGF3qwHAWDumsE=', 'Content-Length': '0'} 2023-09-29T22:50:52.623 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab7a56ed2c019bd-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.623 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.624 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.624 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.624 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.624 DEBUG:boto:Method: GET 2023-09-29T22:50:52.624 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.624 DEBUG:boto:Data: 2023-09-29T22:50:52.624 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.624 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.624 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.624 DEBUG:boto:Params: {} 2023-09-29T22:50:52.625 DEBUG:boto:Token: None 2023-09-29T22:50:52.625 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.625 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.630 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a4545428a776951-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.631 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.631 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.632 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.632 DEBUG:boto:path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.632 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.632 DEBUG:boto:Method: GET 2023-09-29T22:50:52.632 DEBUG:boto:Path: /hcnphz-59/?prefix=obj46 2023-09-29T22:50:52.632 DEBUG:boto:Data: 2023-09-29T22:50:52.632 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.632 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.632 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.633 DEBUG:boto:Params: {} 2023-09-29T22:50:52.633 DEBUG:boto:Token: None 2023-09-29T22:50:52.633 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.638 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f20c3736b7cca3aa-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.639 DEBUG:boto:b'hcnphz-59obj461000falseobj462023-09-29T22:42:07.736Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:50:52.639 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.639 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.639 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.640 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.640 DEBUG:boto:Method: GET 2023-09-29T22:50:52.640 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.640 DEBUG:boto:Data: 2023-09-29T22:50:52.640 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.640 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.640 DEBUG:boto:Params: {} 2023-09-29T22:50:52.640 DEBUG:boto:Token: None 2023-09-29T22:50:52.641 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o= 2023-09-29T22:50:52.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o=', 'Content-Length': '0'} 2023-09-29T22:50:52.643 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088b7731fe4f2ff07-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.643 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.643 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.643 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.643 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.644 DEBUG:boto:Method: GET 2023-09-29T22:50:52.644 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.644 DEBUG:boto:Data: 2023-09-29T22:50:52.644 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.644 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.644 DEBUG:boto:Params: {} 2023-09-29T22:50:52.644 DEBUG:boto:Token: None 2023-09-29T22:50:52.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o= 2023-09-29T22:50:52.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o=', 'Content-Length': '0'} 2023-09-29T22:50:52.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006bafb40889ec1aa8-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.647 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.647 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.647 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.647 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.647 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.647 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.647 DEBUG:boto:Data: 2023-09-29T22:50:52.648 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.648 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.648 DEBUG:boto:Params: {} 2023-09-29T22:50:52.648 DEBUG:boto:Token: None 2023-09-29T22:50:52.648 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66/FiXTviIZj8cobaZ4SwfbNymQ= 2023-09-29T22:50:52.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66/FiXTviIZj8cobaZ4SwfbNymQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.650 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008604f1be45246fee-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.650 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.650 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.651 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.651 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.651 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.651 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.651 DEBUG:boto:Data: 2023-09-29T22:50:52.651 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.651 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.651 DEBUG:boto:Params: {} 2023-09-29T22:50:52.651 DEBUG:boto:Token: None 2023-09-29T22:50:52.652 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:66/FiXTviIZj8cobaZ4SwfbNymQ= 2023-09-29T22:50:52.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:66/FiXTviIZj8cobaZ4SwfbNymQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e7eea8982b4eba5-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:50:52.654 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.655 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.655 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.655 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.655 DEBUG:boto:Method: GET 2023-09-29T22:50:52.655 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.655 DEBUG:boto:Data: 2023-09-29T22:50:52.655 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.655 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.655 DEBUG:boto:Params: {} 2023-09-29T22:50:52.656 DEBUG:boto:Token: None 2023-09-29T22:50:52.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o= 2023-09-29T22:50:52.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o=', 'Content-Length': '0'} 2023-09-29T22:50:52.658 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cb2f0a0fcdca2af-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.702 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.702 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.702 DEBUG:boto:path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.703 DEBUG:boto:auth_path=/hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.703 DEBUG:boto:Method: GET 2023-09-29T22:50:52.703 DEBUG:boto:Path: /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.703 DEBUG:boto:Data: 2023-09-29T22:50:52.703 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.703 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.703 DEBUG:boto:Params: {} 2023-09-29T22:50:52.703 DEBUG:boto:Token: None 2023-09-29T22:50:52.703 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51?versionId=null 2023-09-29T22:50:52.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o= 2023-09-29T22:50:52.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Axc13c4SA1vRX5BeZwcczjZGy5o=', 'Content-Length': '0'} 2023-09-29T22:50:52.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f59655526b21af64-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.708 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.708 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.708 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.708 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.708 DEBUG:boto:Method: GET 2023-09-29T22:50:52.708 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.708 DEBUG:boto:Data: 2023-09-29T22:50:52.708 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.709 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.709 DEBUG:boto:Params: {} 2023-09-29T22:50:52.709 DEBUG:boto:Token: None 2023-09-29T22:50:52.709 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002731f0fe62c24eeb-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.714 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.714 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.714 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.715 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.715 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.715 DEBUG:boto:Method: GET 2023-09-29T22:50:52.715 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.715 DEBUG:boto:Data: 2023-09-29T22:50:52.715 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.715 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.715 DEBUG:boto:Params: {} 2023-09-29T22:50:52.715 DEBUG:boto:Token: None 2023-09-29T22:50:52.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.720 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008cfc9e5d8611fbaf-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.720 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.721 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.721 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.721 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.721 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:50:52.721 DEBUG:boto:Data: 2023-09-29T22:50:52.722 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.722 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.722 DEBUG:boto:Params: {} 2023-09-29T22:50:52.722 DEBUG:boto:Token: None 2023-09-29T22:50:52.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51 2023-09-29T22:50:52.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hd53PDiVBsaEtBitIuq8N2bu2m8= 2023-09-29T22:50:52.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hd53PDiVBsaEtBitIuq8N2bu2m8=', 'Content-Length': '0'} 2023-09-29T22:50:52.724 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc4329cd49d212cb-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.725 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.725 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.725 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.725 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:50:52.725 DEBUG:boto:Data: 2023-09-29T22:50:52.725 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.725 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.725 DEBUG:boto:Params: {} 2023-09-29T22:50:52.726 DEBUG:boto:Token: None 2023-09-29T22:50:52.726 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51 2023-09-29T22:50:52.726 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hd53PDiVBsaEtBitIuq8N2bu2m8= 2023-09-29T22:50:52.726 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hd53PDiVBsaEtBitIuq8N2bu2m8=', 'Content-Length': '0'} 2023-09-29T22:50:52.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b5dc97b6f36947a1-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:50:52.728 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.728 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.728 DEBUG:boto:Method: GET 2023-09-29T22:50:52.728 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:50:52.729 DEBUG:boto:Data: 2023-09-29T22:50:52.729 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.729 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.729 DEBUG:boto:Params: {} 2023-09-29T22:50:52.729 DEBUG:boto:Token: None 2023-09-29T22:50:52.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51 2023-09-29T22:50:52.729 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cnSdvdaILmoclpvPB8jejheV21s= 2023-09-29T22:50:52.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cnSdvdaILmoclpvPB8jejheV21s=', 'Content-Length': '0'} 2023-09-29T22:50:52.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e6232eaa87270c0-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.732 DEBUG:boto:path=/hcnphz-59/obj51 2023-09-29T22:50:52.732 DEBUG:boto:auth_path=/hcnphz-59/obj51 2023-09-29T22:50:52.732 DEBUG:boto:Method: GET 2023-09-29T22:50:52.732 DEBUG:boto:Path: /hcnphz-59/obj51 2023-09-29T22:50:52.732 DEBUG:boto:Data: 2023-09-29T22:50:52.732 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.732 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.732 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.732 DEBUG:boto:Params: {} 2023-09-29T22:50:52.733 DEBUG:boto:Token: None 2023-09-29T22:50:52.733 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj51 2023-09-29T22:50:52.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cnSdvdaILmoclpvPB8jejheV21s= 2023-09-29T22:50:52.733 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cnSdvdaILmoclpvPB8jejheV21s=', 'Content-Length': '0'} 2023-09-29T22:50:52.735 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:05 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a802c7e07d1861eb-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.735 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.735 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.735 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.735 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.735 DEBUG:boto:Method: GET 2023-09-29T22:50:52.736 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.736 DEBUG:boto:Data: 2023-09-29T22:50:52.736 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.736 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.736 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.736 DEBUG:boto:Params: {} 2023-09-29T22:50:52.736 DEBUG:boto:Token: None 2023-09-29T22:50:52.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.740 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de0f678812723b39-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.782 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.783 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.783 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.783 DEBUG:boto:path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.783 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.783 DEBUG:boto:Method: GET 2023-09-29T22:50:52.783 DEBUG:boto:Path: /hcnphz-59/?prefix=obj51 2023-09-29T22:50:52.783 DEBUG:boto:Data: 2023-09-29T22:50:52.783 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.784 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.784 DEBUG:boto:Params: {} 2023-09-29T22:50:52.784 DEBUG:boto:Token: None 2023-09-29T22:50:52.784 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.795 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000157520fe23bddb14-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.795 DEBUG:boto:b'hcnphz-59obj511000falseobj512023-09-29T22:36:05.953Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:50:52.795 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.796 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.796 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.796 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.796 DEBUG:boto:Method: GET 2023-09-29T22:50:52.796 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.796 DEBUG:boto:Data: 2023-09-29T22:50:52.796 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.796 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.796 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.796 DEBUG:boto:Params: {} 2023-09-29T22:50:52.797 DEBUG:boto:Token: None 2023-09-29T22:50:52.797 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.797 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0= 2023-09-29T22:50:52.797 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0=', 'Content-Length': '0'} 2023-09-29T22:50:52.799 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8ef1ca270c8cd11-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.800 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.800 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.800 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.800 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.800 DEBUG:boto:Method: GET 2023-09-29T22:50:52.800 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.800 DEBUG:boto:Data: 2023-09-29T22:50:52.801 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.801 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.801 DEBUG:boto:Params: {} 2023-09-29T22:50:52.801 DEBUG:boto:Token: None 2023-09-29T22:50:52.801 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.801 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0= 2023-09-29T22:50:52.801 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0=', 'Content-Length': '0'} 2023-09-29T22:50:52.803 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091380265a36d0256-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.846 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.846 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.847 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.847 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.847 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.847 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.847 DEBUG:boto:Data: 2023-09-29T22:50:52.847 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.847 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.847 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.847 DEBUG:boto:Params: {} 2023-09-29T22:50:52.847 DEBUG:boto:Token: None 2023-09-29T22:50:52.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.848 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HW1viEmKYDYeZHaMZ84OYp+U7XA= 2023-09-29T22:50:52.848 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HW1viEmKYDYeZHaMZ84OYp+U7XA=', 'Content-Length': '0'} 2023-09-29T22:50:52.850 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000421140236ccfbd7a-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.851 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.851 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.851 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.851 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.851 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.851 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.851 DEBUG:boto:Data: 2023-09-29T22:50:52.851 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.852 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.852 DEBUG:boto:Params: {} 2023-09-29T22:50:52.852 DEBUG:boto:Token: None 2023-09-29T22:50:52.852 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HW1viEmKYDYeZHaMZ84OYp+U7XA= 2023-09-29T22:50:52.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HW1viEmKYDYeZHaMZ84OYp+U7XA=', 'Content-Length': '0'} 2023-09-29T22:50:52.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d7285e39d9ebfc3-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:50:52.856 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.856 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.856 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.856 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.856 DEBUG:boto:Method: GET 2023-09-29T22:50:52.856 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.856 DEBUG:boto:Data: 2023-09-29T22:50:52.856 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.856 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.857 DEBUG:boto:Params: {} 2023-09-29T22:50:52.857 DEBUG:boto:Token: None 2023-09-29T22:50:52.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0= 2023-09-29T22:50:52.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0=', 'Content-Length': '0'} 2023-09-29T22:50:52.860 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000014ff7e3a4f72c399-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.860 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.860 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.860 DEBUG:boto:path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.860 DEBUG:boto:auth_path=/hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.860 DEBUG:boto:Method: GET 2023-09-29T22:50:52.861 DEBUG:boto:Path: /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.861 DEBUG:boto:Data: 2023-09-29T22:50:52.861 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.861 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.861 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.861 DEBUG:boto:Params: {} 2023-09-29T22:50:52.861 DEBUG:boto:Token: None 2023-09-29T22:50:52.861 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55?versionId=null 2023-09-29T22:50:52.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0= 2023-09-29T22:50:52.862 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i3GZm63EYbZTCnnd4UEh7U1vse0=', 'Content-Length': '0'} 2023-09-29T22:50:52.864 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000866d384dbf644454-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.864 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.865 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.865 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.865 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.865 DEBUG:boto:Method: GET 2023-09-29T22:50:52.865 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.865 DEBUG:boto:Data: 2023-09-29T22:50:52.865 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.865 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.865 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.865 DEBUG:boto:Params: {} 2023-09-29T22:50:52.866 DEBUG:boto:Token: None 2023-09-29T22:50:52.866 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.866 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.866 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.871 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1732f02bc5ce313-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.918 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.919 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.919 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.919 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.919 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.919 DEBUG:boto:Method: GET 2023-09-29T22:50:52.919 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.919 DEBUG:boto:Data: 2023-09-29T22:50:52.920 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.920 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.920 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.920 DEBUG:boto:Params: {} 2023-09-29T22:50:52.920 DEBUG:boto:Token: None 2023-09-29T22:50:52.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.929 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a2c6c80845d49e77-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.930 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.930 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.930 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.930 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.930 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:50:52.931 DEBUG:boto:Data: 2023-09-29T22:50:52.931 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.931 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.931 DEBUG:boto:Params: {} 2023-09-29T22:50:52.931 DEBUG:boto:Token: None 2023-09-29T22:50:52.931 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55 2023-09-29T22:50:52.931 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hGoQLZruXLCt7hTSHb/4DZYj3dg= 2023-09-29T22:50:52.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hGoQLZruXLCt7hTSHb/4DZYj3dg=', 'Content-Length': '0'} 2023-09-29T22:50:52.934 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049ef418a4db3e252-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.934 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.934 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.934 DEBUG:boto:Method: HEAD 2023-09-29T22:50:52.934 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:50:52.935 DEBUG:boto:Data: 2023-09-29T22:50:52.935 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.935 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.935 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.935 DEBUG:boto:Params: {} 2023-09-29T22:50:52.935 DEBUG:boto:Token: None 2023-09-29T22:50:52.935 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55 2023-09-29T22:50:52.935 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hGoQLZruXLCt7hTSHb/4DZYj3dg= 2023-09-29T22:50:52.936 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hGoQLZruXLCt7hTSHb/4DZYj3dg=', 'Content-Length': '0'} 2023-09-29T22:50:52.938 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f76430ab47e1a3f8-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:50:52.938 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.938 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.938 DEBUG:boto:Method: GET 2023-09-29T22:50:52.938 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:50:52.939 DEBUG:boto:Data: 2023-09-29T22:50:52.939 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.939 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.939 DEBUG:boto:Params: {} 2023-09-29T22:50:52.939 DEBUG:boto:Token: None 2023-09-29T22:50:52.939 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55 2023-09-29T22:50:52.939 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:40uzh7ogS9kZWHEamDPB2J/hQHU= 2023-09-29T22:50:52.939 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:40uzh7ogS9kZWHEamDPB2J/hQHU=', 'Content-Length': '0'} 2023-09-29T22:50:52.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478f72b09728c884-00651754cc-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.942 DEBUG:boto:path=/hcnphz-59/obj55 2023-09-29T22:50:52.942 DEBUG:boto:auth_path=/hcnphz-59/obj55 2023-09-29T22:50:52.942 DEBUG:boto:Method: GET 2023-09-29T22:50:52.942 DEBUG:boto:Path: /hcnphz-59/obj55 2023-09-29T22:50:52.942 DEBUG:boto:Data: 2023-09-29T22:50:52.942 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.942 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.943 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.943 DEBUG:boto:Params: {} 2023-09-29T22:50:52.943 DEBUG:boto:Token: None 2023-09-29T22:50:52.943 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/obj55 2023-09-29T22:50:52.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:40uzh7ogS9kZWHEamDPB2J/hQHU= 2023-09-29T22:50:52.943 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:40uzh7ogS9kZWHEamDPB2J/hQHU=', 'Content-Length': '0'} 2023-09-29T22:50:52.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:35:44 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a562d60be21c78a7-00651754cc-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.990 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.991 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.991 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.991 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.991 DEBUG:boto:Method: GET 2023-09-29T22:50:52.991 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.991 DEBUG:boto:Data: 2023-09-29T22:50:52.991 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:52.991 DEBUG:boto:Port: 8001 2023-09-29T22:50:52.992 DEBUG:boto:Params: {} 2023-09-29T22:50:52.992 DEBUG:boto:Token: None 2023-09-29T22:50:52.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:52.997 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f1b88c6d4f52925-00651754cc-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:52.997 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:52.997 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:52.998 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:52.998 DEBUG:boto:path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.998 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.998 DEBUG:boto:Method: GET 2023-09-29T22:50:52.998 DEBUG:boto:Path: /hcnphz-59/?prefix=obj55 2023-09-29T22:50:52.998 DEBUG:boto:Data: 2023-09-29T22:50:52.998 DEBUG:boto:Headers: {} 2023-09-29T22:50:52.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:52.998 DEBUG:boto:Port: 8000 2023-09-29T22:50:52.999 DEBUG:boto:Params: {} 2023-09-29T22:50:52.999 DEBUG:boto:Token: None 2023-09-29T22:50:52.999 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:52 GMT /hcnphz-59/ 2023-09-29T22:50:52.999 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ= 2023-09-29T22:50:52.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SOxu75id9itnurbuIBfhvO+4sDQ=', 'Content-Length': '0'} 2023-09-29T22:50:53.006 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002568b1f2e3533143-00651754cc-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.006 DEBUG:boto:b'hcnphz-59obj551000falseobj552023-09-29T22:35:44.478Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:50:53.007 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.007 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.007 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.007 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.007 DEBUG:boto:Method: GET 2023-09-29T22:50:53.007 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.008 DEBUG:boto:Data: 2023-09-29T22:50:53.008 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.008 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.008 DEBUG:boto:Params: {} 2023-09-29T22:50:53.008 DEBUG:boto:Token: None 2023-09-29T22:50:53.008 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.008 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ= 2023-09-29T22:50:53.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ=', 'Content-Length': '0'} 2023-09-29T22:50:53.010 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac93614c4d66c22c-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.011 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.011 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.011 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.011 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.011 DEBUG:boto:Method: GET 2023-09-29T22:50:53.011 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.012 DEBUG:boto:Data: 2023-09-29T22:50:53.012 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.012 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.012 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.012 DEBUG:boto:Params: {} 2023-09-29T22:50:53.012 DEBUG:boto:Token: None 2023-09-29T22:50:53.012 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.012 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ= 2023-09-29T22:50:53.012 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ=', 'Content-Length': '0'} 2023-09-29T22:50:53.015 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d9d2e359e15035ef-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.015 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.015 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.015 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.015 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.015 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.016 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.016 DEBUG:boto:Data: 2023-09-29T22:50:53.016 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.016 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.016 DEBUG:boto:Params: {} 2023-09-29T22:50:53.016 DEBUG:boto:Token: None 2023-09-29T22:50:53.016 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O5a3A8OdWQxrNvDIXBvkXCti3yk= 2023-09-29T22:50:53.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O5a3A8OdWQxrNvDIXBvkXCti3yk=', 'Content-Length': '0'} 2023-09-29T22:50:53.019 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003f2e6c970501872-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.019 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.019 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.019 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.019 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.019 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.020 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.020 DEBUG:boto:Data: 2023-09-29T22:50:53.020 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.020 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.020 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.020 DEBUG:boto:Params: {} 2023-09-29T22:50:53.020 DEBUG:boto:Token: None 2023-09-29T22:50:53.020 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O5a3A8OdWQxrNvDIXBvkXCti3yk= 2023-09-29T22:50:53.021 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O5a3A8OdWQxrNvDIXBvkXCti3yk=', 'Content-Length': '0'} 2023-09-29T22:50:53.023 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042fe86a91c7cb32a-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.023 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:50:53.023 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.023 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.023 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.023 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.024 DEBUG:boto:Method: GET 2023-09-29T22:50:53.024 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.024 DEBUG:boto:Data: 2023-09-29T22:50:53.024 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.024 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.024 DEBUG:boto:Params: {} 2023-09-29T22:50:53.024 DEBUG:boto:Token: None 2023-09-29T22:50:53.024 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.025 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ= 2023-09-29T22:50:53.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ=', 'Content-Length': '0'} 2023-09-29T22:50:53.027 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000737abcdeb0223282-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.074 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.075 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.075 DEBUG:boto:path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.075 DEBUG:boto:auth_path=/hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.075 DEBUG:boto:Method: GET 2023-09-29T22:50:53.075 DEBUG:boto:Path: /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.075 DEBUG:boto:Data: 2023-09-29T22:50:53.075 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.075 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.076 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.076 DEBUG:boto:Params: {} 2023-09-29T22:50:53.076 DEBUG:boto:Token: None 2023-09-29T22:50:53.076 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57?versionId=null 2023-09-29T22:50:53.076 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ= 2023-09-29T22:50:53.076 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rsE1TGH0/n2hYD69vS2+6Z37hTQ=', 'Content-Length': '0'} 2023-09-29T22:50:53.079 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ac8cdcd3f5c657f-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.079 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.079 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.080 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.080 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.080 DEBUG:boto:Method: GET 2023-09-29T22:50:53.080 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.080 DEBUG:boto:Data: 2023-09-29T22:50:53.080 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.080 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.080 DEBUG:boto:Params: {} 2023-09-29T22:50:53.081 DEBUG:boto:Token: None 2023-09-29T22:50:53.081 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.081 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.081 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.086 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000284cc1dced64cd84-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.086 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.087 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.087 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.087 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.087 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.087 DEBUG:boto:Method: GET 2023-09-29T22:50:53.087 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.087 DEBUG:boto:Data: 2023-09-29T22:50:53.087 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.088 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.088 DEBUG:boto:Params: {} 2023-09-29T22:50:53.088 DEBUG:boto:Token: None 2023-09-29T22:50:53.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.088 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.097 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000954ebbf272f2d6e-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.097 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.097 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.098 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.098 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.098 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:50:53.098 DEBUG:boto:Data: 2023-09-29T22:50:53.098 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.098 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.098 DEBUG:boto:Params: {} 2023-09-29T22:50:53.099 DEBUG:boto:Token: None 2023-09-29T22:50:53.099 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57 2023-09-29T22:50:53.099 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QXOV4HTkP+ozjkFnn/vrsvvrdDw= 2023-09-29T22:50:53.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QXOV4HTkP+ozjkFnn/vrsvvrdDw=', 'Content-Length': '0'} 2023-09-29T22:50:53.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4db08443c8de5f1-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.101 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.102 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.102 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.102 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:50:53.102 DEBUG:boto:Data: 2023-09-29T22:50:53.102 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.102 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.102 DEBUG:boto:Params: {} 2023-09-29T22:50:53.103 DEBUG:boto:Token: None 2023-09-29T22:50:53.103 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57 2023-09-29T22:50:53.103 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QXOV4HTkP+ozjkFnn/vrsvvrdDw= 2023-09-29T22:50:53.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QXOV4HTkP+ozjkFnn/vrsvvrdDw=', 'Content-Length': '0'} 2023-09-29T22:50:53.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042048f50eaa8e914-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:50:53.106 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.106 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.106 DEBUG:boto:Method: GET 2023-09-29T22:50:53.106 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:50:53.106 DEBUG:boto:Data: 2023-09-29T22:50:53.106 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.106 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.106 DEBUG:boto:Params: {} 2023-09-29T22:50:53.107 DEBUG:boto:Token: None 2023-09-29T22:50:53.107 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57 2023-09-29T22:50:53.107 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KUl/OMRNFbtlpqF7bLQsYVXkJvc= 2023-09-29T22:50:53.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KUl/OMRNFbtlpqF7bLQsYVXkJvc=', 'Content-Length': '0'} 2023-09-29T22:50:53.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1dfe2affbba5a84-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.109 DEBUG:boto:path=/hcnphz-59/obj57 2023-09-29T22:50:53.109 DEBUG:boto:auth_path=/hcnphz-59/obj57 2023-09-29T22:50:53.110 DEBUG:boto:Method: GET 2023-09-29T22:50:53.110 DEBUG:boto:Path: /hcnphz-59/obj57 2023-09-29T22:50:53.110 DEBUG:boto:Data: 2023-09-29T22:50:53.110 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.110 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.110 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.110 DEBUG:boto:Params: {} 2023-09-29T22:50:53.110 DEBUG:boto:Token: None 2023-09-29T22:50:53.111 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj57 2023-09-29T22:50:53.111 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KUl/OMRNFbtlpqF7bLQsYVXkJvc= 2023-09-29T22:50:53.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KUl/OMRNFbtlpqF7bLQsYVXkJvc=', 'Content-Length': '0'} 2023-09-29T22:50:53.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052b4a42b583b1281-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.113 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.113 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.113 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.113 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.113 DEBUG:boto:Method: GET 2023-09-29T22:50:53.114 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.114 DEBUG:boto:Data: 2023-09-29T22:50:53.114 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.114 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.114 DEBUG:boto:Params: {} 2023-09-29T22:50:53.114 DEBUG:boto:Token: None 2023-09-29T22:50:53.114 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.119 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000398eeac4c5736a48-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.119 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.119 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.120 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.120 DEBUG:boto:path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.120 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.120 DEBUG:boto:Method: GET 2023-09-29T22:50:53.120 DEBUG:boto:Path: /hcnphz-59/?prefix=obj57 2023-09-29T22:50:53.120 DEBUG:boto:Data: 2023-09-29T22:50:53.120 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.120 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.121 DEBUG:boto:Params: {} 2023-09-29T22:50:53.121 DEBUG:boto:Token: None 2023-09-29T22:50:53.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.128 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009dabdf4006eeb5c5-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.128 DEBUG:boto:b'hcnphz-59obj571000falseobj572023-09-29T22:38:48.874Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:50:53.129 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.129 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.129 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.129 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.129 DEBUG:boto:Method: GET 2023-09-29T22:50:53.129 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.129 DEBUG:boto:Data: 2023-09-29T22:50:53.129 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.130 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.130 DEBUG:boto:Params: {} 2023-09-29T22:50:53.130 DEBUG:boto:Token: None 2023-09-29T22:50:53.130 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.130 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E= 2023-09-29T22:50:53.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E=', 'Content-Length': '0'} 2023-09-29T22:50:53.133 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a42d20c40cd27a3f-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.174 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.175 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.175 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.175 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.175 DEBUG:boto:Method: GET 2023-09-29T22:50:53.175 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.175 DEBUG:boto:Data: 2023-09-29T22:50:53.175 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.175 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.176 DEBUG:boto:Params: {} 2023-09-29T22:50:53.176 DEBUG:boto:Token: None 2023-09-29T22:50:53.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E= 2023-09-29T22:50:53.177 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E=', 'Content-Length': '0'} 2023-09-29T22:50:53.180 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5c23bb38ed6091-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.180 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.180 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.180 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.180 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.180 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.181 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.181 DEBUG:boto:Data: 2023-09-29T22:50:53.181 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.181 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.181 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.181 DEBUG:boto:Params: {} 2023-09-29T22:50:53.181 DEBUG:boto:Token: None 2023-09-29T22:50:53.181 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.182 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FHAttdCVPFXJc4Or+PkBkotFP6A= 2023-09-29T22:50:53.182 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FHAttdCVPFXJc4Or+PkBkotFP6A=', 'Content-Length': '0'} 2023-09-29T22:50:53.185 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6cfbeca26b29556-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.185 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.185 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.185 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.185 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.186 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.186 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.186 DEBUG:boto:Data: 2023-09-29T22:50:53.186 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.186 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.186 DEBUG:boto:Params: {} 2023-09-29T22:50:53.186 DEBUG:boto:Token: None 2023-09-29T22:50:53.187 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FHAttdCVPFXJc4Or+PkBkotFP6A= 2023-09-29T22:50:53.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FHAttdCVPFXJc4Or+PkBkotFP6A=', 'Content-Length': '0'} 2023-09-29T22:50:53.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000034b12bcae4a04306-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:50:53.190 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.190 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.190 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.190 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.190 DEBUG:boto:Method: GET 2023-09-29T22:50:53.190 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.190 DEBUG:boto:Data: 2023-09-29T22:50:53.191 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.191 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.191 DEBUG:boto:Params: {} 2023-09-29T22:50:53.191 DEBUG:boto:Token: None 2023-09-29T22:50:53.191 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E= 2023-09-29T22:50:53.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E=', 'Content-Length': '0'} 2023-09-29T22:50:53.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009cca15f7bbaaae0e-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.194 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.194 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.195 DEBUG:boto:path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.195 DEBUG:boto:auth_path=/hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.195 DEBUG:boto:Method: GET 2023-09-29T22:50:53.195 DEBUG:boto:Path: /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.195 DEBUG:boto:Data: 2023-09-29T22:50:53.195 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.195 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.195 DEBUG:boto:Params: {} 2023-09-29T22:50:53.195 DEBUG:boto:Token: None 2023-09-29T22:50:53.196 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58?versionId=null 2023-09-29T22:50:53.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E= 2023-09-29T22:50:53.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GnBz3qzX/fFiW6vWBnS6gPqK92E=', 'Content-Length': '0'} 2023-09-29T22:50:53.198 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654130ea20e191be-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.242 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.242 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.243 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.243 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.243 DEBUG:boto:Method: GET 2023-09-29T22:50:53.243 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.243 DEBUG:boto:Data: 2023-09-29T22:50:53.243 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.243 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.243 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.243 DEBUG:boto:Params: {} 2023-09-29T22:50:53.244 DEBUG:boto:Token: None 2023-09-29T22:50:53.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007c9203a2ba133e23-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.249 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.250 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.250 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.250 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.250 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.250 DEBUG:boto:Method: GET 2023-09-29T22:50:53.250 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.250 DEBUG:boto:Data: 2023-09-29T22:50:53.250 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.251 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.251 DEBUG:boto:Params: {} 2023-09-29T22:50:53.251 DEBUG:boto:Token: None 2023-09-29T22:50:53.251 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.251 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.251 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000082ac989c17fe42ac-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.260 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.260 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.261 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.261 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.261 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:50:53.261 DEBUG:boto:Data: 2023-09-29T22:50:53.261 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.261 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.261 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.261 DEBUG:boto:Params: {} 2023-09-29T22:50:53.261 DEBUG:boto:Token: None 2023-09-29T22:50:53.262 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58 2023-09-29T22:50:53.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FlO7N1nblPq84LjYMQlYtJS3jZo= 2023-09-29T22:50:53.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FlO7N1nblPq84LjYMQlYtJS3jZo=', 'Content-Length': '0'} 2023-09-29T22:50:53.264 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d0ce9be0158ef5-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.264 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.264 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.264 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.264 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:50:53.265 DEBUG:boto:Data: 2023-09-29T22:50:53.265 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.265 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.265 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.265 DEBUG:boto:Params: {} 2023-09-29T22:50:53.265 DEBUG:boto:Token: None 2023-09-29T22:50:53.265 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58 2023-09-29T22:50:53.265 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FlO7N1nblPq84LjYMQlYtJS3jZo= 2023-09-29T22:50:53.265 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FlO7N1nblPq84LjYMQlYtJS3jZo=', 'Content-Length': '0'} 2023-09-29T22:50:53.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008bbce4400d691560-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:50:53.268 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.268 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.268 DEBUG:boto:Method: GET 2023-09-29T22:50:53.268 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:50:53.268 DEBUG:boto:Data: 2023-09-29T22:50:53.269 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.269 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.269 DEBUG:boto:Params: {} 2023-09-29T22:50:53.269 DEBUG:boto:Token: None 2023-09-29T22:50:53.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58 2023-09-29T22:50:53.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:751vBASX/bmEE4yAqKwgYrpQtN4= 2023-09-29T22:50:53.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:751vBASX/bmEE4yAqKwgYrpQtN4=', 'Content-Length': '0'} 2023-09-29T22:50:53.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccc614a60dd64c84-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.271 DEBUG:boto:path=/hcnphz-59/obj58 2023-09-29T22:50:53.272 DEBUG:boto:auth_path=/hcnphz-59/obj58 2023-09-29T22:50:53.272 DEBUG:boto:Method: GET 2023-09-29T22:50:53.272 DEBUG:boto:Path: /hcnphz-59/obj58 2023-09-29T22:50:53.272 DEBUG:boto:Data: 2023-09-29T22:50:53.272 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.272 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.272 DEBUG:boto:Params: {} 2023-09-29T22:50:53.272 DEBUG:boto:Token: None 2023-09-29T22:50:53.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj58 2023-09-29T22:50:53.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:751vBASX/bmEE4yAqKwgYrpQtN4= 2023-09-29T22:50:53.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:751vBASX/bmEE4yAqKwgYrpQtN4=', 'Content-Length': '0'} 2023-09-29T22:50:53.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047643b23ed3dd43b-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.322 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.323 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.323 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.323 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.323 DEBUG:boto:Method: GET 2023-09-29T22:50:53.323 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.323 DEBUG:boto:Data: 2023-09-29T22:50:53.323 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.323 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.324 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.324 DEBUG:boto:Params: {} 2023-09-29T22:50:53.324 DEBUG:boto:Token: None 2023-09-29T22:50:53.324 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.324 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009133f0920d20a0af-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.330 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.330 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.330 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.330 DEBUG:boto:path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.330 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.331 DEBUG:boto:Method: GET 2023-09-29T22:50:53.331 DEBUG:boto:Path: /hcnphz-59/?prefix=obj58 2023-09-29T22:50:53.331 DEBUG:boto:Data: 2023-09-29T22:50:53.331 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.331 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.331 DEBUG:boto:Params: {} 2023-09-29T22:50:53.331 DEBUG:boto:Token: None 2023-09-29T22:50:53.331 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.340 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0e005ee63cdd71f-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.340 DEBUG:boto:b'hcnphz-59obj581000falseobj582023-09-29T22:43:47.224Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:50:53.340 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.340 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.341 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.341 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.341 DEBUG:boto:Method: GET 2023-09-29T22:50:53.341 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.341 DEBUG:boto:Data: 2023-09-29T22:50:53.341 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.341 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.341 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.341 DEBUG:boto:Params: {} 2023-09-29T22:50:53.341 DEBUG:boto:Token: None 2023-09-29T22:50:53.342 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.342 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc= 2023-09-29T22:50:53.342 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc=', 'Content-Length': '0'} 2023-09-29T22:50:53.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ac11c2e2d3e5e72-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.386 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.387 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.387 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.387 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.387 DEBUG:boto:Method: GET 2023-09-29T22:50:53.387 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.387 DEBUG:boto:Data: 2023-09-29T22:50:53.387 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.387 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.387 DEBUG:boto:Params: {} 2023-09-29T22:50:53.388 DEBUG:boto:Token: None 2023-09-29T22:50:53.388 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc= 2023-09-29T22:50:53.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc=', 'Content-Length': '0'} 2023-09-29T22:50:53.391 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000618721f7d7732452-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.392 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.392 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.392 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.392 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.392 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.392 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.392 DEBUG:boto:Data: 2023-09-29T22:50:53.393 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.393 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.393 DEBUG:boto:Params: {} 2023-09-29T22:50:53.393 DEBUG:boto:Token: None 2023-09-29T22:50:53.393 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n0xuOUaVfq5qXXXyC1Rm1K92kv4= 2023-09-29T22:50:53.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n0xuOUaVfq5qXXXyC1Rm1K92kv4=', 'Content-Length': '0'} 2023-09-29T22:50:53.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8b4ccfc492aef14-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.396 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.396 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.396 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.396 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.397 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.397 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.397 DEBUG:boto:Data: 2023-09-29T22:50:53.397 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.397 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.397 DEBUG:boto:Params: {} 2023-09-29T22:50:53.397 DEBUG:boto:Token: None 2023-09-29T22:50:53.397 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n0xuOUaVfq5qXXXyC1Rm1K92kv4= 2023-09-29T22:50:53.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n0xuOUaVfq5qXXXyC1Rm1K92kv4=', 'Content-Length': '0'} 2023-09-29T22:50:53.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b0ac4ead3e18a51-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:50:53.401 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.401 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.401 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.401 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.401 DEBUG:boto:Method: GET 2023-09-29T22:50:53.401 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.401 DEBUG:boto:Data: 2023-09-29T22:50:53.402 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.402 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.402 DEBUG:boto:Params: {} 2023-09-29T22:50:53.402 DEBUG:boto:Token: None 2023-09-29T22:50:53.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc= 2023-09-29T22:50:53.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc=', 'Content-Length': '0'} 2023-09-29T22:50:53.405 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000678e55f845247426-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.406 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.406 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.406 DEBUG:boto:path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.406 DEBUG:boto:auth_path=/hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.406 DEBUG:boto:Method: GET 2023-09-29T22:50:53.406 DEBUG:boto:Path: /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.406 DEBUG:boto:Data: 2023-09-29T22:50:53.406 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.406 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.407 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.407 DEBUG:boto:Params: {} 2023-09-29T22:50:53.407 DEBUG:boto:Token: None 2023-09-29T22:50:53.407 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62?versionId=null 2023-09-29T22:50:53.407 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc= 2023-09-29T22:50:53.407 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nuWtQNbfepICVeEvU8Xp2I2WXKc=', 'Content-Length': '0'} 2023-09-29T22:50:53.410 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000134f2ce318db7812-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.450 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.450 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.451 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.451 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.451 DEBUG:boto:Method: GET 2023-09-29T22:50:53.451 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.451 DEBUG:boto:Data: 2023-09-29T22:50:53.451 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.451 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.451 DEBUG:boto:Params: {} 2023-09-29T22:50:53.451 DEBUG:boto:Token: None 2023-09-29T22:50:53.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe9714556717787e-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.457 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.457 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.458 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.458 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.458 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.458 DEBUG:boto:Method: GET 2023-09-29T22:50:53.458 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.458 DEBUG:boto:Data: 2023-09-29T22:50:53.458 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.458 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.459 DEBUG:boto:Params: {} 2023-09-29T22:50:53.459 DEBUG:boto:Token: None 2023-09-29T22:50:53.459 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.459 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.466 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3809e735445ff43-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.466 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.466 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.466 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.467 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.467 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:50:53.467 DEBUG:boto:Data: 2023-09-29T22:50:53.467 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.467 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.467 DEBUG:boto:Params: {} 2023-09-29T22:50:53.467 DEBUG:boto:Token: None 2023-09-29T22:50:53.467 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62 2023-09-29T22:50:53.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cxHo0vCgEAICF4uWJjo8LSFIrB0= 2023-09-29T22:50:53.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cxHo0vCgEAICF4uWJjo8LSFIrB0=', 'Content-Length': '0'} 2023-09-29T22:50:53.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007084f6c490b79df5-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.470 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.470 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.470 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.470 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:50:53.471 DEBUG:boto:Data: 2023-09-29T22:50:53.471 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.471 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.471 DEBUG:boto:Params: {} 2023-09-29T22:50:53.471 DEBUG:boto:Token: None 2023-09-29T22:50:53.471 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62 2023-09-29T22:50:53.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cxHo0vCgEAICF4uWJjo8LSFIrB0= 2023-09-29T22:50:53.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cxHo0vCgEAICF4uWJjo8LSFIrB0=', 'Content-Length': '0'} 2023-09-29T22:50:53.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbd1f92cdd376f90-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:50:53.474 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.474 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.474 DEBUG:boto:Method: GET 2023-09-29T22:50:53.475 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:50:53.475 DEBUG:boto:Data: 2023-09-29T22:50:53.475 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.475 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.475 DEBUG:boto:Params: {} 2023-09-29T22:50:53.475 DEBUG:boto:Token: None 2023-09-29T22:50:53.475 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62 2023-09-29T22:50:53.475 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lFUQeGuyo1aNV4rBgmvTtx77Fy4= 2023-09-29T22:50:53.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lFUQeGuyo1aNV4rBgmvTtx77Fy4=', 'Content-Length': '0'} 2023-09-29T22:50:53.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ec4c5cdf619e858-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.518 DEBUG:boto:path=/hcnphz-59/obj62 2023-09-29T22:50:53.518 DEBUG:boto:auth_path=/hcnphz-59/obj62 2023-09-29T22:50:53.519 DEBUG:boto:Method: GET 2023-09-29T22:50:53.519 DEBUG:boto:Path: /hcnphz-59/obj62 2023-09-29T22:50:53.519 DEBUG:boto:Data: 2023-09-29T22:50:53.519 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.519 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.519 DEBUG:boto:Params: {} 2023-09-29T22:50:53.519 DEBUG:boto:Token: None 2023-09-29T22:50:53.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj62 2023-09-29T22:50:53.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lFUQeGuyo1aNV4rBgmvTtx77Fy4= 2023-09-29T22:50:53.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lFUQeGuyo1aNV4rBgmvTtx77Fy4=', 'Content-Length': '0'} 2023-09-29T22:50:53.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ba4cb9549d06dddb-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.523 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.523 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.523 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.523 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.523 DEBUG:boto:Method: GET 2023-09-29T22:50:53.523 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.523 DEBUG:boto:Data: 2023-09-29T22:50:53.524 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.524 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.524 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.524 DEBUG:boto:Params: {} 2023-09-29T22:50:53.524 DEBUG:boto:Token: None 2023-09-29T22:50:53.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.529 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000221ee74b2d4ae2f9-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.529 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.529 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.530 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.530 DEBUG:boto:path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.530 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.530 DEBUG:boto:Method: GET 2023-09-29T22:50:53.530 DEBUG:boto:Path: /hcnphz-59/?prefix=obj62 2023-09-29T22:50:53.530 DEBUG:boto:Data: 2023-09-29T22:50:53.530 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.530 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.530 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.531 DEBUG:boto:Params: {} 2023-09-29T22:50:53.531 DEBUG:boto:Token: None 2023-09-29T22:50:53.531 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.531 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.531 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.538 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000078d54d2814285be-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.582 DEBUG:boto:b'hcnphz-59obj621000falseobj622023-09-29T22:46:48.012Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:50:53.583 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.583 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.583 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.583 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.583 DEBUG:boto:Method: GET 2023-09-29T22:50:53.583 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.584 DEBUG:boto:Data: 2023-09-29T22:50:53.584 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.584 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.584 DEBUG:boto:Params: {} 2023-09-29T22:50:53.584 DEBUG:boto:Token: None 2023-09-29T22:50:53.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8= 2023-09-29T22:50:53.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8=', 'Content-Length': '0'} 2023-09-29T22:50:53.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc750f09808b9774-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.588 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.588 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.588 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.588 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.588 DEBUG:boto:Method: GET 2023-09-29T22:50:53.589 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.589 DEBUG:boto:Data: 2023-09-29T22:50:53.589 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.589 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.589 DEBUG:boto:Params: {} 2023-09-29T22:50:53.589 DEBUG:boto:Token: None 2023-09-29T22:50:53.589 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8= 2023-09-29T22:50:53.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8=', 'Content-Length': '0'} 2023-09-29T22:50:53.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006db3639781d3090f-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.593 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.593 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.593 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.593 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.593 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.593 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.593 DEBUG:boto:Data: 2023-09-29T22:50:53.593 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.594 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.594 DEBUG:boto:Params: {} 2023-09-29T22:50:53.594 DEBUG:boto:Token: None 2023-09-29T22:50:53.594 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.594 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HdTHxotdsa2PeJtZ7wNq3ykjW64= 2023-09-29T22:50:53.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HdTHxotdsa2PeJtZ7wNq3ykjW64=', 'Content-Length': '0'} 2023-09-29T22:50:53.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099499bc4f9ff3eb8-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.597 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.597 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.598 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.598 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.598 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.598 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.598 DEBUG:boto:Data: 2023-09-29T22:50:53.598 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.598 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.598 DEBUG:boto:Params: {} 2023-09-29T22:50:53.599 DEBUG:boto:Token: None 2023-09-29T22:50:53.599 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HdTHxotdsa2PeJtZ7wNq3ykjW64= 2023-09-29T22:50:53.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HdTHxotdsa2PeJtZ7wNq3ykjW64=', 'Content-Length': '0'} 2023-09-29T22:50:53.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c8358df74737498-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:50:53.602 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.602 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.602 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.602 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.603 DEBUG:boto:Method: GET 2023-09-29T22:50:53.603 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.603 DEBUG:boto:Data: 2023-09-29T22:50:53.603 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.603 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.603 DEBUG:boto:Params: {} 2023-09-29T22:50:53.603 DEBUG:boto:Token: None 2023-09-29T22:50:53.603 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.604 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8= 2023-09-29T22:50:53.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8=', 'Content-Length': '0'} 2023-09-29T22:50:53.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085880e1cb88efe3f-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.607 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.607 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.607 DEBUG:boto:path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.607 DEBUG:boto:auth_path=/hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.608 DEBUG:boto:Method: GET 2023-09-29T22:50:53.608 DEBUG:boto:Path: /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.608 DEBUG:boto:Data: 2023-09-29T22:50:53.608 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.608 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.608 DEBUG:boto:Params: {} 2023-09-29T22:50:53.608 DEBUG:boto:Token: None 2023-09-29T22:50:53.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68?versionId=null 2023-09-29T22:50:53.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8= 2023-09-29T22:50:53.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lnYwReQkXmTkAIG4SUP8MAdGhV8=', 'Content-Length': '0'} 2023-09-29T22:50:53.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000028544e1fcc596044-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.612 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.612 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.612 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.612 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.612 DEBUG:boto:Method: GET 2023-09-29T22:50:53.612 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.612 DEBUG:boto:Data: 2023-09-29T22:50:53.612 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.612 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.613 DEBUG:boto:Params: {} 2023-09-29T22:50:53.613 DEBUG:boto:Token: None 2023-09-29T22:50:53.613 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.613 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.618 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000cc7b937e58380f2-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.658 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.659 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.659 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.659 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.659 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.659 DEBUG:boto:Method: GET 2023-09-29T22:50:53.659 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.659 DEBUG:boto:Data: 2023-09-29T22:50:53.660 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.660 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.660 DEBUG:boto:Params: {} 2023-09-29T22:50:53.660 DEBUG:boto:Token: None 2023-09-29T22:50:53.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.670 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077c1a72db688a061-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.670 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.670 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.671 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.671 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.671 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:50:53.671 DEBUG:boto:Data: 2023-09-29T22:50:53.671 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.671 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.671 DEBUG:boto:Params: {} 2023-09-29T22:50:53.671 DEBUG:boto:Token: None 2023-09-29T22:50:53.672 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68 2023-09-29T22:50:53.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0Gq/MBGp3KcXw7eUYNeZ7HUVnAk= 2023-09-29T22:50:53.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0Gq/MBGp3KcXw7eUYNeZ7HUVnAk=', 'Content-Length': '0'} 2023-09-29T22:50:53.674 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b44c6937b7f0d851-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.674 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.674 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.674 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.675 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:50:53.675 DEBUG:boto:Data: 2023-09-29T22:50:53.675 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.675 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.675 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.675 DEBUG:boto:Params: {} 2023-09-29T22:50:53.675 DEBUG:boto:Token: None 2023-09-29T22:50:53.675 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68 2023-09-29T22:50:53.675 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0Gq/MBGp3KcXw7eUYNeZ7HUVnAk= 2023-09-29T22:50:53.676 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0Gq/MBGp3KcXw7eUYNeZ7HUVnAk=', 'Content-Length': '0'} 2023-09-29T22:50:53.678 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000083cdd82d6f21d6be-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:50:53.678 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.678 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.679 DEBUG:boto:Method: GET 2023-09-29T22:50:53.679 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:50:53.679 DEBUG:boto:Data: 2023-09-29T22:50:53.679 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.679 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.679 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.679 DEBUG:boto:Params: {} 2023-09-29T22:50:53.679 DEBUG:boto:Token: None 2023-09-29T22:50:53.679 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68 2023-09-29T22:50:53.680 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Lv8VFLOs3XdNL/Zg3l4gfZq0Z8= 2023-09-29T22:50:53.680 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Lv8VFLOs3XdNL/Zg3l4gfZq0Z8=', 'Content-Length': '0'} 2023-09-29T22:50:53.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003616b5f2c1acac05-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.682 DEBUG:boto:path=/hcnphz-59/obj68 2023-09-29T22:50:53.682 DEBUG:boto:auth_path=/hcnphz-59/obj68 2023-09-29T22:50:53.682 DEBUG:boto:Method: GET 2023-09-29T22:50:53.682 DEBUG:boto:Path: /hcnphz-59/obj68 2023-09-29T22:50:53.682 DEBUG:boto:Data: 2023-09-29T22:50:53.682 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.683 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.683 DEBUG:boto:Params: {} 2023-09-29T22:50:53.683 DEBUG:boto:Token: None 2023-09-29T22:50:53.683 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj68 2023-09-29T22:50:53.683 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Lv8VFLOs3XdNL/Zg3l4gfZq0Z8= 2023-09-29T22:50:53.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Lv8VFLOs3XdNL/Zg3l4gfZq0Z8=', 'Content-Length': '0'} 2023-09-29T22:50:53.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000077f581edce761c16-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.730 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.731 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.731 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.731 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.731 DEBUG:boto:Method: GET 2023-09-29T22:50:53.731 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.731 DEBUG:boto:Data: 2023-09-29T22:50:53.731 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.732 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.732 DEBUG:boto:Params: {} 2023-09-29T22:50:53.732 DEBUG:boto:Token: None 2023-09-29T22:50:53.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c8d374e841f8720-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.737 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.737 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.738 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.738 DEBUG:boto:path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.738 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.738 DEBUG:boto:Method: GET 2023-09-29T22:50:53.738 DEBUG:boto:Path: /hcnphz-59/?prefix=obj68 2023-09-29T22:50:53.738 DEBUG:boto:Data: 2023-09-29T22:50:53.738 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.738 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.739 DEBUG:boto:Params: {} 2023-09-29T22:50:53.739 DEBUG:boto:Token: None 2023-09-29T22:50:53.739 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.746 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cf75336ff13693a9-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.747 DEBUG:boto:b'hcnphz-59obj681000falseobj682023-09-29T22:36:06.006Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:50:53.747 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.747 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.747 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.748 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.748 DEBUG:boto:Method: GET 2023-09-29T22:50:53.748 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.748 DEBUG:boto:Data: 2023-09-29T22:50:53.748 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.748 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.748 DEBUG:boto:Params: {} 2023-09-29T22:50:53.748 DEBUG:boto:Token: None 2023-09-29T22:50:53.748 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.749 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4= 2023-09-29T22:50:53.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4=', 'Content-Length': '0'} 2023-09-29T22:50:53.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7db19a60378c9e2-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.751 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.751 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.751 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.751 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.752 DEBUG:boto:Method: GET 2023-09-29T22:50:53.752 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.752 DEBUG:boto:Data: 2023-09-29T22:50:53.752 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.752 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.752 DEBUG:boto:Params: {} 2023-09-29T22:50:53.752 DEBUG:boto:Token: None 2023-09-29T22:50:53.752 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4= 2023-09-29T22:50:53.753 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4=', 'Content-Length': '0'} 2023-09-29T22:50:53.755 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071bcd53c01b51440-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.755 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.755 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.755 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.755 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.756 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.756 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.756 DEBUG:boto:Data: 2023-09-29T22:50:53.756 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.756 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.756 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.756 DEBUG:boto:Params: {} 2023-09-29T22:50:53.756 DEBUG:boto:Token: None 2023-09-29T22:50:53.756 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.757 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y1ugwt82fRvfCgvHxt6mDKDzPFU= 2023-09-29T22:50:53.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y1ugwt82fRvfCgvHxt6mDKDzPFU=', 'Content-Length': '0'} 2023-09-29T22:50:53.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c2a53b7cd1ec09-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.759 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.759 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.759 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.759 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.759 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.759 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.760 DEBUG:boto:Data: 2023-09-29T22:50:53.760 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.760 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.760 DEBUG:boto:Params: {} 2023-09-29T22:50:53.760 DEBUG:boto:Token: None 2023-09-29T22:50:53.760 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.760 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y1ugwt82fRvfCgvHxt6mDKDzPFU= 2023-09-29T22:50:53.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y1ugwt82fRvfCgvHxt6mDKDzPFU=', 'Content-Length': '0'} 2023-09-29T22:50:53.762 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000064a165692d8dbad0-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:50:53.763 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.763 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.763 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.763 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.763 DEBUG:boto:Method: GET 2023-09-29T22:50:53.763 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.763 DEBUG:boto:Data: 2023-09-29T22:50:53.763 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.764 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.764 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.764 DEBUG:boto:Params: {} 2023-09-29T22:50:53.764 DEBUG:boto:Token: None 2023-09-29T22:50:53.764 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.764 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4= 2023-09-29T22:50:53.764 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4=', 'Content-Length': '0'} 2023-09-29T22:50:53.766 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e1a1237caddb5c7-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.810 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.811 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.811 DEBUG:boto:path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.811 DEBUG:boto:auth_path=/hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.811 DEBUG:boto:Method: GET 2023-09-29T22:50:53.811 DEBUG:boto:Path: /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.811 DEBUG:boto:Data: 2023-09-29T22:50:53.811 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.811 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.811 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.811 DEBUG:boto:Params: {} 2023-09-29T22:50:53.812 DEBUG:boto:Token: None 2023-09-29T22:50:53.812 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69?versionId=null 2023-09-29T22:50:53.812 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4= 2023-09-29T22:50:53.812 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MpOeNzNghcgeYhA69g8gDQpson4=', 'Content-Length': '0'} 2023-09-29T22:50:53.815 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b6a58d0b830ab20-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.815 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.815 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.815 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.816 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.816 DEBUG:boto:Method: GET 2023-09-29T22:50:53.816 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.816 DEBUG:boto:Data: 2023-09-29T22:50:53.816 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.816 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.816 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.816 DEBUG:boto:Params: {} 2023-09-29T22:50:53.816 DEBUG:boto:Token: None 2023-09-29T22:50:53.816 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.817 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.817 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.821 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e4aa374f3398622-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.821 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.822 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.822 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.822 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.822 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.822 DEBUG:boto:Method: GET 2023-09-29T22:50:53.822 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.822 DEBUG:boto:Data: 2023-09-29T22:50:53.822 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.823 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.823 DEBUG:boto:Params: {} 2023-09-29T22:50:53.823 DEBUG:boto:Token: None 2023-09-29T22:50:53.823 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.823 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.823 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.832 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005207290460108389-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.832 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.833 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.833 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.833 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.833 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:50:53.833 DEBUG:boto:Data: 2023-09-29T22:50:53.833 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.834 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.834 DEBUG:boto:Params: {} 2023-09-29T22:50:53.834 DEBUG:boto:Token: None 2023-09-29T22:50:53.834 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69 2023-09-29T22:50:53.834 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vr15oOGb6B6+t6enhIDDVSmRph8= 2023-09-29T22:50:53.834 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vr15oOGb6B6+t6enhIDDVSmRph8=', 'Content-Length': '0'} 2023-09-29T22:50:53.836 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc5f7c2bd0273721-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.836 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.836 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.837 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.837 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:50:53.837 DEBUG:boto:Data: 2023-09-29T22:50:53.837 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.837 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.837 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.837 DEBUG:boto:Params: {} 2023-09-29T22:50:53.837 DEBUG:boto:Token: None 2023-09-29T22:50:53.837 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69 2023-09-29T22:50:53.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Vr15oOGb6B6+t6enhIDDVSmRph8= 2023-09-29T22:50:53.838 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Vr15oOGb6B6+t6enhIDDVSmRph8=', 'Content-Length': '0'} 2023-09-29T22:50:53.840 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f8e9efd9acebc1-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:50:53.840 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.840 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.841 DEBUG:boto:Method: GET 2023-09-29T22:50:53.841 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:50:53.841 DEBUG:boto:Data: 2023-09-29T22:50:53.841 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.841 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.841 DEBUG:boto:Params: {} 2023-09-29T22:50:53.841 DEBUG:boto:Token: None 2023-09-29T22:50:53.842 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69 2023-09-29T22:50:53.842 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5ZuZrvzPuCjD3zvWfoqWFsJBV8= 2023-09-29T22:50:53.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5ZuZrvzPuCjD3zvWfoqWFsJBV8=', 'Content-Length': '0'} 2023-09-29T22:50:53.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052087abe0d440fa9-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.844 DEBUG:boto:path=/hcnphz-59/obj69 2023-09-29T22:50:53.844 DEBUG:boto:auth_path=/hcnphz-59/obj69 2023-09-29T22:50:53.844 DEBUG:boto:Method: GET 2023-09-29T22:50:53.844 DEBUG:boto:Path: /hcnphz-59/obj69 2023-09-29T22:50:53.845 DEBUG:boto:Data: 2023-09-29T22:50:53.845 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.845 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.845 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.845 DEBUG:boto:Params: {} 2023-09-29T22:50:53.845 DEBUG:boto:Token: None 2023-09-29T22:50:53.845 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj69 2023-09-29T22:50:53.845 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5ZuZrvzPuCjD3zvWfoqWFsJBV8= 2023-09-29T22:50:53.845 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5ZuZrvzPuCjD3zvWfoqWFsJBV8=', 'Content-Length': '0'} 2023-09-29T22:50:53.848 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ab1973bb7b9d8c9-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.848 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.848 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.848 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.848 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.848 DEBUG:boto:Method: GET 2023-09-29T22:50:53.848 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.849 DEBUG:boto:Data: 2023-09-29T22:50:53.849 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.849 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.849 DEBUG:boto:Params: {} 2023-09-29T22:50:53.849 DEBUG:boto:Token: None 2023-09-29T22:50:53.849 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.854 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dc499158d2a8012c-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.854 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.854 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.854 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.855 DEBUG:boto:path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.855 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.855 DEBUG:boto:Method: GET 2023-09-29T22:50:53.855 DEBUG:boto:Path: /hcnphz-59/?prefix=obj69 2023-09-29T22:50:53.855 DEBUG:boto:Data: 2023-09-29T22:50:53.855 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.855 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.855 DEBUG:boto:Params: {} 2023-09-29T22:50:53.856 DEBUG:boto:Token: None 2023-09-29T22:50:53.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.863 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027feb6816705b241-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.910 DEBUG:boto:b'hcnphz-59obj691000falseobj692023-09-29T22:42:07.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:50:53.911 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:53.911 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:53.911 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.911 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.911 DEBUG:boto:Method: GET 2023-09-29T22:50:53.911 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.912 DEBUG:boto:Data: 2023-09-29T22:50:53.912 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.912 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.912 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.912 DEBUG:boto:Params: {} 2023-09-29T22:50:53.912 DEBUG:boto:Token: None 2023-09-29T22:50:53.912 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI= 2023-09-29T22:50:53.913 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI=', 'Content-Length': '0'} 2023-09-29T22:50:53.915 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c0c6746d49b576-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.916 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:53.916 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:53.916 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.916 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.916 DEBUG:boto:Method: GET 2023-09-29T22:50:53.916 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.916 DEBUG:boto:Data: 2023-09-29T22:50:53.917 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.917 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.917 DEBUG:boto:Params: {} 2023-09-29T22:50:53.917 DEBUG:boto:Token: None 2023-09-29T22:50:53.917 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.917 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI= 2023-09-29T22:50:53.917 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI=', 'Content-Length': '0'} 2023-09-29T22:50:53.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f4d85ab37cfc7d5-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.921 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:53.921 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:53.921 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.921 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.921 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.921 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.921 DEBUG:boto:Data: 2023-09-29T22:50:53.921 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.922 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.922 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.922 DEBUG:boto:Params: {} 2023-09-29T22:50:53.922 DEBUG:boto:Token: None 2023-09-29T22:50:53.922 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/qh+pHhu6Awly24hiG2cebWmFOU= 2023-09-29T22:50:53.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/qh+pHhu6Awly24hiG2cebWmFOU=', 'Content-Length': '0'} 2023-09-29T22:50:53.925 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000a5d922860d7457d-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.926 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:53.926 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:53.926 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.926 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.926 DEBUG:boto:Method: HEAD 2023-09-29T22:50:53.926 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.926 DEBUG:boto:Data: 2023-09-29T22:50:53.926 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.927 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.927 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.927 DEBUG:boto:Params: {} 2023-09-29T22:50:53.927 DEBUG:boto:Token: None 2023-09-29T22:50:53.927 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.927 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/qh+pHhu6Awly24hiG2cebWmFOU= 2023-09-29T22:50:53.927 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/qh+pHhu6Awly24hiG2cebWmFOU=', 'Content-Length': '0'} 2023-09-29T22:50:53.930 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5ebd2faa324e9e2-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:50:53.930 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:53.931 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:53.931 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.931 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.931 DEBUG:boto:Method: GET 2023-09-29T22:50:53.931 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.931 DEBUG:boto:Data: 2023-09-29T22:50:53.931 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.932 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.932 DEBUG:boto:Params: {} 2023-09-29T22:50:53.932 DEBUG:boto:Token: None 2023-09-29T22:50:53.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI= 2023-09-29T22:50:53.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI=', 'Content-Length': '0'} 2023-09-29T22:50:53.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000052928c7d5c423012-00651754cd-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.982 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:53.982 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:53.983 DEBUG:boto:path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.983 DEBUG:boto:auth_path=/hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.983 DEBUG:boto:Method: GET 2023-09-29T22:50:53.983 DEBUG:boto:Path: /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.983 DEBUG:boto:Data: 2023-09-29T22:50:53.983 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.983 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.983 DEBUG:boto:Params: {} 2023-09-29T22:50:53.984 DEBUG:boto:Token: None 2023-09-29T22:50:53.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/obj74?versionId=null 2023-09-29T22:50:53.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI= 2023-09-29T22:50:53.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LIX2ubht65GeJ8EGOTk6jWmOxgI=', 'Content-Length': '0'} 2023-09-29T22:50:53.987 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000447a4c34e05a3182-00651754cd-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.987 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.987 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.987 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:53.988 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:53.988 DEBUG:boto:Method: GET 2023-09-29T22:50:53.988 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:50:53.988 DEBUG:boto:Data: 2023-09-29T22:50:53.988 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.988 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:53.988 DEBUG:boto:Port: 8001 2023-09-29T22:50:53.988 DEBUG:boto:Params: {} 2023-09-29T22:50:53.988 DEBUG:boto:Token: None 2023-09-29T22:50:53.989 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.989 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.989 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:53.993 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000043e4e2a9c0278dbb-00651754cd-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:53.993 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:53.994 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:53.994 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:53.994 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:53.994 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:53.994 DEBUG:boto:Method: GET 2023-09-29T22:50:53.994 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:50:53.995 DEBUG:boto:Data: 2023-09-29T22:50:53.995 DEBUG:boto:Headers: {} 2023-09-29T22:50:53.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:53.995 DEBUG:boto:Port: 8000 2023-09-29T22:50:53.995 DEBUG:boto:Params: {} 2023-09-29T22:50:53.995 DEBUG:boto:Token: None 2023-09-29T22:50:53.995 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:53 GMT /hcnphz-59/ 2023-09-29T22:50:53.995 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk= 2023-09-29T22:50:53.996 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cMfDOuUVLpDIjHm/FimfvWCpAnk=', 'Content-Length': '0'} 2023-09-29T22:50:54.003 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000052a490a71998a0a3-00651754cd-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.046 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.047 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:54.047 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:54.047 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.047 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:50:54.047 DEBUG:boto:Data: 2023-09-29T22:50:54.047 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.048 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.048 DEBUG:boto:Params: {} 2023-09-29T22:50:54.048 DEBUG:boto:Token: None 2023-09-29T22:50:54.048 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj74 2023-09-29T22:50:54.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TY4JMv7gtm/vqLSxIps527/tMcQ= 2023-09-29T22:50:54.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TY4JMv7gtm/vqLSxIps527/tMcQ=', 'Content-Length': '0'} 2023-09-29T22:50:54.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a16c644ba0aa5317-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.051 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:54.051 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:54.051 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.052 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:50:54.052 DEBUG:boto:Data: 2023-09-29T22:50:54.052 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.052 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.052 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.052 DEBUG:boto:Params: {} 2023-09-29T22:50:54.052 DEBUG:boto:Token: None 2023-09-29T22:50:54.052 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj74 2023-09-29T22:50:54.052 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TY4JMv7gtm/vqLSxIps527/tMcQ= 2023-09-29T22:50:54.053 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TY4JMv7gtm/vqLSxIps527/tMcQ=', 'Content-Length': '0'} 2023-09-29T22:50:54.055 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000410e311a913c0a7c-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:50:54.056 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:54.056 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:54.056 DEBUG:boto:Method: GET 2023-09-29T22:50:54.056 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:50:54.056 DEBUG:boto:Data: 2023-09-29T22:50:54.056 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.056 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.057 DEBUG:boto:Params: {} 2023-09-29T22:50:54.057 DEBUG:boto:Token: None 2023-09-29T22:50:54.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj74 2023-09-29T22:50:54.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RSDawtbevad2gDr6k4KAvSIhP1c= 2023-09-29T22:50:54.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RSDawtbevad2gDr6k4KAvSIhP1c=', 'Content-Length': '0'} 2023-09-29T22:50:54.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3e72273a774cbca-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.060 DEBUG:boto:path=/hcnphz-59/obj74 2023-09-29T22:50:54.060 DEBUG:boto:auth_path=/hcnphz-59/obj74 2023-09-29T22:50:54.060 DEBUG:boto:Method: GET 2023-09-29T22:50:54.060 DEBUG:boto:Path: /hcnphz-59/obj74 2023-09-29T22:50:54.060 DEBUG:boto:Data: 2023-09-29T22:50:54.060 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.061 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.061 DEBUG:boto:Params: {} 2023-09-29T22:50:54.061 DEBUG:boto:Token: None 2023-09-29T22:50:54.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj74 2023-09-29T22:50:54.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RSDawtbevad2gDr6k4KAvSIhP1c= 2023-09-29T22:50:54.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RSDawtbevad2gDr6k4KAvSIhP1c=', 'Content-Length': '0'} 2023-09-29T22:50:54.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:48:09 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074eb46c8b13d1146-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.064 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.064 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.064 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:54.064 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:54.064 DEBUG:boto:Method: GET 2023-09-29T22:50:54.065 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:50:54.065 DEBUG:boto:Data: 2023-09-29T22:50:54.065 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.065 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.065 DEBUG:boto:Params: {} 2023-09-29T22:50:54.065 DEBUG:boto:Token: None 2023-09-29T22:50:54.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d882fca59d428a1-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.114 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.115 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.115 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.115 DEBUG:boto:path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:54.115 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj74 2023-09-29T22:50:54.115 DEBUG:boto:Method: GET 2023-09-29T22:50:54.115 DEBUG:boto:Path: /hcnphz-59/?prefix=obj74 2023-09-29T22:50:54.115 DEBUG:boto:Data: 2023-09-29T22:50:54.116 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.116 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.116 DEBUG:boto:Params: {} 2023-09-29T22:50:54.116 DEBUG:boto:Token: None 2023-09-29T22:50:54.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.126 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df65c324a649cd55-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.126 DEBUG:boto:b'hcnphz-59obj741000falseobj742023-09-29T22:48:09.358Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:50:54.127 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.127 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.127 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.127 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.127 DEBUG:boto:Method: GET 2023-09-29T22:50:54.128 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.128 DEBUG:boto:Data: 2023-09-29T22:50:54.128 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.128 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.128 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.128 DEBUG:boto:Params: {} 2023-09-29T22:50:54.128 DEBUG:boto:Token: None 2023-09-29T22:50:54.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k= 2023-09-29T22:50:54.129 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k=', 'Content-Length': '0'} 2023-09-29T22:50:54.131 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b32fe4854dce798-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.132 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.132 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.132 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.132 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.132 DEBUG:boto:Method: GET 2023-09-29T22:50:54.132 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.132 DEBUG:boto:Data: 2023-09-29T22:50:54.132 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.133 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.133 DEBUG:boto:Params: {} 2023-09-29T22:50:54.133 DEBUG:boto:Token: None 2023-09-29T22:50:54.133 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k= 2023-09-29T22:50:54.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k=', 'Content-Length': '0'} 2023-09-29T22:50:54.135 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007985a6da05058de8-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.178 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.179 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.179 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.179 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.179 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.179 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.179 DEBUG:boto:Data: 2023-09-29T22:50:54.179 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.179 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.180 DEBUG:boto:Params: {} 2023-09-29T22:50:54.180 DEBUG:boto:Token: None 2023-09-29T22:50:54.180 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tj0WbTGsJYvOve8cJ+/PYtS6jTQ= 2023-09-29T22:50:54.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tj0WbTGsJYvOve8cJ+/PYtS6jTQ=', 'Content-Length': '0'} 2023-09-29T22:50:54.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000926bd100ff204f05-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.183 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.183 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.183 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.183 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.183 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.183 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.183 DEBUG:boto:Data: 2023-09-29T22:50:54.183 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.184 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.184 DEBUG:boto:Params: {} 2023-09-29T22:50:54.184 DEBUG:boto:Token: None 2023-09-29T22:50:54.184 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.184 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tj0WbTGsJYvOve8cJ+/PYtS6jTQ= 2023-09-29T22:50:54.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tj0WbTGsJYvOve8cJ+/PYtS6jTQ=', 'Content-Length': '0'} 2023-09-29T22:50:54.186 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080b6b9353599f7e3-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:50:54.187 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.187 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.187 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.187 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.187 DEBUG:boto:Method: GET 2023-09-29T22:50:54.187 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.187 DEBUG:boto:Data: 2023-09-29T22:50:54.187 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.187 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.188 DEBUG:boto:Params: {} 2023-09-29T22:50:54.188 DEBUG:boto:Token: None 2023-09-29T22:50:54.188 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k= 2023-09-29T22:50:54.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k=', 'Content-Length': '0'} 2023-09-29T22:50:54.190 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003bc0af255f5391ed-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.190 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.191 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.191 DEBUG:boto:path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.191 DEBUG:boto:auth_path=/hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.191 DEBUG:boto:Method: GET 2023-09-29T22:50:54.191 DEBUG:boto:Path: /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.191 DEBUG:boto:Data: 2023-09-29T22:50:54.191 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.191 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.192 DEBUG:boto:Params: {} 2023-09-29T22:50:54.192 DEBUG:boto:Token: None 2023-09-29T22:50:54.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76?versionId=null 2023-09-29T22:50:54.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k= 2023-09-29T22:50:54.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dm9xYW0NCDJVmP+mC4dgV+VfY7k=', 'Content-Length': '0'} 2023-09-29T22:50:54.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000041886e69c1f130cf-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.194 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.194 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.195 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.195 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.195 DEBUG:boto:Method: GET 2023-09-29T22:50:54.195 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.195 DEBUG:boto:Data: 2023-09-29T22:50:54.197 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.197 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.197 DEBUG:boto:Params: {} 2023-09-29T22:50:54.198 DEBUG:boto:Token: None 2023-09-29T22:50:54.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001e166efead48f339-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.242 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.243 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.243 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.243 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.243 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.243 DEBUG:boto:Method: GET 2023-09-29T22:50:54.244 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.244 DEBUG:boto:Data: 2023-09-29T22:50:54.244 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.244 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.244 DEBUG:boto:Params: {} 2023-09-29T22:50:54.244 DEBUG:boto:Token: None 2023-09-29T22:50:54.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.245 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.254 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000054521423600959d9-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.254 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.255 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.255 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.255 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.255 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:50:54.255 DEBUG:boto:Data: 2023-09-29T22:50:54.255 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.256 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.256 DEBUG:boto:Params: {} 2023-09-29T22:50:54.256 DEBUG:boto:Token: None 2023-09-29T22:50:54.256 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76 2023-09-29T22:50:54.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GSMa8oVosWMlYgS8ii/BNlssOVs= 2023-09-29T22:50:54.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GSMa8oVosWMlYgS8ii/BNlssOVs=', 'Content-Length': '0'} 2023-09-29T22:50:54.258 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000580a3feab1ad1bd5-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.258 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.258 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.258 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.259 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:50:54.259 DEBUG:boto:Data: 2023-09-29T22:50:54.259 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.259 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.259 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.259 DEBUG:boto:Params: {} 2023-09-29T22:50:54.259 DEBUG:boto:Token: None 2023-09-29T22:50:54.259 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76 2023-09-29T22:50:54.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GSMa8oVosWMlYgS8ii/BNlssOVs= 2023-09-29T22:50:54.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GSMa8oVosWMlYgS8ii/BNlssOVs=', 'Content-Length': '0'} 2023-09-29T22:50:54.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008729e75a03824433-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:50:54.262 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.262 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.262 DEBUG:boto:Method: GET 2023-09-29T22:50:54.262 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:50:54.262 DEBUG:boto:Data: 2023-09-29T22:50:54.262 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.263 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.263 DEBUG:boto:Params: {} 2023-09-29T22:50:54.263 DEBUG:boto:Token: None 2023-09-29T22:50:54.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76 2023-09-29T22:50:54.263 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3WOcWPE0VXXYglxGqjzm17KAIN8= 2023-09-29T22:50:54.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3WOcWPE0VXXYglxGqjzm17KAIN8=', 'Content-Length': '0'} 2023-09-29T22:50:54.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000160a09cc81ead2de-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.265 DEBUG:boto:path=/hcnphz-59/obj76 2023-09-29T22:50:54.265 DEBUG:boto:auth_path=/hcnphz-59/obj76 2023-09-29T22:50:54.265 DEBUG:boto:Method: GET 2023-09-29T22:50:54.266 DEBUG:boto:Path: /hcnphz-59/obj76 2023-09-29T22:50:54.266 DEBUG:boto:Data: 2023-09-29T22:50:54.266 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.266 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.266 DEBUG:boto:Params: {} 2023-09-29T22:50:54.266 DEBUG:boto:Token: None 2023-09-29T22:50:54.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj76 2023-09-29T22:50:54.266 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3WOcWPE0VXXYglxGqjzm17KAIN8= 2023-09-29T22:50:54.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3WOcWPE0VXXYglxGqjzm17KAIN8=', 'Content-Length': '0'} 2023-09-29T22:50:54.268 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f148c23ffa74fef-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.269 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.269 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.269 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.269 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.269 DEBUG:boto:Method: GET 2023-09-29T22:50:54.269 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.269 DEBUG:boto:Data: 2023-09-29T22:50:54.269 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.270 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.270 DEBUG:boto:Params: {} 2023-09-29T22:50:54.270 DEBUG:boto:Token: None 2023-09-29T22:50:54.270 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.273 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ef32580067997922-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.273 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.274 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.274 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.274 DEBUG:boto:path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.274 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.274 DEBUG:boto:Method: GET 2023-09-29T22:50:54.275 DEBUG:boto:Path: /hcnphz-59/?prefix=obj76 2023-09-29T22:50:54.275 DEBUG:boto:Data: 2023-09-29T22:50:54.275 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.275 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.275 DEBUG:boto:Params: {} 2023-09-29T22:50:54.275 DEBUG:boto:Token: None 2023-09-29T22:50:54.275 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.282 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a4eff22276465bd8-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.282 DEBUG:boto:b'hcnphz-59obj761000falseobj762023-09-29T22:38:48.927Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:50:54.283 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.283 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.283 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.283 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.283 DEBUG:boto:Method: GET 2023-09-29T22:50:54.283 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.283 DEBUG:boto:Data: 2023-09-29T22:50:54.283 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.283 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.283 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.284 DEBUG:boto:Params: {} 2023-09-29T22:50:54.284 DEBUG:boto:Token: None 2023-09-29T22:50:54.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs= 2023-09-29T22:50:54.284 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs=', 'Content-Length': '0'} 2023-09-29T22:50:54.286 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3629588ed076bdb-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.287 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.287 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.287 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.287 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.287 DEBUG:boto:Method: GET 2023-09-29T22:50:54.287 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.287 DEBUG:boto:Data: 2023-09-29T22:50:54.287 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.288 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.288 DEBUG:boto:Params: {} 2023-09-29T22:50:54.288 DEBUG:boto:Token: None 2023-09-29T22:50:54.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs= 2023-09-29T22:50:54.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs=', 'Content-Length': '0'} 2023-09-29T22:50:54.290 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2094c0398cf160-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.334 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.335 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.335 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.335 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.335 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.336 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.336 DEBUG:boto:Data: 2023-09-29T22:50:54.336 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.336 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.336 DEBUG:boto:Params: {} 2023-09-29T22:50:54.337 DEBUG:boto:Token: None 2023-09-29T22:50:54.337 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8BfYwHpuXT3vDHZ/ZQarlj0wp5k= 2023-09-29T22:50:54.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8BfYwHpuXT3vDHZ/ZQarlj0wp5k=', 'Content-Length': '0'} 2023-09-29T22:50:54.340 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000082b42e3ffbc4bef-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.340 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.341 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.341 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.341 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.341 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.341 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.342 DEBUG:boto:Data: 2023-09-29T22:50:54.342 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.342 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.342 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.342 DEBUG:boto:Params: {} 2023-09-29T22:50:54.343 DEBUG:boto:Token: None 2023-09-29T22:50:54.343 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.343 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8BfYwHpuXT3vDHZ/ZQarlj0wp5k= 2023-09-29T22:50:54.343 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8BfYwHpuXT3vDHZ/ZQarlj0wp5k=', 'Content-Length': '0'} 2023-09-29T22:50:54.346 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094a667ff9f54325f-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:50:54.346 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.346 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.347 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.347 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.347 DEBUG:boto:Method: GET 2023-09-29T22:50:54.347 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.347 DEBUG:boto:Data: 2023-09-29T22:50:54.348 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.348 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.348 DEBUG:boto:Params: {} 2023-09-29T22:50:54.348 DEBUG:boto:Token: None 2023-09-29T22:50:54.349 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs= 2023-09-29T22:50:54.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs=', 'Content-Length': '0'} 2023-09-29T22:50:54.352 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000720cdde28d44c833-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.352 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.352 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.352 DEBUG:boto:path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.353 DEBUG:boto:auth_path=/hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.353 DEBUG:boto:Method: GET 2023-09-29T22:50:54.353 DEBUG:boto:Path: /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.353 DEBUG:boto:Data: 2023-09-29T22:50:54.353 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.354 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.354 DEBUG:boto:Params: {} 2023-09-29T22:50:54.354 DEBUG:boto:Token: None 2023-09-29T22:50:54.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85?versionId=null 2023-09-29T22:50:54.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs= 2023-09-29T22:50:54.355 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zrfmjNYG4o+qXN2eC+vfSRNPXhs=', 'Content-Length': '0'} 2023-09-29T22:50:54.357 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007df51575d2d55f8d-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.358 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.358 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.358 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.359 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.359 DEBUG:boto:Method: GET 2023-09-29T22:50:54.359 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.359 DEBUG:boto:Data: 2023-09-29T22:50:54.359 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.360 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.360 DEBUG:boto:Params: {} 2023-09-29T22:50:54.360 DEBUG:boto:Token: None 2023-09-29T22:50:54.360 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e78de3e28aba111-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.406 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.407 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.407 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.407 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.408 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.408 DEBUG:boto:Method: GET 2023-09-29T22:50:54.408 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.408 DEBUG:boto:Data: 2023-09-29T22:50:54.408 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.409 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.409 DEBUG:boto:Params: {} 2023-09-29T22:50:54.409 DEBUG:boto:Token: None 2023-09-29T22:50:54.409 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.410 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.410 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.418 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000068e72e5f9f8ff9f3-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.418 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.418 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.419 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.419 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.419 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:50:54.419 DEBUG:boto:Data: 2023-09-29T22:50:54.419 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.419 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.419 DEBUG:boto:Params: {} 2023-09-29T22:50:54.420 DEBUG:boto:Token: None 2023-09-29T22:50:54.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85 2023-09-29T22:50:54.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rM6au+YtwHddHsLbTlmPjzIiBuw= 2023-09-29T22:50:54.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rM6au+YtwHddHsLbTlmPjzIiBuw=', 'Content-Length': '0'} 2023-09-29T22:50:54.422 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b52a94ed289d284-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.422 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.423 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.423 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.423 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:50:54.423 DEBUG:boto:Data: 2023-09-29T22:50:54.423 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.424 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.424 DEBUG:boto:Params: {} 2023-09-29T22:50:54.424 DEBUG:boto:Token: None 2023-09-29T22:50:54.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85 2023-09-29T22:50:54.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rM6au+YtwHddHsLbTlmPjzIiBuw= 2023-09-29T22:50:54.425 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rM6au+YtwHddHsLbTlmPjzIiBuw=', 'Content-Length': '0'} 2023-09-29T22:50:54.427 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6c2b7bc9851ee5a-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:50:54.428 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.428 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.428 DEBUG:boto:Method: GET 2023-09-29T22:50:54.428 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:50:54.429 DEBUG:boto:Data: 2023-09-29T22:50:54.429 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.429 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.429 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.429 DEBUG:boto:Params: {} 2023-09-29T22:50:54.430 DEBUG:boto:Token: None 2023-09-29T22:50:54.430 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85 2023-09-29T22:50:54.430 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HHWSqwlyZ4vdy8guCaCsDKc1lRU= 2023-09-29T22:50:54.430 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HHWSqwlyZ4vdy8guCaCsDKc1lRU=', 'Content-Length': '0'} 2023-09-29T22:50:54.432 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005182d6c303b93792-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.433 DEBUG:boto:path=/hcnphz-59/obj85 2023-09-29T22:50:54.433 DEBUG:boto:auth_path=/hcnphz-59/obj85 2023-09-29T22:50:54.433 DEBUG:boto:Method: GET 2023-09-29T22:50:54.433 DEBUG:boto:Path: /hcnphz-59/obj85 2023-09-29T22:50:54.433 DEBUG:boto:Data: 2023-09-29T22:50:54.433 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.434 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.434 DEBUG:boto:Params: {} 2023-09-29T22:50:54.434 DEBUG:boto:Token: None 2023-09-29T22:50:54.434 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj85 2023-09-29T22:50:54.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HHWSqwlyZ4vdy8guCaCsDKc1lRU= 2023-09-29T22:50:54.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HHWSqwlyZ4vdy8guCaCsDKc1lRU=', 'Content-Length': '0'} 2023-09-29T22:50:54.436 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:36:06 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae3a0cbb914e6099-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.478 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.479 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.479 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.479 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.479 DEBUG:boto:Method: GET 2023-09-29T22:50:54.479 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.479 DEBUG:boto:Data: 2023-09-29T22:50:54.479 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.479 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.480 DEBUG:boto:Params: {} 2023-09-29T22:50:54.480 DEBUG:boto:Token: None 2023-09-29T22:50:54.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.485 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b3cc43707f90d5a2-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.485 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.485 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.485 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.486 DEBUG:boto:path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.486 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.486 DEBUG:boto:Method: GET 2023-09-29T22:50:54.486 DEBUG:boto:Path: /hcnphz-59/?prefix=obj85 2023-09-29T22:50:54.486 DEBUG:boto:Data: 2023-09-29T22:50:54.486 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.486 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.486 DEBUG:boto:Params: {} 2023-09-29T22:50:54.487 DEBUG:boto:Token: None 2023-09-29T22:50:54.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.494 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ce0a4f3455862682-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.494 DEBUG:boto:b'hcnphz-59obj851000falseobj852023-09-29T22:36:06.023Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:50:54.495 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.495 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.495 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.495 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.495 DEBUG:boto:Method: GET 2023-09-29T22:50:54.495 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.495 DEBUG:boto:Data: 2023-09-29T22:50:54.496 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.496 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.496 DEBUG:boto:Params: {} 2023-09-29T22:50:54.496 DEBUG:boto:Token: None 2023-09-29T22:50:54.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA= 2023-09-29T22:50:54.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA=', 'Content-Length': '0'} 2023-09-29T22:50:54.498 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c44628caa330267-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.499 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.499 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.499 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.499 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.499 DEBUG:boto:Method: GET 2023-09-29T22:50:54.499 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.499 DEBUG:boto:Data: 2023-09-29T22:50:54.499 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.500 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.500 DEBUG:boto:Params: {} 2023-09-29T22:50:54.500 DEBUG:boto:Token: None 2023-09-29T22:50:54.500 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA= 2023-09-29T22:50:54.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA=', 'Content-Length': '0'} 2023-09-29T22:50:54.502 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a93627139faf6a1-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.503 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.503 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.503 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.503 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.503 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.503 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.504 DEBUG:boto:Data: 2023-09-29T22:50:54.504 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.504 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.504 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.504 DEBUG:boto:Params: {} 2023-09-29T22:50:54.504 DEBUG:boto:Token: None 2023-09-29T22:50:54.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O3Nx4yLw/mzCpustCaKwh8K+k7M= 2023-09-29T22:50:54.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O3Nx4yLw/mzCpustCaKwh8K+k7M=', 'Content-Length': '0'} 2023-09-29T22:50:54.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bea57248c55134d8-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.506 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.507 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.507 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.507 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.507 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.507 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.507 DEBUG:boto:Data: 2023-09-29T22:50:54.507 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.507 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.508 DEBUG:boto:Params: {} 2023-09-29T22:50:54.508 DEBUG:boto:Token: None 2023-09-29T22:50:54.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O3Nx4yLw/mzCpustCaKwh8K+k7M= 2023-09-29T22:50:54.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O3Nx4yLw/mzCpustCaKwh8K+k7M=', 'Content-Length': '0'} 2023-09-29T22:50:54.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000056b30f16b07c812d-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:50:54.511 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.511 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.511 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.511 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.511 DEBUG:boto:Method: GET 2023-09-29T22:50:54.511 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.511 DEBUG:boto:Data: 2023-09-29T22:50:54.511 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.512 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.512 DEBUG:boto:Params: {} 2023-09-29T22:50:54.512 DEBUG:boto:Token: None 2023-09-29T22:50:54.512 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA= 2023-09-29T22:50:54.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA=', 'Content-Length': '0'} 2023-09-29T22:50:54.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d0875eff8cdafde-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.562 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.563 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.564 DEBUG:boto:path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.564 DEBUG:boto:auth_path=/hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.564 DEBUG:boto:Method: GET 2023-09-29T22:50:54.564 DEBUG:boto:Path: /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.564 DEBUG:boto:Data: 2023-09-29T22:50:54.564 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.564 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.564 DEBUG:boto:Params: {} 2023-09-29T22:50:54.565 DEBUG:boto:Token: None 2023-09-29T22:50:54.565 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87?versionId=null 2023-09-29T22:50:54.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA= 2023-09-29T22:50:54.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MssNiNP6nLgl0P0rAtwErazAomA=', 'Content-Length': '0'} 2023-09-29T22:50:54.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e23821609d51c192-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.568 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.568 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.568 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.568 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.568 DEBUG:boto:Method: GET 2023-09-29T22:50:54.568 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.569 DEBUG:boto:Data: 2023-09-29T22:50:54.569 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.569 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.569 DEBUG:boto:Params: {} 2023-09-29T22:50:54.569 DEBUG:boto:Token: None 2023-09-29T22:50:54.569 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e06287cb47fb985-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.574 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.575 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.575 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.575 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.575 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.575 DEBUG:boto:Method: GET 2023-09-29T22:50:54.576 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.576 DEBUG:boto:Data: 2023-09-29T22:50:54.576 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.576 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.576 DEBUG:boto:Params: {} 2023-09-29T22:50:54.576 DEBUG:boto:Token: None 2023-09-29T22:50:54.577 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.577 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.577 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b1932690a147da9a-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.584 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.585 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.585 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.585 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.585 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:50:54.585 DEBUG:boto:Data: 2023-09-29T22:50:54.585 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.585 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.586 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.586 DEBUG:boto:Params: {} 2023-09-29T22:50:54.586 DEBUG:boto:Token: None 2023-09-29T22:50:54.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87 2023-09-29T22:50:54.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGxpdy7YnOD7F8s3Y8vzQuoAXYg= 2023-09-29T22:50:54.586 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGxpdy7YnOD7F8s3Y8vzQuoAXYg=', 'Content-Length': '0'} 2023-09-29T22:50:54.588 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a05ac09dcacef7b2-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.589 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.589 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.589 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.589 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:50:54.589 DEBUG:boto:Data: 2023-09-29T22:50:54.589 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.590 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.590 DEBUG:boto:Params: {} 2023-09-29T22:50:54.590 DEBUG:boto:Token: None 2023-09-29T22:50:54.590 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87 2023-09-29T22:50:54.590 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JGxpdy7YnOD7F8s3Y8vzQuoAXYg= 2023-09-29T22:50:54.590 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JGxpdy7YnOD7F8s3Y8vzQuoAXYg=', 'Content-Length': '0'} 2023-09-29T22:50:54.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091c0cf22cbbd2a2a-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:50:54.593 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.593 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.593 DEBUG:boto:Method: GET 2023-09-29T22:50:54.593 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:50:54.593 DEBUG:boto:Data: 2023-09-29T22:50:54.593 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.593 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.593 DEBUG:boto:Params: {} 2023-09-29T22:50:54.594 DEBUG:boto:Token: None 2023-09-29T22:50:54.594 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87 2023-09-29T22:50:54.594 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOXGmpsgPPOGW5MmDQd0i4hKH7w= 2023-09-29T22:50:54.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOXGmpsgPPOGW5MmDQd0i4hKH7w=', 'Content-Length': '0'} 2023-09-29T22:50:54.596 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000881120f798d48527-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.596 DEBUG:boto:path=/hcnphz-59/obj87 2023-09-29T22:50:54.596 DEBUG:boto:auth_path=/hcnphz-59/obj87 2023-09-29T22:50:54.596 DEBUG:boto:Method: GET 2023-09-29T22:50:54.597 DEBUG:boto:Path: /hcnphz-59/obj87 2023-09-29T22:50:54.597 DEBUG:boto:Data: 2023-09-29T22:50:54.597 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.597 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.597 DEBUG:boto:Params: {} 2023-09-29T22:50:54.597 DEBUG:boto:Token: None 2023-09-29T22:50:54.597 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj87 2023-09-29T22:50:54.598 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QOXGmpsgPPOGW5MmDQd0i4hKH7w= 2023-09-29T22:50:54.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QOXGmpsgPPOGW5MmDQd0i4hKH7w=', 'Content-Length': '0'} 2023-09-29T22:50:54.600 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:43:47 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c38739aab7b3168-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.600 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.600 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.600 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.600 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.601 DEBUG:boto:Method: GET 2023-09-29T22:50:54.601 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.601 DEBUG:boto:Data: 2023-09-29T22:50:54.601 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.601 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.601 DEBUG:boto:Params: {} 2023-09-29T22:50:54.601 DEBUG:boto:Token: None 2023-09-29T22:50:54.601 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.602 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.605 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046f1fe75572abc2f-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.605 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.606 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.606 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.606 DEBUG:boto:path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.606 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.606 DEBUG:boto:Method: GET 2023-09-29T22:50:54.606 DEBUG:boto:Path: /hcnphz-59/?prefix=obj87 2023-09-29T22:50:54.606 DEBUG:boto:Data: 2023-09-29T22:50:54.606 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.607 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.607 DEBUG:boto:Params: {} 2023-09-29T22:50:54.607 DEBUG:boto:Token: None 2023-09-29T22:50:54.607 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.607 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.615 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000e967099331b27d6-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.662 DEBUG:boto:b'hcnphz-59obj871000falseobj872023-09-29T22:43:47.242Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:50:54.663 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.663 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.663 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.663 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.664 DEBUG:boto:Method: GET 2023-09-29T22:50:54.664 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.664 DEBUG:boto:Data: 2023-09-29T22:50:54.664 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.664 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.664 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.664 DEBUG:boto:Params: {} 2023-09-29T22:50:54.664 DEBUG:boto:Token: None 2023-09-29T22:50:54.664 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4= 2023-09-29T22:50:54.665 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4=', 'Content-Length': '0'} 2023-09-29T22:50:54.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053c911cc9c3c2726-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.668 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.668 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.668 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.668 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.669 DEBUG:boto:Method: GET 2023-09-29T22:50:54.669 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.669 DEBUG:boto:Data: 2023-09-29T22:50:54.669 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.669 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.669 DEBUG:boto:Params: {} 2023-09-29T22:50:54.669 DEBUG:boto:Token: None 2023-09-29T22:50:54.669 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4= 2023-09-29T22:50:54.670 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4=', 'Content-Length': '0'} 2023-09-29T22:50:54.672 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf3878554be9ee42-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.673 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.673 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.673 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.673 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.673 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.673 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.673 DEBUG:boto:Data: 2023-09-29T22:50:54.674 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.674 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.674 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.674 DEBUG:boto:Params: {} 2023-09-29T22:50:54.674 DEBUG:boto:Token: None 2023-09-29T22:50:54.674 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pXj2QyrCSjTsv03KlM7lYp6GSFI= 2023-09-29T22:50:54.675 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXj2QyrCSjTsv03KlM7lYp6GSFI=', 'Content-Length': '0'} 2023-09-29T22:50:54.677 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f415f941ccea9f95-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.677 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.678 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.678 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.678 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.678 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.678 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.678 DEBUG:boto:Data: 2023-09-29T22:50:54.678 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.678 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.678 DEBUG:boto:Params: {} 2023-09-29T22:50:54.679 DEBUG:boto:Token: None 2023-09-29T22:50:54.679 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.679 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pXj2QyrCSjTsv03KlM7lYp6GSFI= 2023-09-29T22:50:54.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXj2QyrCSjTsv03KlM7lYp6GSFI=', 'Content-Length': '0'} 2023-09-29T22:50:54.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b214a15208f109-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:50:54.682 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.682 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.682 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.682 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.682 DEBUG:boto:Method: GET 2023-09-29T22:50:54.682 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.682 DEBUG:boto:Data: 2023-09-29T22:50:54.682 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.683 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.683 DEBUG:boto:Params: {} 2023-09-29T22:50:54.683 DEBUG:boto:Token: None 2023-09-29T22:50:54.683 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.683 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4= 2023-09-29T22:50:54.683 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4=', 'Content-Length': '0'} 2023-09-29T22:50:54.686 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f914eb4bb4823d2-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.726 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.726 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.727 DEBUG:boto:path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.727 DEBUG:boto:auth_path=/hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.727 DEBUG:boto:Method: GET 2023-09-29T22:50:54.727 DEBUG:boto:Path: /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.727 DEBUG:boto:Data: 2023-09-29T22:50:54.727 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.727 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.727 DEBUG:boto:Params: {} 2023-09-29T22:50:54.727 DEBUG:boto:Token: None 2023-09-29T22:50:54.728 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92?versionId=null 2023-09-29T22:50:54.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4= 2023-09-29T22:50:54.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WyzLhDFCSIuf/A+lQf66jvhGbu4=', 'Content-Length': '0'} 2023-09-29T22:50:54.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000198558a79ce9f464-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.732 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.732 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.732 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.732 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.732 DEBUG:boto:Method: GET 2023-09-29T22:50:54.732 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.733 DEBUG:boto:Data: 2023-09-29T22:50:54.733 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.733 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.733 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.733 DEBUG:boto:Params: {} 2023-09-29T22:50:54.733 DEBUG:boto:Token: None 2023-09-29T22:50:54.733 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.733 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.734 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.737 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000542992e99a3350eb-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.737 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.737 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.738 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.738 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.738 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.738 DEBUG:boto:Method: GET 2023-09-29T22:50:54.738 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.738 DEBUG:boto:Data: 2023-09-29T22:50:54.738 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.738 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.738 DEBUG:boto:Params: {} 2023-09-29T22:50:54.739 DEBUG:boto:Token: None 2023-09-29T22:50:54.739 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.745 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000212ca707f061c3fd-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.745 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.746 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.746 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.746 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.746 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:50:54.746 DEBUG:boto:Data: 2023-09-29T22:50:54.746 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.746 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.746 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.746 DEBUG:boto:Params: {} 2023-09-29T22:50:54.747 DEBUG:boto:Token: None 2023-09-29T22:50:54.747 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92 2023-09-29T22:50:54.747 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FgVJ4w3EbFYxy/xLY9butgpRPQY= 2023-09-29T22:50:54.747 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FgVJ4w3EbFYxy/xLY9butgpRPQY=', 'Content-Length': '0'} 2023-09-29T22:50:54.749 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ec6a74c6d0adf9b-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.749 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.749 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.749 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.749 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:50:54.749 DEBUG:boto:Data: 2023-09-29T22:50:54.749 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.750 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.750 DEBUG:boto:Params: {} 2023-09-29T22:50:54.750 DEBUG:boto:Token: None 2023-09-29T22:50:54.750 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92 2023-09-29T22:50:54.750 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FgVJ4w3EbFYxy/xLY9butgpRPQY= 2023-09-29T22:50:54.750 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FgVJ4w3EbFYxy/xLY9butgpRPQY=', 'Content-Length': '0'} 2023-09-29T22:50:54.752 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca3e57365ca97931-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:50:54.753 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.753 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.753 DEBUG:boto:Method: GET 2023-09-29T22:50:54.753 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:50:54.753 DEBUG:boto:Data: 2023-09-29T22:50:54.753 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.753 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.753 DEBUG:boto:Params: {} 2023-09-29T22:50:54.754 DEBUG:boto:Token: None 2023-09-29T22:50:54.754 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92 2023-09-29T22:50:54.754 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPBWu/gjTH9Bmq6/KGSgGGUg6QE= 2023-09-29T22:50:54.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPBWu/gjTH9Bmq6/KGSgGGUg6QE=', 'Content-Length': '0'} 2023-09-29T22:50:54.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003621d19ff40c3312-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.756 DEBUG:boto:path=/hcnphz-59/obj92 2023-09-29T22:50:54.756 DEBUG:boto:auth_path=/hcnphz-59/obj92 2023-09-29T22:50:54.756 DEBUG:boto:Method: GET 2023-09-29T22:50:54.756 DEBUG:boto:Path: /hcnphz-59/obj92 2023-09-29T22:50:54.757 DEBUG:boto:Data: 2023-09-29T22:50:54.757 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.757 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.757 DEBUG:boto:Params: {} 2023-09-29T22:50:54.757 DEBUG:boto:Token: None 2023-09-29T22:50:54.757 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj92 2023-09-29T22:50:54.757 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPBWu/gjTH9Bmq6/KGSgGGUg6QE= 2023-09-29T22:50:54.757 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPBWu/gjTH9Bmq6/KGSgGGUg6QE=', 'Content-Length': '0'} 2023-09-29T22:50:54.759 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:42:07 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000fa92d1018fa611-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.802 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.803 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.803 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.803 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.803 DEBUG:boto:Method: GET 2023-09-29T22:50:54.803 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.803 DEBUG:boto:Data: 2023-09-29T22:50:54.803 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.803 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.803 DEBUG:boto:Params: {} 2023-09-29T22:50:54.804 DEBUG:boto:Token: None 2023-09-29T22:50:54.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.809 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000091f24574a995126b-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.809 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.809 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.810 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.810 DEBUG:boto:path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.810 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.810 DEBUG:boto:Method: GET 2023-09-29T22:50:54.810 DEBUG:boto:Path: /hcnphz-59/?prefix=obj92 2023-09-29T22:50:54.810 DEBUG:boto:Data: 2023-09-29T22:50:54.810 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.810 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.811 DEBUG:boto:Params: {} 2023-09-29T22:50:54.811 DEBUG:boto:Token: None 2023-09-29T22:50:54.811 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.819 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006dfd1088ca9e9baa-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.819 DEBUG:boto:b'hcnphz-59obj921000falseobj922023-09-29T22:42:07.813Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:50:54.819 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.819 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.820 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.820 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.820 DEBUG:boto:Method: GET 2023-09-29T22:50:54.820 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.820 DEBUG:boto:Data: 2023-09-29T22:50:54.820 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.820 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.820 DEBUG:boto:Params: {} 2023-09-29T22:50:54.820 DEBUG:boto:Token: None 2023-09-29T22:50:54.821 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY= 2023-09-29T22:50:54.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY=', 'Content-Length': '0'} 2023-09-29T22:50:54.824 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000360e8efc9361453b-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.866 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.867 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.867 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.867 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.867 DEBUG:boto:Method: GET 2023-09-29T22:50:54.867 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.867 DEBUG:boto:Data: 2023-09-29T22:50:54.867 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.867 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.868 DEBUG:boto:Params: {} 2023-09-29T22:50:54.868 DEBUG:boto:Token: None 2023-09-29T22:50:54.868 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY= 2023-09-29T22:50:54.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY=', 'Content-Length': '0'} 2023-09-29T22:50:54.871 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b9796fd14155527f-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.871 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.871 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.871 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.871 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.872 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.872 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.872 DEBUG:boto:Data: 2023-09-29T22:50:54.872 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.872 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.872 DEBUG:boto:Params: {} 2023-09-29T22:50:54.872 DEBUG:boto:Token: None 2023-09-29T22:50:54.872 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.873 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MRTQBQls0qTWUERpfBKFD7yBhTg= 2023-09-29T22:50:54.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MRTQBQls0qTWUERpfBKFD7yBhTg=', 'Content-Length': '0'} 2023-09-29T22:50:54.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048e15f2be4da5812-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.875 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.876 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.876 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.876 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.876 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.876 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.876 DEBUG:boto:Data: 2023-09-29T22:50:54.876 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.876 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.876 DEBUG:boto:Params: {} 2023-09-29T22:50:54.877 DEBUG:boto:Token: None 2023-09-29T22:50:54.877 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MRTQBQls0qTWUERpfBKFD7yBhTg= 2023-09-29T22:50:54.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MRTQBQls0qTWUERpfBKFD7yBhTg=', 'Content-Length': '0'} 2023-09-29T22:50:54.879 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037ec0889f299d28c-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:50:54.880 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.880 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.880 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.880 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.880 DEBUG:boto:Method: GET 2023-09-29T22:50:54.880 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.880 DEBUG:boto:Data: 2023-09-29T22:50:54.880 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.881 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.881 DEBUG:boto:Params: {} 2023-09-29T22:50:54.881 DEBUG:boto:Token: None 2023-09-29T22:50:54.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY= 2023-09-29T22:50:54.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY=', 'Content-Length': '0'} 2023-09-29T22:50:54.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002faf26ba6973559e-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.884 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.884 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.884 DEBUG:boto:path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.884 DEBUG:boto:auth_path=/hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.885 DEBUG:boto:Method: GET 2023-09-29T22:50:54.885 DEBUG:boto:Path: /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.885 DEBUG:boto:Data: 2023-09-29T22:50:54.885 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.885 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.885 DEBUG:boto:Params: {} 2023-09-29T22:50:54.885 DEBUG:boto:Token: None 2023-09-29T22:50:54.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93?versionId=null 2023-09-29T22:50:54.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY= 2023-09-29T22:50:54.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XL+Y2QQxASKfaZ/CbCCZMNornkY=', 'Content-Length': '0'} 2023-09-29T22:50:54.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbfa2ab5da2e489d-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.930 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.931 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.931 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:54.931 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:54.931 DEBUG:boto:Method: GET 2023-09-29T22:50:54.931 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:50:54.931 DEBUG:boto:Data: 2023-09-29T22:50:54.931 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.931 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.932 DEBUG:boto:Params: {} 2023-09-29T22:50:54.932 DEBUG:boto:Token: None 2023-09-29T22:50:54.932 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.937 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee1eb8fc0d8e333a-00651754ce-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.938 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.938 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:54.938 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:54.938 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:54.938 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:54.939 DEBUG:boto:Method: GET 2023-09-29T22:50:54.939 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:50:54.939 DEBUG:boto:Data: 2023-09-29T22:50:54.939 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.939 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.939 DEBUG:boto:Params: {} 2023-09-29T22:50:54.939 DEBUG:boto:Token: None 2023-09-29T22:50:54.939 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/ 2023-09-29T22:50:54.940 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY= 2023-09-29T22:50:54.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SqIF1ulutJruDP2FfTqBiy9mQCY=', 'Content-Length': '0'} 2023-09-29T22:50:54.947 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a39ca2bb728fb1b6-00651754ce-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.947 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:54.948 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.948 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.948 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.948 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:50:54.948 DEBUG:boto:Data: 2023-09-29T22:50:54.948 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.949 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.949 DEBUG:boto:Params: {} 2023-09-29T22:50:54.949 DEBUG:boto:Token: None 2023-09-29T22:50:54.949 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93 2023-09-29T22:50:54.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DNrfAqdgaJ1TgAK7j+SQ6lzbAgQ= 2023-09-29T22:50:54.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DNrfAqdgaJ1TgAK7j+SQ6lzbAgQ=', 'Content-Length': '0'} 2023-09-29T22:50:54.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fff7e0e6d5c11077-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.952 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.952 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.952 DEBUG:boto:Method: HEAD 2023-09-29T22:50:54.952 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:50:54.952 DEBUG:boto:Data: 2023-09-29T22:50:54.952 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:54.953 DEBUG:boto:Port: 8000 2023-09-29T22:50:54.953 DEBUG:boto:Params: {} 2023-09-29T22:50:54.953 DEBUG:boto:Token: None 2023-09-29T22:50:54.953 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93 2023-09-29T22:50:54.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DNrfAqdgaJ1TgAK7j+SQ6lzbAgQ= 2023-09-29T22:50:54.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DNrfAqdgaJ1TgAK7j+SQ6lzbAgQ=', 'Content-Length': '0'} 2023-09-29T22:50:54.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088588852375bb519-00651754ce-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:54.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:50:54.956 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:54.956 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:54.957 DEBUG:boto:Method: GET 2023-09-29T22:50:54.957 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:50:54.957 DEBUG:boto:Data: 2023-09-29T22:50:54.957 DEBUG:boto:Headers: {} 2023-09-29T22:50:54.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:54.957 DEBUG:boto:Port: 8001 2023-09-29T22:50:54.957 DEBUG:boto:Params: {} 2023-09-29T22:50:54.957 DEBUG:boto:Token: None 2023-09-29T22:50:54.957 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:54 GMT /hcnphz-59/obj93 2023-09-29T22:50:54.958 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ke1yWXEhhTPEGt7AEeBK4kvhkSo= 2023-09-29T22:50:54.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ke1yWXEhhTPEGt7AEeBK4kvhkSo=', 'Content-Length': '0'} 2023-09-29T22:50:54.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8db99ab2f1fa6d2-00651754ce-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.002 DEBUG:boto:path=/hcnphz-59/obj93 2023-09-29T22:50:55.003 DEBUG:boto:auth_path=/hcnphz-59/obj93 2023-09-29T22:50:55.003 DEBUG:boto:Method: GET 2023-09-29T22:50:55.003 DEBUG:boto:Path: /hcnphz-59/obj93 2023-09-29T22:50:55.003 DEBUG:boto:Data: 2023-09-29T22:50:55.003 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.003 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.003 DEBUG:boto:Params: {} 2023-09-29T22:50:55.004 DEBUG:boto:Token: None 2023-09-29T22:50:55.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj93 2023-09-29T22:50:55.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XJyf4A9SzJ6XDba/pcKf09uwtic= 2023-09-29T22:50:55.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XJyf4A9SzJ6XDba/pcKf09uwtic=', 'Content-Length': '0'} 2023-09-29T22:50:55.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:46:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab51f90178207bda-00651754cf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.007 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:55.007 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:55.008 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:55.008 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:55.008 DEBUG:boto:Method: GET 2023-09-29T22:50:55.008 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:50:55.008 DEBUG:boto:Data: 2023-09-29T22:50:55.008 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.008 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.008 DEBUG:boto:Params: {} 2023-09-29T22:50:55.009 DEBUG:boto:Token: None 2023-09-29T22:50:55.009 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/ 2023-09-29T22:50:55.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk= 2023-09-29T22:50:55.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk=', 'Content-Length': '0'} 2023-09-29T22:50:55.013 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007f72c2ce10ddc62a-00651754cf-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.014 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:55.014 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:55.014 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:55.014 DEBUG:boto:path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:55.014 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj93 2023-09-29T22:50:55.015 DEBUG:boto:Method: GET 2023-09-29T22:50:55.015 DEBUG:boto:Path: /hcnphz-59/?prefix=obj93 2023-09-29T22:50:55.015 DEBUG:boto:Data: 2023-09-29T22:50:55.015 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.015 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.015 DEBUG:boto:Params: {} 2023-09-29T22:50:55.015 DEBUG:boto:Token: None 2023-09-29T22:50:55.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/ 2023-09-29T22:50:55.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk= 2023-09-29T22:50:55.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk=', 'Content-Length': '0'} 2023-09-29T22:50:55.023 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000886e6de3f229c764-00651754cf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.066 DEBUG:boto:b'hcnphz-59obj931000falseobj932023-09-29T22:46:48.031Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:55.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:50:55.067 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.067 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.067 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.067 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.068 DEBUG:boto:Method: GET 2023-09-29T22:50:55.068 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.068 DEBUG:boto:Data: 2023-09-29T22:50:55.068 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.068 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.068 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.068 DEBUG:boto:Params: {} 2023-09-29T22:50:55.069 DEBUG:boto:Token: None 2023-09-29T22:50:55.069 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.069 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w= 2023-09-29T22:50:55.069 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w=', 'Content-Length': '0'} 2023-09-29T22:50:55.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c6ded1fb01df8008-00651754cf-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.072 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.072 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.072 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.073 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.073 DEBUG:boto:Method: GET 2023-09-29T22:50:55.073 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.073 DEBUG:boto:Data: 2023-09-29T22:50:55.073 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.073 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.073 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.073 DEBUG:boto:Params: {} 2023-09-29T22:50:55.073 DEBUG:boto:Token: None 2023-09-29T22:50:55.074 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.074 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w= 2023-09-29T22:50:55.074 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w=', 'Content-Length': '0'} 2023-09-29T22:50:55.077 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4edefcc6e80b392-00651754cf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.077 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.078 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.078 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.078 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.078 DEBUG:boto:Method: HEAD 2023-09-29T22:50:55.078 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.078 DEBUG:boto:Data: 2023-09-29T22:50:55.078 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.078 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.079 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.079 DEBUG:boto:Params: {} 2023-09-29T22:50:55.079 DEBUG:boto:Token: None 2023-09-29T22:50:55.079 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.079 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BP6u7lpLP2S7adBlwwMfHCeFR6Q= 2023-09-29T22:50:55.079 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BP6u7lpLP2S7adBlwwMfHCeFR6Q=', 'Content-Length': '0'} 2023-09-29T22:50:55.082 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bcd73acab95552d-00651754cf-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.082 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.082 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.083 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.083 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.083 DEBUG:boto:Method: HEAD 2023-09-29T22:50:55.083 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.083 DEBUG:boto:Data: 2023-09-29T22:50:55.083 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.083 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.083 DEBUG:boto:Params: {} 2023-09-29T22:50:55.084 DEBUG:boto:Token: None 2023-09-29T22:50:55.084 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BP6u7lpLP2S7adBlwwMfHCeFR6Q= 2023-09-29T22:50:55.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BP6u7lpLP2S7adBlwwMfHCeFR6Q=', 'Content-Length': '0'} 2023-09-29T22:50:55.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000540a00f5e96385e8-00651754cf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:50:55.087 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.087 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.087 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.087 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.087 DEBUG:boto:Method: GET 2023-09-29T22:50:55.088 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.088 DEBUG:boto:Data: 2023-09-29T22:50:55.088 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.088 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.088 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.088 DEBUG:boto:Params: {} 2023-09-29T22:50:55.088 DEBUG:boto:Token: None 2023-09-29T22:50:55.088 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.088 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w= 2023-09-29T22:50:55.089 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w=', 'Content-Length': '0'} 2023-09-29T22:50:55.091 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078e51fb2b17586fe-00651754cf-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.091 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.092 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.092 DEBUG:boto:path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.092 DEBUG:boto:auth_path=/hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.092 DEBUG:boto:Method: GET 2023-09-29T22:50:55.092 DEBUG:boto:Path: /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.092 DEBUG:boto:Data: 2023-09-29T22:50:55.092 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.092 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.093 DEBUG:boto:Params: {} 2023-09-29T22:50:55.093 DEBUG:boto:Token: None 2023-09-29T22:50:55.093 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95?versionId=null 2023-09-29T22:50:55.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w= 2023-09-29T22:50:55.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IvMv4mFle00B1QOPoi+YMv0jy7w=', 'Content-Length': '0'} 2023-09-29T22:50:55.096 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5da5ad44d9ccd3e-00651754cf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.096 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:55.096 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:55.097 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.097 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.097 DEBUG:boto:Method: GET 2023-09-29T22:50:55.097 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.097 DEBUG:boto:Data: 2023-09-29T22:50:55.097 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.097 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.097 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.097 DEBUG:boto:Params: {} 2023-09-29T22:50:55.098 DEBUG:boto:Token: None 2023-09-29T22:50:55.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/ 2023-09-29T22:50:55.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk= 2023-09-29T22:50:55.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk=', 'Content-Length': '0'} 2023-09-29T22:50:55.102 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000036a36fcded7576e3-00651754cf-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.142 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:55.143 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:55.143 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:55.143 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.143 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.143 DEBUG:boto:Method: GET 2023-09-29T22:50:55.143 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.143 DEBUG:boto:Data: 2023-09-29T22:50:55.144 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.144 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.144 DEBUG:boto:Params: {} 2023-09-29T22:50:55.144 DEBUG:boto:Token: None 2023-09-29T22:50:55.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/ 2023-09-29T22:50:55.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk= 2023-09-29T22:50:55.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk=', 'Content-Length': '0'} 2023-09-29T22:50:55.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000094eb8bd614a75227-00651754cf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.153 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:55.153 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.153 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.153 DEBUG:boto:Method: HEAD 2023-09-29T22:50:55.153 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:50:55.153 DEBUG:boto:Data: 2023-09-29T22:50:55.154 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.154 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.154 DEBUG:boto:Params: {} 2023-09-29T22:50:55.154 DEBUG:boto:Token: None 2023-09-29T22:50:55.154 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95 2023-09-29T22:50:55.154 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XZWZFVOca5lFRkkhLSc7RmZKBFQ= 2023-09-29T22:50:55.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XZWZFVOca5lFRkkhLSc7RmZKBFQ=', 'Content-Length': '0'} 2023-09-29T22:50:55.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003eabe46a139349de-00651754cf-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.157 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.157 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.157 DEBUG:boto:Method: HEAD 2023-09-29T22:50:55.157 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:50:55.158 DEBUG:boto:Data: 2023-09-29T22:50:55.158 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.158 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.158 DEBUG:boto:Params: {} 2023-09-29T22:50:55.158 DEBUG:boto:Token: None 2023-09-29T22:50:55.158 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95 2023-09-29T22:50:55.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XZWZFVOca5lFRkkhLSc7RmZKBFQ= 2023-09-29T22:50:55.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XZWZFVOca5lFRkkhLSc7RmZKBFQ=', 'Content-Length': '0'} 2023-09-29T22:50:55.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035a39b9239e7aa2a-00651754cf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:50:55.161 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.161 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.161 DEBUG:boto:Method: GET 2023-09-29T22:50:55.161 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:50:55.161 DEBUG:boto:Data: 2023-09-29T22:50:55.161 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.162 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.162 DEBUG:boto:Params: {} 2023-09-29T22:50:55.162 DEBUG:boto:Token: None 2023-09-29T22:50:55.162 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95 2023-09-29T22:50:55.162 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JOFQofCd3gWayLqqwYDXeufe6zI= 2023-09-29T22:50:55.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JOFQofCd3gWayLqqwYDXeufe6zI=', 'Content-Length': '0'} 2023-09-29T22:50:55.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-59:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.24'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d603dad7561cf4ce-00651754cf-1da5-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.165 DEBUG:boto:path=/hcnphz-59/obj95 2023-09-29T22:50:55.165 DEBUG:boto:auth_path=/hcnphz-59/obj95 2023-09-29T22:50:55.165 DEBUG:boto:Method: GET 2023-09-29T22:50:55.165 DEBUG:boto:Path: /hcnphz-59/obj95 2023-09-29T22:50:55.165 DEBUG:boto:Data: 2023-09-29T22:50:55.165 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.165 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.165 DEBUG:boto:Params: {} 2023-09-29T22:50:55.166 DEBUG:boto:Token: None 2023-09-29T22:50:55.166 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/obj95 2023-09-29T22:50:55.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JOFQofCd3gWayLqqwYDXeufe6zI= 2023-09-29T22:50:55.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JOFQofCd3gWayLqqwYDXeufe6zI=', 'Content-Length': '0'} 2023-09-29T22:50:55.168 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:38:48 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000085a01b708e54d4a0-00651754cf-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.210 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:55.211 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:55.211 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.211 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.211 DEBUG:boto:Method: GET 2023-09-29T22:50:55.211 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.211 DEBUG:boto:Data: 2023-09-29T22:50:55.211 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:50:55.211 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.212 DEBUG:boto:Params: {} 2023-09-29T22:50:55.212 DEBUG:boto:Token: None 2023-09-29T22:50:55.212 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/ 2023-09-29T22:50:55.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk= 2023-09-29T22:50:55.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk=', 'Content-Length': '0'} 2023-09-29T22:50:55.217 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004379dbe1dcbf3f7d-00651754cf-1da5-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.217 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:55.218 DEBUG:boto:path=/hcnphz-59/ 2023-09-29T22:50:55.218 DEBUG:boto:auth_path=/hcnphz-59/ 2023-09-29T22:50:55.218 DEBUG:boto:path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.218 DEBUG:boto:auth_path=/hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.218 DEBUG:boto:Method: GET 2023-09-29T22:50:55.218 DEBUG:boto:Path: /hcnphz-59/?prefix=obj95 2023-09-29T22:50:55.218 DEBUG:boto:Data: 2023-09-29T22:50:55.218 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:50:55.219 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.219 DEBUG:boto:Params: {} 2023-09-29T22:50:55.219 DEBUG:boto:Token: None 2023-09-29T22:50:55.219 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:50:55 GMT /hcnphz-59/ 2023-09-29T22:50:55.219 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk= 2023-09-29T22:50:55.219 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:50:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PCvKP1lSBJwvWUxN7vGxdup27yk=', 'Content-Length': '0'} 2023-09-29T22:50:55.226 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b03d9a4077c05a1c-00651754cf-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.227 DEBUG:boto:b'hcnphz-59obj951000falseobj952023-09-29T22:38:48.945Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:50:55.227 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-59 zones={a2, a1} 2023-09-29T22:50:55.228 INFO:tasks.rgw_multisite_tests:Create several generations of objects, trash them, then run bucket sync init ... ok 2023-09-29T22:50:55.232 INFO:rgw_multi.tests:create role zone=a1 name=roles-hcnphz-1 2023-09-29T22:50:55.232 DEBUG:boto:Method: POST 2023-09-29T22:50:55.232 DEBUG:boto:Path: / 2023-09-29T22:50:55.232 DEBUG:boto:Data: 2023-09-29T22:50:55.232 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-09-29T22:50:55.232 DEBUG:boto:Port: 8000 2023-09-29T22:50:55.232 DEBUG:boto:Params: {'RoleName': 'roles-hcnphz-1', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-29T22:50:55.233 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:50:55.233 DEBUG:boto:Token: None 2023-09-29T22:50:55.233 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8000 x-amz-date:20230929T225055Z host;x-amz-date 5d28dec600224e48a9bee0c246f687a63940005396b07a62d8356a449c233d25 2023-09-29T22:50:55.233 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T225055Z 20230929/front/smithi112/aws4_request 60fc2ee79b8d268fe7d631f97e14e2fe6b572aef96a207f89e246f2c16c2762f 2023-09-29T22:50:55.233 DEBUG:boto:Signature: 1afb9270450fa5d658b553291e96d5ebf1fcd0d84638c2b67c7ea89043bf03d4 2023-09-29T22:50:55.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T225055Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=P5GVQP2RXA6372D10DSL/20230929/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=1afb9270450fa5d658b553291e96d5ebf1fcd0d84638c2b67c7ea89043bf03d4', 'Host': 'smithi112.front.sepia.ceph.com:8000'} 2023-09-29T22:50:55.253 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000068ed98bde9d914cb-00651754cf-1648-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.253 DEBUG:boto:b'9151dd95-5c2e-42e6-8be7-4f33ca6cc94froles-hcnphz-1/arn:aws:iam:::role/roles-hcnphz-12023-09-29T22:50:55.240Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx0000068ed98bde9d914cb-00651754cf-1648-a1' 2023-09-29T22:50:55.254 INFO:rgw_multi.tests:create role zone=a2 name=roles-hcnphz-2 2023-09-29T22:50:55.254 DEBUG:boto:Method: POST 2023-09-29T22:50:55.254 DEBUG:boto:Path: / 2023-09-29T22:50:55.254 DEBUG:boto:Data: 2023-09-29T22:50:55.254 DEBUG:boto:Headers: {} 2023-09-29T22:50:55.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-09-29T22:50:55.255 DEBUG:boto:Port: 8001 2023-09-29T22:50:55.255 DEBUG:boto:Params: {'RoleName': 'roles-hcnphz-2', 'AssumeRolePolicyDocument': '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}', 'Path': '', 'Action': 'CreateRole', 'Version': '2010-05-08'} 2023-09-29T22:50:55.255 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:50:55.255 DEBUG:boto:Token: None 2023-09-29T22:50:55.255 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8001 x-amz-date:20230929T225055Z host;x-amz-date 7d036aa4d1c2a0a4bb86e234625bef6796a7ac6d3b743a292c602c81cfe45b7d 2023-09-29T22:50:55.255 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T225055Z 20230929/front/smithi112/aws4_request 28005935658dae13104d9abcd26341d9fbd1008050884200521af2798e19ee6b 2023-09-29T22:50:55.256 DEBUG:boto:Signature: 7f5b8e65bb2c5cf1602296cb28fa044cec8aeabf19051c64a0421454c782f1b4 2023-09-29T22:50:55.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T225055Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '325', 'Authorization': 'AWS4-HMAC-SHA256 Credential=P5GVQP2RXA6372D10DSL/20230929/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=7f5b8e65bb2c5cf1602296cb28fa044cec8aeabf19051c64a0421454c782f1b4', 'Host': 'smithi112.front.sepia.ceph.com:8001'} 2023-09-29T22:50:55.292 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000c7b6654e9093d188-00651754cf-1da5-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '723'), ('Date', 'Fri, 29 Sep 2023 22:50:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:50:55.293 DEBUG:boto:b'0ad95f98-9899-4e17-ad62-eca46c521b91roles-hcnphz-2/arn:aws:iam:::role/roles-hcnphz-22023-09-29T22:50:55.284Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}tx00000c7b6654e9093d188-00651754cf-1da5-a2' 2023-09-29T22:50:55.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:55.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:55.294 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:55.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:55.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:55.394 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:50:55.421 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:50:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:50:55.421 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:48:09.223323Z" 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:50:55.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:50:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:50:55.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:50:55.423 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:50:55.424 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:50:55.424 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:50:55.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:55.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:50:55.425 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:50:55.510 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:50:55.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:50:55.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:55.562+0000 7fe47096af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:55.566 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:50:55.562+0000 7fe47096af40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:50:55.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:50:55.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:50:55.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:50:55.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:50:55.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:50:55.598 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:50:55Z" 2023-09-29T22:50:55.599 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:50:55.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:50:55Z" } 2023-09-29T22:50:55.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:50:55.601 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:50:55.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:50:55.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696024631.846323_86.1 2023-09-29T22:51:00.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:00.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:00.602 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:00.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:00.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:00.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:00.702+0000 7f91aa33bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:00.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:00.706+0000 7f91aa33bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:00.731 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:00.731 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:00.732 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:00.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:00.734 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:00.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:00.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:00.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:00.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:00.739 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:00Z" 2023-09-29T22:51:00.739 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:00.740 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:00Z" } 2023-09-29T22:51:00.740 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:00.741 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:00.741 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:00.741 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696024631.846323_86.1 2023-09-29T22:51:05.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:05.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:05.742 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:05.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:05.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:05.842+0000 7f1319781f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:05.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:05.846+0000 7f1319781f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:05.874 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:05.874 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:05.875 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:05.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:05.877 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:05.878 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:05.879 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:05.880 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:05.881 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:05Z" 2023-09-29T22:51:05.881 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:05.882 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:05Z" } 2023-09-29T22:51:05.882 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:05.882 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:05.882 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:05.882 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696024631.846323_86.1 2023-09-29T22:51:10.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:10.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:10.884 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:10.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:10.987 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:10.986+0000 7fc027fc1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:10.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:10.986+0000 7fc027fc1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:11.016 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:11.017 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:11.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:11.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:11.020 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:11.021 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:11.022 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:10Z" 2023-09-29T22:51:11.023 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:11.024 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:10Z" } 2023-09-29T22:51:11.024 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:11.025 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:11.025 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:11.025 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696024631.846323_86.1 2023-09-29T22:51:16.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:16.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:16.026 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:16.071 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:16.071 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:16.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:16.126+0000 7f4fbe306f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:16.131 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:16.130+0000 7f4fbe306f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:16.164 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:16.165 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:51:16.166 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:16.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:16.168 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:16.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:16.170 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:16Z" 2023-09-29T22:51:16.171 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:16.172 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:16Z" } 2023-09-29T22:51:16.173 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:16.173 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:16.173 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:16.173 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696024631.846323_86.1 2023-09-29T22:51:21.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:21.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:21.175 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:21.223 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:21.224 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:21.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:21.274+0000 7f7672804f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:21.278 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:21.274+0000 7f7672804f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:21.306 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696024631.846323_86.1", 2023-09-29T22:51:21.307 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:21.308 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:21.309 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:21.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:21.311 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:21.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:21Z" 2023-09-29T22:51:21.313 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:21.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696024631.846323_86.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T21:57:11.846323Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:21Z" } 2023-09-29T22:51:21.314 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:21.314 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:21.314 DEBUG:rgw_multi.tests:sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:21.314 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696024631.846323_86.1 2023-09-29T22:51:26.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:26.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:26.315 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:26.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:26.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:26.412 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:26.409+0000 7fb958010f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:26.414 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:26.409+0000 7fb958010f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:26.440 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.240770_225.1", 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:26.441 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.240770Z", 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:26.442 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:26.443 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:26.444 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:26.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:26.446 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:26.446 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:26.446 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:26.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:26.446 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:26Z" 2023-09-29T22:51:26.446 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:26.447 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.240770_225.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.240770Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:26Z" } 2023-09-29T22:51:26.447 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.240770_225.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.240770Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:26.447 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:26.447 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.240770_225.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:26.448 WARNING:rgw_multi.tests:zone a2 behind master: shard=0 master=1_1696027855.273178_228.1 target=1_1696027855.240770_225.1 2023-09-29T22:51:31.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:31.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:31.449 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:31.496 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:31.496 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:31.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:31.545+0000 7f0dc96d1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:31.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:31.549+0000 7f0dc96d1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:31.577 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.578 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T22:51:31.579 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027689.223323_538.1", 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.580 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.223323Z", 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.581 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:31.583 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:31.584 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.585 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:31Z" 2023-09-29T22:51:31.585 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:31.586 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027689.223323_538.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.223323Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:31Z" } 2023-09-29T22:51:31.586 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027689.223323_538.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.223323Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:31.586 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:31.586 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696027689.223323_538.1', 2: '', 3: ''} 2023-09-29T22:51:31.586 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:51:31.587 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-hcnphz-1 zones={a2, a1} 2023-09-29T22:51:31.587 DEBUG:boto:Method: POST 2023-09-29T22:51:31.587 DEBUG:boto:Path: / 2023-09-29T22:51:31.587 DEBUG:boto:Data: 2023-09-29T22:51:31.587 DEBUG:boto:Headers: {} 2023-09-29T22:51:31.587 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-09-29T22:51:31.587 DEBUG:boto:Port: 8001 2023-09-29T22:51:31.587 DEBUG:boto:Params: {'RoleName': 'roles-hcnphz-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T22:51:31.588 DEBUG:boto:Token: None 2023-09-29T22:51:31.588 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8001 x-amz-date:20230929T225131Z host;x-amz-date fe30d89552f1e62cd97743d6cbfebcd34b54585ec968f4e4e39b5212cc9cc762 2023-09-29T22:51:31.588 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T225131Z 20230929/front/smithi112/aws4_request 3e853526b18aa588441497be225bf1abbc12efef3f92a01fed89a68cecfe7863 2023-09-29T22:51:31.588 DEBUG:boto:Signature: 9e07abb193acb999cab65a742e95bebabd69f701a3b31b8dd75ac40b48b35075 2023-09-29T22:51:31.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T225131Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=P5GVQP2RXA6372D10DSL/20230929/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=9e07abb193acb999cab65a742e95bebabd69f701a3b31b8dd75ac40b48b35075', 'Host': 'smithi112.front.sepia.ceph.com:8001'} 2023-09-29T22:51:31.592 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000d957476bfcca58a4-00651754f3-1da5-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 29 Sep 2023 22:51:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:31.592 DEBUG:boto:b'tx00000d957476bfcca58a4-00651754f3-1da5-a29151dd95-5c2e-42e6-8be7-4f33ca6cc94froles-hcnphz-1/arn:aws:iam:::role/roles-hcnphz-12023-09-29T22:50:55.240Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T22:51:31.593 DEBUG:boto:Method: POST 2023-09-29T22:51:31.593 DEBUG:boto:Path: / 2023-09-29T22:51:31.593 DEBUG:boto:Data: 2023-09-29T22:51:31.593 DEBUG:boto:Headers: {} 2023-09-29T22:51:31.593 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-09-29T22:51:31.593 DEBUG:boto:Port: 8000 2023-09-29T22:51:31.593 DEBUG:boto:Params: {'RoleName': 'roles-hcnphz-1', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T22:51:31.594 DEBUG:boto:Token: None 2023-09-29T22:51:31.594 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8000 x-amz-date:20230929T225131Z host;x-amz-date fe30d89552f1e62cd97743d6cbfebcd34b54585ec968f4e4e39b5212cc9cc762 2023-09-29T22:51:31.594 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T225131Z 20230929/front/smithi112/aws4_request 4790235ca637acd9d8692953ee103411ffdac61a69734aea4532ab78f20e4058 2023-09-29T22:51:31.594 DEBUG:boto:Signature: 05c510e5a6d80fd31465e897b71d68b59451cb57dd2928032bc89d774fe85837 2023-09-29T22:51:31.594 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T225131Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=P5GVQP2RXA6372D10DSL/20230929/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=05c510e5a6d80fd31465e897b71d68b59451cb57dd2928032bc89d774fe85837', 'Host': 'smithi112.front.sepia.ceph.com:8000'} 2023-09-29T22:51:31.596 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000025588fad9d1ffa41-00651754f3-1648-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 29 Sep 2023 22:51:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:31.596 DEBUG:boto:b'tx0000025588fad9d1ffa41-00651754f3-1648-a19151dd95-5c2e-42e6-8be7-4f33ca6cc94froles-hcnphz-1/arn:aws:iam:::role/roles-hcnphz-12023-09-29T22:50:55.240Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T22:51:31.596 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-hcnphz-1 2023-09-29T22:51:31.596 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-hcnphz-1 zones={a2, a1} 2023-09-29T22:51:31.597 INFO:tasks.rgw_multi.zone_rados:comparing role=roles-hcnphz-2 zones={a1, a2} 2023-09-29T22:51:31.597 DEBUG:boto:Method: POST 2023-09-29T22:51:31.597 DEBUG:boto:Path: / 2023-09-29T22:51:31.597 DEBUG:boto:Data: 2023-09-29T22:51:31.597 DEBUG:boto:Headers: {} 2023-09-29T22:51:31.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-09-29T22:51:31.597 DEBUG:boto:Port: 8000 2023-09-29T22:51:31.597 DEBUG:boto:Params: {'RoleName': 'roles-hcnphz-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T22:51:31.597 DEBUG:boto:Token: None 2023-09-29T22:51:31.598 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8000 x-amz-date:20230929T225131Z host;x-amz-date 0ca5cc7558f50d44d78a3d5fdf09e95d0b90095c720575e32528c141f7217ac3 2023-09-29T22:51:31.598 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T225131Z 20230929/front/smithi112/aws4_request a4044141323838413f1eadbd67b8c9e8bf81ab373e2de370a5cd1a43a393ac76 2023-09-29T22:51:31.598 DEBUG:boto:Signature: a18d46ba412075433f1be70398df165608e122383cd889ef0fd5432342be83bd 2023-09-29T22:51:31.598 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T225131Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=P5GVQP2RXA6372D10DSL/20230929/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=a18d46ba412075433f1be70398df165608e122383cd889ef0fd5432342be83bd', 'Host': 'smithi112.front.sepia.ceph.com:8000'} 2023-09-29T22:51:31.599 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fced19de70ca2b63-00651754f3-1648-a1'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 29 Sep 2023 22:51:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:31.600 DEBUG:boto:b'tx00000fced19de70ca2b63-00651754f3-1648-a10ad95f98-9899-4e17-ad62-eca46c521b91roles-hcnphz-2/arn:aws:iam:::role/roles-hcnphz-22023-09-29T22:50:55.266Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T22:51:31.600 DEBUG:boto:Method: POST 2023-09-29T22:51:31.600 DEBUG:boto:Path: / 2023-09-29T22:51:31.600 DEBUG:boto:Data: 2023-09-29T22:51:31.601 DEBUG:boto:Headers: {} 2023-09-29T22:51:31.601 DEBUG:boto:Host: smithi112.front.sepia.ceph.com 2023-09-29T22:51:31.601 DEBUG:boto:Port: 8001 2023-09-29T22:51:31.601 DEBUG:boto:Params: {'RoleName': 'roles-hcnphz-2', 'Action': 'GetRole', 'Version': '2010-05-08'} 2023-09-29T22:51:31.601 DEBUG:boto:Token: None 2023-09-29T22:51:31.601 DEBUG:boto:CanonicalRequest: POST / host:smithi112.front.sepia.ceph.com:8001 x-amz-date:20230929T225131Z host;x-amz-date 0ca5cc7558f50d44d78a3d5fdf09e95d0b90095c720575e32528c141f7217ac3 2023-09-29T22:51:31.601 DEBUG:boto:StringToSign: AWS4-HMAC-SHA256 20230929T225131Z 20230929/front/smithi112/aws4_request 99759f14ffff9c33f9baf7422a8ee160f48b163acc50b1198dfa4140e9734b50 2023-09-29T22:51:31.602 DEBUG:boto:Signature: 330a958f55a6480af7b0fe2522240f0c86d3175c6fc3e9fc194a8ca625963c6b 2023-09-29T22:51:31.602 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'X-Amz-Date': '20230929T225131Z', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Content-Length': '57', 'Authorization': 'AWS4-HMAC-SHA256 Credential=P5GVQP2RXA6372D10DSL/20230929/front/smithi112/aws4_request,SignedHeaders=host;x-amz-date,Signature=330a958f55a6480af7b0fe2522240f0c86d3175c6fc3e9fc194a8ca625963c6b', 'Host': 'smithi112.front.sepia.ceph.com:8001'} 2023-09-29T22:51:31.604 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000f5d7a40f694d2949-00651754f3-1da5-a2'), ('Content-Type', 'application/xml'), ('Content-Length', '711'), ('Date', 'Fri, 29 Sep 2023 22:51:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:31.604 DEBUG:boto:b'tx00000f5d7a40f694d2949-00651754f3-1da5-a20ad95f98-9899-4e17-ad62-eca46c521b91roles-hcnphz-2/arn:aws:iam:::role/roles-hcnphz-22023-09-29T22:50:55.266Z3600{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam:::user/testuser"]},"Action":["sts:AssumeRole"]}]}' 2023-09-29T22:51:31.605 DEBUG:tasks.rgw_multi.zone_rados:comparing role name=roles-hcnphz-2 2023-09-29T22:51:31.605 INFO:tasks.rgw_multi.zone_rados:success, role identical: role=roles-hcnphz-2 zones={a1, a2} 2023-09-29T22:51:31.606 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_role_sync ... ok 2023-09-29T22:51:31.606 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:51:31.607 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:51:31.607 DEBUG:boto:Method: PUT 2023-09-29T22:51:31.607 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:51:31.607 DEBUG:boto:Data: 2023-09-29T22:51:31.607 DEBUG:boto:Headers: {} 2023-09-29T22:51:31.607 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:31.607 DEBUG:boto:Port: 8000 2023-09-29T22:51:31.607 DEBUG:boto:Params: {} 2023-09-29T22:51:31.608 DEBUG:boto:Token: None 2023-09-29T22:51:31.608 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:51:31 GMT /hcnphz-60/ 2023-09-29T22:51:31.608 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tsAmAF3PMnRSih32aye1lpAOZps= 2023-09-29T22:51:31.608 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:51:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tsAmAF3PMnRSih32aye1lpAOZps=', 'Content-Length': '0'} 2023-09-29T22:51:31.638 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000000cc8720b444ce6f0-00651754f3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:51:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:31.638 DEBUG:rgw_multi.tests:created bucket=hcnphz-60 2023-09-29T22:51:31.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:31.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:31.639 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:31.680 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:31.680 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:31.736 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:51:31.766 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:51:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:51:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T22:51:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027891.637715_543.1", 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:51:31.637715Z" 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:31.767 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:51:31.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.768 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:51:31.769 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696027891.637715_543.1', 2: '', 3: ''} 2023-09-29T22:51:31.769 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:51:31.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:31.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:51:31.769 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:51:31.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:31.857 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:31.916 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:31.913+0000 7fe4e1d61f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:31.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:31.913+0000 7fe4e1d61f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:51:31.949 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:51:31.949 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:51:31.949 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:51:31.949 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:51:31.949 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.950 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:51:31.951 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027891.637715_543.1", 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:51:31.637715Z", 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.952 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.953 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:51:31.954 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:51:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:51:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:51:31Z" 2023-09-29T22:51:31.956 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:51:31.956 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696027891.637715_543.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:51:31.637715Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:51:31Z" } 2023-09-29T22:51:31.957 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696027891.637715_543.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:51:31.637715Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:51:31.957 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696027891.637715_543.1', 2: '', 3: ''} 2023-09-29T22:51:31.957 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696027891.637715_543.1', 2: '', 3: ''} 2023-09-29T22:51:31.957 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:51:31.957 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T22:51:31.958 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:51:31.958 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:51:31.953+0000 7f72e61f7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 73878) UID: 0 2023-09-29T22:51:31.959 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:51:31.953+0000 7f72e9a56f80 -1 shutting down 2023-09-29T22:51:33.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T22:51:38.059 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T22:51:38.059 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:51:38.059 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:51:38.059 DEBUG:boto:Method: HEAD 2023-09-29T22:51:38.060 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:51:38.060 DEBUG:boto:Data: 2023-09-29T22:51:38.060 DEBUG:boto:Headers: {} 2023-09-29T22:51:38.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.060 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.060 DEBUG:boto:Params: {} 2023-09-29T22:51:38.060 DEBUG:boto:Token: None 2023-09-29T22:51:38.061 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/ 2023-09-29T22:51:38.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI= 2023-09-29T22:51:38.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI=', 'Content-Length': '0'} 2023-09-29T22:51:38.065 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f98349954e9e97c9-00651754fa-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.065 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:51:38.066 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:51:38.066 DEBUG:boto:Method: PUT 2023-09-29T22:51:38.066 DEBUG:boto:Path: /hcnphz-60/obj11 2023-09-29T22:51:38.066 DEBUG:boto:Data: 2023-09-29T22:51:38.066 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:51:38.066 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.066 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.066 DEBUG:boto:Params: {} 2023-09-29T22:51:38.066 DEBUG:boto:Token: None 2023-09-29T22:51:38.067 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/obj11 2023-09-29T22:51:38.067 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6bVZrRm0w4Czd3wIM6sfAaPXwcg= 2023-09-29T22:51:38.067 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6bVZrRm0w4Czd3wIM6sfAaPXwcg='} 2023-09-29T22:51:38.080 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000034ba4f4d14700ca3-00651754fa-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.080 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:51:38.080 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:51:38.080 DEBUG:boto:Method: HEAD 2023-09-29T22:51:38.080 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:51:38.081 DEBUG:boto:Data: 2023-09-29T22:51:38.081 DEBUG:boto:Headers: {} 2023-09-29T22:51:38.081 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.081 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.081 DEBUG:boto:Params: {} 2023-09-29T22:51:38.081 DEBUG:boto:Token: None 2023-09-29T22:51:38.081 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/ 2023-09-29T22:51:38.082 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI= 2023-09-29T22:51:38.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI=', 'Content-Length': '0'} 2023-09-29T22:51:38.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f769571cec0ad54c-00651754fa-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.085 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:51:38.086 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:51:38.086 DEBUG:boto:Method: PUT 2023-09-29T22:51:38.086 DEBUG:boto:Path: /hcnphz-60/obj22 2023-09-29T22:51:38.086 DEBUG:boto:Data: 2023-09-29T22:51:38.086 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:51:38.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.086 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.086 DEBUG:boto:Params: {} 2023-09-29T22:51:38.086 DEBUG:boto:Token: None 2023-09-29T22:51:38.087 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/obj22 2023-09-29T22:51:38.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:REhDlPDdlVhFxNMaINJlcaTinec= 2023-09-29T22:51:38.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:REhDlPDdlVhFxNMaINJlcaTinec='} 2023-09-29T22:51:38.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000930e6d7ce434ff84-00651754fa-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.131 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:51:38.131 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:51:38.131 DEBUG:boto:Method: HEAD 2023-09-29T22:51:38.131 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:51:38.131 DEBUG:boto:Data: 2023-09-29T22:51:38.131 DEBUG:boto:Headers: {} 2023-09-29T22:51:38.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.131 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.132 DEBUG:boto:Params: {} 2023-09-29T22:51:38.132 DEBUG:boto:Token: None 2023-09-29T22:51:38.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/ 2023-09-29T22:51:38.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI= 2023-09-29T22:51:38.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI=', 'Content-Length': '0'} 2023-09-29T22:51:38.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b49e4783b45bfe9d-00651754fa-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.136 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:51:38.136 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:51:38.136 DEBUG:boto:Method: PUT 2023-09-29T22:51:38.136 DEBUG:boto:Path: /hcnphz-60/obj33 2023-09-29T22:51:38.136 DEBUG:boto:Data: 2023-09-29T22:51:38.136 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:51:38.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.137 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.137 DEBUG:boto:Params: {} 2023-09-29T22:51:38.137 DEBUG:boto:Token: None 2023-09-29T22:51:38.137 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/obj33 2023-09-29T22:51:38.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d+h+NSfDQNSIZziCZ3Owb9Xhnq8= 2023-09-29T22:51:38.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d+h+NSfDQNSIZziCZ3Owb9Xhnq8='} 2023-09-29T22:51:38.154 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003e8d7eb27cbcba8e-00651754fa-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.154 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:51:38.155 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:51:38.155 DEBUG:boto:Method: HEAD 2023-09-29T22:51:38.155 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:51:38.155 DEBUG:boto:Data: 2023-09-29T22:51:38.155 DEBUG:boto:Headers: {} 2023-09-29T22:51:38.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.155 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.155 DEBUG:boto:Params: {} 2023-09-29T22:51:38.155 DEBUG:boto:Token: None 2023-09-29T22:51:38.156 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/ 2023-09-29T22:51:38.156 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI= 2023-09-29T22:51:38.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI=', 'Content-Length': '0'} 2023-09-29T22:51:38.158 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ebcbf5d987c7215f-00651754fa-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.158 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:51:38.158 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:51:38.158 DEBUG:boto:Method: PUT 2023-09-29T22:51:38.159 DEBUG:boto:Path: /hcnphz-60/obj44 2023-09-29T22:51:38.159 DEBUG:boto:Data: 2023-09-29T22:51:38.159 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:51:38.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.159 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.159 DEBUG:boto:Params: {} 2023-09-29T22:51:38.159 DEBUG:boto:Token: None 2023-09-29T22:51:38.159 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/obj44 2023-09-29T22:51:38.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U82iygE4LAchswTUu/31797w500= 2023-09-29T22:51:38.160 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U82iygE4LAchswTUu/31797w500='} 2023-09-29T22:51:38.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003a4895f5fdb8fa30-00651754fa-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.207 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:51:38.207 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:51:38.207 DEBUG:boto:Method: HEAD 2023-09-29T22:51:38.207 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:51:38.207 DEBUG:boto:Data: 2023-09-29T22:51:38.207 DEBUG:boto:Headers: {} 2023-09-29T22:51:38.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.207 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.208 DEBUG:boto:Params: {} 2023-09-29T22:51:38.208 DEBUG:boto:Token: None 2023-09-29T22:51:38.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/ 2023-09-29T22:51:38.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI= 2023-09-29T22:51:38.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERiMM+EpgWd6v6YruEmXQ86yctI=', 'Content-Length': '0'} 2023-09-29T22:51:38.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000064c1829ea5c88ce8-00651754fa-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.212 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:51:38.212 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:51:38.212 DEBUG:boto:Method: PUT 2023-09-29T22:51:38.212 DEBUG:boto:Path: /hcnphz-60/obj55 2023-09-29T22:51:38.213 DEBUG:boto:Data: 2023-09-29T22:51:38.213 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:51:38.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:51:38.213 DEBUG:boto:Port: 8000 2023-09-29T22:51:38.213 DEBUG:boto:Params: {} 2023-09-29T22:51:38.213 DEBUG:boto:Token: None 2023-09-29T22:51:38.213 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:51:38 GMT /hcnphz-60/obj55 2023-09-29T22:51:38.213 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I7WJFu5sA1n7D4nPBKbETtnpZaQ= 2023-09-29T22:51:38.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:51:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I7WJFu5sA1n7D4nPBKbETtnpZaQ='} 2023-09-29T22:51:38.222 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ac1f9b19ed83e142-00651754fa-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:51:38.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:51:38.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:51:38.223 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:51:38.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:38.268 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:38.324 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:38.321+0000 7f73ea3c6f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-29T22:51:38.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:38.321+0000 7f73ea3c6f40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-29T22:51:38.366 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:51:38.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:51:38.368 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:51:38.368 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:51:38.384 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:51:38.384 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:51:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:51:38.387 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:51:38.387 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T22:51:39.388 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:51:39.405 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:51:39.406 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:51:39.408 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T22:51:39.409 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T22:51:39.409 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-60 2023-09-29T22:51:39.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:51:39.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-60', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:51:39.410 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-60 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:51:39.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:51:39.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:51:39.522 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:39.517+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:51:39.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T22:51:44.523 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:44.525+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:51:49.524 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:49.525+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:51:54.525 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:54.525+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:51:59.526 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:51:59.525+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:04.527 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:04.529+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:09.528 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:09.528+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:10.213 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:52:10.206+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:52:14.529 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:14.528+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:19.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:19.528+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:24.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:24.532+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:29.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:29.532+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:34.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:34.532+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:39.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:39.532+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:44.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:44.536+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:49.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:49.535+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:54.537 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:54.535+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:52:59.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:52:59.535+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:04.539 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:04.535+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:09.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:09.539+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:14.541 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:14.539+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:19.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:19.539+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:24.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:24.539+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:29.544 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:29.542+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:34.545 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:34.542+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:39.546 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:39.542+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:40.173 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:53:40.170+0000 7f97aadc8700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T22:53:44.547 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:44.542+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:49.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:49.546+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:54.549 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:54.546+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:53:59.551 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:53:59.546+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:04.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:04.550+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:09.552 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:09.549+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:14.553 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:14.549+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:19.555 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:19.557+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:24.556 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:24.557+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:29.557 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:29.557+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:34.558 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:34.557+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:39.559 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:39.561+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:44.560 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:44.561+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:49.562 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:49.560+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:54.563 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:54.564+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:54:59.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:54:59.564+0000 7f298605bf40 1 waiting to reach incremental sync.. 2023-09-29T22:55:04.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:04.568+0000 7f298605bf40 1 bucket sync caught up with source: 2023-09-29T22:55:04.568 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.49.6, 00000000002.64.6, 00000000002.88.6] 2023-09-29T22:55:04.568 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.49.6, 00000000002.64.6, 00000000002.88.6] 2023-09-29T22:55:04.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:04.568+0000 7f298605bf40 0 bucket checkpoint complete 2023-09-29T22:55:04.598 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-60 2023-09-29T22:55:04.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:55:04.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-60', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:55:04.599 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-60 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:55:04.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:04.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:04.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:04.712+0000 7f3222a5df40 1 bucket sync caught up with source: 2023-09-29T22:55:04.713 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.316.6, 00000000002.183.6, 00000000002.330.6] 2023-09-29T22:55:04.713 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.316.6, 00000000002.183.6, 00000000002.330.6] 2023-09-29T22:55:04.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:04.712+0000 7f3222a5df40 0 bucket checkpoint complete 2023-09-29T22:55:04.739 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-60 zones={a2, a1} 2023-09-29T22:55:04.740 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.740 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.740 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.741 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:55:04.741 DEBUG:boto:Data: 2023-09-29T22:55:04.741 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.741 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.741 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.741 DEBUG:boto:Params: {} 2023-09-29T22:55:04.742 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:55:04.742 DEBUG:boto:Token: None 2023-09-29T22:55:04.742 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/ 2023-09-29T22:55:04.742 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1mnYCj7tcXKQ0XOZGRIMpAJ4rq4= 2023-09-29T22:55:04.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1mnYCj7tcXKQ0XOZGRIMpAJ4rq4=', 'Content-Length': '0'} 2023-09-29T22:55:04.751 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007d14373080178e0f-00651755c8-1e1c-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.751 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.751 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.751 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.752 DEBUG:boto:Path: /hcnphz-60/ 2023-09-29T22:55:04.752 DEBUG:boto:Data: 2023-09-29T22:55:04.752 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.752 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.752 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.752 DEBUG:boto:Params: {} 2023-09-29T22:55:04.753 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:55:04.753 DEBUG:boto:Token: None 2023-09-29T22:55:04.753 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/ 2023-09-29T22:55:04.754 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1mnYCj7tcXKQ0XOZGRIMpAJ4rq4= 2023-09-29T22:55:04.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1mnYCj7tcXKQ0XOZGRIMpAJ4rq4=', 'Content-Length': '0'} 2023-09-29T22:55:04.758 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000099199d1a5cce61be-00651755c8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.758 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:55:04.759 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.759 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.759 DEBUG:boto:path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.759 DEBUG:boto:auth_path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.759 DEBUG:boto:Method: GET 2023-09-29T22:55:04.759 DEBUG:boto:Path: /hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.759 DEBUG:boto:Data: 2023-09-29T22:55:04.759 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.760 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.760 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.760 DEBUG:boto:Params: {} 2023-09-29T22:55:04.760 DEBUG:boto:Token: None 2023-09-29T22:55:04.760 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/?versions 2023-09-29T22:55:04.761 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI= 2023-09-29T22:55:04.761 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI=', 'Content-Length': '0'} 2023-09-29T22:55:04.764 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fed494ea71aae2cf-00651755c8-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.764 DEBUG:boto:b'hcnphz-60999falseobj11nulltrue2023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:55:04.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:55:04.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:55:04.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:55:04.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:55:04.765 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:55:04.766 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.766 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.766 DEBUG:boto:path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.766 DEBUG:boto:auth_path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.766 DEBUG:boto:Method: GET 2023-09-29T22:55:04.766 DEBUG:boto:Path: /hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.766 DEBUG:boto:Data: 2023-09-29T22:55:04.766 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.766 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.766 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.767 DEBUG:boto:Params: {} 2023-09-29T22:55:04.767 DEBUG:boto:Token: None 2023-09-29T22:55:04.767 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/?versions 2023-09-29T22:55:04.767 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI= 2023-09-29T22:55:04.767 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI=', 'Content-Length': '0'} 2023-09-29T22:55:04.771 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000781549c51ae8e55c-00651755c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.771 DEBUG:boto:b'hcnphz-60999falseobj11nulltrue2023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:55:04.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:55:04.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:55:04.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:55:04.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:55:04.772 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.772 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.772 DEBUG:boto:path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.773 DEBUG:boto:auth_path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.773 DEBUG:boto:Method: GET 2023-09-29T22:55:04.773 DEBUG:boto:Path: /hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.773 DEBUG:boto:Data: 2023-09-29T22:55:04.773 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.773 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.773 DEBUG:boto:Params: {} 2023-09-29T22:55:04.774 DEBUG:boto:Token: None 2023-09-29T22:55:04.774 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/?versions 2023-09-29T22:55:04.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI= 2023-09-29T22:55:04.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI=', 'Content-Length': '0'} 2023-09-29T22:55:04.777 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000114dfce368f3d7a9-00651755c8-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.777 DEBUG:boto:b'hcnphz-60999falseobj11nulltrue2023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.778 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.778 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.778 DEBUG:boto:path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.778 DEBUG:boto:auth_path=/hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.779 DEBUG:boto:Method: GET 2023-09-29T22:55:04.779 DEBUG:boto:Path: /hcnphz-60/?versions&max-keys=999 2023-09-29T22:55:04.779 DEBUG:boto:Data: 2023-09-29T22:55:04.779 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.779 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.779 DEBUG:boto:Params: {} 2023-09-29T22:55:04.779 DEBUG:boto:Token: None 2023-09-29T22:55:04.779 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/?versions 2023-09-29T22:55:04.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI= 2023-09-29T22:55:04.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AVbeHodaNV2EpWIR6u2b68r3pbI=', 'Content-Length': '0'} 2023-09-29T22:55:04.784 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003cd434a52c48aa08-00651755c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.784 DEBUG:boto:b'hcnphz-60999falseobj11nulltrue2023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:55:04.785 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.785 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.785 DEBUG:boto:path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.785 DEBUG:boto:auth_path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.785 DEBUG:boto:Method: GET 2023-09-29T22:55:04.785 DEBUG:boto:Path: /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.785 DEBUG:boto:Data: 2023-09-29T22:55:04.785 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.786 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.786 DEBUG:boto:Params: {} 2023-09-29T22:55:04.786 DEBUG:boto:Token: None 2023-09-29T22:55:04.786 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU= 2023-09-29T22:55:04.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU=', 'Content-Length': '0'} 2023-09-29T22:55:04.789 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000721a2c2b602cc45b-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.790 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.790 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.790 DEBUG:boto:path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.790 DEBUG:boto:auth_path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.790 DEBUG:boto:Method: GET 2023-09-29T22:55:04.790 DEBUG:boto:Path: /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.790 DEBUG:boto:Data: 2023-09-29T22:55:04.791 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.791 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.791 DEBUG:boto:Params: {} 2023-09-29T22:55:04.791 DEBUG:boto:Token: None 2023-09-29T22:55:04.791 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.791 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU= 2023-09-29T22:55:04.791 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU=', 'Content-Length': '0'} 2023-09-29T22:55:04.794 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c793371605c18b8f-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.794 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.794 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.795 DEBUG:boto:path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.795 DEBUG:boto:auth_path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.795 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.795 DEBUG:boto:Path: /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.795 DEBUG:boto:Data: 2023-09-29T22:55:04.795 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.795 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.795 DEBUG:boto:Params: {} 2023-09-29T22:55:04.796 DEBUG:boto:Token: None 2023-09-29T22:55:04.796 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bQb602muufysRUE35NdmEKKaBGQ= 2023-09-29T22:55:04.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bQb602muufysRUE35NdmEKKaBGQ=', 'Content-Length': '0'} 2023-09-29T22:55:04.798 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c773efc47b383960-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.799 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.799 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.799 DEBUG:boto:path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.799 DEBUG:boto:auth_path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.799 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.799 DEBUG:boto:Path: /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.799 DEBUG:boto:Data: 2023-09-29T22:55:04.799 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.799 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.800 DEBUG:boto:Params: {} 2023-09-29T22:55:04.800 DEBUG:boto:Token: None 2023-09-29T22:55:04.800 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bQb602muufysRUE35NdmEKKaBGQ= 2023-09-29T22:55:04.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bQb602muufysRUE35NdmEKKaBGQ=', 'Content-Length': '0'} 2023-09-29T22:55:04.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbbfd66bb030b11d-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:55:04.803 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.803 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.803 DEBUG:boto:path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.803 DEBUG:boto:auth_path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.803 DEBUG:boto:Method: GET 2023-09-29T22:55:04.803 DEBUG:boto:Path: /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.803 DEBUG:boto:Data: 2023-09-29T22:55:04.803 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.804 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.804 DEBUG:boto:Params: {} 2023-09-29T22:55:04.804 DEBUG:boto:Token: None 2023-09-29T22:55:04.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU= 2023-09-29T22:55:04.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU=', 'Content-Length': '0'} 2023-09-29T22:55:04.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071f127bed70df754-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.850 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.851 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.851 DEBUG:boto:path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.851 DEBUG:boto:auth_path=/hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.851 DEBUG:boto:Method: GET 2023-09-29T22:55:04.851 DEBUG:boto:Path: /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.851 DEBUG:boto:Data: 2023-09-29T22:55:04.851 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.851 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.852 DEBUG:boto:Params: {} 2023-09-29T22:55:04.852 DEBUG:boto:Token: None 2023-09-29T22:55:04.852 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11?versionId=null 2023-09-29T22:55:04.852 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU= 2023-09-29T22:55:04.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/HhywhfJho0vpxC6068hXk6pwIU=', 'Content-Length': '0'} 2023-09-29T22:55:04.855 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063862a26ec5b90f4-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.855 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.855 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.855 DEBUG:boto:path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.856 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.856 DEBUG:boto:Method: GET 2023-09-29T22:55:04.856 DEBUG:boto:Path: /hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.856 DEBUG:boto:Data: 2023-09-29T22:55:04.856 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.856 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.856 DEBUG:boto:Params: {} 2023-09-29T22:55:04.856 DEBUG:boto:Token: None 2023-09-29T22:55:04.857 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/ 2023-09-29T22:55:04.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8= 2023-09-29T22:55:04.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8=', 'Content-Length': '0'} 2023-09-29T22:55:04.862 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f680d1ab2c8aa9c-00651755c8-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.862 DEBUG:boto:b'hcnphz-60obj111000falseobj112023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.863 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.863 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.863 DEBUG:boto:path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.863 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.863 DEBUG:boto:Method: GET 2023-09-29T22:55:04.863 DEBUG:boto:Path: /hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.863 DEBUG:boto:Data: 2023-09-29T22:55:04.863 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.863 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.863 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.864 DEBUG:boto:Params: {} 2023-09-29T22:55:04.864 DEBUG:boto:Token: None 2023-09-29T22:55:04.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/ 2023-09-29T22:55:04.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8= 2023-09-29T22:55:04.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8=', 'Content-Length': '0'} 2023-09-29T22:55:04.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eca54cadfe08f92b-00651755c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.869 DEBUG:boto:b'hcnphz-60obj111000falseobj112023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.869 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.869 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.869 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.869 DEBUG:boto:Path: /hcnphz-60/obj11 2023-09-29T22:55:04.869 DEBUG:boto:Data: 2023-09-29T22:55:04.870 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.870 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.870 DEBUG:boto:Params: {} 2023-09-29T22:55:04.870 DEBUG:boto:Token: None 2023-09-29T22:55:04.870 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11 2023-09-29T22:55:04.870 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4MsSzlxQB9R8fEwImDkOgBe1pNQ= 2023-09-29T22:55:04.870 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4MsSzlxQB9R8fEwImDkOgBe1pNQ=', 'Content-Length': '0'} 2023-09-29T22:55:04.873 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091af65f45de1f727-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.873 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.873 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.873 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.873 DEBUG:boto:Path: /hcnphz-60/obj11 2023-09-29T22:55:04.873 DEBUG:boto:Data: 2023-09-29T22:55:04.874 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.874 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.874 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.874 DEBUG:boto:Params: {} 2023-09-29T22:55:04.874 DEBUG:boto:Token: None 2023-09-29T22:55:04.874 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11 2023-09-29T22:55:04.874 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4MsSzlxQB9R8fEwImDkOgBe1pNQ= 2023-09-29T22:55:04.874 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4MsSzlxQB9R8fEwImDkOgBe1pNQ=', 'Content-Length': '0'} 2023-09-29T22:55:04.876 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043581d0f0e585f02-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:55:04.877 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.877 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.877 DEBUG:boto:Method: GET 2023-09-29T22:55:04.877 DEBUG:boto:Path: /hcnphz-60/obj11 2023-09-29T22:55:04.877 DEBUG:boto:Data: 2023-09-29T22:55:04.877 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.878 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.878 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.878 DEBUG:boto:Params: {} 2023-09-29T22:55:04.878 DEBUG:boto:Token: None 2023-09-29T22:55:04.878 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11 2023-09-29T22:55:04.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w2xTfgIIDERmgK4eMgNwH/4eHuE= 2023-09-29T22:55:04.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w2xTfgIIDERmgK4eMgNwH/4eHuE=', 'Content-Length': '0'} 2023-09-29T22:55:04.881 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d6cbef6a72a056b-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.881 DEBUG:boto:path=/hcnphz-60/obj11 2023-09-29T22:55:04.882 DEBUG:boto:auth_path=/hcnphz-60/obj11 2023-09-29T22:55:04.882 DEBUG:boto:Method: GET 2023-09-29T22:55:04.882 DEBUG:boto:Path: /hcnphz-60/obj11 2023-09-29T22:55:04.882 DEBUG:boto:Data: 2023-09-29T22:55:04.882 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.882 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.883 DEBUG:boto:Params: {} 2023-09-29T22:55:04.883 DEBUG:boto:Token: None 2023-09-29T22:55:04.883 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj11 2023-09-29T22:55:04.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w2xTfgIIDERmgK4eMgNwH/4eHuE= 2023-09-29T22:55:04.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w2xTfgIIDERmgK4eMgNwH/4eHuE=', 'Content-Length': '0'} 2023-09-29T22:55:04.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf67c465ed81c901-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.887 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.887 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.887 DEBUG:boto:path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.887 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.887 DEBUG:boto:Method: GET 2023-09-29T22:55:04.887 DEBUG:boto:Path: /hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.887 DEBUG:boto:Data: 2023-09-29T22:55:04.887 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.887 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.888 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.888 DEBUG:boto:Params: {} 2023-09-29T22:55:04.888 DEBUG:boto:Token: None 2023-09-29T22:55:04.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/ 2023-09-29T22:55:04.888 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8= 2023-09-29T22:55:04.888 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8=', 'Content-Length': '0'} 2023-09-29T22:55:04.892 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004757d78812b03516-00651755c8-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.892 DEBUG:boto:b'hcnphz-60obj111000falseobj112023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.893 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:04.893 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:04.893 DEBUG:boto:path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.893 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.893 DEBUG:boto:Method: GET 2023-09-29T22:55:04.893 DEBUG:boto:Path: /hcnphz-60/?prefix=obj11 2023-09-29T22:55:04.894 DEBUG:boto:Data: 2023-09-29T22:55:04.894 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.894 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.894 DEBUG:boto:Params: {} 2023-09-29T22:55:04.894 DEBUG:boto:Token: None 2023-09-29T22:55:04.894 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/ 2023-09-29T22:55:04.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8= 2023-09-29T22:55:04.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KbwXEQuQmzntRldLI+dLCXUN0f8=', 'Content-Length': '0'} 2023-09-29T22:55:04.899 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf97fc2627f96858-00651755c8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.946 DEBUG:boto:b'hcnphz-60obj111000falseobj112023-09-29T22:51:38.072Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:04.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:55:04.947 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:04.947 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:04.947 DEBUG:boto:path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.947 DEBUG:boto:auth_path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.947 DEBUG:boto:Method: GET 2023-09-29T22:55:04.947 DEBUG:boto:Path: /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.948 DEBUG:boto:Data: 2023-09-29T22:55:04.948 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.948 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.948 DEBUG:boto:Params: {} 2023-09-29T22:55:04.948 DEBUG:boto:Token: None 2023-09-29T22:55:04.948 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.948 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:woF9Su9Zkisu2xygPWtRmhUkIMo= 2023-09-29T22:55:04.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:woF9Su9Zkisu2xygPWtRmhUkIMo=', 'Content-Length': '0'} 2023-09-29T22:55:04.951 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fff5ebf6d4fd183-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.952 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:04.952 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:04.952 DEBUG:boto:path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.952 DEBUG:boto:auth_path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.952 DEBUG:boto:Method: GET 2023-09-29T22:55:04.952 DEBUG:boto:Path: /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.952 DEBUG:boto:Data: 2023-09-29T22:55:04.952 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.953 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.953 DEBUG:boto:Params: {} 2023-09-29T22:55:04.953 DEBUG:boto:Token: None 2023-09-29T22:55:04.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:woF9Su9Zkisu2xygPWtRmhUkIMo= 2023-09-29T22:55:04.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:woF9Su9Zkisu2xygPWtRmhUkIMo=', 'Content-Length': '0'} 2023-09-29T22:55:04.956 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007337075176a8907b-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.956 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:04.956 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:04.956 DEBUG:boto:path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.956 DEBUG:boto:auth_path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.957 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.957 DEBUG:boto:Path: /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.957 DEBUG:boto:Data: 2023-09-29T22:55:04.957 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.957 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.957 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.957 DEBUG:boto:Params: {} 2023-09-29T22:55:04.957 DEBUG:boto:Token: None 2023-09-29T22:55:04.958 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.958 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pE0a7vxfEXc1d7YaMbg3JxdILYE= 2023-09-29T22:55:04.958 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pE0a7vxfEXc1d7YaMbg3JxdILYE=', 'Content-Length': '0'} 2023-09-29T22:55:04.960 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000104e72f9db837ca4-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.961 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:04.961 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:04.961 DEBUG:boto:path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.961 DEBUG:boto:auth_path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.961 DEBUG:boto:Method: HEAD 2023-09-29T22:55:04.962 DEBUG:boto:Path: /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.962 DEBUG:boto:Data: 2023-09-29T22:55:04.962 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:04.962 DEBUG:boto:Port: 8000 2023-09-29T22:55:04.962 DEBUG:boto:Params: {} 2023-09-29T22:55:04.962 DEBUG:boto:Token: None 2023-09-29T22:55:04.962 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.963 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pE0a7vxfEXc1d7YaMbg3JxdILYE= 2023-09-29T22:55:04.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pE0a7vxfEXc1d7YaMbg3JxdILYE=', 'Content-Length': '0'} 2023-09-29T22:55:04.966 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f11febd308efccc-00651755c8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:04.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:55:04.966 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:04.967 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:04.967 DEBUG:boto:path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.967 DEBUG:boto:auth_path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.967 DEBUG:boto:Method: GET 2023-09-29T22:55:04.967 DEBUG:boto:Path: /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.967 DEBUG:boto:Data: 2023-09-29T22:55:04.967 DEBUG:boto:Headers: {} 2023-09-29T22:55:04.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:04.967 DEBUG:boto:Port: 8001 2023-09-29T22:55:04.968 DEBUG:boto:Params: {} 2023-09-29T22:55:04.968 DEBUG:boto:Token: None 2023-09-29T22:55:04.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:04 GMT /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:04.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:woF9Su9Zkisu2xygPWtRmhUkIMo= 2023-09-29T22:55:04.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:04 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:woF9Su9Zkisu2xygPWtRmhUkIMo=', 'Content-Length': '0'} 2023-09-29T22:55:04.971 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a6c6c89bee371a6-00651755c8-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:04 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.018 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:05.019 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:05.019 DEBUG:boto:path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:05.019 DEBUG:boto:auth_path=/hcnphz-60/obj22?versionId=null 2023-09-29T22:55:05.019 DEBUG:boto:Method: GET 2023-09-29T22:55:05.019 DEBUG:boto:Path: /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:05.019 DEBUG:boto:Data: 2023-09-29T22:55:05.019 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.020 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.020 DEBUG:boto:Params: {} 2023-09-29T22:55:05.020 DEBUG:boto:Token: None 2023-09-29T22:55:05.020 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj22?versionId=null 2023-09-29T22:55:05.020 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HSSKTVf1bJIKBeJE3o+HwilpBvk= 2023-09-29T22:55:05.020 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HSSKTVf1bJIKBeJE3o+HwilpBvk=', 'Content-Length': '0'} 2023-09-29T22:55:05.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ff08d6073e2fa75-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.024 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.024 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.024 DEBUG:boto:path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.024 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.025 DEBUG:boto:Method: GET 2023-09-29T22:55:05.025 DEBUG:boto:Path: /hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.025 DEBUG:boto:Data: 2023-09-29T22:55:05.025 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.025 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.025 DEBUG:boto:Params: {} 2023-09-29T22:55:05.025 DEBUG:boto:Token: None 2023-09-29T22:55:05.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fbe9b13c67430585-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.030 DEBUG:boto:b'hcnphz-60obj221000falseobj222023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.031 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.031 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.031 DEBUG:boto:path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.031 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.031 DEBUG:boto:Method: GET 2023-09-29T22:55:05.032 DEBUG:boto:Path: /hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.032 DEBUG:boto:Data: 2023-09-29T22:55:05.032 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.032 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.032 DEBUG:boto:Params: {} 2023-09-29T22:55:05.033 DEBUG:boto:Token: None 2023-09-29T22:55:05.033 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.033 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.033 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.037 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008f0244bf7fdacc9-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.079 DEBUG:boto:b'hcnphz-60obj221000falseobj222023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.079 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:05.079 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:05.080 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.080 DEBUG:boto:Path: /hcnphz-60/obj22 2023-09-29T22:55:05.080 DEBUG:boto:Data: 2023-09-29T22:55:05.080 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.080 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.080 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.080 DEBUG:boto:Params: {} 2023-09-29T22:55:05.081 DEBUG:boto:Token: None 2023-09-29T22:55:05.081 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj22 2023-09-29T22:55:05.082 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XcM0ipcQoj5oGRuFLzAAnBpTb04= 2023-09-29T22:55:05.082 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XcM0ipcQoj5oGRuFLzAAnBpTb04=', 'Content-Length': '0'} 2023-09-29T22:55:05.085 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8c2c45df2cb9777-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.085 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:05.085 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:05.086 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.086 DEBUG:boto:Path: /hcnphz-60/obj22 2023-09-29T22:55:05.086 DEBUG:boto:Data: 2023-09-29T22:55:05.086 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.086 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.086 DEBUG:boto:Params: {} 2023-09-29T22:55:05.086 DEBUG:boto:Token: None 2023-09-29T22:55:05.087 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj22 2023-09-29T22:55:05.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XcM0ipcQoj5oGRuFLzAAnBpTb04= 2023-09-29T22:55:05.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XcM0ipcQoj5oGRuFLzAAnBpTb04=', 'Content-Length': '0'} 2023-09-29T22:55:05.089 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef2588b655d2afb1-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:55:05.090 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:05.090 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:05.090 DEBUG:boto:Method: GET 2023-09-29T22:55:05.090 DEBUG:boto:Path: /hcnphz-60/obj22 2023-09-29T22:55:05.090 DEBUG:boto:Data: 2023-09-29T22:55:05.090 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.091 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.091 DEBUG:boto:Params: {} 2023-09-29T22:55:05.091 DEBUG:boto:Token: None 2023-09-29T22:55:05.091 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj22 2023-09-29T22:55:05.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tOCOPG0UZpbHMBmAgevbsROc0mc= 2023-09-29T22:55:05.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tOCOPG0UZpbHMBmAgevbsROc0mc=', 'Content-Length': '0'} 2023-09-29T22:55:05.094 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f65b0af14cee8d24-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.094 DEBUG:boto:path=/hcnphz-60/obj22 2023-09-29T22:55:05.095 DEBUG:boto:auth_path=/hcnphz-60/obj22 2023-09-29T22:55:05.095 DEBUG:boto:Method: GET 2023-09-29T22:55:05.095 DEBUG:boto:Path: /hcnphz-60/obj22 2023-09-29T22:55:05.095 DEBUG:boto:Data: 2023-09-29T22:55:05.096 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.096 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.096 DEBUG:boto:Params: {} 2023-09-29T22:55:05.096 DEBUG:boto:Token: None 2023-09-29T22:55:05.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj22 2023-09-29T22:55:05.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tOCOPG0UZpbHMBmAgevbsROc0mc= 2023-09-29T22:55:05.098 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tOCOPG0UZpbHMBmAgevbsROc0mc=', 'Content-Length': '0'} 2023-09-29T22:55:05.100 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007bbcab5dd03c6a56-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.101 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.101 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.101 DEBUG:boto:path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.101 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.101 DEBUG:boto:Method: GET 2023-09-29T22:55:05.101 DEBUG:boto:Path: /hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.102 DEBUG:boto:Data: 2023-09-29T22:55:05.102 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.102 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.102 DEBUG:boto:Params: {} 2023-09-29T22:55:05.102 DEBUG:boto:Token: None 2023-09-29T22:55:05.102 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.107 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ecf57bff1d6d19e7-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.108 DEBUG:boto:b'hcnphz-60obj221000falseobj222023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.108 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.109 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.109 DEBUG:boto:path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.109 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.109 DEBUG:boto:Method: GET 2023-09-29T22:55:05.109 DEBUG:boto:Path: /hcnphz-60/?prefix=obj22 2023-09-29T22:55:05.109 DEBUG:boto:Data: 2023-09-29T22:55:05.109 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.109 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.110 DEBUG:boto:Params: {} 2023-09-29T22:55:05.110 DEBUG:boto:Token: None 2023-09-29T22:55:05.110 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.110 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.114 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089cd82adff492930-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.115 DEBUG:boto:b'hcnphz-60obj221000falseobj222023-09-29T22:51:38.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:55:05.115 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.115 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.115 DEBUG:boto:path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.115 DEBUG:boto:auth_path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.116 DEBUG:boto:Method: GET 2023-09-29T22:55:05.116 DEBUG:boto:Path: /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.116 DEBUG:boto:Data: 2023-09-29T22:55:05.116 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.116 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.116 DEBUG:boto:Params: {} 2023-09-29T22:55:05.116 DEBUG:boto:Token: None 2023-09-29T22:55:05.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI= 2023-09-29T22:55:05.117 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI=', 'Content-Length': '0'} 2023-09-29T22:55:05.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002d2a22009d94c744-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.162 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.163 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.163 DEBUG:boto:path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.163 DEBUG:boto:auth_path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.163 DEBUG:boto:Method: GET 2023-09-29T22:55:05.163 DEBUG:boto:Path: /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.163 DEBUG:boto:Data: 2023-09-29T22:55:05.163 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.164 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.164 DEBUG:boto:Params: {} 2023-09-29T22:55:05.164 DEBUG:boto:Token: None 2023-09-29T22:55:05.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI= 2023-09-29T22:55:05.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI=', 'Content-Length': '0'} 2023-09-29T22:55:05.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000122a73daf98ee296-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.168 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.168 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.168 DEBUG:boto:path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.168 DEBUG:boto:auth_path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.168 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.168 DEBUG:boto:Path: /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.168 DEBUG:boto:Data: 2023-09-29T22:55:05.168 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.169 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.169 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.169 DEBUG:boto:Params: {} 2023-09-29T22:55:05.169 DEBUG:boto:Token: None 2023-09-29T22:55:05.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WazcMvoj8qQ9CweUC5t7X0rE26w= 2023-09-29T22:55:05.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WazcMvoj8qQ9CweUC5t7X0rE26w=', 'Content-Length': '0'} 2023-09-29T22:55:05.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021f1a13af55a3085-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.172 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.173 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.173 DEBUG:boto:path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.173 DEBUG:boto:auth_path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.173 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.173 DEBUG:boto:Path: /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.173 DEBUG:boto:Data: 2023-09-29T22:55:05.173 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.174 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.174 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.174 DEBUG:boto:Params: {} 2023-09-29T22:55:05.174 DEBUG:boto:Token: None 2023-09-29T22:55:05.174 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.175 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WazcMvoj8qQ9CweUC5t7X0rE26w= 2023-09-29T22:55:05.175 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WazcMvoj8qQ9CweUC5t7X0rE26w=', 'Content-Length': '0'} 2023-09-29T22:55:05.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dca6617debff1b3d-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:55:05.178 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.178 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.178 DEBUG:boto:path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.178 DEBUG:boto:auth_path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.178 DEBUG:boto:Method: GET 2023-09-29T22:55:05.178 DEBUG:boto:Path: /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.179 DEBUG:boto:Data: 2023-09-29T22:55:05.179 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.179 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.179 DEBUG:boto:Params: {} 2023-09-29T22:55:05.179 DEBUG:boto:Token: None 2023-09-29T22:55:05.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.179 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI= 2023-09-29T22:55:05.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI=', 'Content-Length': '0'} 2023-09-29T22:55:05.182 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e7043e12f3016997-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.183 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.183 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.183 DEBUG:boto:path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.183 DEBUG:boto:auth_path=/hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.183 DEBUG:boto:Method: GET 2023-09-29T22:55:05.184 DEBUG:boto:Path: /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.184 DEBUG:boto:Data: 2023-09-29T22:55:05.184 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.184 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.184 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.185 DEBUG:boto:Params: {} 2023-09-29T22:55:05.185 DEBUG:boto:Token: None 2023-09-29T22:55:05.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33?versionId=null 2023-09-29T22:55:05.185 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI= 2023-09-29T22:55:05.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMGtSACCpab9AtzvxqDESCLKgYI=', 'Content-Length': '0'} 2023-09-29T22:55:05.189 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b6f90f665706552-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.189 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.190 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.190 DEBUG:boto:path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.190 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.190 DEBUG:boto:Method: GET 2023-09-29T22:55:05.190 DEBUG:boto:Path: /hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.191 DEBUG:boto:Data: 2023-09-29T22:55:05.191 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.191 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.192 DEBUG:boto:Params: {} 2023-09-29T22:55:05.192 DEBUG:boto:Token: None 2023-09-29T22:55:05.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.198 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000748fb3f73aa67fa-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.242 DEBUG:boto:b'hcnphz-60obj331000falseobj332023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.243 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.243 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.243 DEBUG:boto:path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.243 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.243 DEBUG:boto:Method: GET 2023-09-29T22:55:05.243 DEBUG:boto:Path: /hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.243 DEBUG:boto:Data: 2023-09-29T22:55:05.244 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.244 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.244 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.244 DEBUG:boto:Params: {} 2023-09-29T22:55:05.244 DEBUG:boto:Token: None 2023-09-29T22:55:05.244 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.245 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.249 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000518b5a80e1998687-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.249 DEBUG:boto:b'hcnphz-60obj331000falseobj332023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.250 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.250 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.250 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.250 DEBUG:boto:Path: /hcnphz-60/obj33 2023-09-29T22:55:05.251 DEBUG:boto:Data: 2023-09-29T22:55:05.251 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.251 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.251 DEBUG:boto:Params: {} 2023-09-29T22:55:05.251 DEBUG:boto:Token: None 2023-09-29T22:55:05.251 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33 2023-09-29T22:55:05.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WY6/gcWJNb3Wu7aZDY+u8vkCztA= 2023-09-29T22:55:05.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WY6/gcWJNb3Wu7aZDY+u8vkCztA=', 'Content-Length': '0'} 2023-09-29T22:55:05.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000573e8776b7666d7c-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.255 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.256 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.256 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.257 DEBUG:boto:Path: /hcnphz-60/obj33 2023-09-29T22:55:05.257 DEBUG:boto:Data: 2023-09-29T22:55:05.257 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.257 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.257 DEBUG:boto:Params: {} 2023-09-29T22:55:05.257 DEBUG:boto:Token: None 2023-09-29T22:55:05.257 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33 2023-09-29T22:55:05.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WY6/gcWJNb3Wu7aZDY+u8vkCztA= 2023-09-29T22:55:05.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WY6/gcWJNb3Wu7aZDY+u8vkCztA=', 'Content-Length': '0'} 2023-09-29T22:55:05.260 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af9a953bdd0f98a8-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:55:05.261 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.261 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.261 DEBUG:boto:Method: GET 2023-09-29T22:55:05.261 DEBUG:boto:Path: /hcnphz-60/obj33 2023-09-29T22:55:05.261 DEBUG:boto:Data: 2023-09-29T22:55:05.261 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.262 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.262 DEBUG:boto:Params: {} 2023-09-29T22:55:05.262 DEBUG:boto:Token: None 2023-09-29T22:55:05.262 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33 2023-09-29T22:55:05.262 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E8dF8UhbzcbhgvzB29uYG93PXrQ= 2023-09-29T22:55:05.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E8dF8UhbzcbhgvzB29uYG93PXrQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.265 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002fabab56204d3d0d-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.266 DEBUG:boto:path=/hcnphz-60/obj33 2023-09-29T22:55:05.266 DEBUG:boto:auth_path=/hcnphz-60/obj33 2023-09-29T22:55:05.266 DEBUG:boto:Method: GET 2023-09-29T22:55:05.266 DEBUG:boto:Path: /hcnphz-60/obj33 2023-09-29T22:55:05.266 DEBUG:boto:Data: 2023-09-29T22:55:05.266 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.266 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.266 DEBUG:boto:Params: {} 2023-09-29T22:55:05.267 DEBUG:boto:Token: None 2023-09-29T22:55:05.267 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj33 2023-09-29T22:55:05.267 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E8dF8UhbzcbhgvzB29uYG93PXrQ= 2023-09-29T22:55:05.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E8dF8UhbzcbhgvzB29uYG93PXrQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.270 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cc97da20d89eea8-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.314 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.315 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.315 DEBUG:boto:path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.315 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.315 DEBUG:boto:Method: GET 2023-09-29T22:55:05.315 DEBUG:boto:Path: /hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.315 DEBUG:boto:Data: 2023-09-29T22:55:05.315 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.315 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.315 DEBUG:boto:Params: {} 2023-09-29T22:55:05.316 DEBUG:boto:Token: None 2023-09-29T22:55:05.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.323 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000047f5f874ab3ebf95-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.323 DEBUG:boto:b'hcnphz-60obj331000falseobj332023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.323 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.323 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.324 DEBUG:boto:path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.324 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.324 DEBUG:boto:Method: GET 2023-09-29T22:55:05.324 DEBUG:boto:Path: /hcnphz-60/?prefix=obj33 2023-09-29T22:55:05.324 DEBUG:boto:Data: 2023-09-29T22:55:05.324 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.324 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.324 DEBUG:boto:Params: {} 2023-09-29T22:55:05.325 DEBUG:boto:Token: None 2023-09-29T22:55:05.325 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.329 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aab549789d02a29f-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.329 DEBUG:boto:b'hcnphz-60obj331000falseobj332023-09-29T22:51:38.142Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:55:05.330 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.330 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.331 DEBUG:boto:path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.331 DEBUG:boto:auth_path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.331 DEBUG:boto:Method: GET 2023-09-29T22:55:05.331 DEBUG:boto:Path: /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.331 DEBUG:boto:Data: 2023-09-29T22:55:05.332 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.332 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.332 DEBUG:boto:Params: {} 2023-09-29T22:55:05.332 DEBUG:boto:Token: None 2023-09-29T22:55:05.333 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.333 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8= 2023-09-29T22:55:05.333 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8=', 'Content-Length': '0'} 2023-09-29T22:55:05.336 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d712011661fad3-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.378 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.379 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.379 DEBUG:boto:path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.379 DEBUG:boto:auth_path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.379 DEBUG:boto:Method: GET 2023-09-29T22:55:05.380 DEBUG:boto:Path: /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.380 DEBUG:boto:Data: 2023-09-29T22:55:05.380 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.381 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.381 DEBUG:boto:Params: {} 2023-09-29T22:55:05.381 DEBUG:boto:Token: None 2023-09-29T22:55:05.381 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8= 2023-09-29T22:55:05.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8=', 'Content-Length': '0'} 2023-09-29T22:55:05.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c540ef9bbf653044-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.385 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.385 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.385 DEBUG:boto:path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.385 DEBUG:boto:auth_path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.385 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.386 DEBUG:boto:Path: /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.386 DEBUG:boto:Data: 2023-09-29T22:55:05.386 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.386 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.386 DEBUG:boto:Params: {} 2023-09-29T22:55:05.386 DEBUG:boto:Token: None 2023-09-29T22:55:05.386 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nz1Oq1sZS0Kx5bXGVl610BeGtB8= 2023-09-29T22:55:05.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nz1Oq1sZS0Kx5bXGVl610BeGtB8=', 'Content-Length': '0'} 2023-09-29T22:55:05.389 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000bdf26a39f472396-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.390 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.390 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.390 DEBUG:boto:path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.390 DEBUG:boto:auth_path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.390 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.390 DEBUG:boto:Path: /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.390 DEBUG:boto:Data: 2023-09-29T22:55:05.391 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.391 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.391 DEBUG:boto:Params: {} 2023-09-29T22:55:05.391 DEBUG:boto:Token: None 2023-09-29T22:55:05.391 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nz1Oq1sZS0Kx5bXGVl610BeGtB8= 2023-09-29T22:55:05.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nz1Oq1sZS0Kx5bXGVl610BeGtB8=', 'Content-Length': '0'} 2023-09-29T22:55:05.394 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e92b7020a10a63d-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:55:05.395 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.395 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.395 DEBUG:boto:path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.395 DEBUG:boto:auth_path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.395 DEBUG:boto:Method: GET 2023-09-29T22:55:05.395 DEBUG:boto:Path: /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.395 DEBUG:boto:Data: 2023-09-29T22:55:05.396 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.396 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.396 DEBUG:boto:Params: {} 2023-09-29T22:55:05.396 DEBUG:boto:Token: None 2023-09-29T22:55:05.397 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8= 2023-09-29T22:55:05.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8=', 'Content-Length': '0'} 2023-09-29T22:55:05.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d91e2490af70341c-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.400 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.400 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.401 DEBUG:boto:path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.401 DEBUG:boto:auth_path=/hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.401 DEBUG:boto:Method: GET 2023-09-29T22:55:05.401 DEBUG:boto:Path: /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.402 DEBUG:boto:Data: 2023-09-29T22:55:05.402 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.402 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.402 DEBUG:boto:Params: {} 2023-09-29T22:55:05.402 DEBUG:boto:Token: None 2023-09-29T22:55:05.403 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44?versionId=null 2023-09-29T22:55:05.403 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8= 2023-09-29T22:55:05.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HjTLuLEYyNVDiPZMSEn1+agIYm8=', 'Content-Length': '0'} 2023-09-29T22:55:05.406 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b14e81306ccfec94-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.450 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.451 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.451 DEBUG:boto:path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.451 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.451 DEBUG:boto:Method: GET 2023-09-29T22:55:05.451 DEBUG:boto:Path: /hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.451 DEBUG:boto:Data: 2023-09-29T22:55:05.451 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.452 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.452 DEBUG:boto:Params: {} 2023-09-29T22:55:05.452 DEBUG:boto:Token: None 2023-09-29T22:55:05.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000525572a6065f7108-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.458 DEBUG:boto:b'hcnphz-60obj441000falseobj442023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.458 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.459 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.459 DEBUG:boto:path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.459 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.459 DEBUG:boto:Method: GET 2023-09-29T22:55:05.459 DEBUG:boto:Path: /hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.459 DEBUG:boto:Data: 2023-09-29T22:55:05.459 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.459 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.459 DEBUG:boto:Params: {} 2023-09-29T22:55:05.460 DEBUG:boto:Token: None 2023-09-29T22:55:05.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.460 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.464 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020f649558eb89852-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.465 DEBUG:boto:b'hcnphz-60obj441000falseobj442023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.465 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.466 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.466 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.466 DEBUG:boto:Path: /hcnphz-60/obj44 2023-09-29T22:55:05.466 DEBUG:boto:Data: 2023-09-29T22:55:05.467 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.467 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.467 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.467 DEBUG:boto:Params: {} 2023-09-29T22:55:05.467 DEBUG:boto:Token: None 2023-09-29T22:55:05.468 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44 2023-09-29T22:55:05.468 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JbSmtUZXscTQrCEA2pLAaLYB6VI= 2023-09-29T22:55:05.468 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JbSmtUZXscTQrCEA2pLAaLYB6VI=', 'Content-Length': '0'} 2023-09-29T22:55:05.470 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030e20d58ce2d1c2a-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.470 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.471 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.471 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.471 DEBUG:boto:Path: /hcnphz-60/obj44 2023-09-29T22:55:05.471 DEBUG:boto:Data: 2023-09-29T22:55:05.471 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.471 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.471 DEBUG:boto:Params: {} 2023-09-29T22:55:05.471 DEBUG:boto:Token: None 2023-09-29T22:55:05.472 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44 2023-09-29T22:55:05.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JbSmtUZXscTQrCEA2pLAaLYB6VI= 2023-09-29T22:55:05.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JbSmtUZXscTQrCEA2pLAaLYB6VI=', 'Content-Length': '0'} 2023-09-29T22:55:05.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058e3545c05e2d85d-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:55:05.475 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.475 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.475 DEBUG:boto:Method: GET 2023-09-29T22:55:05.475 DEBUG:boto:Path: /hcnphz-60/obj44 2023-09-29T22:55:05.475 DEBUG:boto:Data: 2023-09-29T22:55:05.475 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.476 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.476 DEBUG:boto:Params: {} 2023-09-29T22:55:05.476 DEBUG:boto:Token: None 2023-09-29T22:55:05.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44 2023-09-29T22:55:05.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7kLhCcuGfuHzmlUn8PNICaEyu40= 2023-09-29T22:55:05.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7kLhCcuGfuHzmlUn8PNICaEyu40=', 'Content-Length': '0'} 2023-09-29T22:55:05.479 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a7a8c3a0d2e6670e-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.479 DEBUG:boto:path=/hcnphz-60/obj44 2023-09-29T22:55:05.479 DEBUG:boto:auth_path=/hcnphz-60/obj44 2023-09-29T22:55:05.479 DEBUG:boto:Method: GET 2023-09-29T22:55:05.479 DEBUG:boto:Path: /hcnphz-60/obj44 2023-09-29T22:55:05.479 DEBUG:boto:Data: 2023-09-29T22:55:05.480 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.480 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.480 DEBUG:boto:Params: {} 2023-09-29T22:55:05.480 DEBUG:boto:Token: None 2023-09-29T22:55:05.480 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj44 2023-09-29T22:55:05.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7kLhCcuGfuHzmlUn8PNICaEyu40= 2023-09-29T22:55:05.481 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7kLhCcuGfuHzmlUn8PNICaEyu40=', 'Content-Length': '0'} 2023-09-29T22:55:05.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008abed89d44054e3d-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.526 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.527 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.527 DEBUG:boto:path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.527 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.527 DEBUG:boto:Method: GET 2023-09-29T22:55:05.527 DEBUG:boto:Path: /hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.527 DEBUG:boto:Data: 2023-09-29T22:55:05.527 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.527 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.528 DEBUG:boto:Params: {} 2023-09-29T22:55:05.528 DEBUG:boto:Token: None 2023-09-29T22:55:05.528 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.532 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013cd63fb5388ef29-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.533 DEBUG:boto:b'hcnphz-60obj441000falseobj442023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.533 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.533 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.534 DEBUG:boto:path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.534 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.534 DEBUG:boto:Method: GET 2023-09-29T22:55:05.534 DEBUG:boto:Path: /hcnphz-60/?prefix=obj44 2023-09-29T22:55:05.534 DEBUG:boto:Data: 2023-09-29T22:55:05.534 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.534 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.534 DEBUG:boto:Params: {} 2023-09-29T22:55:05.535 DEBUG:boto:Token: None 2023-09-29T22:55:05.535 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.540 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000046df9ff96b8239ea-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.540 DEBUG:boto:b'hcnphz-60obj441000falseobj442023-09-29T22:51:38.163Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.540 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:55:05.541 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.541 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.541 DEBUG:boto:path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.541 DEBUG:boto:auth_path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.541 DEBUG:boto:Method: GET 2023-09-29T22:55:05.541 DEBUG:boto:Path: /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.541 DEBUG:boto:Data: 2023-09-29T22:55:05.542 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.542 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.542 DEBUG:boto:Params: {} 2023-09-29T22:55:05.542 DEBUG:boto:Token: None 2023-09-29T22:55:05.542 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U= 2023-09-29T22:55:05.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U=', 'Content-Length': '0'} 2023-09-29T22:55:05.545 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d51ef0a9f01bfa3c-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.586 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.587 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.587 DEBUG:boto:path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.587 DEBUG:boto:auth_path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.587 DEBUG:boto:Method: GET 2023-09-29T22:55:05.587 DEBUG:boto:Path: /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.587 DEBUG:boto:Data: 2023-09-29T22:55:05.588 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.588 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.588 DEBUG:boto:Params: {} 2023-09-29T22:55:05.588 DEBUG:boto:Token: None 2023-09-29T22:55:05.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U= 2023-09-29T22:55:05.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U=', 'Content-Length': '0'} 2023-09-29T22:55:05.592 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003528d0a54091ba2a-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.592 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.592 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.593 DEBUG:boto:path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.593 DEBUG:boto:auth_path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.593 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.593 DEBUG:boto:Path: /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.594 DEBUG:boto:Data: 2023-09-29T22:55:05.594 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.594 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.594 DEBUG:boto:Params: {} 2023-09-29T22:55:05.595 DEBUG:boto:Token: None 2023-09-29T22:55:05.595 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.595 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xc5izahC43kgdOVzFYK0ebR9T2s= 2023-09-29T22:55:05.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xc5izahC43kgdOVzFYK0ebR9T2s=', 'Content-Length': '0'} 2023-09-29T22:55:05.598 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa91008121303d16-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.598 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.598 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.598 DEBUG:boto:path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.599 DEBUG:boto:auth_path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.599 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.599 DEBUG:boto:Path: /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.599 DEBUG:boto:Data: 2023-09-29T22:55:05.599 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.599 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.599 DEBUG:boto:Params: {} 2023-09-29T22:55:05.600 DEBUG:boto:Token: None 2023-09-29T22:55:05.600 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xc5izahC43kgdOVzFYK0ebR9T2s= 2023-09-29T22:55:05.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xc5izahC43kgdOVzFYK0ebR9T2s=', 'Content-Length': '0'} 2023-09-29T22:55:05.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb70c19a61adda35-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:55:05.603 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.604 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.604 DEBUG:boto:path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.604 DEBUG:boto:auth_path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.604 DEBUG:boto:Method: GET 2023-09-29T22:55:05.604 DEBUG:boto:Path: /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.604 DEBUG:boto:Data: 2023-09-29T22:55:05.604 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.605 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.605 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.605 DEBUG:boto:Params: {} 2023-09-29T22:55:05.605 DEBUG:boto:Token: None 2023-09-29T22:55:05.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.605 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U= 2023-09-29T22:55:05.606 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U=', 'Content-Length': '0'} 2023-09-29T22:55:05.608 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079a25b50bdaa9d6c-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.609 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.609 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.609 DEBUG:boto:path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.609 DEBUG:boto:auth_path=/hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.609 DEBUG:boto:Method: GET 2023-09-29T22:55:05.609 DEBUG:boto:Path: /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.610 DEBUG:boto:Data: 2023-09-29T22:55:05.610 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.610 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.610 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.610 DEBUG:boto:Params: {} 2023-09-29T22:55:05.610 DEBUG:boto:Token: None 2023-09-29T22:55:05.610 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55?versionId=null 2023-09-29T22:55:05.611 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U= 2023-09-29T22:55:05.611 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oj4KQMDZur/t/DHwIr7/UzkZD6U=', 'Content-Length': '0'} 2023-09-29T22:55:05.613 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007915ac6c9781214e-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.658 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.659 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.659 DEBUG:boto:path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.659 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.659 DEBUG:boto:Method: GET 2023-09-29T22:55:05.659 DEBUG:boto:Path: /hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.659 DEBUG:boto:Data: 2023-09-29T22:55:05.659 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.659 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.659 DEBUG:boto:Params: {} 2023-09-29T22:55:05.660 DEBUG:boto:Token: None 2023-09-29T22:55:05.660 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.660 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.660 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.665 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ea25d24d09a32f3-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.665 DEBUG:boto:b'hcnphz-60obj551000falseobj552023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.665 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.665 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.666 DEBUG:boto:path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.666 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.666 DEBUG:boto:Method: GET 2023-09-29T22:55:05.666 DEBUG:boto:Path: /hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.666 DEBUG:boto:Data: 2023-09-29T22:55:05.666 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.666 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.666 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.666 DEBUG:boto:Params: {} 2023-09-29T22:55:05.667 DEBUG:boto:Token: None 2023-09-29T22:55:05.667 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.667 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.667 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.672 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006d31f0201e51817f-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.672 DEBUG:boto:b'hcnphz-60obj551000falseobj552023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.673 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.673 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.673 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.673 DEBUG:boto:Path: /hcnphz-60/obj55 2023-09-29T22:55:05.673 DEBUG:boto:Data: 2023-09-29T22:55:05.673 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.673 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.674 DEBUG:boto:Params: {} 2023-09-29T22:55:05.674 DEBUG:boto:Token: None 2023-09-29T22:55:05.674 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55 2023-09-29T22:55:05.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4xV+54Uxhr7h7rHBBpZ0nxZbupw= 2023-09-29T22:55:05.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4xV+54Uxhr7h7rHBBpZ0nxZbupw=', 'Content-Length': '0'} 2023-09-29T22:55:05.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051a88d275af55e46-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.677 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.677 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.677 DEBUG:boto:Method: HEAD 2023-09-29T22:55:05.677 DEBUG:boto:Path: /hcnphz-60/obj55 2023-09-29T22:55:05.677 DEBUG:boto:Data: 2023-09-29T22:55:05.677 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.677 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.678 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.678 DEBUG:boto:Params: {} 2023-09-29T22:55:05.678 DEBUG:boto:Token: None 2023-09-29T22:55:05.678 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55 2023-09-29T22:55:05.678 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4xV+54Uxhr7h7rHBBpZ0nxZbupw= 2023-09-29T22:55:05.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4xV+54Uxhr7h7rHBBpZ0nxZbupw=', 'Content-Length': '0'} 2023-09-29T22:55:05.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004673fbd53766196e-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:55:05.681 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.681 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.681 DEBUG:boto:Method: GET 2023-09-29T22:55:05.681 DEBUG:boto:Path: /hcnphz-60/obj55 2023-09-29T22:55:05.681 DEBUG:boto:Data: 2023-09-29T22:55:05.682 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.682 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.682 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.682 DEBUG:boto:Params: {} 2023-09-29T22:55:05.682 DEBUG:boto:Token: None 2023-09-29T22:55:05.682 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55 2023-09-29T22:55:05.682 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MuEoF+oXJd43oVylUTW5V5T8AiI= 2023-09-29T22:55:05.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MuEoF+oXJd43oVylUTW5V5T8AiI=', 'Content-Length': '0'} 2023-09-29T22:55:05.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-60:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.25'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9f5036a0194f629-00651755c9-1e1c-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.685 DEBUG:boto:path=/hcnphz-60/obj55 2023-09-29T22:55:05.685 DEBUG:boto:auth_path=/hcnphz-60/obj55 2023-09-29T22:55:05.685 DEBUG:boto:Method: GET 2023-09-29T22:55:05.685 DEBUG:boto:Path: /hcnphz-60/obj55 2023-09-29T22:55:05.685 DEBUG:boto:Data: 2023-09-29T22:55:05.685 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.686 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.686 DEBUG:boto:Params: {} 2023-09-29T22:55:05.686 DEBUG:boto:Token: None 2023-09-29T22:55:05.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/obj55 2023-09-29T22:55:05.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MuEoF+oXJd43oVylUTW5V5T8AiI= 2023-09-29T22:55:05.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MuEoF+oXJd43oVylUTW5V5T8AiI=', 'Content-Length': '0'} 2023-09-29T22:55:05.689 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:51:38 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cebd7540b5da030-00651755c9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.689 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.689 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.690 DEBUG:boto:path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.690 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.690 DEBUG:boto:Method: GET 2023-09-29T22:55:05.690 DEBUG:boto:Path: /hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.690 DEBUG:boto:Data: 2023-09-29T22:55:05.690 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:05.691 DEBUG:boto:Port: 8001 2023-09-29T22:55:05.691 DEBUG:boto:Params: {} 2023-09-29T22:55:05.691 DEBUG:boto:Token: None 2023-09-29T22:55:05.691 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.691 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.695 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000059a32000628bb65-00651755c9-1e1c-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.696 DEBUG:boto:b'hcnphz-60obj551000falseobj552023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.696 DEBUG:boto:path=/hcnphz-60/ 2023-09-29T22:55:05.696 DEBUG:boto:auth_path=/hcnphz-60/ 2023-09-29T22:55:05.696 DEBUG:boto:path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.697 DEBUG:boto:auth_path=/hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.697 DEBUG:boto:Method: GET 2023-09-29T22:55:05.697 DEBUG:boto:Path: /hcnphz-60/?prefix=obj55 2023-09-29T22:55:05.697 DEBUG:boto:Data: 2023-09-29T22:55:05.697 DEBUG:boto:Headers: {} 2023-09-29T22:55:05.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:05.697 DEBUG:boto:Port: 8000 2023-09-29T22:55:05.697 DEBUG:boto:Params: {} 2023-09-29T22:55:05.697 DEBUG:boto:Token: None 2023-09-29T22:55:05.698 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:55:05 GMT /hcnphz-60/ 2023-09-29T22:55:05.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ= 2023-09-29T22:55:05.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+5PEc0pHOdJBva4E8SDzLE2XJXQ=', 'Content-Length': '0'} 2023-09-29T22:55:05.702 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ff2d742e05be190-00651755c9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:55:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:05.702 DEBUG:boto:b'hcnphz-60obj551000falseobj552023-09-29T22:51:38.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:55:05.703 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-60 zones={a2, a1} 2023-09-29T22:55:05.703 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:55:05.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:05.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:05.703 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:05.744 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.745 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.845 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000030737", 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:51:38.167764Z" 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000027628", 2023-09-29T22:55:05.846 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:51:41.636851Z" 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000025975", 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:51:41.640556Z" 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000018331", 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:48:09.290769Z" 2023-09-29T22:55:05.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:05.848 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:55:05.849 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000030737', 1: '00000000000000000000:00000000000000027628', 2: '00000000000000000000:00000000000000025975', 3: '00000000000000000000:00000000000000018331'} 2023-09-29T22:55:05.849 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:55:05.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:05.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:05.850 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:05.897 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:05.897 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:05.986 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:55:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:55:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:55:05.986 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 8560915926801402562 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:05.987 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000030737", 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 51, 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:51:38.167764Z" 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000027628", 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:51:41.636851Z" 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:05.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000025975", 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:05.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:51:41.640556Z" 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000018331", 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:05.991 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:48:09.290769Z" 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 60, 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 60 2023-09-29T22:55:05.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:05.993 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:55:05Z" 2023-09-29T22:55:05.993 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:55:05.994 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 8560915926801402562 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000030737", "next_step_marker": "", "total_entries": 51, "pos": 0, "timestamp": "2023-09-29T22:51:38.167764Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000027628", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T22:51:41.636851Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000025975", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:51:41.640556Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000018331", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:48:09.290769Z" } } ] }, "full_sync": { "total": 60, "complete": 60 }, "current_time": "2023-09-29T22:55:05Z" } 2023-09-29T22:55:05.994 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000030737', 'next_step_marker': '', 'total_entries': 51, 'pos': 0, 'timestamp': '2023-09-29T22:51:38.167764Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000027628', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T22:51:41.636851Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000025975', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:51:41.640556Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000018331', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:48:09.290769Z'}}] 2023-09-29T22:55:05.994 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000030737', 1: '00000000000000000000:00000000000000027628', 2: '00000000000000000000:00000000000000025975', 3: '00000000000000000000:00000000000000018331'} 2023-09-29T22:55:05.994 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000030737', 1: '00000000000000000000:00000000000000027628', 2: '00000000000000000000:00000000000000025975', 3: '00000000000000000000:00000000000000018331'} 2023-09-29T22:55:05.994 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:55:05.994 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:55:05.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:05.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:05.995 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:06.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:06.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:06.133 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:55:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014883", 2023-09-29T22:55:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:53:39.206430Z" 2023-09-29T22:55:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.134 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013194", 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:53:50.502744Z" 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000010687", 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:53:50.506126Z" 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005022", 2023-09-29T22:55:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:49:46.598219Z" 2023-09-29T22:55:06.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.136 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:55:06.137 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000014883', 1: '00000000000000000000:00000000000000013194', 2: '00000000000000000000:00000000000000010687', 3: '00000000000000000000:00000000000000005022'} 2023-09-29T22:55:06.137 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:55:06.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:06.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:06.138 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:06.186 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:06.186 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:06.264 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:55:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:55:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:55:06.264 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014883", 2023-09-29T22:55:06.265 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:53:39.206430Z" 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:55:06.266 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013194", 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:53:50.502744Z" 2023-09-29T22:55:06.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000010687", 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.268 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:53:50.506126Z" 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:55:06.269 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005022", 2023-09-29T22:55:06.270 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.270 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.270 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.270 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:49:46.598219Z" 2023-09-29T22:55:06.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.270 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:55:06Z" 2023-09-29T22:55:06.271 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:55:06.273 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014883", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:53:39.206430Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013194", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:53:50.502744Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000010687", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:53:50.506126Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005022", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:49:46.598219Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:55:06Z" } 2023-09-29T22:55:06.273 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014883', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:53:39.206430Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013194', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:53:50.502744Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000010687', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:53:50.506126Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005022', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:49:46.598219Z'}}] 2023-09-29T22:55:06.273 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000014883', 1: '00000000000000000000:00000000000000013194', 2: '00000000000000000000:00000000000000010687', 3: '00000000000000000000:00000000000000005022'} 2023-09-29T22:55:06.274 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000014883', 1: '00000000000000000000:00000000000000013194', 2: '00000000000000000000:00000000000000010687', 3: '00000000000000000000:00000000000000005022'} 2023-09-29T22:55:06.274 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:55:06.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_full_sync_after_data_sync_init ... ok 2023-09-29T22:55:06.275 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:06.275 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:06.275 DEBUG:boto:Method: PUT 2023-09-29T22:55:06.275 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:06.276 DEBUG:boto:Data: 2023-09-29T22:55:06.276 DEBUG:boto:Headers: {} 2023-09-29T22:55:06.276 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:06.276 DEBUG:boto:Port: 8000 2023-09-29T22:55:06.276 DEBUG:boto:Params: {} 2023-09-29T22:55:06.276 DEBUG:boto:Token: None 2023-09-29T22:55:06.277 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:55:06 GMT /hcnphz-61/ 2023-09-29T22:55:06.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7LDYXFO+i+lftI735miGb53Jfk0= 2023-09-29T22:55:06.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:06 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7LDYXFO+i+lftI735miGb53Jfk0=', 'Content-Length': '0'} 2023-09-29T22:55:06.305 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000fa4e0908c9320a7c-00651755ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:06 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:06.306 DEBUG:rgw_multi.tests:created bucket=hcnphz-61 2023-09-29T22:55:06.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:06.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:06.306 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:06.347 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:06.349 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:06.403 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:55:06.432 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:55:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:55:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T22:55:06.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028106.305142_555.1", 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:55:06.305142Z" 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:55:06.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:55:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:55:06.434 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.434 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:55:06.435 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028106.305142_555.1', 2: '', 3: ''} 2023-09-29T22:55:06.435 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:55:06.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:06.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:55:06.436 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:55:06.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:06.518 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:06.572 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:06.572+0000 7f3d4c64bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:06.574 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:06.572+0000 7f3d4c64bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:55:06.601 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:55:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:55:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:55:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:55:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:55:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:55:06.602 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:55:06.603 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028106.305142_555.1", 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:55:06.305142Z", 2023-09-29T22:55:06.605 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:55:06.606 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:55:06.608 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:55:06.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:55:06.610 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:55:06Z" 2023-09-29T22:55:06.610 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:55:06.611 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028106.305142_555.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:06.305142Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:55:06Z" } 2023-09-29T22:55:06.611 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028106.305142_555.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:06.305142Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:55:06.611 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028106.305142_555.1', 2: '', 3: ''} 2023-09-29T22:55:06.611 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028106.305142_555.1', 2: '', 3: ''} 2023-09-29T22:55:06.611 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:55:06.612 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T22:55:06.612 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T22:55:06.613 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:55:06.612+0000 7f97b5dde700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 75746) UID: 0 2023-09-29T22:55:06.613 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T22:55:06.612+0000 7f97b963df80 -1 shutting down 2023-09-29T22:55:10.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T22:55:12.713 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T22:55:12.713 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:12.713 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:12.714 DEBUG:boto:Method: HEAD 2023-09-29T22:55:12.714 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:12.714 DEBUG:boto:Data: 2023-09-29T22:55:12.714 DEBUG:boto:Headers: {} 2023-09-29T22:55:12.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.714 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.714 DEBUG:boto:Params: {} 2023-09-29T22:55:12.714 DEBUG:boto:Token: None 2023-09-29T22:55:12.715 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/ 2023-09-29T22:55:12.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0= 2023-09-29T22:55:12.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0=', 'Content-Length': '0'} 2023-09-29T22:55:12.719 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004cb5e27a3f793a8b-00651755d0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.719 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:55:12.719 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:55:12.720 DEBUG:boto:Method: PUT 2023-09-29T22:55:12.720 DEBUG:boto:Path: /hcnphz-61/obj11 2023-09-29T22:55:12.720 DEBUG:boto:Data: 2023-09-29T22:55:12.720 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:12.720 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.720 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.720 DEBUG:boto:Params: {} 2023-09-29T22:55:12.720 DEBUG:boto:Token: None 2023-09-29T22:55:12.721 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/obj11 2023-09-29T22:55:12.721 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UQB8+/uczma1a8L3Eed5tCMNsPE= 2023-09-29T22:55:12.721 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UQB8+/uczma1a8L3Eed5tCMNsPE='} 2023-09-29T22:55:12.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e03e2a16a1856627-00651755d0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.767 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:12.767 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:12.767 DEBUG:boto:Method: HEAD 2023-09-29T22:55:12.767 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:12.767 DEBUG:boto:Data: 2023-09-29T22:55:12.767 DEBUG:boto:Headers: {} 2023-09-29T22:55:12.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.768 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.768 DEBUG:boto:Params: {} 2023-09-29T22:55:12.768 DEBUG:boto:Token: None 2023-09-29T22:55:12.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/ 2023-09-29T22:55:12.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0= 2023-09-29T22:55:12.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0=', 'Content-Length': '0'} 2023-09-29T22:55:12.772 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000985b4f2543e77929-00651755d0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.772 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:55:12.772 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:55:12.773 DEBUG:boto:Method: PUT 2023-09-29T22:55:12.773 DEBUG:boto:Path: /hcnphz-61/obj22 2023-09-29T22:55:12.773 DEBUG:boto:Data: 2023-09-29T22:55:12.773 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:12.773 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.773 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.773 DEBUG:boto:Params: {} 2023-09-29T22:55:12.773 DEBUG:boto:Token: None 2023-09-29T22:55:12.773 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/obj22 2023-09-29T22:55:12.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eF8kh2QSGBtsK+sHLg7Ui2Fki+s= 2023-09-29T22:55:12.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eF8kh2QSGBtsK+sHLg7Ui2Fki+s='} 2023-09-29T22:55:12.782 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000422155d0ac8dca32-00651755d0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.782 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:12.782 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:12.782 DEBUG:boto:Method: HEAD 2023-09-29T22:55:12.782 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:12.783 DEBUG:boto:Data: 2023-09-29T22:55:12.783 DEBUG:boto:Headers: {} 2023-09-29T22:55:12.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.783 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.783 DEBUG:boto:Params: {} 2023-09-29T22:55:12.783 DEBUG:boto:Token: None 2023-09-29T22:55:12.783 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/ 2023-09-29T22:55:12.783 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0= 2023-09-29T22:55:12.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0=', 'Content-Length': '0'} 2023-09-29T22:55:12.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002d9c197190252163-00651755d0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.787 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:55:12.788 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:55:12.788 DEBUG:boto:Method: PUT 2023-09-29T22:55:12.788 DEBUG:boto:Path: /hcnphz-61/obj33 2023-09-29T22:55:12.788 DEBUG:boto:Data: 2023-09-29T22:55:12.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:12.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.788 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.788 DEBUG:boto:Params: {} 2023-09-29T22:55:12.788 DEBUG:boto:Token: None 2023-09-29T22:55:12.789 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/obj33 2023-09-29T22:55:12.789 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yuvjwLHeN+4gLtPVkd+wzh7ca5Y= 2023-09-29T22:55:12.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yuvjwLHeN+4gLtPVkd+wzh7ca5Y='} 2023-09-29T22:55:12.835 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a10ef5aab60bf52d-00651755d0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.835 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:12.835 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:12.835 DEBUG:boto:Method: HEAD 2023-09-29T22:55:12.835 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:12.835 DEBUG:boto:Data: 2023-09-29T22:55:12.835 DEBUG:boto:Headers: {} 2023-09-29T22:55:12.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.836 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.836 DEBUG:boto:Params: {} 2023-09-29T22:55:12.836 DEBUG:boto:Token: None 2023-09-29T22:55:12.836 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/ 2023-09-29T22:55:12.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0= 2023-09-29T22:55:12.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0=', 'Content-Length': '0'} 2023-09-29T22:55:12.840 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000652f975c9a37c645-00651755d0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.840 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:55:12.841 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:55:12.841 DEBUG:boto:Method: PUT 2023-09-29T22:55:12.841 DEBUG:boto:Path: /hcnphz-61/obj44 2023-09-29T22:55:12.841 DEBUG:boto:Data: 2023-09-29T22:55:12.841 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:12.841 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.841 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.841 DEBUG:boto:Params: {} 2023-09-29T22:55:12.842 DEBUG:boto:Token: None 2023-09-29T22:55:12.842 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/obj44 2023-09-29T22:55:12.842 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ySggG/Sf9Q0bQazCGnamZfyF/64= 2023-09-29T22:55:12.842 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ySggG/Sf9Q0bQazCGnamZfyF/64='} 2023-09-29T22:55:12.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000003a2dc4463cabd08-00651755d0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.854 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:12.854 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:12.855 DEBUG:boto:Method: HEAD 2023-09-29T22:55:12.855 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:12.855 DEBUG:boto:Data: 2023-09-29T22:55:12.855 DEBUG:boto:Headers: {} 2023-09-29T22:55:12.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.855 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.855 DEBUG:boto:Params: {} 2023-09-29T22:55:12.855 DEBUG:boto:Token: None 2023-09-29T22:55:12.856 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/ 2023-09-29T22:55:12.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0= 2023-09-29T22:55:12.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VLrAh63fPQfE8gi1yUADHNBgjr0=', 'Content-Length': '0'} 2023-09-29T22:55:12.859 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000077592cda21bc47ad-00651755d0-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.860 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:55:12.860 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:55:12.860 DEBUG:boto:Method: PUT 2023-09-29T22:55:12.860 DEBUG:boto:Path: /hcnphz-61/obj55 2023-09-29T22:55:12.860 DEBUG:boto:Data: 2023-09-29T22:55:12.860 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:12.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:12.860 DEBUG:boto:Port: 8000 2023-09-29T22:55:12.861 DEBUG:boto:Params: {} 2023-09-29T22:55:12.861 DEBUG:boto:Token: None 2023-09-29T22:55:12.861 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:12 GMT /hcnphz-61/obj55 2023-09-29T22:55:12.861 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mkkWIhNi18w7XkkPokuTRUiJUrQ= 2023-09-29T22:55:12.861 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:12 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mkkWIhNi18w7XkkPokuTRUiJUrQ='} 2023-09-29T22:55:12.906 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ead838a3077f8b24-00651755d0-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:12.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:12.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:12.907 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-61 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:12.955 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:12.956 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:13.038 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:55:13.038 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-61 2023-09-29T22:55:13.046 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-09-29T22:55:13.118 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.118 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.119 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.119 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.119 DEBUG:boto:Data: 2023-09-29T22:55:13.119 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.120 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.120 DEBUG:boto:Params: {} 2023-09-29T22:55:13.120 DEBUG:boto:Token: None 2023-09-29T22:55:13.120 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.124 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d2b00bfe7d0a1baa-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.124 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:55:13.124 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:55:13.124 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.125 DEBUG:boto:Path: /hcnphz-61/obj17 2023-09-29T22:55:13.125 DEBUG:boto:Data: 2023-09-29T22:55:13.125 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.125 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.125 DEBUG:boto:Params: {} 2023-09-29T22:55:13.125 DEBUG:boto:Token: None 2023-09-29T22:55:13.125 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj17 2023-09-29T22:55:13.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Uy46nx9URnAz4v5/eTnixwlEkI= 2023-09-29T22:55:13.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Uy46nx9URnAz4v5/eTnixwlEkI='} 2023-09-29T22:55:13.135 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009f4143bdee8a3dd1-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.136 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.136 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.136 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.136 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.136 DEBUG:boto:Data: 2023-09-29T22:55:13.137 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.137 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.137 DEBUG:boto:Params: {} 2023-09-29T22:55:13.137 DEBUG:boto:Token: None 2023-09-29T22:55:13.137 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.137 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.141 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003eed09fb7ca1590a-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.142 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:55:13.142 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:55:13.142 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.142 DEBUG:boto:Path: /hcnphz-61/obj34 2023-09-29T22:55:13.143 DEBUG:boto:Data: 2023-09-29T22:55:13.143 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.143 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.143 DEBUG:boto:Params: {} 2023-09-29T22:55:13.143 DEBUG:boto:Token: None 2023-09-29T22:55:13.144 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj34 2023-09-29T22:55:13.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rdqfK3o8Niwmf4W11Hi71jyD1Y4= 2023-09-29T22:55:13.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rdqfK3o8Niwmf4W11Hi71jyD1Y4='} 2023-09-29T22:55:13.186 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3517c91a84fcf64-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.187 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.187 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.187 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.187 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.187 DEBUG:boto:Data: 2023-09-29T22:55:13.187 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.188 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.188 DEBUG:boto:Params: {} 2023-09-29T22:55:13.188 DEBUG:boto:Token: None 2023-09-29T22:55:13.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.193 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001939432a17ec43a9-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.193 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:55:13.193 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:55:13.193 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.194 DEBUG:boto:Path: /hcnphz-61/obj51 2023-09-29T22:55:13.194 DEBUG:boto:Data: 2023-09-29T22:55:13.194 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.194 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.194 DEBUG:boto:Params: {} 2023-09-29T22:55:13.195 DEBUG:boto:Token: None 2023-09-29T22:55:13.196 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj51 2023-09-29T22:55:13.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6Z7Qg5vwFYv2stNqj4iClDzIkqc= 2023-09-29T22:55:13.196 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6Z7Qg5vwFYv2stNqj4iClDzIkqc='} 2023-09-29T22:55:13.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba2921d45c365111-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.206 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.206 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.207 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.207 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.207 DEBUG:boto:Data: 2023-09-29T22:55:13.207 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.207 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.207 DEBUG:boto:Params: {} 2023-09-29T22:55:13.208 DEBUG:boto:Token: None 2023-09-29T22:55:13.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.212 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd15860febdf1be-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.212 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:55:13.212 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:55:13.213 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.213 DEBUG:boto:Path: /hcnphz-61/obj68 2023-09-29T22:55:13.213 DEBUG:boto:Data: 2023-09-29T22:55:13.213 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.213 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.213 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.213 DEBUG:boto:Params: {} 2023-09-29T22:55:13.213 DEBUG:boto:Token: None 2023-09-29T22:55:13.214 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj68 2023-09-29T22:55:13.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k3DdRs16ssB75Huac+qhWMbwn8U= 2023-09-29T22:55:13.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k3DdRs16ssB75Huac+qhWMbwn8U='} 2023-09-29T22:55:13.259 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4f8ef28b259e3ac-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.259 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.259 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.259 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.259 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.259 DEBUG:boto:Data: 2023-09-29T22:55:13.260 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.260 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.260 DEBUG:boto:Params: {} 2023-09-29T22:55:13.260 DEBUG:boto:Token: None 2023-09-29T22:55:13.260 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.260 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.260 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.265 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c107d323cd51893f-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.265 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:55:13.266 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:55:13.266 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.266 DEBUG:boto:Path: /hcnphz-61/obj85 2023-09-29T22:55:13.266 DEBUG:boto:Data: 2023-09-29T22:55:13.266 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.266 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.266 DEBUG:boto:Params: {} 2023-09-29T22:55:13.267 DEBUG:boto:Token: None 2023-09-29T22:55:13.267 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj85 2023-09-29T22:55:13.267 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IqyfK+krnD7RuJOvpZK6Lavu1Vc= 2023-09-29T22:55:13.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IqyfK+krnD7RuJOvpZK6Lavu1Vc='} 2023-09-29T22:55:13.277 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fa371f4808f4b59a-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:13.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:13.278 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-61 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:13.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:13.324 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:13.407 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:55:13.407 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-61 2023-09-29T22:55:13.420 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-09-29T22:55:13.517 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.517 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.517 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.517 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.517 DEBUG:boto:Data: 2023-09-29T22:55:13.517 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.517 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.518 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.518 DEBUG:boto:Params: {} 2023-09-29T22:55:13.518 DEBUG:boto:Token: None 2023-09-29T22:55:13.518 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.518 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.518 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.525 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000078e304869cf22844-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.526 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:55:13.526 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:55:13.526 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.526 DEBUG:boto:Path: /hcnphz-61/obj19 2023-09-29T22:55:13.527 DEBUG:boto:Data: 2023-09-29T22:55:13.527 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.527 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.527 DEBUG:boto:Params: {} 2023-09-29T22:55:13.527 DEBUG:boto:Token: None 2023-09-29T22:55:13.527 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj19 2023-09-29T22:55:13.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AnoDFudN5U++uh8DdqXUI9f85Y8= 2023-09-29T22:55:13.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AnoDFudN5U++uh8DdqXUI9f85Y8='} 2023-09-29T22:55:13.571 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000007f51df6bb301690-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.571 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.571 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.572 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.572 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.572 DEBUG:boto:Data: 2023-09-29T22:55:13.572 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.573 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.573 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.573 DEBUG:boto:Params: {} 2023-09-29T22:55:13.573 DEBUG:boto:Token: None 2023-09-29T22:55:13.574 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.574 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.580 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000033b85d2b4af8e96c-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.581 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:55:13.581 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:55:13.581 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.581 DEBUG:boto:Path: /hcnphz-61/obj38 2023-09-29T22:55:13.581 DEBUG:boto:Data: 2023-09-29T22:55:13.581 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.582 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.582 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.582 DEBUG:boto:Params: {} 2023-09-29T22:55:13.582 DEBUG:boto:Token: None 2023-09-29T22:55:13.582 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj38 2023-09-29T22:55:13.582 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v6PrZfq3mdWLGcZ3o4lPc6d8nuk= 2023-09-29T22:55:13.582 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v6PrZfq3mdWLGcZ3o4lPc6d8nuk='} 2023-09-29T22:55:13.590 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a87516369c4d4b7-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.590 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.590 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.590 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.591 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.591 DEBUG:boto:Data: 2023-09-29T22:55:13.591 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.591 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.591 DEBUG:boto:Params: {} 2023-09-29T22:55:13.591 DEBUG:boto:Token: None 2023-09-29T22:55:13.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.595 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047b3edf94cca851e-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.596 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:55:13.596 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:55:13.596 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.596 DEBUG:boto:Path: /hcnphz-61/obj57 2023-09-29T22:55:13.596 DEBUG:boto:Data: 2023-09-29T22:55:13.596 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.597 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.597 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.597 DEBUG:boto:Params: {} 2023-09-29T22:55:13.597 DEBUG:boto:Token: None 2023-09-29T22:55:13.597 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj57 2023-09-29T22:55:13.597 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MJqWFrrLC7VvWY9CMN0vtxSX7MA= 2023-09-29T22:55:13.597 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MJqWFrrLC7VvWY9CMN0vtxSX7MA='} 2023-09-29T22:55:13.643 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003338762432c74a3d-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.643 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.643 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.643 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.644 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.644 DEBUG:boto:Data: 2023-09-29T22:55:13.644 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.644 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.644 DEBUG:boto:Params: {} 2023-09-29T22:55:13.645 DEBUG:boto:Token: None 2023-09-29T22:55:13.645 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.645 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.645 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.650 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000617850d3be2543db-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.651 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:55:13.651 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:55:13.651 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.652 DEBUG:boto:Path: /hcnphz-61/obj76 2023-09-29T22:55:13.652 DEBUG:boto:Data: 2023-09-29T22:55:13.652 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.652 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.652 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.652 DEBUG:boto:Params: {} 2023-09-29T22:55:13.653 DEBUG:boto:Token: None 2023-09-29T22:55:13.653 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj76 2023-09-29T22:55:13.653 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2Rhm2nMQehMmyzthGfHLNEHQRc0= 2023-09-29T22:55:13.654 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2Rhm2nMQehMmyzthGfHLNEHQRc0='} 2023-09-29T22:55:13.664 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093b610cadd3b1ebc-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.664 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.664 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.664 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.664 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.665 DEBUG:boto:Data: 2023-09-29T22:55:13.665 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.665 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.665 DEBUG:boto:Params: {} 2023-09-29T22:55:13.665 DEBUG:boto:Token: None 2023-09-29T22:55:13.665 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.665 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.670 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057ca78aafc47c313-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.670 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:55:13.670 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:55:13.670 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.670 DEBUG:boto:Path: /hcnphz-61/obj95 2023-09-29T22:55:13.671 DEBUG:boto:Data: 2023-09-29T22:55:13.671 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.671 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.671 DEBUG:boto:Params: {} 2023-09-29T22:55:13.671 DEBUG:boto:Token: None 2023-09-29T22:55:13.671 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj95 2023-09-29T22:55:13.671 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dUKf0tyn2mHHYnFAeQDMjAJlseE= 2023-09-29T22:55:13.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dUKf0tyn2mHHYnFAeQDMjAJlseE='} 2023-09-29T22:55:13.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f4bcd4355f4485b-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:13.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:13.715 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-61 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:13.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:13.767 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:13.846 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:55:13.847 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-61 2023-09-29T22:55:13.861 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-09-29T22:55:13.961 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.961 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.961 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.961 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.962 DEBUG:boto:Data: 2023-09-29T22:55:13.962 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.962 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.962 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.962 DEBUG:boto:Params: {} 2023-09-29T22:55:13.962 DEBUG:boto:Token: None 2023-09-29T22:55:13.962 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.963 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.963 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.968 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005170388977600e60-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.968 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:55:13.968 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:55:13.969 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.969 DEBUG:boto:Path: /hcnphz-61/obj23 2023-09-29T22:55:13.969 DEBUG:boto:Data: 2023-09-29T22:55:13.969 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.969 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.969 DEBUG:boto:Params: {} 2023-09-29T22:55:13.969 DEBUG:boto:Token: None 2023-09-29T22:55:13.969 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj23 2023-09-29T22:55:13.970 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ycnsarjQo75cewR8a2erMTjINE0= 2023-09-29T22:55:13.970 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ycnsarjQo75cewR8a2erMTjINE0='} 2023-09-29T22:55:13.979 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000130d891f6f1516cc-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.979 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:13.980 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:13.980 DEBUG:boto:Method: HEAD 2023-09-29T22:55:13.980 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:13.980 DEBUG:boto:Data: 2023-09-29T22:55:13.980 DEBUG:boto:Headers: {} 2023-09-29T22:55:13.980 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.980 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.980 DEBUG:boto:Params: {} 2023-09-29T22:55:13.981 DEBUG:boto:Token: None 2023-09-29T22:55:13.981 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/ 2023-09-29T22:55:13.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM= 2023-09-29T22:55:13.981 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dIruEPD25Xz6eB9w5fwweLaRvuM=', 'Content-Length': '0'} 2023-09-29T22:55:13.986 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021903d3d8308d200-00651755d1-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:13.986 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:55:13.986 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:55:13.986 DEBUG:boto:Method: PUT 2023-09-29T22:55:13.986 DEBUG:boto:Path: /hcnphz-61/obj46 2023-09-29T22:55:13.987 DEBUG:boto:Data: 2023-09-29T22:55:13.987 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:13.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:13.987 DEBUG:boto:Port: 8000 2023-09-29T22:55:13.987 DEBUG:boto:Params: {} 2023-09-29T22:55:13.988 DEBUG:boto:Token: None 2023-09-29T22:55:13.988 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:13 GMT /hcnphz-61/obj46 2023-09-29T22:55:13.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WPp6nMB4uyFrrLzwrr4vEo6QGB0= 2023-09-29T22:55:13.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:13 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WPp6nMB4uyFrrLzwrr4vEo6QGB0='} 2023-09-29T22:55:14.030 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ae60c3f4ec41c3e0-00651755d1-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.031 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.031 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.031 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.031 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.031 DEBUG:boto:Data: 2023-09-29T22:55:14.031 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.032 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.032 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.032 DEBUG:boto:Params: {} 2023-09-29T22:55:14.032 DEBUG:boto:Token: None 2023-09-29T22:55:14.032 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.038 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b2d744d05a735a67-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.038 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:55:14.039 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:55:14.039 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.039 DEBUG:boto:Path: /hcnphz-61/obj69 2023-09-29T22:55:14.039 DEBUG:boto:Data: 2023-09-29T22:55:14.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.040 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.040 DEBUG:boto:Params: {} 2023-09-29T22:55:14.040 DEBUG:boto:Token: None 2023-09-29T22:55:14.040 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj69 2023-09-29T22:55:14.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iXgmmt/R2+Gd8Ip7GWEZw5Ed3MY= 2023-09-29T22:55:14.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iXgmmt/R2+Gd8Ip7GWEZw5Ed3MY='} 2023-09-29T22:55:14.052 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000032540f1c10086903-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.052 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.052 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.052 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.053 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.053 DEBUG:boto:Data: 2023-09-29T22:55:14.053 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.053 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.053 DEBUG:boto:Params: {} 2023-09-29T22:55:14.053 DEBUG:boto:Token: None 2023-09-29T22:55:14.053 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.054 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.059 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000007222f718e291446-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.059 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:55:14.059 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:55:14.060 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.060 DEBUG:boto:Path: /hcnphz-61/obj92 2023-09-29T22:55:14.060 DEBUG:boto:Data: 2023-09-29T22:55:14.060 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.060 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.060 DEBUG:boto:Params: {} 2023-09-29T22:55:14.060 DEBUG:boto:Token: None 2023-09-29T22:55:14.061 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj92 2023-09-29T22:55:14.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dmZyxjJZ7+idp4nLRG0PTxxa7Rc= 2023-09-29T22:55:14.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dmZyxjJZ7+idp4nLRG0PTxxa7Rc='} 2023-09-29T22:55:14.106 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f7cd65be3c73b4e2-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.107 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.107 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.107 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.107 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.107 DEBUG:boto:Data: 2023-09-29T22:55:14.107 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.108 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.108 DEBUG:boto:Params: {} 2023-09-29T22:55:14.108 DEBUG:boto:Token: None 2023-09-29T22:55:14.108 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.114 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e47ac8bae03931e7-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.115 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:55:14.115 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:55:14.115 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.115 DEBUG:boto:Path: /hcnphz-61/obj115 2023-09-29T22:55:14.115 DEBUG:boto:Data: 2023-09-29T22:55:14.115 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.115 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.116 DEBUG:boto:Params: {} 2023-09-29T22:55:14.116 DEBUG:boto:Token: None 2023-09-29T22:55:14.116 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj115 2023-09-29T22:55:14.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xTuQk4Cav/lWCl321HiUGiwiKAY= 2023-09-29T22:55:14.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xTuQk4Cav/lWCl321HiUGiwiKAY='} 2023-09-29T22:55:14.126 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ce462d5f09841e88-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:14.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:14.127 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-61 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:14.174 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:14.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:14.248 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:55:14.248 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-61 2023-09-29T22:55:14.264 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-09-29T22:55:14.369 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.370 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.370 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.370 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.370 DEBUG:boto:Data: 2023-09-29T22:55:14.370 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.370 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.371 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.371 DEBUG:boto:Params: {} 2023-09-29T22:55:14.371 DEBUG:boto:Token: None 2023-09-29T22:55:14.371 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.377 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007a75e4eaba3a32c9-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.378 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:55:14.378 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:55:14.378 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.378 DEBUG:boto:Path: /hcnphz-61/obj29 2023-09-29T22:55:14.378 DEBUG:boto:Data: 2023-09-29T22:55:14.378 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.378 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.378 DEBUG:boto:Params: {} 2023-09-29T22:55:14.379 DEBUG:boto:Token: None 2023-09-29T22:55:14.379 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj29 2023-09-29T22:55:14.379 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:d6vu5LPNmqNe+NEyAmyuq9zY/Ek= 2023-09-29T22:55:14.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:d6vu5LPNmqNe+NEyAmyuq9zY/Ek='} 2023-09-29T22:55:14.423 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000041720313c28927dc-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.423 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.423 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.423 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.424 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.424 DEBUG:boto:Data: 2023-09-29T22:55:14.424 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.424 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.425 DEBUG:boto:Params: {} 2023-09-29T22:55:14.425 DEBUG:boto:Token: None 2023-09-29T22:55:14.425 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.425 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.432 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000039d02002b68b8877-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.432 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:55:14.432 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:55:14.432 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.433 DEBUG:boto:Path: /hcnphz-61/obj58 2023-09-29T22:55:14.433 DEBUG:boto:Data: 2023-09-29T22:55:14.433 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.433 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.433 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.433 DEBUG:boto:Params: {} 2023-09-29T22:55:14.434 DEBUG:boto:Token: None 2023-09-29T22:55:14.434 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj58 2023-09-29T22:55:14.434 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h6sm9JK4EHQLpVyDKKf7n68HuqM= 2023-09-29T22:55:14.434 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h6sm9JK4EHQLpVyDKKf7n68HuqM='} 2023-09-29T22:55:14.444 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005ee2f89d12d5ddf4-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.444 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.444 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.444 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.445 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.445 DEBUG:boto:Data: 2023-09-29T22:55:14.445 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.445 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.445 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.446 DEBUG:boto:Params: {} 2023-09-29T22:55:14.446 DEBUG:boto:Token: None 2023-09-29T22:55:14.446 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.447 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.447 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.452 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003e5d2dc2b55ac6a2-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.453 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:55:14.453 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:55:14.453 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.453 DEBUG:boto:Path: /hcnphz-61/obj87 2023-09-29T22:55:14.453 DEBUG:boto:Data: 2023-09-29T22:55:14.453 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.453 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.454 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.454 DEBUG:boto:Params: {} 2023-09-29T22:55:14.454 DEBUG:boto:Token: None 2023-09-29T22:55:14.454 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj87 2023-09-29T22:55:14.454 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cc6MwPCcOEDP/c8ac2X9YTRos/Y= 2023-09-29T22:55:14.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cc6MwPCcOEDP/c8ac2X9YTRos/Y='} 2023-09-29T22:55:14.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006bc67ec53a412abc-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.499 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.499 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.499 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.499 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.499 DEBUG:boto:Data: 2023-09-29T22:55:14.500 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.500 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.500 DEBUG:boto:Params: {} 2023-09-29T22:55:14.500 DEBUG:boto:Token: None 2023-09-29T22:55:14.500 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.500 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.509 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006e6937dbc2013171-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.509 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:55:14.509 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:55:14.509 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.510 DEBUG:boto:Path: /hcnphz-61/obj116 2023-09-29T22:55:14.510 DEBUG:boto:Data: 2023-09-29T22:55:14.510 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.510 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.510 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.510 DEBUG:boto:Params: {} 2023-09-29T22:55:14.510 DEBUG:boto:Token: None 2023-09-29T22:55:14.510 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj116 2023-09-29T22:55:14.511 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KFqloZRH3M8QnM/xMCLTVaWaFpY= 2023-09-29T22:55:14.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KFqloZRH3M8QnM/xMCLTVaWaFpY='} 2023-09-29T22:55:14.518 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e803c1617d0e8ace-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.519 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.519 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.519 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.519 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.519 DEBUG:boto:Data: 2023-09-29T22:55:14.519 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.519 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.520 DEBUG:boto:Params: {} 2023-09-29T22:55:14.520 DEBUG:boto:Token: None 2023-09-29T22:55:14.520 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.524 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f2df2ca3685e5d05-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.525 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:55:14.525 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:55:14.526 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.526 DEBUG:boto:Path: /hcnphz-61/obj145 2023-09-29T22:55:14.526 DEBUG:boto:Data: 2023-09-29T22:55:14.526 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.527 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.527 DEBUG:boto:Params: {} 2023-09-29T22:55:14.527 DEBUG:boto:Token: None 2023-09-29T22:55:14.528 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj145 2023-09-29T22:55:14.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tW6quxjBCefdMAKdQhHgqCNQ3yQ= 2023-09-29T22:55:14.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tW6quxjBCefdMAKdQhHgqCNQ3yQ='} 2023-09-29T22:55:14.570 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fed02f17304a7acd-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:14.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:14.571 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-61 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:14.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:14.621 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:14.705 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:55:14.705 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-61 2023-09-29T22:55:14.725 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-09-29T22:55:14.847 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.847 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.847 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.847 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.848 DEBUG:boto:Data: 2023-09-29T22:55:14.848 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.848 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.848 DEBUG:boto:Params: {} 2023-09-29T22:55:14.848 DEBUG:boto:Token: None 2023-09-29T22:55:14.848 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fa7b877c9e0df433-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.854 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:55:14.854 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:55:14.855 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.855 DEBUG:boto:Path: /hcnphz-61/obj31 2023-09-29T22:55:14.855 DEBUG:boto:Data: 2023-09-29T22:55:14.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.855 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.855 DEBUG:boto:Params: {} 2023-09-29T22:55:14.855 DEBUG:boto:Token: None 2023-09-29T22:55:14.856 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj31 2023-09-29T22:55:14.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PqYH9SB1y2mvwLxXspJbUQZYVF8= 2023-09-29T22:55:14.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PqYH9SB1y2mvwLxXspJbUQZYVF8='} 2023-09-29T22:55:14.866 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006fe9a52de9120455-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.867 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.867 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.867 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.867 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.867 DEBUG:boto:Data: 2023-09-29T22:55:14.867 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.867 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.868 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.868 DEBUG:boto:Params: {} 2023-09-29T22:55:14.868 DEBUG:boto:Token: None 2023-09-29T22:55:14.868 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.868 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.868 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.873 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057a6c9d42f7f756e-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.874 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:55:14.874 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:55:14.874 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.874 DEBUG:boto:Path: /hcnphz-61/obj62 2023-09-29T22:55:14.874 DEBUG:boto:Data: 2023-09-29T22:55:14.874 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.875 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.875 DEBUG:boto:Params: {} 2023-09-29T22:55:14.875 DEBUG:boto:Token: None 2023-09-29T22:55:14.875 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj62 2023-09-29T22:55:14.875 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y56KOPCgYQJdjqEAif1bB2lSHgE= 2023-09-29T22:55:14.875 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y56KOPCgYQJdjqEAif1bB2lSHgE='} 2023-09-29T22:55:14.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000df59e6e19125d8eb-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.919 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.919 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.919 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.919 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.919 DEBUG:boto:Data: 2023-09-29T22:55:14.919 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.920 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.920 DEBUG:boto:Params: {} 2023-09-29T22:55:14.920 DEBUG:boto:Token: None 2023-09-29T22:55:14.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000067e4d0719ed0081e-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.927 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:55:14.927 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:55:14.927 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.928 DEBUG:boto:Path: /hcnphz-61/obj93 2023-09-29T22:55:14.928 DEBUG:boto:Data: 2023-09-29T22:55:14.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.928 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.928 DEBUG:boto:Params: {} 2023-09-29T22:55:14.928 DEBUG:boto:Token: None 2023-09-29T22:55:14.928 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj93 2023-09-29T22:55:14.929 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/0f1zkK7n7OW6Bfn+yny13ytQKM= 2023-09-29T22:55:14.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/0f1zkK7n7OW6Bfn+yny13ytQKM='} 2023-09-29T22:55:14.938 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd4e429144b281ed-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.938 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.938 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.938 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.938 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.939 DEBUG:boto:Data: 2023-09-29T22:55:14.939 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.939 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.939 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.939 DEBUG:boto:Params: {} 2023-09-29T22:55:14.939 DEBUG:boto:Token: None 2023-09-29T22:55:14.939 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.939 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.940 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.944 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008afb3ab53babf16b-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.944 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:55:14.944 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:55:14.945 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.945 DEBUG:boto:Path: /hcnphz-61/obj124 2023-09-29T22:55:14.945 DEBUG:boto:Data: 2023-09-29T22:55:14.945 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:14.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.945 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.945 DEBUG:boto:Params: {} 2023-09-29T22:55:14.945 DEBUG:boto:Token: None 2023-09-29T22:55:14.946 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/obj124 2023-09-29T22:55:14.946 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RxN3ou2F6nXMM+5nB9lAi+MeG7w= 2023-09-29T22:55:14.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RxN3ou2F6nXMM+5nB9lAi+MeG7w='} 2023-09-29T22:55:14.990 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e3b4f8436d74177-00651755d2-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.991 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:14.991 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:14.991 DEBUG:boto:Method: HEAD 2023-09-29T22:55:14.991 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:14.991 DEBUG:boto:Data: 2023-09-29T22:55:14.991 DEBUG:boto:Headers: {} 2023-09-29T22:55:14.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:14.992 DEBUG:boto:Port: 8000 2023-09-29T22:55:14.992 DEBUG:boto:Params: {} 2023-09-29T22:55:14.992 DEBUG:boto:Token: None 2023-09-29T22:55:14.992 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:14 GMT /hcnphz-61/ 2023-09-29T22:55:14.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE= 2023-09-29T22:55:14.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Va/3A33TBe32qHKbqrjp3JYPiFE=', 'Content-Length': '0'} 2023-09-29T22:55:14.999 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cf144403baacd868-00651755d2-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:14.999 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:55:14.999 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:55:14.999 DEBUG:boto:Method: PUT 2023-09-29T22:55:14.999 DEBUG:boto:Path: /hcnphz-61/obj155 2023-09-29T22:55:15.000 DEBUG:boto:Data: 2023-09-29T22:55:15.000 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:15.000 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.000 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.000 DEBUG:boto:Params: {} 2023-09-29T22:55:15.000 DEBUG:boto:Token: None 2023-09-29T22:55:15.000 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/obj155 2023-09-29T22:55:15.000 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KLSbYo2ows+kg09agE5HJE/gLnU= 2023-09-29T22:55:15.001 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KLSbYo2ows+kg09agE5HJE/gLnU='} 2023-09-29T22:55:15.010 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000917a68b530f94415-00651755d3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:15.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-61', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:15.011 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-61 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:15.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:15.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:15.144 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T22:55:15.145 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-61 2023-09-29T22:55:15.170 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-09-29T22:55:15.293 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:15.293 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:15.293 DEBUG:boto:Method: HEAD 2023-09-29T22:55:15.293 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:15.294 DEBUG:boto:Data: 2023-09-29T22:55:15.294 DEBUG:boto:Headers: {} 2023-09-29T22:55:15.294 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.294 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.294 DEBUG:boto:Params: {} 2023-09-29T22:55:15.294 DEBUG:boto:Token: None 2023-09-29T22:55:15.294 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/ 2023-09-29T22:55:15.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg= 2023-09-29T22:55:15.295 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg=', 'Content-Length': '0'} 2023-09-29T22:55:15.301 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008697dfa19f589ec8-00651755d3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.302 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:55:15.302 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:55:15.302 DEBUG:boto:Method: PUT 2023-09-29T22:55:15.302 DEBUG:boto:Path: /hcnphz-61/obj37 2023-09-29T22:55:15.302 DEBUG:boto:Data: 2023-09-29T22:55:15.302 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:15.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.302 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.303 DEBUG:boto:Params: {} 2023-09-29T22:55:15.303 DEBUG:boto:Token: None 2023-09-29T22:55:15.303 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/obj37 2023-09-29T22:55:15.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dt0z+Jku4TzEtc6Vb9zVQx8o44w= 2023-09-29T22:55:15.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dt0z+Jku4TzEtc6Vb9zVQx8o44w='} 2023-09-29T22:55:15.346 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b96cb04b37a463b6-00651755d3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.347 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:15.347 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:15.347 DEBUG:boto:Method: HEAD 2023-09-29T22:55:15.347 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:15.347 DEBUG:boto:Data: 2023-09-29T22:55:15.347 DEBUG:boto:Headers: {} 2023-09-29T22:55:15.347 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.348 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.348 DEBUG:boto:Params: {} 2023-09-29T22:55:15.348 DEBUG:boto:Token: None 2023-09-29T22:55:15.348 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/ 2023-09-29T22:55:15.348 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg= 2023-09-29T22:55:15.348 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg=', 'Content-Length': '0'} 2023-09-29T22:55:15.355 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000790bc11b8138a8da-00651755d3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.356 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:55:15.356 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:55:15.356 DEBUG:boto:Method: PUT 2023-09-29T22:55:15.357 DEBUG:boto:Path: /hcnphz-61/obj74 2023-09-29T22:55:15.357 DEBUG:boto:Data: 2023-09-29T22:55:15.357 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:15.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.357 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.357 DEBUG:boto:Params: {} 2023-09-29T22:55:15.357 DEBUG:boto:Token: None 2023-09-29T22:55:15.358 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/obj74 2023-09-29T22:55:15.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qFBaf1ImBUJBdYpLw674AIDiGzs= 2023-09-29T22:55:15.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qFBaf1ImBUJBdYpLw674AIDiGzs='} 2023-09-29T22:55:15.367 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ea140a786b2a7292-00651755d3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.368 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:15.368 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:15.368 DEBUG:boto:Method: HEAD 2023-09-29T22:55:15.368 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:15.368 DEBUG:boto:Data: 2023-09-29T22:55:15.368 DEBUG:boto:Headers: {} 2023-09-29T22:55:15.368 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.369 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.369 DEBUG:boto:Params: {} 2023-09-29T22:55:15.369 DEBUG:boto:Token: None 2023-09-29T22:55:15.369 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/ 2023-09-29T22:55:15.369 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg= 2023-09-29T22:55:15.369 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg=', 'Content-Length': '0'} 2023-09-29T22:55:15.375 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000062328ce540b79629-00651755d3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.375 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:55:15.375 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:55:15.375 DEBUG:boto:Method: PUT 2023-09-29T22:55:15.375 DEBUG:boto:Path: /hcnphz-61/obj111 2023-09-29T22:55:15.376 DEBUG:boto:Data: 2023-09-29T22:55:15.376 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:15.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.376 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.376 DEBUG:boto:Params: {} 2023-09-29T22:55:15.376 DEBUG:boto:Token: None 2023-09-29T22:55:15.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/obj111 2023-09-29T22:55:15.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A9sj57XaM2r7/YFpT6YelQNt5xY= 2023-09-29T22:55:15.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A9sj57XaM2r7/YFpT6YelQNt5xY='} 2023-09-29T22:55:15.422 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033e347614d3a3f34-00651755d3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.423 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:15.423 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:15.423 DEBUG:boto:Method: HEAD 2023-09-29T22:55:15.423 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:15.423 DEBUG:boto:Data: 2023-09-29T22:55:15.423 DEBUG:boto:Headers: {} 2023-09-29T22:55:15.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.424 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.424 DEBUG:boto:Params: {} 2023-09-29T22:55:15.424 DEBUG:boto:Token: None 2023-09-29T22:55:15.424 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/ 2023-09-29T22:55:15.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg= 2023-09-29T22:55:15.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg=', 'Content-Length': '0'} 2023-09-29T22:55:15.431 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041f0b83bbddcfdee-00651755d3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.431 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:55:15.432 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:55:15.432 DEBUG:boto:Method: PUT 2023-09-29T22:55:15.432 DEBUG:boto:Path: /hcnphz-61/obj148 2023-09-29T22:55:15.432 DEBUG:boto:Data: 2023-09-29T22:55:15.432 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:15.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.432 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.432 DEBUG:boto:Params: {} 2023-09-29T22:55:15.432 DEBUG:boto:Token: None 2023-09-29T22:55:15.433 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/obj148 2023-09-29T22:55:15.433 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yM23Anv3yyH49/7d2UCuQ7Re7bo= 2023-09-29T22:55:15.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yM23Anv3yyH49/7d2UCuQ7Re7bo='} 2023-09-29T22:55:15.443 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015df17327b5be42e-00651755d3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.443 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:55:15.443 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:55:15.443 DEBUG:boto:Method: HEAD 2023-09-29T22:55:15.443 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:55:15.444 DEBUG:boto:Data: 2023-09-29T22:55:15.444 DEBUG:boto:Headers: {} 2023-09-29T22:55:15.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.444 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.444 DEBUG:boto:Params: {} 2023-09-29T22:55:15.444 DEBUG:boto:Token: None 2023-09-29T22:55:15.444 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/ 2023-09-29T22:55:15.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg= 2023-09-29T22:55:15.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rIzQHjs5j+zKhEYbsgkClX0/WOg=', 'Content-Length': '0'} 2023-09-29T22:55:15.450 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007c680a793370fb79-00651755d3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.451 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:55:15.451 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:55:15.451 DEBUG:boto:Method: PUT 2023-09-29T22:55:15.451 DEBUG:boto:Path: /hcnphz-61/obj185 2023-09-29T22:55:15.451 DEBUG:boto:Data: 2023-09-29T22:55:15.451 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:55:15.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:55:15.452 DEBUG:boto:Port: 8000 2023-09-29T22:55:15.452 DEBUG:boto:Params: {} 2023-09-29T22:55:15.452 DEBUG:boto:Token: None 2023-09-29T22:55:15.452 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:55:15 GMT /hcnphz-61/obj185 2023-09-29T22:55:15.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AiR5aIBrujGHYLVirvQH/DVf6BA= 2023-09-29T22:55:15.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:55:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AiR5aIBrujGHYLVirvQH/DVf6BA='} 2023-09-29T22:55:15.498 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000679bf2143ff5e84c-00651755d3-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:55:15.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:15.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T22:55:15.499 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T22:55:15.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:15.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:15.605 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:15.604+0000 7f7f3abf0f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-29T22:55:15.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:15.608+0000 7f7f3abf0f40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-29T22:55:15.644 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T22:55:15.645 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T22:55:15.692 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T22:55:15.693 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:15.709 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:55:15.709 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:55:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T22:55:15.710 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T22:55:15.711 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T22:55:16.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T22:55:16.712 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T22:55:16.731 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T22:55:16.731 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T22:55:16.732 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T22:55:16.732 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T22:55:16.733 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-61 2023-09-29T22:55:16.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:55:16.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-61', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:55:16.733 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-61 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:55:16.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:55:16.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:55:16.843 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:16.844+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:21.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:21.844+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:26.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:26.843+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:31.846 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:31.843+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:36.847 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:36.847+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:41.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:41.847+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:46.849 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:46.847+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:51.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:51.847+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:55:56.851 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:55:56.851+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:01.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:01.851+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:06.853 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:06.850+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:11.854 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:11.850+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:16.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:16.854+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:21.856 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:21.854+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:26.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:26.854+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:31.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:31.854+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:36.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:36.854+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:41.860 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:41.858+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:46.861 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:46.857+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:51.862 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:51.857+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:56:56.864 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:56:56.861+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:01.865 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:01.865+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:06.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:06.865+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:10.217 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T22:57:10.216+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T22:57:11.867 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:11.869+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:16.868 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:16.869+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:21.869 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:21.869+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:26.871 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:26.872+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:31.872 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:31.872+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:36.873 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:36.872+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:41.874 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:41.872+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:46.875 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:46.876+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:51.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:51.876+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:57:56.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:57:56.876+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:01.879 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:01.880+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:06.880 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:06.879+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:11.881 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:11.879+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:16.882 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:16.879+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:21.884 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:21.883+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:26.885 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:26.883+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:31.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:31.883+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:36.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:36.887+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:41.888 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:41.887+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:46.889 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:46.890+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:51.890 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:51.886+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:58:56.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:58:56.890+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:01.892 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:01.890+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:06.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:06.894+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:11.894 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:11.894+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:16.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:16.894+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:21.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:21.894+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:26.898 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:26.897+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:31.899 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:31.897+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:36.900 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:36.901+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:41.901 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:41.901+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:46.902 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:46.901+0000 7f4e9d629f40 1 waiting to reach incremental sync.. 2023-09-29T22:59:51.909 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:51.909+0000 7f4e9d629f40 1 bucket sync caught up with source: 2023-09-29T22:59:51.910 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.134.6, , , , , , 00000000002.57.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.77.6, , , , 00000000001.134.6] 2023-09-29T22:59:51.910 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.134.6, , , , , , 00000000002.57.6, , , , , 00000000002.75.6, , , , , , , , , , , , , , , 00000000001.77.6, , , , 00000000001.134.6] 2023-09-29T22:59:51.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:51.909+0000 7f4e9d629f40 0 bucket checkpoint complete 2023-09-29T22:59:51.934 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-61 2023-09-29T22:59:51.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:59:51.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-61', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:59:51.934 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-61 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:59:51.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:51.985 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:52.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:52.053+0000 7f63d6f11f40 1 bucket sync caught up with source: 2023-09-29T22:59:52.053 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, 00000000003.203.6, 00000000002.140.6, 00000000002.335.6, 00000000005.378.6, 00000000004.343.6, 00000000001.213.6, 00000000001.193.6, 00000000004.206.6, 00000000007.222.6, 00000000006.632.6] 2023-09-29T22:59:52.053 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, 00000000003.203.6, 00000000002.140.6, 00000000002.335.6, 00000000005.378.6, 00000000004.343.6, 00000000001.213.6, 00000000001.193.6, 00000000004.206.6, 00000000007.222.6, 00000000006.632.6] 2023-09-29T22:59:52.053 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:52.053+0000 7f63d6f11f40 0 bucket checkpoint complete 2023-09-29T22:59:52.081 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-61 zones={a2, a1} 2023-09-29T22:59:52.081 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.081 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.081 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.081 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:59:52.082 DEBUG:boto:Data: 2023-09-29T22:59:52.082 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.082 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.082 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.082 DEBUG:boto:Params: {} 2023-09-29T22:59:52.083 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T22:59:52.083 DEBUG:boto:Token: None 2023-09-29T22:59:52.083 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.083 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qEj7EgzKj+1dXKEbwmVA303KB9U= 2023-09-29T22:59:52.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qEj7EgzKj+1dXKEbwmVA303KB9U=', 'Content-Length': '0'} 2023-09-29T22:59:52.091 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cbaf22597f371bc9-00651756e8-1e9a-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.091 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.092 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.092 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.092 DEBUG:boto:Path: /hcnphz-61/ 2023-09-29T22:59:52.092 DEBUG:boto:Data: 2023-09-29T22:59:52.092 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.093 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.093 DEBUG:boto:Params: {} 2023-09-29T22:59:52.093 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T22:59:52.093 DEBUG:boto:Token: None 2023-09-29T22:59:52.093 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qEj7EgzKj+1dXKEbwmVA303KB9U= 2023-09-29T22:59:52.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qEj7EgzKj+1dXKEbwmVA303KB9U=', 'Content-Length': '0'} 2023-09-29T22:59:52.105 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '35'), ('X-RGW-Bytes-Used', '105'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000212f53e0cf9ac5ba-00651756e8-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.106 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T22:59:52.106 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.106 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.106 DEBUG:boto:path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.106 DEBUG:boto:auth_path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.106 DEBUG:boto:Method: GET 2023-09-29T22:59:52.107 DEBUG:boto:Path: /hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.107 DEBUG:boto:Data: 2023-09-29T22:59:52.107 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.107 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.107 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.107 DEBUG:boto:Params: {} 2023-09-29T22:59:52.107 DEBUG:boto:Token: None 2023-09-29T22:59:52.107 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/?versions 2023-09-29T22:59:52.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc= 2023-09-29T22:59:52.108 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc=', 'Content-Length': '0'} 2023-09-29T22:59:52.113 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000075877dd7ce91a93c-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.114 DEBUG:boto:b'hcnphz-61999falseobj11nulltrue2023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:59:52.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-29T22:59:52.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:59:52.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:59:52.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:59:52.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:59:52.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T22:59:52.120 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.120 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.120 DEBUG:boto:path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.120 DEBUG:boto:auth_path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.120 DEBUG:boto:Method: GET 2023-09-29T22:59:52.120 DEBUG:boto:Path: /hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.120 DEBUG:boto:Data: 2023-09-29T22:59:52.120 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.121 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.121 DEBUG:boto:Params: {} 2023-09-29T22:59:52.121 DEBUG:boto:Token: None 2023-09-29T22:59:52.122 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/?versions 2023-09-29T22:59:52.122 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc= 2023-09-29T22:59:52.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc=', 'Content-Length': '0'} 2023-09-29T22:59:52.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000638b01f468301ae0-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.130 DEBUG:boto:b'hcnphz-61999falseobj11nulltrue2023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T22:59:52.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj111 2023-09-29T22:59:52.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj115 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj116 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj124 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj145 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj148 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj155 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj17 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj185 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj19 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj23 2023-09-29T22:59:52.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj29 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj31 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj34 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj37 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj38 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj46 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj51 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T22:59:52.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj57 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj58 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj62 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj68 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj69 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj74 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj76 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj85 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj87 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj92 2023-09-29T22:59:52.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj93 2023-09-29T22:59:52.136 DEBUG:tasks.rgw_multi.zone_rados:o=obj95 2023-09-29T22:59:52.136 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.136 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.136 DEBUG:boto:path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.136 DEBUG:boto:auth_path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.136 DEBUG:boto:Method: GET 2023-09-29T22:59:52.137 DEBUG:boto:Path: /hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.137 DEBUG:boto:Data: 2023-09-29T22:59:52.137 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.137 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.137 DEBUG:boto:Params: {} 2023-09-29T22:59:52.137 DEBUG:boto:Token: None 2023-09-29T22:59:52.137 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/?versions 2023-09-29T22:59:52.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc= 2023-09-29T22:59:52.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc=', 'Content-Length': '0'} 2023-09-29T22:59:52.141 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007bf231d0c38220cb-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.141 DEBUG:boto:b'hcnphz-61999falseobj11nulltrue2023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.143 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.144 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.144 DEBUG:boto:path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.144 DEBUG:boto:auth_path=/hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.144 DEBUG:boto:Method: GET 2023-09-29T22:59:52.144 DEBUG:boto:Path: /hcnphz-61/?versions&max-keys=999 2023-09-29T22:59:52.144 DEBUG:boto:Data: 2023-09-29T22:59:52.144 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.145 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.145 DEBUG:boto:Params: {} 2023-09-29T22:59:52.145 DEBUG:boto:Token: None 2023-09-29T22:59:52.145 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/?versions 2023-09-29T22:59:52.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc= 2023-09-29T22:59:52.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0LuHFUeopoUN7iBQQJJbgoHqTTc=', 'Content-Length': '0'} 2023-09-29T22:59:52.152 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000382de26eb822fe02-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.152 DEBUG:boto:b'hcnphz-61999falseobj11nulltrue2023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj111nulltrue2023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj115nulltrue2023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj116nulltrue2023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj124nulltrue2023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj145nulltrue2023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj148nulltrue2023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj155nulltrue2023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj17nulltrue2023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj185nulltrue2023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj19nulltrue2023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj23nulltrue2023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj29nulltrue2023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj31nulltrue2023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj34nulltrue2023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj37nulltrue2023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj38nulltrue2023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj46nulltrue2023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj51nulltrue2023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj57nulltrue2023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj58nulltrue2023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj62nulltrue2023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj68nulltrue2023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj69nulltrue2023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj74nulltrue2023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj76nulltrue2023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj85nulltrue2023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj87nulltrue2023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj92nulltrue2023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj93nulltrue2023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj95nulltrue2023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:59:52.155 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.155 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.155 DEBUG:boto:path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.155 DEBUG:boto:auth_path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.155 DEBUG:boto:Method: GET 2023-09-29T22:59:52.155 DEBUG:boto:Path: /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.155 DEBUG:boto:Data: 2023-09-29T22:59:52.155 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.156 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.156 DEBUG:boto:Params: {} 2023-09-29T22:59:52.156 DEBUG:boto:Token: None 2023-09-29T22:59:52.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.156 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM= 2023-09-29T22:59:52.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM=', 'Content-Length': '0'} 2023-09-29T22:59:52.160 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000024c50384f1c0bc75-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.161 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.161 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.161 DEBUG:boto:path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.161 DEBUG:boto:auth_path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.161 DEBUG:boto:Method: GET 2023-09-29T22:59:52.161 DEBUG:boto:Path: /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.161 DEBUG:boto:Data: 2023-09-29T22:59:52.162 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.162 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.164 DEBUG:boto:Params: {} 2023-09-29T22:59:52.164 DEBUG:boto:Token: None 2023-09-29T22:59:52.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM= 2023-09-29T22:59:52.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM=', 'Content-Length': '0'} 2023-09-29T22:59:52.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ca06934fc353964-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.210 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.211 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.211 DEBUG:boto:path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.211 DEBUG:boto:auth_path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.211 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.211 DEBUG:boto:Path: /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.211 DEBUG:boto:Data: 2023-09-29T22:59:52.211 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.211 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.212 DEBUG:boto:Params: {} 2023-09-29T22:59:52.212 DEBUG:boto:Token: None 2023-09-29T22:59:52.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NFXZ3DOcgFc6bo5LbEH7Qv6KQ3s= 2023-09-29T22:59:52.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NFXZ3DOcgFc6bo5LbEH7Qv6KQ3s=', 'Content-Length': '0'} 2023-09-29T22:59:52.215 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d14216404e80d747-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.215 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.215 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.215 DEBUG:boto:path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.215 DEBUG:boto:auth_path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.216 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.216 DEBUG:boto:Path: /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.216 DEBUG:boto:Data: 2023-09-29T22:59:52.216 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.216 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.216 DEBUG:boto:Params: {} 2023-09-29T22:59:52.216 DEBUG:boto:Token: None 2023-09-29T22:59:52.216 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NFXZ3DOcgFc6bo5LbEH7Qv6KQ3s= 2023-09-29T22:59:52.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NFXZ3DOcgFc6bo5LbEH7Qv6KQ3s=', 'Content-Length': '0'} 2023-09-29T22:59:52.220 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf6b72f870e7ca94-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:59:52.220 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.220 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.220 DEBUG:boto:path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.220 DEBUG:boto:auth_path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.221 DEBUG:boto:Method: GET 2023-09-29T22:59:52.221 DEBUG:boto:Path: /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.221 DEBUG:boto:Data: 2023-09-29T22:59:52.221 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.221 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.221 DEBUG:boto:Params: {} 2023-09-29T22:59:52.221 DEBUG:boto:Token: None 2023-09-29T22:59:52.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.222 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM= 2023-09-29T22:59:52.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM=', 'Content-Length': '0'} 2023-09-29T22:59:52.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047578b56eb9d1bec-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.266 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.267 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.267 DEBUG:boto:path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.267 DEBUG:boto:auth_path=/hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.267 DEBUG:boto:Method: GET 2023-09-29T22:59:52.267 DEBUG:boto:Path: /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.267 DEBUG:boto:Data: 2023-09-29T22:59:52.267 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.267 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.267 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.267 DEBUG:boto:Params: {} 2023-09-29T22:59:52.268 DEBUG:boto:Token: None 2023-09-29T22:59:52.268 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11?versionId=null 2023-09-29T22:59:52.268 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM= 2023-09-29T22:59:52.268 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nZuccwcKYwMqpUkO2/2RMmIJGyM=', 'Content-Length': '0'} 2023-09-29T22:59:52.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f295b306b9ff832-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.272 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.272 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.272 DEBUG:boto:path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.272 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.272 DEBUG:boto:Method: GET 2023-09-29T22:59:52.272 DEBUG:boto:Path: /hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.272 DEBUG:boto:Data: 2023-09-29T22:59:52.272 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.273 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.273 DEBUG:boto:Params: {} 2023-09-29T22:59:52.273 DEBUG:boto:Token: None 2023-09-29T22:59:52.273 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.277 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f6ec855e68a1b83b-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.278 DEBUG:boto:b'hcnphz-61obj111000falseobj112023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.278 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.278 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.278 DEBUG:boto:path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.278 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.279 DEBUG:boto:Method: GET 2023-09-29T22:59:52.279 DEBUG:boto:Path: /hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.279 DEBUG:boto:Data: 2023-09-29T22:59:52.279 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.279 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.279 DEBUG:boto:Params: {} 2023-09-29T22:59:52.279 DEBUG:boto:Token: None 2023-09-29T22:59:52.279 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.288 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e2caf1bee8621754-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.288 DEBUG:boto:b'hcnphz-61obj111000falseobj112023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.289 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.289 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.289 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.289 DEBUG:boto:Path: /hcnphz-61/obj11 2023-09-29T22:59:52.289 DEBUG:boto:Data: 2023-09-29T22:59:52.290 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.290 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.290 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.290 DEBUG:boto:Params: {} 2023-09-29T22:59:52.290 DEBUG:boto:Token: None 2023-09-29T22:59:52.290 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11 2023-09-29T22:59:52.290 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GSCal8bn4ua9WfIfgkw5onbBqzM= 2023-09-29T22:59:52.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GSCal8bn4ua9WfIfgkw5onbBqzM=', 'Content-Length': '0'} 2023-09-29T22:59:52.292 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000467d6f121019a01a-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.293 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.293 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.293 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.293 DEBUG:boto:Path: /hcnphz-61/obj11 2023-09-29T22:59:52.293 DEBUG:boto:Data: 2023-09-29T22:59:52.293 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.293 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.293 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.294 DEBUG:boto:Params: {} 2023-09-29T22:59:52.294 DEBUG:boto:Token: None 2023-09-29T22:59:52.294 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11 2023-09-29T22:59:52.294 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GSCal8bn4ua9WfIfgkw5onbBqzM= 2023-09-29T22:59:52.294 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GSCal8bn4ua9WfIfgkw5onbBqzM=', 'Content-Length': '0'} 2023-09-29T22:59:52.296 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aa7647c04ceab989-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T22:59:52.296 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.297 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.297 DEBUG:boto:Method: GET 2023-09-29T22:59:52.297 DEBUG:boto:Path: /hcnphz-61/obj11 2023-09-29T22:59:52.297 DEBUG:boto:Data: 2023-09-29T22:59:52.297 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.297 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.297 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.297 DEBUG:boto:Params: {} 2023-09-29T22:59:52.297 DEBUG:boto:Token: None 2023-09-29T22:59:52.298 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11 2023-09-29T22:59:52.298 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YlYNAmjD9BOmXbZK++rrdMzWF90= 2023-09-29T22:59:52.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YlYNAmjD9BOmXbZK++rrdMzWF90=', 'Content-Length': '0'} 2023-09-29T22:59:52.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049d68f6da36bd07c-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.300 DEBUG:boto:path=/hcnphz-61/obj11 2023-09-29T22:59:52.301 DEBUG:boto:auth_path=/hcnphz-61/obj11 2023-09-29T22:59:52.301 DEBUG:boto:Method: GET 2023-09-29T22:59:52.301 DEBUG:boto:Path: /hcnphz-61/obj11 2023-09-29T22:59:52.301 DEBUG:boto:Data: 2023-09-29T22:59:52.301 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.301 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.301 DEBUG:boto:Params: {} 2023-09-29T22:59:52.301 DEBUG:boto:Token: None 2023-09-29T22:59:52.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj11 2023-09-29T22:59:52.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YlYNAmjD9BOmXbZK++rrdMzWF90= 2023-09-29T22:59:52.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YlYNAmjD9BOmXbZK++rrdMzWF90=', 'Content-Length': '0'} 2023-09-29T22:59:52.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1145dc09bdf5df8-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.347 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.347 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.347 DEBUG:boto:path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.347 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.347 DEBUG:boto:Method: GET 2023-09-29T22:59:52.347 DEBUG:boto:Path: /hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.347 DEBUG:boto:Data: 2023-09-29T22:59:52.348 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.348 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.348 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.348 DEBUG:boto:Params: {} 2023-09-29T22:59:52.348 DEBUG:boto:Token: None 2023-09-29T22:59:52.348 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.349 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.349 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.353 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a197cf05e4d45b07-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.353 DEBUG:boto:b'hcnphz-61obj111000falseobj112023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.354 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.354 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.354 DEBUG:boto:path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.354 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.354 DEBUG:boto:Method: GET 2023-09-29T22:59:52.355 DEBUG:boto:Path: /hcnphz-61/?prefix=obj11 2023-09-29T22:59:52.355 DEBUG:boto:Data: 2023-09-29T22:59:52.355 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.355 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.355 DEBUG:boto:Params: {} 2023-09-29T22:59:52.355 DEBUG:boto:Token: None 2023-09-29T22:59:52.355 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.355 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.364 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb29a40108f94c2b-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.364 DEBUG:boto:b'hcnphz-61obj111000falseobj112023-09-29T22:55:12.727Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:59:52.365 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.365 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.366 DEBUG:boto:path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.366 DEBUG:boto:auth_path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.366 DEBUG:boto:Method: GET 2023-09-29T22:59:52.366 DEBUG:boto:Path: /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.366 DEBUG:boto:Data: 2023-09-29T22:59:52.366 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.366 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.366 DEBUG:boto:Params: {} 2023-09-29T22:59:52.367 DEBUG:boto:Token: None 2023-09-29T22:59:52.367 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.367 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU= 2023-09-29T22:59:52.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU=', 'Content-Length': '0'} 2023-09-29T22:59:52.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000053b7aa4f2f429b6-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.414 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.415 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.415 DEBUG:boto:path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.415 DEBUG:boto:auth_path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.415 DEBUG:boto:Method: GET 2023-09-29T22:59:52.415 DEBUG:boto:Path: /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.415 DEBUG:boto:Data: 2023-09-29T22:59:52.415 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.415 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.416 DEBUG:boto:Params: {} 2023-09-29T22:59:52.416 DEBUG:boto:Token: None 2023-09-29T22:59:52.416 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU= 2023-09-29T22:59:52.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU=', 'Content-Length': '0'} 2023-09-29T22:59:52.419 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cdb50bc91b60a697-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.419 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.420 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.420 DEBUG:boto:path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.420 DEBUG:boto:auth_path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.420 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.420 DEBUG:boto:Path: /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.420 DEBUG:boto:Data: 2023-09-29T22:59:52.420 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.420 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.420 DEBUG:boto:Params: {} 2023-09-29T22:59:52.421 DEBUG:boto:Token: None 2023-09-29T22:59:52.421 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.421 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5BgjRd/kFD8oJjUohUSU9EyqgjM= 2023-09-29T22:59:52.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5BgjRd/kFD8oJjUohUSU9EyqgjM=', 'Content-Length': '0'} 2023-09-29T22:59:52.423 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4ba5c88db3c454b-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.424 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.424 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.424 DEBUG:boto:path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.424 DEBUG:boto:auth_path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.425 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.425 DEBUG:boto:Path: /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.425 DEBUG:boto:Data: 2023-09-29T22:59:52.425 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.425 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.426 DEBUG:boto:Params: {} 2023-09-29T22:59:52.426 DEBUG:boto:Token: None 2023-09-29T22:59:52.426 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.427 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5BgjRd/kFD8oJjUohUSU9EyqgjM= 2023-09-29T22:59:52.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5BgjRd/kFD8oJjUohUSU9EyqgjM=', 'Content-Length': '0'} 2023-09-29T22:59:52.430 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee933a454d642f71-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:59:52.430 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.430 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.430 DEBUG:boto:path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.430 DEBUG:boto:auth_path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.431 DEBUG:boto:Method: GET 2023-09-29T22:59:52.431 DEBUG:boto:Path: /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.431 DEBUG:boto:Data: 2023-09-29T22:59:52.432 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.432 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.432 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.432 DEBUG:boto:Params: {} 2023-09-29T22:59:52.432 DEBUG:boto:Token: None 2023-09-29T22:59:52.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU= 2023-09-29T22:59:52.433 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU=', 'Content-Length': '0'} 2023-09-29T22:59:52.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007671d1befb3d1ff2-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.436 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.436 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.436 DEBUG:boto:path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.436 DEBUG:boto:auth_path=/hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.437 DEBUG:boto:Method: GET 2023-09-29T22:59:52.437 DEBUG:boto:Path: /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.437 DEBUG:boto:Data: 2023-09-29T22:59:52.437 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.437 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.438 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.438 DEBUG:boto:Params: {} 2023-09-29T22:59:52.438 DEBUG:boto:Token: None 2023-09-29T22:59:52.438 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111?versionId=null 2023-09-29T22:59:52.439 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU= 2023-09-29T22:59:52.439 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CepP1Pkxd0hYz4QiR7VfSExFsZU=', 'Content-Length': '0'} 2023-09-29T22:59:52.442 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004968c4df5508575a-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.443 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.443 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.443 DEBUG:boto:path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.444 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.444 DEBUG:boto:Method: GET 2023-09-29T22:59:52.444 DEBUG:boto:Path: /hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.444 DEBUG:boto:Data: 2023-09-29T22:59:52.444 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.444 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.444 DEBUG:boto:Params: {} 2023-09-29T22:59:52.445 DEBUG:boto:Token: None 2023-09-29T22:59:52.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.451 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a210d00f735684c2-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.494 DEBUG:boto:b'hcnphz-61obj1111000falseobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.495 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.495 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.495 DEBUG:boto:path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.495 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.495 DEBUG:boto:Method: GET 2023-09-29T22:59:52.496 DEBUG:boto:Path: /hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.496 DEBUG:boto:Data: 2023-09-29T22:59:52.496 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.496 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.496 DEBUG:boto:Params: {} 2023-09-29T22:59:52.496 DEBUG:boto:Token: None 2023-09-29T22:59:52.496 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.506 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000077ba2725ddd98609-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.506 DEBUG:boto:b'hcnphz-61obj1111000falseobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.507 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.507 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.507 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.507 DEBUG:boto:Path: /hcnphz-61/obj111 2023-09-29T22:59:52.507 DEBUG:boto:Data: 2023-09-29T22:59:52.507 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.507 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.508 DEBUG:boto:Params: {} 2023-09-29T22:59:52.508 DEBUG:boto:Token: None 2023-09-29T22:59:52.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111 2023-09-29T22:59:52.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uQIc1ZCJsGV+YymQ6bK62nVpaCA= 2023-09-29T22:59:52.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uQIc1ZCJsGV+YymQ6bK62nVpaCA=', 'Content-Length': '0'} 2023-09-29T22:59:52.510 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0974e2978bf9d6a-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.511 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.511 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.511 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.511 DEBUG:boto:Path: /hcnphz-61/obj111 2023-09-29T22:59:52.511 DEBUG:boto:Data: 2023-09-29T22:59:52.511 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.511 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.511 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.512 DEBUG:boto:Params: {} 2023-09-29T22:59:52.512 DEBUG:boto:Token: None 2023-09-29T22:59:52.512 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111 2023-09-29T22:59:52.512 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uQIc1ZCJsGV+YymQ6bK62nVpaCA= 2023-09-29T22:59:52.512 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uQIc1ZCJsGV+YymQ6bK62nVpaCA=', 'Content-Length': '0'} 2023-09-29T22:59:52.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e31055eafeb3905-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj111 2023-09-29T22:59:52.514 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.515 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.515 DEBUG:boto:Method: GET 2023-09-29T22:59:52.515 DEBUG:boto:Path: /hcnphz-61/obj111 2023-09-29T22:59:52.515 DEBUG:boto:Data: 2023-09-29T22:59:52.515 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.515 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.515 DEBUG:boto:Params: {} 2023-09-29T22:59:52.515 DEBUG:boto:Token: None 2023-09-29T22:59:52.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111 2023-09-29T22:59:52.516 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZqTJ75bD/YYLbKWC9/1E9hUmBWo= 2023-09-29T22:59:52.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZqTJ75bD/YYLbKWC9/1E9hUmBWo=', 'Content-Length': '0'} 2023-09-29T22:59:52.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000452e000a09526e72-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.518 DEBUG:boto:path=/hcnphz-61/obj111 2023-09-29T22:59:52.518 DEBUG:boto:auth_path=/hcnphz-61/obj111 2023-09-29T22:59:52.518 DEBUG:boto:Method: GET 2023-09-29T22:59:52.518 DEBUG:boto:Path: /hcnphz-61/obj111 2023-09-29T22:59:52.518 DEBUG:boto:Data: 2023-09-29T22:59:52.518 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.518 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.519 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.519 DEBUG:boto:Params: {} 2023-09-29T22:59:52.519 DEBUG:boto:Token: None 2023-09-29T22:59:52.519 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj111 2023-09-29T22:59:52.519 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZqTJ75bD/YYLbKWC9/1E9hUmBWo= 2023-09-29T22:59:52.519 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZqTJ75bD/YYLbKWC9/1E9hUmBWo=', 'Content-Length': '0'} 2023-09-29T22:59:52.521 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006ba3cfd270b48100-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.562 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.563 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.563 DEBUG:boto:path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.563 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.563 DEBUG:boto:Method: GET 2023-09-29T22:59:52.563 DEBUG:boto:Path: /hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.563 DEBUG:boto:Data: 2023-09-29T22:59:52.563 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.563 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.563 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.564 DEBUG:boto:Params: {} 2023-09-29T22:59:52.564 DEBUG:boto:Token: None 2023-09-29T22:59:52.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.564 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f4f08299e064c51-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.569 DEBUG:boto:b'hcnphz-61obj1111000falseobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.569 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.569 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.569 DEBUG:boto:path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.570 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.570 DEBUG:boto:Method: GET 2023-09-29T22:59:52.570 DEBUG:boto:Path: /hcnphz-61/?prefix=obj111 2023-09-29T22:59:52.570 DEBUG:boto:Data: 2023-09-29T22:59:52.570 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.570 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.570 DEBUG:boto:Params: {} 2023-09-29T22:59:52.570 DEBUG:boto:Token: None 2023-09-29T22:59:52.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.579 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003867ce4c6c9384c4-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.579 DEBUG:boto:b'hcnphz-61obj1111000falseobj1112023-09-29T22:55:15.381Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.580 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:59:52.580 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.580 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.580 DEBUG:boto:path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.580 DEBUG:boto:auth_path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.580 DEBUG:boto:Method: GET 2023-09-29T22:59:52.580 DEBUG:boto:Path: /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.580 DEBUG:boto:Data: 2023-09-29T22:59:52.581 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.581 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.581 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.581 DEBUG:boto:Params: {} 2023-09-29T22:59:52.581 DEBUG:boto:Token: None 2023-09-29T22:59:52.581 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs= 2023-09-29T22:59:52.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs=', 'Content-Length': '0'} 2023-09-29T22:59:52.584 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfa0aeae9dd8dd56-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.626 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.627 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.627 DEBUG:boto:path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.627 DEBUG:boto:auth_path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.627 DEBUG:boto:Method: GET 2023-09-29T22:59:52.627 DEBUG:boto:Path: /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.627 DEBUG:boto:Data: 2023-09-29T22:59:52.627 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.627 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.628 DEBUG:boto:Params: {} 2023-09-29T22:59:52.628 DEBUG:boto:Token: None 2023-09-29T22:59:52.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs= 2023-09-29T22:59:52.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs=', 'Content-Length': '0'} 2023-09-29T22:59:52.632 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001306a2dee54386a0-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.632 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.632 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.632 DEBUG:boto:path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.632 DEBUG:boto:auth_path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.632 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.632 DEBUG:boto:Path: /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.633 DEBUG:boto:Data: 2023-09-29T22:59:52.633 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.633 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.633 DEBUG:boto:Params: {} 2023-09-29T22:59:52.633 DEBUG:boto:Token: None 2023-09-29T22:59:52.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jyI8RsUEMOMly/EF2Fq5WPtZaQM= 2023-09-29T22:59:52.634 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jyI8RsUEMOMly/EF2Fq5WPtZaQM=', 'Content-Length': '0'} 2023-09-29T22:59:52.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006074953a0a117bf2-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.637 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.637 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.637 DEBUG:boto:path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.637 DEBUG:boto:auth_path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.637 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.637 DEBUG:boto:Path: /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.637 DEBUG:boto:Data: 2023-09-29T22:59:52.637 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.638 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.638 DEBUG:boto:Params: {} 2023-09-29T22:59:52.638 DEBUG:boto:Token: None 2023-09-29T22:59:52.638 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jyI8RsUEMOMly/EF2Fq5WPtZaQM= 2023-09-29T22:59:52.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jyI8RsUEMOMly/EF2Fq5WPtZaQM=', 'Content-Length': '0'} 2023-09-29T22:59:52.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f4fda9a49f3981c-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:59:52.642 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.642 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.642 DEBUG:boto:path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.642 DEBUG:boto:auth_path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.642 DEBUG:boto:Method: GET 2023-09-29T22:59:52.643 DEBUG:boto:Path: /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.643 DEBUG:boto:Data: 2023-09-29T22:59:52.643 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.643 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.643 DEBUG:boto:Params: {} 2023-09-29T22:59:52.643 DEBUG:boto:Token: None 2023-09-29T22:59:52.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs= 2023-09-29T22:59:52.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs=', 'Content-Length': '0'} 2023-09-29T22:59:52.646 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000071069baa8d92faf6-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.647 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.647 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.648 DEBUG:boto:path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.648 DEBUG:boto:auth_path=/hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.648 DEBUG:boto:Method: GET 2023-09-29T22:59:52.648 DEBUG:boto:Path: /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.648 DEBUG:boto:Data: 2023-09-29T22:59:52.649 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.649 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.649 DEBUG:boto:Params: {} 2023-09-29T22:59:52.649 DEBUG:boto:Token: None 2023-09-29T22:59:52.650 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115?versionId=null 2023-09-29T22:59:52.650 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs= 2023-09-29T22:59:52.650 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SPNPrgga2YRMV2talwrL2s8xaIs=', 'Content-Length': '0'} 2023-09-29T22:59:52.653 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000053dda00a26f6bd82-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.653 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.654 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.654 DEBUG:boto:path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.654 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.654 DEBUG:boto:Method: GET 2023-09-29T22:59:52.654 DEBUG:boto:Path: /hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.655 DEBUG:boto:Data: 2023-09-29T22:59:52.655 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.655 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.656 DEBUG:boto:Params: {} 2023-09-29T22:59:52.656 DEBUG:boto:Token: None 2023-09-29T22:59:52.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.657 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a95604f0776739e7-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.702 DEBUG:boto:b'hcnphz-61obj1151000falseobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.703 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.703 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.703 DEBUG:boto:path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.703 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.703 DEBUG:boto:Method: GET 2023-09-29T22:59:52.703 DEBUG:boto:Path: /hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.704 DEBUG:boto:Data: 2023-09-29T22:59:52.704 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.704 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.704 DEBUG:boto:Params: {} 2023-09-29T22:59:52.704 DEBUG:boto:Token: None 2023-09-29T22:59:52.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.714 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000069080199f523f87e-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.714 DEBUG:boto:b'hcnphz-61obj1151000falseobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.715 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.715 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.715 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.715 DEBUG:boto:Path: /hcnphz-61/obj115 2023-09-29T22:59:52.715 DEBUG:boto:Data: 2023-09-29T22:59:52.715 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.715 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.715 DEBUG:boto:Params: {} 2023-09-29T22:59:52.716 DEBUG:boto:Token: None 2023-09-29T22:59:52.716 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115 2023-09-29T22:59:52.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DWgouPpTmFYsh+3CIcuGYOVvMhE= 2023-09-29T22:59:52.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DWgouPpTmFYsh+3CIcuGYOVvMhE=', 'Content-Length': '0'} 2023-09-29T22:59:52.718 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004acab77f5a809cab-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.718 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.719 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.719 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.719 DEBUG:boto:Path: /hcnphz-61/obj115 2023-09-29T22:59:52.719 DEBUG:boto:Data: 2023-09-29T22:59:52.719 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.719 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.719 DEBUG:boto:Params: {} 2023-09-29T22:59:52.719 DEBUG:boto:Token: None 2023-09-29T22:59:52.720 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115 2023-09-29T22:59:52.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DWgouPpTmFYsh+3CIcuGYOVvMhE= 2023-09-29T22:59:52.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DWgouPpTmFYsh+3CIcuGYOVvMhE=', 'Content-Length': '0'} 2023-09-29T22:59:52.722 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d9d921eb15d3f3-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj115 2023-09-29T22:59:52.722 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.723 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.723 DEBUG:boto:Method: GET 2023-09-29T22:59:52.723 DEBUG:boto:Path: /hcnphz-61/obj115 2023-09-29T22:59:52.723 DEBUG:boto:Data: 2023-09-29T22:59:52.723 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.723 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.723 DEBUG:boto:Params: {} 2023-09-29T22:59:52.723 DEBUG:boto:Token: None 2023-09-29T22:59:52.724 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115 2023-09-29T22:59:52.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KkTnWdPnAbUQxmlqao6nfBJz/3k= 2023-09-29T22:59:52.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KkTnWdPnAbUQxmlqao6nfBJz/3k=', 'Content-Length': '0'} 2023-09-29T22:59:52.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000768c050b6b1019aa-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.726 DEBUG:boto:path=/hcnphz-61/obj115 2023-09-29T22:59:52.726 DEBUG:boto:auth_path=/hcnphz-61/obj115 2023-09-29T22:59:52.726 DEBUG:boto:Method: GET 2023-09-29T22:59:52.726 DEBUG:boto:Path: /hcnphz-61/obj115 2023-09-29T22:59:52.727 DEBUG:boto:Data: 2023-09-29T22:59:52.727 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.727 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.727 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.727 DEBUG:boto:Params: {} 2023-09-29T22:59:52.727 DEBUG:boto:Token: None 2023-09-29T22:59:52.727 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj115 2023-09-29T22:59:52.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KkTnWdPnAbUQxmlqao6nfBJz/3k= 2023-09-29T22:59:52.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KkTnWdPnAbUQxmlqao6nfBJz/3k=', 'Content-Length': '0'} 2023-09-29T22:59:52.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d0724006b7dff40-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.770 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.771 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.771 DEBUG:boto:path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.771 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.771 DEBUG:boto:Method: GET 2023-09-29T22:59:52.771 DEBUG:boto:Path: /hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.772 DEBUG:boto:Data: 2023-09-29T22:59:52.772 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.772 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.772 DEBUG:boto:Params: {} 2023-09-29T22:59:52.772 DEBUG:boto:Token: None 2023-09-29T22:59:52.772 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.773 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.773 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.776 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf7ca1d7610f932-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.777 DEBUG:boto:b'hcnphz-61obj1151000falseobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.777 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.777 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.778 DEBUG:boto:path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.778 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.778 DEBUG:boto:Method: GET 2023-09-29T22:59:52.778 DEBUG:boto:Path: /hcnphz-61/?prefix=obj115 2023-09-29T22:59:52.779 DEBUG:boto:Data: 2023-09-29T22:59:52.779 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.779 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.779 DEBUG:boto:Params: {} 2023-09-29T22:59:52.779 DEBUG:boto:Token: None 2023-09-29T22:59:52.779 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.779 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.788 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000431e2265d2512d47-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.788 DEBUG:boto:b'hcnphz-61obj1151000falseobj1152023-09-29T22:55:14.121Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:59:52.788 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.789 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.789 DEBUG:boto:path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.789 DEBUG:boto:auth_path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.789 DEBUG:boto:Method: GET 2023-09-29T22:59:52.789 DEBUG:boto:Path: /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.789 DEBUG:boto:Data: 2023-09-29T22:59:52.789 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.789 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.790 DEBUG:boto:Params: {} 2023-09-29T22:59:52.790 DEBUG:boto:Token: None 2023-09-29T22:59:52.790 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4= 2023-09-29T22:59:52.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4=', 'Content-Length': '0'} 2023-09-29T22:59:52.792 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002264fd479ecd73be-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.793 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.793 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.793 DEBUG:boto:path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.793 DEBUG:boto:auth_path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.793 DEBUG:boto:Method: GET 2023-09-29T22:59:52.793 DEBUG:boto:Path: /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.793 DEBUG:boto:Data: 2023-09-29T22:59:52.793 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.793 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.794 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.794 DEBUG:boto:Params: {} 2023-09-29T22:59:52.794 DEBUG:boto:Token: None 2023-09-29T22:59:52.794 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.794 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4= 2023-09-29T22:59:52.794 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4=', 'Content-Length': '0'} 2023-09-29T22:59:52.796 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000983e2fb2d464ca72-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.797 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.797 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.797 DEBUG:boto:path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.797 DEBUG:boto:auth_path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.797 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.797 DEBUG:boto:Path: /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.797 DEBUG:boto:Data: 2023-09-29T22:59:52.797 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.797 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.798 DEBUG:boto:Params: {} 2023-09-29T22:59:52.798 DEBUG:boto:Token: None 2023-09-29T22:59:52.798 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.798 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ROw7CZw4+n+h3iOSDU9sZMPMCho= 2023-09-29T22:59:52.798 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ROw7CZw4+n+h3iOSDU9sZMPMCho=', 'Content-Length': '0'} 2023-09-29T22:59:52.800 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3ebced78d04c61f-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.800 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.801 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.801 DEBUG:boto:path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.801 DEBUG:boto:auth_path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.801 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.801 DEBUG:boto:Path: /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.801 DEBUG:boto:Data: 2023-09-29T22:59:52.801 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.801 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.801 DEBUG:boto:Params: {} 2023-09-29T22:59:52.802 DEBUG:boto:Token: None 2023-09-29T22:59:52.802 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ROw7CZw4+n+h3iOSDU9sZMPMCho= 2023-09-29T22:59:52.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ROw7CZw4+n+h3iOSDU9sZMPMCho=', 'Content-Length': '0'} 2023-09-29T22:59:52.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a1c3986c598c0655-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:59:52.805 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.805 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.805 DEBUG:boto:path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.805 DEBUG:boto:auth_path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.805 DEBUG:boto:Method: GET 2023-09-29T22:59:52.805 DEBUG:boto:Path: /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.805 DEBUG:boto:Data: 2023-09-29T22:59:52.805 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.806 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.806 DEBUG:boto:Params: {} 2023-09-29T22:59:52.806 DEBUG:boto:Token: None 2023-09-29T22:59:52.806 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4= 2023-09-29T22:59:52.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4=', 'Content-Length': '0'} 2023-09-29T22:59:52.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091654760dacfa5b8-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.809 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.809 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.809 DEBUG:boto:path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.809 DEBUG:boto:auth_path=/hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.809 DEBUG:boto:Method: GET 2023-09-29T22:59:52.809 DEBUG:boto:Path: /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.809 DEBUG:boto:Data: 2023-09-29T22:59:52.809 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.810 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.810 DEBUG:boto:Params: {} 2023-09-29T22:59:52.810 DEBUG:boto:Token: None 2023-09-29T22:59:52.810 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116?versionId=null 2023-09-29T22:59:52.810 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4= 2023-09-29T22:59:52.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:irzYXHH3bKYSFQZU1lMbtDWdZv4=', 'Content-Length': '0'} 2023-09-29T22:59:52.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f16dd8b7ffaa96f7-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.813 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.813 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.813 DEBUG:boto:path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.813 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.813 DEBUG:boto:Method: GET 2023-09-29T22:59:52.813 DEBUG:boto:Path: /hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.814 DEBUG:boto:Data: 2023-09-29T22:59:52.814 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.814 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.814 DEBUG:boto:Params: {} 2023-09-29T22:59:52.814 DEBUG:boto:Token: None 2023-09-29T22:59:52.814 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.814 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b67cd8fb2321f1ed-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.819 DEBUG:boto:b'hcnphz-61obj1161000falseobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.819 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.819 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.819 DEBUG:boto:path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.820 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.820 DEBUG:boto:Method: GET 2023-09-29T22:59:52.820 DEBUG:boto:Path: /hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.820 DEBUG:boto:Data: 2023-09-29T22:59:52.820 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.820 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.820 DEBUG:boto:Params: {} 2023-09-29T22:59:52.820 DEBUG:boto:Token: None 2023-09-29T22:59:52.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.821 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.829 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000240e6bd2cd588aa0-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.829 DEBUG:boto:b'hcnphz-61obj1161000falseobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.830 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.830 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.830 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.830 DEBUG:boto:Path: /hcnphz-61/obj116 2023-09-29T22:59:52.830 DEBUG:boto:Data: 2023-09-29T22:59:52.831 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.831 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.831 DEBUG:boto:Params: {} 2023-09-29T22:59:52.831 DEBUG:boto:Token: None 2023-09-29T22:59:52.831 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116 2023-09-29T22:59:52.831 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Gite4X4AIj/DecctLXuJQmISgIw= 2023-09-29T22:59:52.831 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Gite4X4AIj/DecctLXuJQmISgIw=', 'Content-Length': '0'} 2023-09-29T22:59:52.833 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d71c2b1218321656-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.833 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.834 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.834 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.834 DEBUG:boto:Path: /hcnphz-61/obj116 2023-09-29T22:59:52.834 DEBUG:boto:Data: 2023-09-29T22:59:52.834 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.834 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.834 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.834 DEBUG:boto:Params: {} 2023-09-29T22:59:52.835 DEBUG:boto:Token: None 2023-09-29T22:59:52.835 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116 2023-09-29T22:59:52.835 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Gite4X4AIj/DecctLXuJQmISgIw= 2023-09-29T22:59:52.835 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Gite4X4AIj/DecctLXuJQmISgIw=', 'Content-Length': '0'} 2023-09-29T22:59:52.837 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9ff9b001037a091-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj116 2023-09-29T22:59:52.837 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.837 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.837 DEBUG:boto:Method: GET 2023-09-29T22:59:52.838 DEBUG:boto:Path: /hcnphz-61/obj116 2023-09-29T22:59:52.838 DEBUG:boto:Data: 2023-09-29T22:59:52.838 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.838 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.838 DEBUG:boto:Params: {} 2023-09-29T22:59:52.838 DEBUG:boto:Token: None 2023-09-29T22:59:52.838 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116 2023-09-29T22:59:52.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QTvVFPCQwQvyrS2OPmevp+YRAng= 2023-09-29T22:59:52.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QTvVFPCQwQvyrS2OPmevp+YRAng=', 'Content-Length': '0'} 2023-09-29T22:59:52.841 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038cc1a115c573b1d-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.882 DEBUG:boto:path=/hcnphz-61/obj116 2023-09-29T22:59:52.883 DEBUG:boto:auth_path=/hcnphz-61/obj116 2023-09-29T22:59:52.883 DEBUG:boto:Method: GET 2023-09-29T22:59:52.883 DEBUG:boto:Path: /hcnphz-61/obj116 2023-09-29T22:59:52.883 DEBUG:boto:Data: 2023-09-29T22:59:52.883 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.883 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.883 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.883 DEBUG:boto:Params: {} 2023-09-29T22:59:52.883 DEBUG:boto:Token: None 2023-09-29T22:59:52.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj116 2023-09-29T22:59:52.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QTvVFPCQwQvyrS2OPmevp+YRAng= 2023-09-29T22:59:52.884 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QTvVFPCQwQvyrS2OPmevp+YRAng=', 'Content-Length': '0'} 2023-09-29T22:59:52.886 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b2f223a2223cf791-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.887 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.887 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.887 DEBUG:boto:path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.887 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.887 DEBUG:boto:Method: GET 2023-09-29T22:59:52.888 DEBUG:boto:Path: /hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.888 DEBUG:boto:Data: 2023-09-29T22:59:52.888 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.888 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.888 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.888 DEBUG:boto:Params: {} 2023-09-29T22:59:52.888 DEBUG:boto:Token: None 2023-09-29T22:59:52.888 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea70f0f10ff4bb46-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.894 DEBUG:boto:b'hcnphz-61obj1161000falseobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.894 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.894 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.894 DEBUG:boto:path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.895 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.895 DEBUG:boto:Method: GET 2023-09-29T22:59:52.895 DEBUG:boto:Path: /hcnphz-61/?prefix=obj116 2023-09-29T22:59:52.895 DEBUG:boto:Data: 2023-09-29T22:59:52.895 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.895 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.895 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.895 DEBUG:boto:Params: {} 2023-09-29T22:59:52.895 DEBUG:boto:Token: None 2023-09-29T22:59:52.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.903 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062dd935ef677e41d-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.904 DEBUG:boto:b'hcnphz-61obj1161000falseobj1162023-09-29T22:55:14.515Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:59:52.904 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.904 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.904 DEBUG:boto:path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.904 DEBUG:boto:auth_path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.905 DEBUG:boto:Method: GET 2023-09-29T22:59:52.905 DEBUG:boto:Path: /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.905 DEBUG:boto:Data: 2023-09-29T22:59:52.905 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.905 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.905 DEBUG:boto:Params: {} 2023-09-29T22:59:52.905 DEBUG:boto:Token: None 2023-09-29T22:59:52.905 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.906 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc= 2023-09-29T22:59:52.906 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc=', 'Content-Length': '0'} 2023-09-29T22:59:52.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3e36f3621aa3170-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.908 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.908 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.908 DEBUG:boto:path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.909 DEBUG:boto:auth_path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.909 DEBUG:boto:Method: GET 2023-09-29T22:59:52.909 DEBUG:boto:Path: /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.909 DEBUG:boto:Data: 2023-09-29T22:59:52.909 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.909 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.909 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.909 DEBUG:boto:Params: {} 2023-09-29T22:59:52.909 DEBUG:boto:Token: None 2023-09-29T22:59:52.910 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc= 2023-09-29T22:59:52.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc=', 'Content-Length': '0'} 2023-09-29T22:59:52.912 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000358f73d7e3c355c-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.912 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.913 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.913 DEBUG:boto:path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.913 DEBUG:boto:auth_path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.913 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.913 DEBUG:boto:Path: /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.913 DEBUG:boto:Data: 2023-09-29T22:59:52.913 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.913 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.913 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.913 DEBUG:boto:Params: {} 2023-09-29T22:59:52.914 DEBUG:boto:Token: None 2023-09-29T22:59:52.914 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.914 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uj4kJt4uENfimgXTvQWWoEmYiqs= 2023-09-29T22:59:52.914 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uj4kJt4uENfimgXTvQWWoEmYiqs=', 'Content-Length': '0'} 2023-09-29T22:59:52.916 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ce55f16f9f70368-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.916 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.916 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.916 DEBUG:boto:path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.916 DEBUG:boto:auth_path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.916 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.917 DEBUG:boto:Path: /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.917 DEBUG:boto:Data: 2023-09-29T22:59:52.917 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.917 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.917 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.917 DEBUG:boto:Params: {} 2023-09-29T22:59:52.917 DEBUG:boto:Token: None 2023-09-29T22:59:52.917 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.918 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uj4kJt4uENfimgXTvQWWoEmYiqs= 2023-09-29T22:59:52.918 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uj4kJt4uENfimgXTvQWWoEmYiqs=', 'Content-Length': '0'} 2023-09-29T22:59:52.920 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e61df353582f8d2a-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:59:52.920 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.920 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.920 DEBUG:boto:path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.921 DEBUG:boto:auth_path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.921 DEBUG:boto:Method: GET 2023-09-29T22:59:52.921 DEBUG:boto:Path: /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.921 DEBUG:boto:Data: 2023-09-29T22:59:52.921 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.921 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.921 DEBUG:boto:Params: {} 2023-09-29T22:59:52.921 DEBUG:boto:Token: None 2023-09-29T22:59:52.922 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc= 2023-09-29T22:59:52.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc=', 'Content-Length': '0'} 2023-09-29T22:59:52.924 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000441bc82ee778f565-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.924 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.924 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.925 DEBUG:boto:path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.925 DEBUG:boto:auth_path=/hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.925 DEBUG:boto:Method: GET 2023-09-29T22:59:52.925 DEBUG:boto:Path: /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.925 DEBUG:boto:Data: 2023-09-29T22:59:52.925 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.925 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.925 DEBUG:boto:Params: {} 2023-09-29T22:59:52.925 DEBUG:boto:Token: None 2023-09-29T22:59:52.926 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124?versionId=null 2023-09-29T22:59:52.926 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc= 2023-09-29T22:59:52.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8kbmgGbIAYv5/O5zVKEIrJoJkOc=', 'Content-Length': '0'} 2023-09-29T22:59:52.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000684982bdc328767d-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.970 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.971 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.971 DEBUG:boto:path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:52.971 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:52.971 DEBUG:boto:Method: GET 2023-09-29T22:59:52.971 DEBUG:boto:Path: /hcnphz-61/?prefix=obj124 2023-09-29T22:59:52.971 DEBUG:boto:Data: 2023-09-29T22:59:52.971 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.971 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.972 DEBUG:boto:Params: {} 2023-09-29T22:59:52.972 DEBUG:boto:Token: None 2023-09-29T22:59:52.972 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.977 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba63a5cfdeaa2a80-00651756e8-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.977 DEBUG:boto:b'hcnphz-61obj1241000falseobj1242023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.977 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:52.977 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:52.978 DEBUG:boto:path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:52.978 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:52.978 DEBUG:boto:Method: GET 2023-09-29T22:59:52.978 DEBUG:boto:Path: /hcnphz-61/?prefix=obj124 2023-09-29T22:59:52.978 DEBUG:boto:Data: 2023-09-29T22:59:52.978 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.978 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.978 DEBUG:boto:Params: {} 2023-09-29T22:59:52.979 DEBUG:boto:Token: None 2023-09-29T22:59:52.979 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/ 2023-09-29T22:59:52.979 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw= 2023-09-29T22:59:52.979 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JtXLVHNdKblpyUTdH0is1EoJRpw=', 'Content-Length': '0'} 2023-09-29T22:59:52.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c732cf6994d64dcf-00651756e8-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.987 DEBUG:boto:b'hcnphz-61obj1241000falseobj1242023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:52.988 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.989 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.989 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.989 DEBUG:boto:Path: /hcnphz-61/obj124 2023-09-29T22:59:52.989 DEBUG:boto:Data: 2023-09-29T22:59:52.989 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.990 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:52.990 DEBUG:boto:Port: 8001 2023-09-29T22:59:52.990 DEBUG:boto:Params: {} 2023-09-29T22:59:52.990 DEBUG:boto:Token: None 2023-09-29T22:59:52.991 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124 2023-09-29T22:59:52.991 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HxNGyaU/E5hsAyAEFqByMkJJHts= 2023-09-29T22:59:52.991 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HxNGyaU/E5hsAyAEFqByMkJJHts=', 'Content-Length': '0'} 2023-09-29T22:59:52.993 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c3752f4d773b530-00651756e8-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:52 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.993 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:52.994 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:52.994 DEBUG:boto:Method: HEAD 2023-09-29T22:59:52.994 DEBUG:boto:Path: /hcnphz-61/obj124 2023-09-29T22:59:52.994 DEBUG:boto:Data: 2023-09-29T22:59:52.995 DEBUG:boto:Headers: {} 2023-09-29T22:59:52.995 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:52.995 DEBUG:boto:Port: 8000 2023-09-29T22:59:52.995 DEBUG:boto:Params: {} 2023-09-29T22:59:52.995 DEBUG:boto:Token: None 2023-09-29T22:59:52.996 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:52 GMT /hcnphz-61/obj124 2023-09-29T22:59:52.996 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HxNGyaU/E5hsAyAEFqByMkJJHts= 2023-09-29T22:59:52.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:52 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HxNGyaU/E5hsAyAEFqByMkJJHts=', 'Content-Length': '0'} 2023-09-29T22:59:52.999 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6c016ea7f435784-00651756e8-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:52.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj124 2023-09-29T22:59:52.999 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:53.000 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:53.000 DEBUG:boto:Method: GET 2023-09-29T22:59:53.000 DEBUG:boto:Path: /hcnphz-61/obj124 2023-09-29T22:59:53.000 DEBUG:boto:Data: 2023-09-29T22:59:53.000 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.001 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.001 DEBUG:boto:Params: {} 2023-09-29T22:59:53.001 DEBUG:boto:Token: None 2023-09-29T22:59:53.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj124 2023-09-29T22:59:53.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IMwppcwD3j5iBsrDkapXtL5CQ1c= 2023-09-29T22:59:53.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IMwppcwD3j5iBsrDkapXtL5CQ1c=', 'Content-Length': '0'} 2023-09-29T22:59:53.004 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000027a053c2ff35ab4f-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.046 DEBUG:boto:path=/hcnphz-61/obj124 2023-09-29T22:59:53.047 DEBUG:boto:auth_path=/hcnphz-61/obj124 2023-09-29T22:59:53.047 DEBUG:boto:Method: GET 2023-09-29T22:59:53.047 DEBUG:boto:Path: /hcnphz-61/obj124 2023-09-29T22:59:53.047 DEBUG:boto:Data: 2023-09-29T22:59:53.047 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.047 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.048 DEBUG:boto:Params: {} 2023-09-29T22:59:53.048 DEBUG:boto:Token: None 2023-09-29T22:59:53.048 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj124 2023-09-29T22:59:53.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IMwppcwD3j5iBsrDkapXtL5CQ1c= 2023-09-29T22:59:53.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IMwppcwD3j5iBsrDkapXtL5CQ1c=', 'Content-Length': '0'} 2023-09-29T22:59:53.051 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f9012f0d971987-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.051 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.052 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.052 DEBUG:boto:path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:53.052 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:53.053 DEBUG:boto:Method: GET 2023-09-29T22:59:53.053 DEBUG:boto:Path: /hcnphz-61/?prefix=obj124 2023-09-29T22:59:53.053 DEBUG:boto:Data: 2023-09-29T22:59:53.053 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.053 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.053 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.053 DEBUG:boto:Params: {} 2023-09-29T22:59:53.053 DEBUG:boto:Token: None 2023-09-29T22:59:53.054 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.054 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.054 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.059 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000066d326bcc5dd59b3-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.059 DEBUG:boto:b'hcnphz-61obj1241000falseobj1242023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.059 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.060 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.060 DEBUG:boto:path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:53.060 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj124 2023-09-29T22:59:53.060 DEBUG:boto:Method: GET 2023-09-29T22:59:53.060 DEBUG:boto:Path: /hcnphz-61/?prefix=obj124 2023-09-29T22:59:53.060 DEBUG:boto:Data: 2023-09-29T22:59:53.060 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.060 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.060 DEBUG:boto:Params: {} 2023-09-29T22:59:53.061 DEBUG:boto:Token: None 2023-09-29T22:59:53.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.069 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d488363ca3240d7a-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.110 DEBUG:boto:b'hcnphz-61obj1241000falseobj1242023-09-29T22:55:14.951Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:59:53.112 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.112 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.112 DEBUG:boto:path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.112 DEBUG:boto:auth_path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.113 DEBUG:boto:Method: GET 2023-09-29T22:59:53.113 DEBUG:boto:Path: /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.113 DEBUG:boto:Data: 2023-09-29T22:59:53.113 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.114 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.114 DEBUG:boto:Params: {} 2023-09-29T22:59:53.114 DEBUG:boto:Token: None 2023-09-29T22:59:53.115 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.115 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U= 2023-09-29T22:59:53.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U=', 'Content-Length': '0'} 2023-09-29T22:59:53.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f7ef282f0cc7b01-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.119 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.119 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.120 DEBUG:boto:path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.120 DEBUG:boto:auth_path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.120 DEBUG:boto:Method: GET 2023-09-29T22:59:53.120 DEBUG:boto:Path: /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.120 DEBUG:boto:Data: 2023-09-29T22:59:53.121 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.121 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.121 DEBUG:boto:Params: {} 2023-09-29T22:59:53.121 DEBUG:boto:Token: None 2023-09-29T22:59:53.122 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.122 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U= 2023-09-29T22:59:53.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U=', 'Content-Length': '0'} 2023-09-29T22:59:53.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000432b8dd7342d50a4-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.126 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.126 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.126 DEBUG:boto:path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.126 DEBUG:boto:auth_path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.126 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.126 DEBUG:boto:Path: /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.126 DEBUG:boto:Data: 2023-09-29T22:59:53.127 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.127 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.127 DEBUG:boto:Params: {} 2023-09-29T22:59:53.127 DEBUG:boto:Token: None 2023-09-29T22:59:53.127 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.127 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TaL4z3PIp58hoXTfydPj91lY7yk= 2023-09-29T22:59:53.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TaL4z3PIp58hoXTfydPj91lY7yk=', 'Content-Length': '0'} 2023-09-29T22:59:53.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035d3516831bebc37-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.131 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.131 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.131 DEBUG:boto:path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.131 DEBUG:boto:auth_path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.131 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.132 DEBUG:boto:Path: /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.132 DEBUG:boto:Data: 2023-09-29T22:59:53.132 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.132 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.132 DEBUG:boto:Params: {} 2023-09-29T22:59:53.132 DEBUG:boto:Token: None 2023-09-29T22:59:53.133 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.133 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TaL4z3PIp58hoXTfydPj91lY7yk= 2023-09-29T22:59:53.133 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TaL4z3PIp58hoXTfydPj91lY7yk=', 'Content-Length': '0'} 2023-09-29T22:59:53.136 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edbfd0c9614d3cc0-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:59:53.136 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.136 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.136 DEBUG:boto:path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.137 DEBUG:boto:auth_path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.137 DEBUG:boto:Method: GET 2023-09-29T22:59:53.137 DEBUG:boto:Path: /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.137 DEBUG:boto:Data: 2023-09-29T22:59:53.137 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.137 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.137 DEBUG:boto:Params: {} 2023-09-29T22:59:53.138 DEBUG:boto:Token: None 2023-09-29T22:59:53.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U= 2023-09-29T22:59:53.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U=', 'Content-Length': '0'} 2023-09-29T22:59:53.141 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001468ed6017cd1455-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.142 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.142 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.142 DEBUG:boto:path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.142 DEBUG:boto:auth_path=/hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.143 DEBUG:boto:Method: GET 2023-09-29T22:59:53.143 DEBUG:boto:Path: /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.143 DEBUG:boto:Data: 2023-09-29T22:59:53.143 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.143 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.143 DEBUG:boto:Params: {} 2023-09-29T22:59:53.143 DEBUG:boto:Token: None 2023-09-29T22:59:53.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145?versionId=null 2023-09-29T22:59:53.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U= 2023-09-29T22:59:53.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RQyk6UNGpsbyrFHnwioQxnkxk+U=', 'Content-Length': '0'} 2023-09-29T22:59:53.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f590d3729814849e-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.147 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.147 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.148 DEBUG:boto:path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.148 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.148 DEBUG:boto:Method: GET 2023-09-29T22:59:53.148 DEBUG:boto:Path: /hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.148 DEBUG:boto:Data: 2023-09-29T22:59:53.148 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.149 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.149 DEBUG:boto:Params: {} 2023-09-29T22:59:53.149 DEBUG:boto:Token: None 2023-09-29T22:59:53.149 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.153 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ff75446259e80424-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.154 DEBUG:boto:b'hcnphz-61obj1451000falseobj1452023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.154 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.154 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.155 DEBUG:boto:path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.155 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.155 DEBUG:boto:Method: GET 2023-09-29T22:59:53.155 DEBUG:boto:Path: /hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.155 DEBUG:boto:Data: 2023-09-29T22:59:53.155 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.155 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.155 DEBUG:boto:Params: {} 2023-09-29T22:59:53.156 DEBUG:boto:Token: None 2023-09-29T22:59:53.156 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.156 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.164 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000027415a98ca40e524-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.206 DEBUG:boto:b'hcnphz-61obj1451000falseobj1452023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.207 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.207 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.207 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.208 DEBUG:boto:Path: /hcnphz-61/obj145 2023-09-29T22:59:53.208 DEBUG:boto:Data: 2023-09-29T22:59:53.208 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.209 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.209 DEBUG:boto:Params: {} 2023-09-29T22:59:53.209 DEBUG:boto:Token: None 2023-09-29T22:59:53.209 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145 2023-09-29T22:59:53.210 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a+Hqgco2aK7gJoZ7Fa9WDt2yjDQ= 2023-09-29T22:59:53.210 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a+Hqgco2aK7gJoZ7Fa9WDt2yjDQ=', 'Content-Length': '0'} 2023-09-29T22:59:53.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af0d36bddcd7cdab-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.214 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.214 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.214 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.214 DEBUG:boto:Path: /hcnphz-61/obj145 2023-09-29T22:59:53.214 DEBUG:boto:Data: 2023-09-29T22:59:53.214 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.215 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.215 DEBUG:boto:Params: {} 2023-09-29T22:59:53.215 DEBUG:boto:Token: None 2023-09-29T22:59:53.215 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145 2023-09-29T22:59:53.215 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a+Hqgco2aK7gJoZ7Fa9WDt2yjDQ= 2023-09-29T22:59:53.215 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a+Hqgco2aK7gJoZ7Fa9WDt2yjDQ=', 'Content-Length': '0'} 2023-09-29T22:59:53.218 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007cfe4891c0a578a8-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj145 2023-09-29T22:59:53.219 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.219 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.219 DEBUG:boto:Method: GET 2023-09-29T22:59:53.219 DEBUG:boto:Path: /hcnphz-61/obj145 2023-09-29T22:59:53.219 DEBUG:boto:Data: 2023-09-29T22:59:53.219 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.219 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.220 DEBUG:boto:Params: {} 2023-09-29T22:59:53.220 DEBUG:boto:Token: None 2023-09-29T22:59:53.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145 2023-09-29T22:59:53.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XXLUGTVLHmyOTxRd0jlJ3D2lKgo= 2023-09-29T22:59:53.220 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XXLUGTVLHmyOTxRd0jlJ3D2lKgo=', 'Content-Length': '0'} 2023-09-29T22:59:53.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ca868d63655d99d-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.223 DEBUG:boto:path=/hcnphz-61/obj145 2023-09-29T22:59:53.223 DEBUG:boto:auth_path=/hcnphz-61/obj145 2023-09-29T22:59:53.223 DEBUG:boto:Method: GET 2023-09-29T22:59:53.223 DEBUG:boto:Path: /hcnphz-61/obj145 2023-09-29T22:59:53.224 DEBUG:boto:Data: 2023-09-29T22:59:53.224 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.224 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.224 DEBUG:boto:Params: {} 2023-09-29T22:59:53.224 DEBUG:boto:Token: None 2023-09-29T22:59:53.224 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj145 2023-09-29T22:59:53.224 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XXLUGTVLHmyOTxRd0jlJ3D2lKgo= 2023-09-29T22:59:53.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XXLUGTVLHmyOTxRd0jlJ3D2lKgo=', 'Content-Length': '0'} 2023-09-29T22:59:53.227 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000110ada587adeba36-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.227 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.227 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.228 DEBUG:boto:path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.228 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.228 DEBUG:boto:Method: GET 2023-09-29T22:59:53.228 DEBUG:boto:Path: /hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.228 DEBUG:boto:Data: 2023-09-29T22:59:53.228 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.228 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.229 DEBUG:boto:Params: {} 2023-09-29T22:59:53.229 DEBUG:boto:Token: None 2023-09-29T22:59:53.229 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.233 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ed9b8263c2b0611e-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.278 DEBUG:boto:b'hcnphz-61obj1451000falseobj1452023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.279 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.279 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.279 DEBUG:boto:path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.280 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.280 DEBUG:boto:Method: GET 2023-09-29T22:59:53.280 DEBUG:boto:Path: /hcnphz-61/?prefix=obj145 2023-09-29T22:59:53.280 DEBUG:boto:Data: 2023-09-29T22:59:53.281 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.281 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.281 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.281 DEBUG:boto:Params: {} 2023-09-29T22:59:53.281 DEBUG:boto:Token: None 2023-09-29T22:59:53.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.292 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ff596fe09435a81-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.292 DEBUG:boto:b'hcnphz-61obj1451000falseobj1452023-09-29T22:55:14.533Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:59:53.293 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.293 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.294 DEBUG:boto:path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.294 DEBUG:boto:auth_path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.294 DEBUG:boto:Method: GET 2023-09-29T22:59:53.294 DEBUG:boto:Path: /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.294 DEBUG:boto:Data: 2023-09-29T22:59:53.295 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.295 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.295 DEBUG:boto:Params: {} 2023-09-29T22:59:53.296 DEBUG:boto:Token: None 2023-09-29T22:59:53.296 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8= 2023-09-29T22:59:53.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8=', 'Content-Length': '0'} 2023-09-29T22:59:53.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000975fda9d4e9630d7-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.299 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.299 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.299 DEBUG:boto:path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.300 DEBUG:boto:auth_path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.300 DEBUG:boto:Method: GET 2023-09-29T22:59:53.300 DEBUG:boto:Path: /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.300 DEBUG:boto:Data: 2023-09-29T22:59:53.301 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.301 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.301 DEBUG:boto:Params: {} 2023-09-29T22:59:53.301 DEBUG:boto:Token: None 2023-09-29T22:59:53.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8= 2023-09-29T22:59:53.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8=', 'Content-Length': '0'} 2023-09-29T22:59:53.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000822311498ffdecc6-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.305 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.305 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.305 DEBUG:boto:path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.305 DEBUG:boto:auth_path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.305 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.306 DEBUG:boto:Path: /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.306 DEBUG:boto:Data: 2023-09-29T22:59:53.306 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.306 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.307 DEBUG:boto:Params: {} 2023-09-29T22:59:53.307 DEBUG:boto:Token: None 2023-09-29T22:59:53.307 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C/ywAwyhqIaKFWQrCyaGBqKkzj8= 2023-09-29T22:59:53.308 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C/ywAwyhqIaKFWQrCyaGBqKkzj8=', 'Content-Length': '0'} 2023-09-29T22:59:53.310 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005406e358d0559f83-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.310 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.310 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.310 DEBUG:boto:path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.311 DEBUG:boto:auth_path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.311 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.311 DEBUG:boto:Path: /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.311 DEBUG:boto:Data: 2023-09-29T22:59:53.312 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.312 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.312 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.312 DEBUG:boto:Params: {} 2023-09-29T22:59:53.312 DEBUG:boto:Token: None 2023-09-29T22:59:53.313 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.313 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:C/ywAwyhqIaKFWQrCyaGBqKkzj8= 2023-09-29T22:59:53.313 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:C/ywAwyhqIaKFWQrCyaGBqKkzj8=', 'Content-Length': '0'} 2023-09-29T22:59:53.315 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130f8babf6f7b8eb-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:59:53.316 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.317 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.317 DEBUG:boto:path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.317 DEBUG:boto:auth_path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.317 DEBUG:boto:Method: GET 2023-09-29T22:59:53.318 DEBUG:boto:Path: /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.318 DEBUG:boto:Data: 2023-09-29T22:59:53.318 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.319 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.319 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.319 DEBUG:boto:Params: {} 2023-09-29T22:59:53.319 DEBUG:boto:Token: None 2023-09-29T22:59:53.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8= 2023-09-29T22:59:53.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8=', 'Content-Length': '0'} 2023-09-29T22:59:53.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009d2c273586d622b9-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.324 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.324 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.324 DEBUG:boto:path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.324 DEBUG:boto:auth_path=/hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.325 DEBUG:boto:Method: GET 2023-09-29T22:59:53.325 DEBUG:boto:Path: /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.325 DEBUG:boto:Data: 2023-09-29T22:59:53.325 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.326 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.326 DEBUG:boto:Params: {} 2023-09-29T22:59:53.326 DEBUG:boto:Token: None 2023-09-29T22:59:53.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148?versionId=null 2023-09-29T22:59:53.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8= 2023-09-29T22:59:53.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aEUvIRPLjon8dD9+Q4vf42cjwB8=', 'Content-Length': '0'} 2023-09-29T22:59:53.330 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004950c9cb089e877a-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.370 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.371 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.371 DEBUG:boto:path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.371 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.371 DEBUG:boto:Method: GET 2023-09-29T22:59:53.371 DEBUG:boto:Path: /hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.371 DEBUG:boto:Data: 2023-09-29T22:59:53.371 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.372 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.372 DEBUG:boto:Params: {} 2023-09-29T22:59:53.372 DEBUG:boto:Token: None 2023-09-29T22:59:53.372 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.377 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d01112c8e18faf03-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.377 DEBUG:boto:b'hcnphz-61obj1481000falseobj1482023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.378 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.378 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.378 DEBUG:boto:path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.379 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.379 DEBUG:boto:Method: GET 2023-09-29T22:59:53.379 DEBUG:boto:Path: /hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.379 DEBUG:boto:Data: 2023-09-29T22:59:53.379 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.379 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.379 DEBUG:boto:Params: {} 2023-09-29T22:59:53.380 DEBUG:boto:Token: None 2023-09-29T22:59:53.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.388 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006e24bef8a45a168d-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.388 DEBUG:boto:b'hcnphz-61obj1481000falseobj1482023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.389 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.389 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.389 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.389 DEBUG:boto:Path: /hcnphz-61/obj148 2023-09-29T22:59:53.389 DEBUG:boto:Data: 2023-09-29T22:59:53.389 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.390 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.390 DEBUG:boto:Params: {} 2023-09-29T22:59:53.390 DEBUG:boto:Token: None 2023-09-29T22:59:53.390 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148 2023-09-29T22:59:53.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n+CYndqt2C0Pkbn3Jf6qNRVbAKE= 2023-09-29T22:59:53.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n+CYndqt2C0Pkbn3Jf6qNRVbAKE=', 'Content-Length': '0'} 2023-09-29T22:59:53.393 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a36c7059badf646-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.393 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.393 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.393 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.394 DEBUG:boto:Path: /hcnphz-61/obj148 2023-09-29T22:59:53.394 DEBUG:boto:Data: 2023-09-29T22:59:53.394 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.394 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.394 DEBUG:boto:Params: {} 2023-09-29T22:59:53.394 DEBUG:boto:Token: None 2023-09-29T22:59:53.395 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148 2023-09-29T22:59:53.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n+CYndqt2C0Pkbn3Jf6qNRVbAKE= 2023-09-29T22:59:53.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n+CYndqt2C0Pkbn3Jf6qNRVbAKE=', 'Content-Length': '0'} 2023-09-29T22:59:53.397 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b3ba7a78e0961785-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj148 2023-09-29T22:59:53.397 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.398 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.398 DEBUG:boto:Method: GET 2023-09-29T22:59:53.398 DEBUG:boto:Path: /hcnphz-61/obj148 2023-09-29T22:59:53.398 DEBUG:boto:Data: 2023-09-29T22:59:53.398 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.399 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.399 DEBUG:boto:Params: {} 2023-09-29T22:59:53.399 DEBUG:boto:Token: None 2023-09-29T22:59:53.399 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148 2023-09-29T22:59:53.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFZDEBMqwgFdaGdkH9afLJezDkc= 2023-09-29T22:59:53.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFZDEBMqwgFdaGdkH9afLJezDkc=', 'Content-Length': '0'} 2023-09-29T22:59:53.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb0277eb72178d4c-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.442 DEBUG:boto:path=/hcnphz-61/obj148 2023-09-29T22:59:53.443 DEBUG:boto:auth_path=/hcnphz-61/obj148 2023-09-29T22:59:53.443 DEBUG:boto:Method: GET 2023-09-29T22:59:53.443 DEBUG:boto:Path: /hcnphz-61/obj148 2023-09-29T22:59:53.443 DEBUG:boto:Data: 2023-09-29T22:59:53.444 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.444 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.444 DEBUG:boto:Params: {} 2023-09-29T22:59:53.445 DEBUG:boto:Token: None 2023-09-29T22:59:53.445 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj148 2023-09-29T22:59:53.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFZDEBMqwgFdaGdkH9afLJezDkc= 2023-09-29T22:59:53.446 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFZDEBMqwgFdaGdkH9afLJezDkc=', 'Content-Length': '0'} 2023-09-29T22:59:53.449 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ff3c9752c8f70eef-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.450 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.450 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.450 DEBUG:boto:path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.450 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.451 DEBUG:boto:Method: GET 2023-09-29T22:59:53.451 DEBUG:boto:Path: /hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.451 DEBUG:boto:Data: 2023-09-29T22:59:53.451 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.452 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.452 DEBUG:boto:Params: {} 2023-09-29T22:59:53.452 DEBUG:boto:Token: None 2023-09-29T22:59:53.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000895ce51ebe1b2ec1-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.458 DEBUG:boto:b'hcnphz-61obj1481000falseobj1482023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.458 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.459 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.459 DEBUG:boto:path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.459 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.459 DEBUG:boto:Method: GET 2023-09-29T22:59:53.460 DEBUG:boto:Path: /hcnphz-61/?prefix=obj148 2023-09-29T22:59:53.460 DEBUG:boto:Data: 2023-09-29T22:59:53.460 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.461 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.461 DEBUG:boto:Params: {} 2023-09-29T22:59:53.461 DEBUG:boto:Token: None 2023-09-29T22:59:53.461 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.462 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.462 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.470 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b8b383774046d82-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.471 DEBUG:boto:b'hcnphz-61obj1481000falseobj1482023-09-29T22:55:15.438Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:59:53.472 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.472 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.472 DEBUG:boto:path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.472 DEBUG:boto:auth_path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.472 DEBUG:boto:Method: GET 2023-09-29T22:59:53.472 DEBUG:boto:Path: /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.473 DEBUG:boto:Data: 2023-09-29T22:59:53.473 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.473 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.473 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.473 DEBUG:boto:Params: {} 2023-09-29T22:59:53.473 DEBUG:boto:Token: None 2023-09-29T22:59:53.473 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.474 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM= 2023-09-29T22:59:53.474 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM=', 'Content-Length': '0'} 2023-09-29T22:59:53.476 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000929c0166b5d27eb3-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.477 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.477 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.477 DEBUG:boto:path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.477 DEBUG:boto:auth_path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.477 DEBUG:boto:Method: GET 2023-09-29T22:59:53.478 DEBUG:boto:Path: /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.478 DEBUG:boto:Data: 2023-09-29T22:59:53.478 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.478 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.478 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.478 DEBUG:boto:Params: {} 2023-09-29T22:59:53.478 DEBUG:boto:Token: None 2023-09-29T22:59:53.479 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.479 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM= 2023-09-29T22:59:53.479 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM=', 'Content-Length': '0'} 2023-09-29T22:59:53.481 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005260ee8eea197d6e-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.526 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.527 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.527 DEBUG:boto:path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.527 DEBUG:boto:auth_path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.527 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.527 DEBUG:boto:Path: /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.527 DEBUG:boto:Data: 2023-09-29T22:59:53.527 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.528 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.528 DEBUG:boto:Params: {} 2023-09-29T22:59:53.528 DEBUG:boto:Token: None 2023-09-29T22:59:53.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tzrj2j1vGLFizxjb1bhOfsOL9uw= 2023-09-29T22:59:53.529 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tzrj2j1vGLFizxjb1bhOfsOL9uw=', 'Content-Length': '0'} 2023-09-29T22:59:53.531 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000063963e5aab58040d-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.532 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.532 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.532 DEBUG:boto:path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.532 DEBUG:boto:auth_path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.532 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.533 DEBUG:boto:Path: /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.533 DEBUG:boto:Data: 2023-09-29T22:59:53.533 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.533 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.533 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.533 DEBUG:boto:Params: {} 2023-09-29T22:59:53.533 DEBUG:boto:Token: None 2023-09-29T22:59:53.533 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.534 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Tzrj2j1vGLFizxjb1bhOfsOL9uw= 2023-09-29T22:59:53.534 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tzrj2j1vGLFizxjb1bhOfsOL9uw=', 'Content-Length': '0'} 2023-09-29T22:59:53.537 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f8b6d0bf733a9b-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:59:53.537 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.537 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.537 DEBUG:boto:path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.538 DEBUG:boto:auth_path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.538 DEBUG:boto:Method: GET 2023-09-29T22:59:53.538 DEBUG:boto:Path: /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.538 DEBUG:boto:Data: 2023-09-29T22:59:53.538 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.538 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.538 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.538 DEBUG:boto:Params: {} 2023-09-29T22:59:53.539 DEBUG:boto:Token: None 2023-09-29T22:59:53.539 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.539 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM= 2023-09-29T22:59:53.539 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM=', 'Content-Length': '0'} 2023-09-29T22:59:53.542 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005a5ab3650805ed78-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.582 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.583 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.583 DEBUG:boto:path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.583 DEBUG:boto:auth_path=/hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.583 DEBUG:boto:Method: GET 2023-09-29T22:59:53.583 DEBUG:boto:Path: /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.583 DEBUG:boto:Data: 2023-09-29T22:59:53.583 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.584 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.584 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.584 DEBUG:boto:Params: {} 2023-09-29T22:59:53.584 DEBUG:boto:Token: None 2023-09-29T22:59:53.584 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155?versionId=null 2023-09-29T22:59:53.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM= 2023-09-29T22:59:53.585 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/OIqzaxDirUENjbSjnQYXwwhIfM=', 'Content-Length': '0'} 2023-09-29T22:59:53.587 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d3a623d1325df8a5-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.588 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.588 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.588 DEBUG:boto:path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.588 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.588 DEBUG:boto:Method: GET 2023-09-29T22:59:53.589 DEBUG:boto:Path: /hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.589 DEBUG:boto:Data: 2023-09-29T22:59:53.589 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.589 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.589 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.589 DEBUG:boto:Params: {} 2023-09-29T22:59:53.590 DEBUG:boto:Token: None 2023-09-29T22:59:53.590 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.590 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.596 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006168927e2b327e94-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.597 DEBUG:boto:b'hcnphz-61obj1551000falseobj1552023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.597 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.597 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.598 DEBUG:boto:path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.598 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.598 DEBUG:boto:Method: GET 2023-09-29T22:59:53.599 DEBUG:boto:Path: /hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.599 DEBUG:boto:Data: 2023-09-29T22:59:53.599 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.599 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.600 DEBUG:boto:Params: {} 2023-09-29T22:59:53.600 DEBUG:boto:Token: None 2023-09-29T22:59:53.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.600 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.608 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004ac97ebfba1bf529-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.609 DEBUG:boto:b'hcnphz-61obj1551000falseobj1552023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.609 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.609 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.610 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.610 DEBUG:boto:Path: /hcnphz-61/obj155 2023-09-29T22:59:53.610 DEBUG:boto:Data: 2023-09-29T22:59:53.611 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.611 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.611 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.611 DEBUG:boto:Params: {} 2023-09-29T22:59:53.612 DEBUG:boto:Token: None 2023-09-29T22:59:53.612 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155 2023-09-29T22:59:53.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0mjN7Ovkyt5BE0jwMVDigvDsI4o= 2023-09-29T22:59:53.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0mjN7Ovkyt5BE0jwMVDigvDsI4o=', 'Content-Length': '0'} 2023-09-29T22:59:53.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002ad1d955c2e1076-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.615 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.616 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.616 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.616 DEBUG:boto:Path: /hcnphz-61/obj155 2023-09-29T22:59:53.616 DEBUG:boto:Data: 2023-09-29T22:59:53.617 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.617 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.617 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.617 DEBUG:boto:Params: {} 2023-09-29T22:59:53.617 DEBUG:boto:Token: None 2023-09-29T22:59:53.618 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155 2023-09-29T22:59:53.618 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0mjN7Ovkyt5BE0jwMVDigvDsI4o= 2023-09-29T22:59:53.618 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0mjN7Ovkyt5BE0jwMVDigvDsI4o=', 'Content-Length': '0'} 2023-09-29T22:59:53.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065e0ab49ea9a1222-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj155 2023-09-29T22:59:53.622 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.622 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.622 DEBUG:boto:Method: GET 2023-09-29T22:59:53.622 DEBUG:boto:Path: /hcnphz-61/obj155 2023-09-29T22:59:53.623 DEBUG:boto:Data: 2023-09-29T22:59:53.623 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.623 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.623 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.623 DEBUG:boto:Params: {} 2023-09-29T22:59:53.624 DEBUG:boto:Token: None 2023-09-29T22:59:53.624 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155 2023-09-29T22:59:53.624 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Tx1kCJodu0YT03+cPzArx+sFxg= 2023-09-29T22:59:53.625 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Tx1kCJodu0YT03+cPzArx+sFxg=', 'Content-Length': '0'} 2023-09-29T22:59:53.627 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4c881d57c69228d-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.674 DEBUG:boto:path=/hcnphz-61/obj155 2023-09-29T22:59:53.675 DEBUG:boto:auth_path=/hcnphz-61/obj155 2023-09-29T22:59:53.675 DEBUG:boto:Method: GET 2023-09-29T22:59:53.675 DEBUG:boto:Path: /hcnphz-61/obj155 2023-09-29T22:59:53.676 DEBUG:boto:Data: 2023-09-29T22:59:53.676 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.676 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.676 DEBUG:boto:Params: {} 2023-09-29T22:59:53.677 DEBUG:boto:Token: None 2023-09-29T22:59:53.677 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj155 2023-09-29T22:59:53.677 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Tx1kCJodu0YT03+cPzArx+sFxg= 2023-09-29T22:59:53.678 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Tx1kCJodu0YT03+cPzArx+sFxg=', 'Content-Length': '0'} 2023-09-29T22:59:53.681 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004910810c10cfc302-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.682 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.682 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.682 DEBUG:boto:path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.682 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.682 DEBUG:boto:Method: GET 2023-09-29T22:59:53.683 DEBUG:boto:Path: /hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.683 DEBUG:boto:Data: 2023-09-29T22:59:53.683 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.683 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.683 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.683 DEBUG:boto:Params: {} 2023-09-29T22:59:53.683 DEBUG:boto:Token: None 2023-09-29T22:59:53.684 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.684 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.684 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.688 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000973093728f809d38-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.688 DEBUG:boto:b'hcnphz-61obj1551000falseobj1552023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.689 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.689 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.689 DEBUG:boto:path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.690 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.690 DEBUG:boto:Method: GET 2023-09-29T22:59:53.690 DEBUG:boto:Path: /hcnphz-61/?prefix=obj155 2023-09-29T22:59:53.690 DEBUG:boto:Data: 2023-09-29T22:59:53.690 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.691 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.691 DEBUG:boto:Params: {} 2023-09-29T22:59:53.691 DEBUG:boto:Token: None 2023-09-29T22:59:53.692 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.700 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005cd09bd3140bab70-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.700 DEBUG:boto:b'hcnphz-61obj1551000falseobj1552023-09-29T22:55:15.005Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:59:53.701 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.701 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.701 DEBUG:boto:path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.701 DEBUG:boto:auth_path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.702 DEBUG:boto:Method: GET 2023-09-29T22:59:53.702 DEBUG:boto:Path: /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.702 DEBUG:boto:Data: 2023-09-29T22:59:53.702 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.703 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.703 DEBUG:boto:Params: {} 2023-09-29T22:59:53.703 DEBUG:boto:Token: None 2023-09-29T22:59:53.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg= 2023-09-29T22:59:53.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg=', 'Content-Length': '0'} 2023-09-29T22:59:53.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d22c91aa0f6f8136-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.707 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.707 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.707 DEBUG:boto:path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.708 DEBUG:boto:auth_path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.708 DEBUG:boto:Method: GET 2023-09-29T22:59:53.708 DEBUG:boto:Path: /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.708 DEBUG:boto:Data: 2023-09-29T22:59:53.708 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.709 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.709 DEBUG:boto:Params: {} 2023-09-29T22:59:53.709 DEBUG:boto:Token: None 2023-09-29T22:59:53.710 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg= 2023-09-29T22:59:53.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg=', 'Content-Length': '0'} 2023-09-29T22:59:53.713 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d256baed2ec6f9a-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.713 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.713 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.714 DEBUG:boto:path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.714 DEBUG:boto:auth_path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.714 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.714 DEBUG:boto:Path: /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.714 DEBUG:boto:Data: 2023-09-29T22:59:53.715 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.715 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.715 DEBUG:boto:Params: {} 2023-09-29T22:59:53.716 DEBUG:boto:Token: None 2023-09-29T22:59:53.716 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HKGlKIEKsLeghx5hHhQjDkNzeR0= 2023-09-29T22:59:53.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HKGlKIEKsLeghx5hHhQjDkNzeR0=', 'Content-Length': '0'} 2023-09-29T22:59:53.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000813f7dabd9ab6630-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.719 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.720 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.720 DEBUG:boto:path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.720 DEBUG:boto:auth_path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.721 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.721 DEBUG:boto:Path: /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.721 DEBUG:boto:Data: 2023-09-29T22:59:53.721 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.722 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.722 DEBUG:boto:Params: {} 2023-09-29T22:59:53.722 DEBUG:boto:Token: None 2023-09-29T22:59:53.723 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HKGlKIEKsLeghx5hHhQjDkNzeR0= 2023-09-29T22:59:53.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HKGlKIEKsLeghx5hHhQjDkNzeR0=', 'Content-Length': '0'} 2023-09-29T22:59:53.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb9e8642da8aec99-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:59:53.726 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.727 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.727 DEBUG:boto:path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.727 DEBUG:boto:auth_path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.727 DEBUG:boto:Method: GET 2023-09-29T22:59:53.728 DEBUG:boto:Path: /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.728 DEBUG:boto:Data: 2023-09-29T22:59:53.728 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.729 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.729 DEBUG:boto:Params: {} 2023-09-29T22:59:53.729 DEBUG:boto:Token: None 2023-09-29T22:59:53.729 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg= 2023-09-29T22:59:53.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg=', 'Content-Length': '0'} 2023-09-29T22:59:53.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078fb014cea3a8b98-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.774 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.775 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.775 DEBUG:boto:path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.775 DEBUG:boto:auth_path=/hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.776 DEBUG:boto:Method: GET 2023-09-29T22:59:53.776 DEBUG:boto:Path: /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.776 DEBUG:boto:Data: 2023-09-29T22:59:53.776 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.777 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.777 DEBUG:boto:Params: {} 2023-09-29T22:59:53.777 DEBUG:boto:Token: None 2023-09-29T22:59:53.778 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17?versionId=null 2023-09-29T22:59:53.778 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg= 2023-09-29T22:59:53.778 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5zLuu98K/m1F7F0plAqBTdGaryg=', 'Content-Length': '0'} 2023-09-29T22:59:53.781 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000aecae0e457bc6e6-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.782 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.782 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.782 DEBUG:boto:path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.782 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.782 DEBUG:boto:Method: GET 2023-09-29T22:59:53.782 DEBUG:boto:Path: /hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.782 DEBUG:boto:Data: 2023-09-29T22:59:53.782 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.783 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.783 DEBUG:boto:Params: {} 2023-09-29T22:59:53.783 DEBUG:boto:Token: None 2023-09-29T22:59:53.783 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.783 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.787 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018cdcc57fb0e7487-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.788 DEBUG:boto:b'hcnphz-61obj171000falseobj172023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.788 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.788 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.788 DEBUG:boto:path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.789 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.789 DEBUG:boto:Method: GET 2023-09-29T22:59:53.789 DEBUG:boto:Path: /hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.789 DEBUG:boto:Data: 2023-09-29T22:59:53.789 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.789 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.789 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.789 DEBUG:boto:Params: {} 2023-09-29T22:59:53.789 DEBUG:boto:Token: None 2023-09-29T22:59:53.789 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.790 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.790 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.798 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000050851c9c641fb3ed-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.798 DEBUG:boto:b'hcnphz-61obj171000falseobj172023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.798 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.798 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.799 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.799 DEBUG:boto:Path: /hcnphz-61/obj17 2023-09-29T22:59:53.799 DEBUG:boto:Data: 2023-09-29T22:59:53.799 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.799 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.799 DEBUG:boto:Params: {} 2023-09-29T22:59:53.799 DEBUG:boto:Token: None 2023-09-29T22:59:53.799 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17 2023-09-29T22:59:53.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PUmC4rrknbIRHVdVccqeMj2AIA8= 2023-09-29T22:59:53.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PUmC4rrknbIRHVdVccqeMj2AIA8=', 'Content-Length': '0'} 2023-09-29T22:59:53.802 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006caddc0fe1df8458-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.802 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.802 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.802 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.802 DEBUG:boto:Path: /hcnphz-61/obj17 2023-09-29T22:59:53.803 DEBUG:boto:Data: 2023-09-29T22:59:53.803 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.803 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.803 DEBUG:boto:Params: {} 2023-09-29T22:59:53.803 DEBUG:boto:Token: None 2023-09-29T22:59:53.803 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17 2023-09-29T22:59:53.803 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PUmC4rrknbIRHVdVccqeMj2AIA8= 2023-09-29T22:59:53.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PUmC4rrknbIRHVdVccqeMj2AIA8=', 'Content-Length': '0'} 2023-09-29T22:59:53.806 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006fa07baee3b00854-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj17 2023-09-29T22:59:53.806 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.806 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.807 DEBUG:boto:Method: GET 2023-09-29T22:59:53.807 DEBUG:boto:Path: /hcnphz-61/obj17 2023-09-29T22:59:53.807 DEBUG:boto:Data: 2023-09-29T22:59:53.807 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.808 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.808 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.808 DEBUG:boto:Params: {} 2023-09-29T22:59:53.808 DEBUG:boto:Token: None 2023-09-29T22:59:53.808 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17 2023-09-29T22:59:53.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHhYKa9NQqftxDgjzTolI4opEko= 2023-09-29T22:59:53.809 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHhYKa9NQqftxDgjzTolI4opEko=', 'Content-Length': '0'} 2023-09-29T22:59:53.811 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e387f2065642a9b-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.812 DEBUG:boto:path=/hcnphz-61/obj17 2023-09-29T22:59:53.812 DEBUG:boto:auth_path=/hcnphz-61/obj17 2023-09-29T22:59:53.812 DEBUG:boto:Method: GET 2023-09-29T22:59:53.812 DEBUG:boto:Path: /hcnphz-61/obj17 2023-09-29T22:59:53.812 DEBUG:boto:Data: 2023-09-29T22:59:53.812 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.813 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.813 DEBUG:boto:Params: {} 2023-09-29T22:59:53.813 DEBUG:boto:Token: None 2023-09-29T22:59:53.813 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj17 2023-09-29T22:59:53.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yHhYKa9NQqftxDgjzTolI4opEko= 2023-09-29T22:59:53.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yHhYKa9NQqftxDgjzTolI4opEko=', 'Content-Length': '0'} 2023-09-29T22:59:53.816 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000365852316797d1d9-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.816 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.816 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.816 DEBUG:boto:path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.817 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.817 DEBUG:boto:Method: GET 2023-09-29T22:59:53.817 DEBUG:boto:Path: /hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.817 DEBUG:boto:Data: 2023-09-29T22:59:53.817 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.818 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.818 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.818 DEBUG:boto:Params: {} 2023-09-29T22:59:53.818 DEBUG:boto:Token: None 2023-09-29T22:59:53.819 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.819 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.819 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.823 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000315ddb29cc488e13-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.866 DEBUG:boto:b'hcnphz-61obj171000falseobj172023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.867 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.867 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.867 DEBUG:boto:path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.867 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.868 DEBUG:boto:Method: GET 2023-09-29T22:59:53.868 DEBUG:boto:Path: /hcnphz-61/?prefix=obj17 2023-09-29T22:59:53.868 DEBUG:boto:Data: 2023-09-29T22:59:53.868 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.868 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.868 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.868 DEBUG:boto:Params: {} 2023-09-29T22:59:53.869 DEBUG:boto:Token: None 2023-09-29T22:59:53.869 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.869 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.869 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.878 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005af278b1dddb5d56-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.879 DEBUG:boto:b'hcnphz-61obj171000falseobj172023-09-29T22:55:13.130Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:59:53.879 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:53.880 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:53.880 DEBUG:boto:path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.880 DEBUG:boto:auth_path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.880 DEBUG:boto:Method: GET 2023-09-29T22:59:53.880 DEBUG:boto:Path: /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.880 DEBUG:boto:Data: 2023-09-29T22:59:53.880 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.881 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.881 DEBUG:boto:Params: {} 2023-09-29T22:59:53.881 DEBUG:boto:Token: None 2023-09-29T22:59:53.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.882 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c= 2023-09-29T22:59:53.882 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c=', 'Content-Length': '0'} 2023-09-29T22:59:53.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045d8f90bb82a2e10-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.885 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:53.885 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:53.885 DEBUG:boto:path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.885 DEBUG:boto:auth_path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.885 DEBUG:boto:Method: GET 2023-09-29T22:59:53.885 DEBUG:boto:Path: /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.885 DEBUG:boto:Data: 2023-09-29T22:59:53.886 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.886 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.886 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.886 DEBUG:boto:Params: {} 2023-09-29T22:59:53.886 DEBUG:boto:Token: None 2023-09-29T22:59:53.886 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c= 2023-09-29T22:59:53.887 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c=', 'Content-Length': '0'} 2023-09-29T22:59:53.889 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043a70af388e510e0-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.934 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:53.935 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:53.935 DEBUG:boto:path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.935 DEBUG:boto:auth_path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.936 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.936 DEBUG:boto:Path: /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.936 DEBUG:boto:Data: 2023-09-29T22:59:53.936 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.937 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.937 DEBUG:boto:Params: {} 2023-09-29T22:59:53.937 DEBUG:boto:Token: None 2023-09-29T22:59:53.937 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.938 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIWFFicpyjk3T6ZIduLUAF7eKKA= 2023-09-29T22:59:53.938 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIWFFicpyjk3T6ZIduLUAF7eKKA=', 'Content-Length': '0'} 2023-09-29T22:59:53.941 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f366b343e98f778-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.941 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:53.941 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:53.942 DEBUG:boto:path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.942 DEBUG:boto:auth_path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.942 DEBUG:boto:Method: HEAD 2023-09-29T22:59:53.942 DEBUG:boto:Path: /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.942 DEBUG:boto:Data: 2023-09-29T22:59:53.943 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.943 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.943 DEBUG:boto:Params: {} 2023-09-29T22:59:53.943 DEBUG:boto:Token: None 2023-09-29T22:59:53.943 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.943 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KIWFFicpyjk3T6ZIduLUAF7eKKA= 2023-09-29T22:59:53.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KIWFFicpyjk3T6ZIduLUAF7eKKA=', 'Content-Length': '0'} 2023-09-29T22:59:53.946 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bccaaebb4a03438a-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:59:53.947 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:53.947 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:53.947 DEBUG:boto:path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.947 DEBUG:boto:auth_path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.947 DEBUG:boto:Method: GET 2023-09-29T22:59:53.948 DEBUG:boto:Path: /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.948 DEBUG:boto:Data: 2023-09-29T22:59:53.948 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.948 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.948 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.949 DEBUG:boto:Params: {} 2023-09-29T22:59:53.949 DEBUG:boto:Token: None 2023-09-29T22:59:53.949 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c= 2023-09-29T22:59:53.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c=', 'Content-Length': '0'} 2023-09-29T22:59:53.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef293272b6bcf40c-00651756e9-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.953 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:53.953 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:53.953 DEBUG:boto:path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.953 DEBUG:boto:auth_path=/hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.954 DEBUG:boto:Method: GET 2023-09-29T22:59:53.954 DEBUG:boto:Path: /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.954 DEBUG:boto:Data: 2023-09-29T22:59:53.954 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.955 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.955 DEBUG:boto:Params: {} 2023-09-29T22:59:53.955 DEBUG:boto:Token: None 2023-09-29T22:59:53.955 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/obj185?versionId=null 2023-09-29T22:59:53.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c= 2023-09-29T22:59:53.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VXQ160SQDlyEN3W78YpWlRjT94c=', 'Content-Length': '0'} 2023-09-29T22:59:53.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000072bed058cfda3b-00651756e9-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.959 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.959 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.959 DEBUG:boto:path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:53.960 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:53.960 DEBUG:boto:Method: GET 2023-09-29T22:59:53.960 DEBUG:boto:Path: /hcnphz-61/?prefix=obj185 2023-09-29T22:59:53.960 DEBUG:boto:Data: 2023-09-29T22:59:53.960 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.961 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:53.961 DEBUG:boto:Port: 8001 2023-09-29T22:59:53.961 DEBUG:boto:Params: {} 2023-09-29T22:59:53.961 DEBUG:boto:Token: None 2023-09-29T22:59:53.962 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.962 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.962 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.967 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d456bad7c2b1f4bb-00651756e9-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:53.967 DEBUG:boto:b'hcnphz-61obj1851000falseobj1852023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:53.968 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:53.968 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:53.968 DEBUG:boto:path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:53.968 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:53.968 DEBUG:boto:Method: GET 2023-09-29T22:59:53.969 DEBUG:boto:Path: /hcnphz-61/?prefix=obj185 2023-09-29T22:59:53.969 DEBUG:boto:Data: 2023-09-29T22:59:53.969 DEBUG:boto:Headers: {} 2023-09-29T22:59:53.969 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:53.970 DEBUG:boto:Port: 8000 2023-09-29T22:59:53.970 DEBUG:boto:Params: {} 2023-09-29T22:59:53.970 DEBUG:boto:Token: None 2023-09-29T22:59:53.970 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:53 GMT /hcnphz-61/ 2023-09-29T22:59:53.971 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE= 2023-09-29T22:59:53.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:53 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:988c9C/NbpvGpT0MJgZssuu/JUE=', 'Content-Length': '0'} 2023-09-29T22:59:53.978 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000020526b7f84d5f5ce-00651756e9-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:53 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.022 DEBUG:boto:b'hcnphz-61obj1851000falseobj1852023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.023 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:54.023 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:54.024 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.024 DEBUG:boto:Path: /hcnphz-61/obj185 2023-09-29T22:59:54.024 DEBUG:boto:Data: 2023-09-29T22:59:54.026 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.026 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.026 DEBUG:boto:Params: {} 2023-09-29T22:59:54.026 DEBUG:boto:Token: None 2023-09-29T22:59:54.026 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj185 2023-09-29T22:59:54.027 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gaXhZhBqmHWZ3T9XDrqTgLy8mOo= 2023-09-29T22:59:54.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gaXhZhBqmHWZ3T9XDrqTgLy8mOo=', 'Content-Length': '0'} 2023-09-29T22:59:54.030 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e6a26bd3c829c89-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.030 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:54.030 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:54.030 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.030 DEBUG:boto:Path: /hcnphz-61/obj185 2023-09-29T22:59:54.031 DEBUG:boto:Data: 2023-09-29T22:59:54.031 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.031 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.031 DEBUG:boto:Params: {} 2023-09-29T22:59:54.031 DEBUG:boto:Token: None 2023-09-29T22:59:54.031 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj185 2023-09-29T22:59:54.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gaXhZhBqmHWZ3T9XDrqTgLy8mOo= 2023-09-29T22:59:54.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gaXhZhBqmHWZ3T9XDrqTgLy8mOo=', 'Content-Length': '0'} 2023-09-29T22:59:54.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e92edfa1e4004c26-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj185 2023-09-29T22:59:54.035 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:54.035 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:54.035 DEBUG:boto:Method: GET 2023-09-29T22:59:54.035 DEBUG:boto:Path: /hcnphz-61/obj185 2023-09-29T22:59:54.035 DEBUG:boto:Data: 2023-09-29T22:59:54.035 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.035 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.035 DEBUG:boto:Params: {} 2023-09-29T22:59:54.036 DEBUG:boto:Token: None 2023-09-29T22:59:54.036 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj185 2023-09-29T22:59:54.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lSWEetgyXlfywd5Cz7T2DybZSI4= 2023-09-29T22:59:54.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lSWEetgyXlfywd5Cz7T2DybZSI4=', 'Content-Length': '0'} 2023-09-29T22:59:54.039 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c79cd0aef2213cd3-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.039 DEBUG:boto:path=/hcnphz-61/obj185 2023-09-29T22:59:54.039 DEBUG:boto:auth_path=/hcnphz-61/obj185 2023-09-29T22:59:54.040 DEBUG:boto:Method: GET 2023-09-29T22:59:54.040 DEBUG:boto:Path: /hcnphz-61/obj185 2023-09-29T22:59:54.040 DEBUG:boto:Data: 2023-09-29T22:59:54.041 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.041 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.042 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.042 DEBUG:boto:Params: {} 2023-09-29T22:59:54.042 DEBUG:boto:Token: None 2023-09-29T22:59:54.042 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj185 2023-09-29T22:59:54.042 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lSWEetgyXlfywd5Cz7T2DybZSI4= 2023-09-29T22:59:54.042 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lSWEetgyXlfywd5Cz7T2DybZSI4=', 'Content-Length': '0'} 2023-09-29T22:59:54.045 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000554fca1d7d7ee321-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.046 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.046 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.046 DEBUG:boto:path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:54.046 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:54.046 DEBUG:boto:Method: GET 2023-09-29T22:59:54.046 DEBUG:boto:Path: /hcnphz-61/?prefix=obj185 2023-09-29T22:59:54.046 DEBUG:boto:Data: 2023-09-29T22:59:54.046 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.047 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.047 DEBUG:boto:Params: {} 2023-09-29T22:59:54.047 DEBUG:boto:Token: None 2023-09-29T22:59:54.047 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.047 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.052 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f1ffb34f4caf4dd8-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.094 DEBUG:boto:b'hcnphz-61obj1851000falseobj1852023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.095 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.095 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.095 DEBUG:boto:path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:54.095 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj185 2023-09-29T22:59:54.096 DEBUG:boto:Method: GET 2023-09-29T22:59:54.096 DEBUG:boto:Path: /hcnphz-61/?prefix=obj185 2023-09-29T22:59:54.096 DEBUG:boto:Data: 2023-09-29T22:59:54.096 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.096 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.096 DEBUG:boto:Params: {} 2023-09-29T22:59:54.096 DEBUG:boto:Token: None 2023-09-29T22:59:54.097 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009c60c2ae3799fde3-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.106 DEBUG:boto:b'hcnphz-61obj1851000falseobj1852023-09-29T22:55:15.457Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:59:54.107 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.107 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.107 DEBUG:boto:path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.107 DEBUG:boto:auth_path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.107 DEBUG:boto:Method: GET 2023-09-29T22:59:54.107 DEBUG:boto:Path: /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.108 DEBUG:boto:Data: 2023-09-29T22:59:54.108 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.108 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.108 DEBUG:boto:Params: {} 2023-09-29T22:59:54.108 DEBUG:boto:Token: None 2023-09-29T22:59:54.108 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.108 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA= 2023-09-29T22:59:54.109 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA=', 'Content-Length': '0'} 2023-09-29T22:59:54.110 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000627ebd12d364cc1e-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.111 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.111 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.111 DEBUG:boto:path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.111 DEBUG:boto:auth_path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.111 DEBUG:boto:Method: GET 2023-09-29T22:59:54.111 DEBUG:boto:Path: /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.112 DEBUG:boto:Data: 2023-09-29T22:59:54.112 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.112 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.112 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.112 DEBUG:boto:Params: {} 2023-09-29T22:59:54.112 DEBUG:boto:Token: None 2023-09-29T22:59:54.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA= 2023-09-29T22:59:54.113 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA=', 'Content-Length': '0'} 2023-09-29T22:59:54.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000559a3003a52509e9-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.158 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.159 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.159 DEBUG:boto:path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.159 DEBUG:boto:auth_path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.159 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.159 DEBUG:boto:Path: /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.159 DEBUG:boto:Data: 2023-09-29T22:59:54.159 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.160 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.160 DEBUG:boto:Params: {} 2023-09-29T22:59:54.161 DEBUG:boto:Token: None 2023-09-29T22:59:54.161 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.162 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BAEs+bh2lbhOV3AbLK47kB+pJGE= 2023-09-29T22:59:54.162 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BAEs+bh2lbhOV3AbLK47kB+pJGE=', 'Content-Length': '0'} 2023-09-29T22:59:54.164 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009552d6608da7be1a-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.164 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.164 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.165 DEBUG:boto:path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.165 DEBUG:boto:auth_path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.165 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.165 DEBUG:boto:Path: /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.165 DEBUG:boto:Data: 2023-09-29T22:59:54.165 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.165 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.165 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.165 DEBUG:boto:Params: {} 2023-09-29T22:59:54.166 DEBUG:boto:Token: None 2023-09-29T22:59:54.166 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.166 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BAEs+bh2lbhOV3AbLK47kB+pJGE= 2023-09-29T22:59:54.166 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BAEs+bh2lbhOV3AbLK47kB+pJGE=', 'Content-Length': '0'} 2023-09-29T22:59:54.171 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067a5bfda4717b592-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:59:54.171 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.172 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.172 DEBUG:boto:path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.172 DEBUG:boto:auth_path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.172 DEBUG:boto:Method: GET 2023-09-29T22:59:54.172 DEBUG:boto:Path: /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.172 DEBUG:boto:Data: 2023-09-29T22:59:54.172 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.172 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.173 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.173 DEBUG:boto:Params: {} 2023-09-29T22:59:54.173 DEBUG:boto:Token: None 2023-09-29T22:59:54.173 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.173 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA= 2023-09-29T22:59:54.173 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA=', 'Content-Length': '0'} 2023-09-29T22:59:54.176 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000043f1ae204ffb155-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.177 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.177 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.178 DEBUG:boto:path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.178 DEBUG:boto:auth_path=/hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.178 DEBUG:boto:Method: GET 2023-09-29T22:59:54.178 DEBUG:boto:Path: /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.178 DEBUG:boto:Data: 2023-09-29T22:59:54.178 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.178 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.178 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.179 DEBUG:boto:Params: {} 2023-09-29T22:59:54.179 DEBUG:boto:Token: None 2023-09-29T22:59:54.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19?versionId=null 2023-09-29T22:59:54.179 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA= 2023-09-29T22:59:54.179 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HAf5iPEJpthn7/RQO+LnhSmw+rA=', 'Content-Length': '0'} 2023-09-29T22:59:54.181 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f0e1b806226e0cf3-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.181 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.181 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.182 DEBUG:boto:path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.182 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.182 DEBUG:boto:Method: GET 2023-09-29T22:59:54.182 DEBUG:boto:Path: /hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.182 DEBUG:boto:Data: 2023-09-29T22:59:54.182 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.183 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.183 DEBUG:boto:Params: {} 2023-09-29T22:59:54.183 DEBUG:boto:Token: None 2023-09-29T22:59:54.183 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.183 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.186 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000097edfbb7329c10ed-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.186 DEBUG:boto:b'hcnphz-61obj191000falseobj192023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.187 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.187 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.187 DEBUG:boto:path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.187 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.187 DEBUG:boto:Method: GET 2023-09-29T22:59:54.187 DEBUG:boto:Path: /hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.187 DEBUG:boto:Data: 2023-09-29T22:59:54.188 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.188 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.188 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.188 DEBUG:boto:Params: {} 2023-09-29T22:59:54.188 DEBUG:boto:Token: None 2023-09-29T22:59:54.189 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.189 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.189 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.195 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003a8a7eab4adb1f6b-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.195 DEBUG:boto:b'hcnphz-61obj191000falseobj192023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.195 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.196 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.196 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.196 DEBUG:boto:Path: /hcnphz-61/obj19 2023-09-29T22:59:54.196 DEBUG:boto:Data: 2023-09-29T22:59:54.196 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.196 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.196 DEBUG:boto:Params: {} 2023-09-29T22:59:54.197 DEBUG:boto:Token: None 2023-09-29T22:59:54.197 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19 2023-09-29T22:59:54.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Slitlkldmd/rcBIeuEsPWA+EtU= 2023-09-29T22:59:54.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Slitlkldmd/rcBIeuEsPWA+EtU=', 'Content-Length': '0'} 2023-09-29T22:59:54.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bcb3756860fa0a9-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.199 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.199 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.199 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.200 DEBUG:boto:Path: /hcnphz-61/obj19 2023-09-29T22:59:54.200 DEBUG:boto:Data: 2023-09-29T22:59:54.200 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.200 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.200 DEBUG:boto:Params: {} 2023-09-29T22:59:54.200 DEBUG:boto:Token: None 2023-09-29T22:59:54.201 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19 2023-09-29T22:59:54.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Slitlkldmd/rcBIeuEsPWA+EtU= 2023-09-29T22:59:54.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Slitlkldmd/rcBIeuEsPWA+EtU=', 'Content-Length': '0'} 2023-09-29T22:59:54.203 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079498f23b384910f-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj19 2023-09-29T22:59:54.203 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.204 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.204 DEBUG:boto:Method: GET 2023-09-29T22:59:54.204 DEBUG:boto:Path: /hcnphz-61/obj19 2023-09-29T22:59:54.204 DEBUG:boto:Data: 2023-09-29T22:59:54.204 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.204 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.204 DEBUG:boto:Params: {} 2023-09-29T22:59:54.204 DEBUG:boto:Token: None 2023-09-29T22:59:54.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19 2023-09-29T22:59:54.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p8BLRP364N86Ipiz5fR+7nqt9Js= 2023-09-29T22:59:54.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p8BLRP364N86Ipiz5fR+7nqt9Js=', 'Content-Length': '0'} 2023-09-29T22:59:54.207 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005c4d8a2d5c209eed-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.250 DEBUG:boto:path=/hcnphz-61/obj19 2023-09-29T22:59:54.250 DEBUG:boto:auth_path=/hcnphz-61/obj19 2023-09-29T22:59:54.251 DEBUG:boto:Method: GET 2023-09-29T22:59:54.251 DEBUG:boto:Path: /hcnphz-61/obj19 2023-09-29T22:59:54.251 DEBUG:boto:Data: 2023-09-29T22:59:54.251 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.251 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.251 DEBUG:boto:Params: {} 2023-09-29T22:59:54.251 DEBUG:boto:Token: None 2023-09-29T22:59:54.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj19 2023-09-29T22:59:54.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:p8BLRP364N86Ipiz5fR+7nqt9Js= 2023-09-29T22:59:54.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p8BLRP364N86Ipiz5fR+7nqt9Js=', 'Content-Length': '0'} 2023-09-29T22:59:54.255 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000987fbdec731f53d4-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.255 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.256 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.256 DEBUG:boto:path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.256 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.257 DEBUG:boto:Method: GET 2023-09-29T22:59:54.257 DEBUG:boto:Path: /hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.257 DEBUG:boto:Data: 2023-09-29T22:59:54.257 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.258 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.258 DEBUG:boto:Params: {} 2023-09-29T22:59:54.258 DEBUG:boto:Token: None 2023-09-29T22:59:54.259 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.259 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.264 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a0d5eb0b751f4457-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.264 DEBUG:boto:b'hcnphz-61obj191000falseobj192023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.265 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.265 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.265 DEBUG:boto:path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.265 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.265 DEBUG:boto:Method: GET 2023-09-29T22:59:54.265 DEBUG:boto:Path: /hcnphz-61/?prefix=obj19 2023-09-29T22:59:54.265 DEBUG:boto:Data: 2023-09-29T22:59:54.265 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.266 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.266 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.266 DEBUG:boto:Params: {} 2023-09-29T22:59:54.266 DEBUG:boto:Token: None 2023-09-29T22:59:54.266 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.266 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.267 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf8119576b4beb6a-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.275 DEBUG:boto:b'hcnphz-61obj191000falseobj192023-09-29T22:55:13.532Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:59:54.276 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.276 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.276 DEBUG:boto:path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.276 DEBUG:boto:auth_path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.277 DEBUG:boto:Method: GET 2023-09-29T22:59:54.277 DEBUG:boto:Path: /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.277 DEBUG:boto:Data: 2023-09-29T22:59:54.277 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.277 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.277 DEBUG:boto:Params: {} 2023-09-29T22:59:54.277 DEBUG:boto:Token: None 2023-09-29T22:59:54.278 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.278 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE= 2023-09-29T22:59:54.278 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE=', 'Content-Length': '0'} 2023-09-29T22:59:54.280 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000458a40f9593780e6-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.281 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.281 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.281 DEBUG:boto:path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.281 DEBUG:boto:auth_path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.281 DEBUG:boto:Method: GET 2023-09-29T22:59:54.281 DEBUG:boto:Path: /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.281 DEBUG:boto:Data: 2023-09-29T22:59:54.281 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.282 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.282 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.282 DEBUG:boto:Params: {} 2023-09-29T22:59:54.282 DEBUG:boto:Token: None 2023-09-29T22:59:54.282 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.282 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE= 2023-09-29T22:59:54.282 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE=', 'Content-Length': '0'} 2023-09-29T22:59:54.285 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000050c15f6a4bba6db7-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.286 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.286 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.286 DEBUG:boto:path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.286 DEBUG:boto:auth_path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.286 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.286 DEBUG:boto:Path: /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.286 DEBUG:boto:Data: 2023-09-29T22:59:54.286 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.287 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.287 DEBUG:boto:Params: {} 2023-09-29T22:59:54.287 DEBUG:boto:Token: None 2023-09-29T22:59:54.287 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.287 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J2WCZKtW4HiLdyHeFgMBJd9d980= 2023-09-29T22:59:54.287 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J2WCZKtW4HiLdyHeFgMBJd9d980=', 'Content-Length': '0'} 2023-09-29T22:59:54.289 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001cf7c77caab5007f-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.290 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.290 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.290 DEBUG:boto:path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.290 DEBUG:boto:auth_path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.290 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.290 DEBUG:boto:Path: /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.290 DEBUG:boto:Data: 2023-09-29T22:59:54.290 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.291 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.291 DEBUG:boto:Params: {} 2023-09-29T22:59:54.291 DEBUG:boto:Token: None 2023-09-29T22:59:54.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.291 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J2WCZKtW4HiLdyHeFgMBJd9d980= 2023-09-29T22:59:54.291 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J2WCZKtW4HiLdyHeFgMBJd9d980=', 'Content-Length': '0'} 2023-09-29T22:59:54.294 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ce6d13c3fca866e2-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:59:54.295 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.295 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.295 DEBUG:boto:path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.295 DEBUG:boto:auth_path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.295 DEBUG:boto:Method: GET 2023-09-29T22:59:54.295 DEBUG:boto:Path: /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.296 DEBUG:boto:Data: 2023-09-29T22:59:54.296 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.296 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.296 DEBUG:boto:Params: {} 2023-09-29T22:59:54.296 DEBUG:boto:Token: None 2023-09-29T22:59:54.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE= 2023-09-29T22:59:54.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE=', 'Content-Length': '0'} 2023-09-29T22:59:54.299 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000ac140ec0017a07-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.300 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.300 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.300 DEBUG:boto:path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.300 DEBUG:boto:auth_path=/hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.301 DEBUG:boto:Method: GET 2023-09-29T22:59:54.301 DEBUG:boto:Path: /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.301 DEBUG:boto:Data: 2023-09-29T22:59:54.301 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.301 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.301 DEBUG:boto:Params: {} 2023-09-29T22:59:54.302 DEBUG:boto:Token: None 2023-09-29T22:59:54.302 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22?versionId=null 2023-09-29T22:59:54.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE= 2023-09-29T22:59:54.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rCFvYda7mteyCBARCIf1lI9IufE=', 'Content-Length': '0'} 2023-09-29T22:59:54.304 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003c6d1f89ceaf7d2b-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.305 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.305 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.305 DEBUG:boto:path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.305 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.305 DEBUG:boto:Method: GET 2023-09-29T22:59:54.305 DEBUG:boto:Path: /hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.305 DEBUG:boto:Data: 2023-09-29T22:59:54.306 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.306 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.306 DEBUG:boto:Params: {} 2023-09-29T22:59:54.306 DEBUG:boto:Token: None 2023-09-29T22:59:54.306 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.311 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e6b137725e9d1bb7-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.354 DEBUG:boto:b'hcnphz-61obj221000falseobj222023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.355 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.355 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.355 DEBUG:boto:path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.355 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.355 DEBUG:boto:Method: GET 2023-09-29T22:59:54.355 DEBUG:boto:Path: /hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.356 DEBUG:boto:Data: 2023-09-29T22:59:54.356 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.356 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.356 DEBUG:boto:Params: {} 2023-09-29T22:59:54.356 DEBUG:boto:Token: None 2023-09-29T22:59:54.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.366 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e375520448e57c9a-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.366 DEBUG:boto:b'hcnphz-61obj221000falseobj222023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.367 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.367 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.367 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.367 DEBUG:boto:Path: /hcnphz-61/obj22 2023-09-29T22:59:54.367 DEBUG:boto:Data: 2023-09-29T22:59:54.367 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.367 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.367 DEBUG:boto:Params: {} 2023-09-29T22:59:54.368 DEBUG:boto:Token: None 2023-09-29T22:59:54.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22 2023-09-29T22:59:54.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XOCrVI4SgbMrQli6bq+Nzyd3he0= 2023-09-29T22:59:54.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XOCrVI4SgbMrQli6bq+Nzyd3he0=', 'Content-Length': '0'} 2023-09-29T22:59:54.370 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006008c1ed3dc60041-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.371 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.371 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.371 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.371 DEBUG:boto:Path: /hcnphz-61/obj22 2023-09-29T22:59:54.371 DEBUG:boto:Data: 2023-09-29T22:59:54.371 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.372 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.372 DEBUG:boto:Params: {} 2023-09-29T22:59:54.372 DEBUG:boto:Token: None 2023-09-29T22:59:54.372 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22 2023-09-29T22:59:54.372 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XOCrVI4SgbMrQli6bq+Nzyd3he0= 2023-09-29T22:59:54.372 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XOCrVI4SgbMrQli6bq+Nzyd3he0=', 'Content-Length': '0'} 2023-09-29T22:59:54.374 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfec8b4605a827ef-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T22:59:54.375 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.375 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.375 DEBUG:boto:Method: GET 2023-09-29T22:59:54.375 DEBUG:boto:Path: /hcnphz-61/obj22 2023-09-29T22:59:54.375 DEBUG:boto:Data: 2023-09-29T22:59:54.375 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.375 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.375 DEBUG:boto:Params: {} 2023-09-29T22:59:54.376 DEBUG:boto:Token: None 2023-09-29T22:59:54.376 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22 2023-09-29T22:59:54.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HCDv81yWOv5NYRzc7Iqb4v+lDoo= 2023-09-29T22:59:54.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HCDv81yWOv5NYRzc7Iqb4v+lDoo=', 'Content-Length': '0'} 2023-09-29T22:59:54.378 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086cabb27b461a5b8-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.378 DEBUG:boto:path=/hcnphz-61/obj22 2023-09-29T22:59:54.379 DEBUG:boto:auth_path=/hcnphz-61/obj22 2023-09-29T22:59:54.381 DEBUG:boto:Method: GET 2023-09-29T22:59:54.381 DEBUG:boto:Path: /hcnphz-61/obj22 2023-09-29T22:59:54.381 DEBUG:boto:Data: 2023-09-29T22:59:54.381 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.381 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.382 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.382 DEBUG:boto:Params: {} 2023-09-29T22:59:54.382 DEBUG:boto:Token: None 2023-09-29T22:59:54.382 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj22 2023-09-29T22:59:54.382 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HCDv81yWOv5NYRzc7Iqb4v+lDoo= 2023-09-29T22:59:54.382 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HCDv81yWOv5NYRzc7Iqb4v+lDoo=', 'Content-Length': '0'} 2023-09-29T22:59:54.385 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c153fb2ddbf5df9-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.430 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.431 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.431 DEBUG:boto:path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.431 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.431 DEBUG:boto:Method: GET 2023-09-29T22:59:54.431 DEBUG:boto:Path: /hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.431 DEBUG:boto:Data: 2023-09-29T22:59:54.431 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.432 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.432 DEBUG:boto:Params: {} 2023-09-29T22:59:54.432 DEBUG:boto:Token: None 2023-09-29T22:59:54.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.437 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e9de6126660bc5f-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.437 DEBUG:boto:b'hcnphz-61obj221000falseobj222023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.438 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.438 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.438 DEBUG:boto:path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.438 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.438 DEBUG:boto:Method: GET 2023-09-29T22:59:54.438 DEBUG:boto:Path: /hcnphz-61/?prefix=obj22 2023-09-29T22:59:54.438 DEBUG:boto:Data: 2023-09-29T22:59:54.438 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.438 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.439 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.439 DEBUG:boto:Params: {} 2023-09-29T22:59:54.439 DEBUG:boto:Token: None 2023-09-29T22:59:54.439 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.439 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.440 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.447 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044df0f770d65aaa3-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.448 DEBUG:boto:b'hcnphz-61obj221000falseobj222023-09-29T22:55:12.778Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:59:54.448 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.448 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.448 DEBUG:boto:path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.448 DEBUG:boto:auth_path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.449 DEBUG:boto:Method: GET 2023-09-29T22:59:54.449 DEBUG:boto:Path: /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.449 DEBUG:boto:Data: 2023-09-29T22:59:54.449 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.449 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.449 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.449 DEBUG:boto:Params: {} 2023-09-29T22:59:54.449 DEBUG:boto:Token: None 2023-09-29T22:59:54.450 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.450 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE= 2023-09-29T22:59:54.450 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE=', 'Content-Length': '0'} 2023-09-29T22:59:54.452 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000654c9dd9fceddf2a-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.494 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.495 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.495 DEBUG:boto:path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.495 DEBUG:boto:auth_path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.496 DEBUG:boto:Method: GET 2023-09-29T22:59:54.496 DEBUG:boto:Path: /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.496 DEBUG:boto:Data: 2023-09-29T22:59:54.496 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.496 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.497 DEBUG:boto:Params: {} 2023-09-29T22:59:54.497 DEBUG:boto:Token: None 2023-09-29T22:59:54.497 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE= 2023-09-29T22:59:54.498 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE=', 'Content-Length': '0'} 2023-09-29T22:59:54.500 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033c75f29fd54b88c-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.501 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.501 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.501 DEBUG:boto:path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.501 DEBUG:boto:auth_path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.501 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.502 DEBUG:boto:Path: /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.502 DEBUG:boto:Data: 2023-09-29T22:59:54.502 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.502 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.502 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.502 DEBUG:boto:Params: {} 2023-09-29T22:59:54.503 DEBUG:boto:Token: None 2023-09-29T22:59:54.503 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W4zdKHQEvKzyvLEXJ53Zws96csk= 2023-09-29T22:59:54.503 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W4zdKHQEvKzyvLEXJ53Zws96csk=', 'Content-Length': '0'} 2023-09-29T22:59:54.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002bf0dcc9f49be0c2-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.507 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.507 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.507 DEBUG:boto:path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.507 DEBUG:boto:auth_path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.507 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.508 DEBUG:boto:Path: /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.508 DEBUG:boto:Data: 2023-09-29T22:59:54.508 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.508 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.508 DEBUG:boto:Params: {} 2023-09-29T22:59:54.509 DEBUG:boto:Token: None 2023-09-29T22:59:54.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W4zdKHQEvKzyvLEXJ53Zws96csk= 2023-09-29T22:59:54.510 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W4zdKHQEvKzyvLEXJ53Zws96csk=', 'Content-Length': '0'} 2023-09-29T22:59:54.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009a354e67da8a4bc8-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:59:54.513 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.513 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.513 DEBUG:boto:path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.514 DEBUG:boto:auth_path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.514 DEBUG:boto:Method: GET 2023-09-29T22:59:54.514 DEBUG:boto:Path: /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.514 DEBUG:boto:Data: 2023-09-29T22:59:54.514 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.515 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.515 DEBUG:boto:Params: {} 2023-09-29T22:59:54.516 DEBUG:boto:Token: None 2023-09-29T22:59:54.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.516 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE= 2023-09-29T22:59:54.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE=', 'Content-Length': '0'} 2023-09-29T22:59:54.519 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000477832502a7ecb07-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.519 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.520 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.520 DEBUG:boto:path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.520 DEBUG:boto:auth_path=/hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.520 DEBUG:boto:Method: GET 2023-09-29T22:59:54.520 DEBUG:boto:Path: /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.520 DEBUG:boto:Data: 2023-09-29T22:59:54.521 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.521 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.521 DEBUG:boto:Params: {} 2023-09-29T22:59:54.521 DEBUG:boto:Token: None 2023-09-29T22:59:54.522 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23?versionId=null 2023-09-29T22:59:54.522 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE= 2023-09-29T22:59:54.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fR59emavyoyHawZuCAAsM/jP+NE=', 'Content-Length': '0'} 2023-09-29T22:59:54.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c764fa2dcf38f6aa-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.566 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.567 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.567 DEBUG:boto:path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.567 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.567 DEBUG:boto:Method: GET 2023-09-29T22:59:54.567 DEBUG:boto:Path: /hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.567 DEBUG:boto:Data: 2023-09-29T22:59:54.567 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.567 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.568 DEBUG:boto:Params: {} 2023-09-29T22:59:54.569 DEBUG:boto:Token: None 2023-09-29T22:59:54.569 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.569 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.574 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006ef2cbc850783b81-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.574 DEBUG:boto:b'hcnphz-61obj231000falseobj232023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.575 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.575 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.575 DEBUG:boto:path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.575 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.575 DEBUG:boto:Method: GET 2023-09-29T22:59:54.575 DEBUG:boto:Path: /hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.575 DEBUG:boto:Data: 2023-09-29T22:59:54.576 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.576 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.576 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.576 DEBUG:boto:Params: {} 2023-09-29T22:59:54.576 DEBUG:boto:Token: None 2023-09-29T22:59:54.576 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.576 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.576 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b1b55fef2e90839-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.585 DEBUG:boto:b'hcnphz-61obj231000falseobj232023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.585 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.585 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.585 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.585 DEBUG:boto:Path: /hcnphz-61/obj23 2023-09-29T22:59:54.586 DEBUG:boto:Data: 2023-09-29T22:59:54.586 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.586 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.586 DEBUG:boto:Params: {} 2023-09-29T22:59:54.586 DEBUG:boto:Token: None 2023-09-29T22:59:54.586 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23 2023-09-29T22:59:54.586 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0E3ZQVO/l1ud7ZSRMwfaz2eYnH8= 2023-09-29T22:59:54.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0E3ZQVO/l1ud7ZSRMwfaz2eYnH8=', 'Content-Length': '0'} 2023-09-29T22:59:54.589 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02299f23ecaab85-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.590 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.590 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.590 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.591 DEBUG:boto:Path: /hcnphz-61/obj23 2023-09-29T22:59:54.591 DEBUG:boto:Data: 2023-09-29T22:59:54.591 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.592 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.592 DEBUG:boto:Params: {} 2023-09-29T22:59:54.592 DEBUG:boto:Token: None 2023-09-29T22:59:54.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23 2023-09-29T22:59:54.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0E3ZQVO/l1ud7ZSRMwfaz2eYnH8= 2023-09-29T22:59:54.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0E3ZQVO/l1ud7ZSRMwfaz2eYnH8=', 'Content-Length': '0'} 2023-09-29T22:59:54.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c9a261da610bda9-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj23 2023-09-29T22:59:54.596 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.596 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.597 DEBUG:boto:Method: GET 2023-09-29T22:59:54.597 DEBUG:boto:Path: /hcnphz-61/obj23 2023-09-29T22:59:54.597 DEBUG:boto:Data: 2023-09-29T22:59:54.598 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.598 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.598 DEBUG:boto:Params: {} 2023-09-29T22:59:54.599 DEBUG:boto:Token: None 2023-09-29T22:59:54.599 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23 2023-09-29T22:59:54.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DVJASpOkNFywxy1ABeDGR+hQseM= 2023-09-29T22:59:54.600 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DVJASpOkNFywxy1ABeDGR+hQseM=', 'Content-Length': '0'} 2023-09-29T22:59:54.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e164bdcfb443c5cf-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.602 DEBUG:boto:path=/hcnphz-61/obj23 2023-09-29T22:59:54.602 DEBUG:boto:auth_path=/hcnphz-61/obj23 2023-09-29T22:59:54.602 DEBUG:boto:Method: GET 2023-09-29T22:59:54.602 DEBUG:boto:Path: /hcnphz-61/obj23 2023-09-29T22:59:54.602 DEBUG:boto:Data: 2023-09-29T22:59:54.603 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.603 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.603 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.603 DEBUG:boto:Params: {} 2023-09-29T22:59:54.603 DEBUG:boto:Token: None 2023-09-29T22:59:54.603 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj23 2023-09-29T22:59:54.603 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DVJASpOkNFywxy1ABeDGR+hQseM= 2023-09-29T22:59:54.604 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DVJASpOkNFywxy1ABeDGR+hQseM=', 'Content-Length': '0'} 2023-09-29T22:59:54.606 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005fb1390ed96e9416-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.646 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.647 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.647 DEBUG:boto:path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.647 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.647 DEBUG:boto:Method: GET 2023-09-29T22:59:54.647 DEBUG:boto:Path: /hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.647 DEBUG:boto:Data: 2023-09-29T22:59:54.647 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.647 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.648 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.648 DEBUG:boto:Params: {} 2023-09-29T22:59:54.648 DEBUG:boto:Token: None 2023-09-29T22:59:54.648 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.648 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.648 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.653 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038fa7c902626b103-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.653 DEBUG:boto:b'hcnphz-61obj231000falseobj232023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.654 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.654 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.654 DEBUG:boto:path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.654 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.654 DEBUG:boto:Method: GET 2023-09-29T22:59:54.654 DEBUG:boto:Path: /hcnphz-61/?prefix=obj23 2023-09-29T22:59:54.654 DEBUG:boto:Data: 2023-09-29T22:59:54.655 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.655 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.655 DEBUG:boto:Params: {} 2023-09-29T22:59:54.655 DEBUG:boto:Token: None 2023-09-29T22:59:54.655 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.655 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.664 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b94d26c20b254556-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.664 DEBUG:boto:b'hcnphz-61obj231000falseobj232023-09-29T22:55:13.975Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:59:54.665 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.665 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.665 DEBUG:boto:path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.665 DEBUG:boto:auth_path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.665 DEBUG:boto:Method: GET 2023-09-29T22:59:54.665 DEBUG:boto:Path: /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.665 DEBUG:boto:Data: 2023-09-29T22:59:54.665 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.665 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.666 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.666 DEBUG:boto:Params: {} 2023-09-29T22:59:54.666 DEBUG:boto:Token: None 2023-09-29T22:59:54.666 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.666 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA= 2023-09-29T22:59:54.666 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA=', 'Content-Length': '0'} 2023-09-29T22:59:54.668 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e8d08319ff665da-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.714 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.715 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.715 DEBUG:boto:path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.715 DEBUG:boto:auth_path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.715 DEBUG:boto:Method: GET 2023-09-29T22:59:54.715 DEBUG:boto:Path: /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.715 DEBUG:boto:Data: 2023-09-29T22:59:54.715 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.715 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.715 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.715 DEBUG:boto:Params: {} 2023-09-29T22:59:54.716 DEBUG:boto:Token: None 2023-09-29T22:59:54.716 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA= 2023-09-29T22:59:54.716 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA=', 'Content-Length': '0'} 2023-09-29T22:59:54.719 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060547c1513ec481f-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.719 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.720 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.720 DEBUG:boto:path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.720 DEBUG:boto:auth_path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.720 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.721 DEBUG:boto:Path: /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.721 DEBUG:boto:Data: 2023-09-29T22:59:54.721 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.721 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.722 DEBUG:boto:Params: {} 2023-09-29T22:59:54.722 DEBUG:boto:Token: None 2023-09-29T22:59:54.723 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a5uSm6rfzDaUj3rSmcBytFL83CY= 2023-09-29T22:59:54.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a5uSm6rfzDaUj3rSmcBytFL83CY=', 'Content-Length': '0'} 2023-09-29T22:59:54.726 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a6c390fc47c6dd9-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.726 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.726 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.727 DEBUG:boto:path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.727 DEBUG:boto:auth_path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.727 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.727 DEBUG:boto:Path: /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.727 DEBUG:boto:Data: 2023-09-29T22:59:54.727 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.728 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.728 DEBUG:boto:Params: {} 2023-09-29T22:59:54.728 DEBUG:boto:Token: None 2023-09-29T22:59:54.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a5uSm6rfzDaUj3rSmcBytFL83CY= 2023-09-29T22:59:54.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a5uSm6rfzDaUj3rSmcBytFL83CY=', 'Content-Length': '0'} 2023-09-29T22:59:54.731 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd27fed9c4fc5a4b-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:59:54.732 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.732 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.733 DEBUG:boto:path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.733 DEBUG:boto:auth_path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.733 DEBUG:boto:Method: GET 2023-09-29T22:59:54.733 DEBUG:boto:Path: /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.733 DEBUG:boto:Data: 2023-09-29T22:59:54.734 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.734 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.734 DEBUG:boto:Params: {} 2023-09-29T22:59:54.735 DEBUG:boto:Token: None 2023-09-29T22:59:54.735 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA= 2023-09-29T22:59:54.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA=', 'Content-Length': '0'} 2023-09-29T22:59:54.738 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001565512e07d458e4-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.738 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.738 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.738 DEBUG:boto:path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.738 DEBUG:boto:auth_path=/hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.739 DEBUG:boto:Method: GET 2023-09-29T22:59:54.739 DEBUG:boto:Path: /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.739 DEBUG:boto:Data: 2023-09-29T22:59:54.739 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.739 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.739 DEBUG:boto:Params: {} 2023-09-29T22:59:54.739 DEBUG:boto:Token: None 2023-09-29T22:59:54.740 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29?versionId=null 2023-09-29T22:59:54.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA= 2023-09-29T22:59:54.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cYiY3xBchTTRWiwxvOEnNYKJdmA=', 'Content-Length': '0'} 2023-09-29T22:59:54.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000841bf831801965dc-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.790 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.791 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.791 DEBUG:boto:path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.791 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.791 DEBUG:boto:Method: GET 2023-09-29T22:59:54.791 DEBUG:boto:Path: /hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.791 DEBUG:boto:Data: 2023-09-29T22:59:54.791 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.791 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.792 DEBUG:boto:Params: {} 2023-09-29T22:59:54.792 DEBUG:boto:Token: None 2023-09-29T22:59:54.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.797 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b41217034f043768-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.797 DEBUG:boto:b'hcnphz-61obj291000falseobj292023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.798 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.798 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.798 DEBUG:boto:path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.798 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.798 DEBUG:boto:Method: GET 2023-09-29T22:59:54.798 DEBUG:boto:Path: /hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.798 DEBUG:boto:Data: 2023-09-29T22:59:54.798 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.799 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.799 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.799 DEBUG:boto:Params: {} 2023-09-29T22:59:54.799 DEBUG:boto:Token: None 2023-09-29T22:59:54.799 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.800 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.800 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000023885185071f4730-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.809 DEBUG:boto:b'hcnphz-61obj291000falseobj292023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.809 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.810 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.810 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.810 DEBUG:boto:Path: /hcnphz-61/obj29 2023-09-29T22:59:54.810 DEBUG:boto:Data: 2023-09-29T22:59:54.810 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.810 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.810 DEBUG:boto:Params: {} 2023-09-29T22:59:54.811 DEBUG:boto:Token: None 2023-09-29T22:59:54.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29 2023-09-29T22:59:54.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVSqyYe+jbUoWBuOZeQ87Lc1nMI= 2023-09-29T22:59:54.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVSqyYe+jbUoWBuOZeQ87Lc1nMI=', 'Content-Length': '0'} 2023-09-29T22:59:54.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d8a8ebc31198101-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.813 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.813 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.814 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.814 DEBUG:boto:Path: /hcnphz-61/obj29 2023-09-29T22:59:54.814 DEBUG:boto:Data: 2023-09-29T22:59:54.815 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.815 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.815 DEBUG:boto:Params: {} 2023-09-29T22:59:54.816 DEBUG:boto:Token: None 2023-09-29T22:59:54.816 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29 2023-09-29T22:59:54.816 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lVSqyYe+jbUoWBuOZeQ87Lc1nMI= 2023-09-29T22:59:54.816 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lVSqyYe+jbUoWBuOZeQ87Lc1nMI=', 'Content-Length': '0'} 2023-09-29T22:59:54.819 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000727323d1809e4cb2-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj29 2023-09-29T22:59:54.819 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.819 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.819 DEBUG:boto:Method: GET 2023-09-29T22:59:54.819 DEBUG:boto:Path: /hcnphz-61/obj29 2023-09-29T22:59:54.819 DEBUG:boto:Data: 2023-09-29T22:59:54.820 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.820 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.820 DEBUG:boto:Params: {} 2023-09-29T22:59:54.820 DEBUG:boto:Token: None 2023-09-29T22:59:54.820 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29 2023-09-29T22:59:54.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YVe7Z7QT/POjIay9M564pyG2Nvw= 2023-09-29T22:59:54.821 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YVe7Z7QT/POjIay9M564pyG2Nvw=', 'Content-Length': '0'} 2023-09-29T22:59:54.823 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001bfbeee282115353-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.823 DEBUG:boto:path=/hcnphz-61/obj29 2023-09-29T22:59:54.823 DEBUG:boto:auth_path=/hcnphz-61/obj29 2023-09-29T22:59:54.823 DEBUG:boto:Method: GET 2023-09-29T22:59:54.823 DEBUG:boto:Path: /hcnphz-61/obj29 2023-09-29T22:59:54.824 DEBUG:boto:Data: 2023-09-29T22:59:54.824 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.824 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.824 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.824 DEBUG:boto:Params: {} 2023-09-29T22:59:54.824 DEBUG:boto:Token: None 2023-09-29T22:59:54.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj29 2023-09-29T22:59:54.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YVe7Z7QT/POjIay9M564pyG2Nvw= 2023-09-29T22:59:54.825 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YVe7Z7QT/POjIay9M564pyG2Nvw=', 'Content-Length': '0'} 2023-09-29T22:59:54.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ac6dfef6214966f-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.874 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.875 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.875 DEBUG:boto:path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.875 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.875 DEBUG:boto:Method: GET 2023-09-29T22:59:54.875 DEBUG:boto:Path: /hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.875 DEBUG:boto:Data: 2023-09-29T22:59:54.875 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.875 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.875 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.876 DEBUG:boto:Params: {} 2023-09-29T22:59:54.876 DEBUG:boto:Token: None 2023-09-29T22:59:54.876 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.876 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.876 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.880 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000062cf521f26f5478e-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.881 DEBUG:boto:b'hcnphz-61obj291000falseobj292023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.881 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.881 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.881 DEBUG:boto:path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.882 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.882 DEBUG:boto:Method: GET 2023-09-29T22:59:54.882 DEBUG:boto:Path: /hcnphz-61/?prefix=obj29 2023-09-29T22:59:54.882 DEBUG:boto:Data: 2023-09-29T22:59:54.882 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.882 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.882 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.882 DEBUG:boto:Params: {} 2023-09-29T22:59:54.882 DEBUG:boto:Token: None 2023-09-29T22:59:54.883 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.883 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.883 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.890 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f35f614fa8619b40-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.891 DEBUG:boto:b'hcnphz-61obj291000falseobj292023-09-29T22:55:14.384Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:59:54.892 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:54.892 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:54.893 DEBUG:boto:path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.893 DEBUG:boto:auth_path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.893 DEBUG:boto:Method: GET 2023-09-29T22:59:54.893 DEBUG:boto:Path: /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.893 DEBUG:boto:Data: 2023-09-29T22:59:54.894 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.894 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.894 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.894 DEBUG:boto:Params: {} 2023-09-29T22:59:54.894 DEBUG:boto:Token: None 2023-09-29T22:59:54.895 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.895 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc= 2023-09-29T22:59:54.895 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc=', 'Content-Length': '0'} 2023-09-29T22:59:54.898 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000928ff81009904680-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.942 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:54.943 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:54.943 DEBUG:boto:path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.943 DEBUG:boto:auth_path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.943 DEBUG:boto:Method: GET 2023-09-29T22:59:54.943 DEBUG:boto:Path: /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.943 DEBUG:boto:Data: 2023-09-29T22:59:54.943 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.943 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.943 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.943 DEBUG:boto:Params: {} 2023-09-29T22:59:54.944 DEBUG:boto:Token: None 2023-09-29T22:59:54.944 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.944 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc= 2023-09-29T22:59:54.944 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc=', 'Content-Length': '0'} 2023-09-29T22:59:54.947 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d60ca641d8bf4120-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.948 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:54.948 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:54.948 DEBUG:boto:path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.948 DEBUG:boto:auth_path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.948 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.948 DEBUG:boto:Path: /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.948 DEBUG:boto:Data: 2023-09-29T22:59:54.948 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.949 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.949 DEBUG:boto:Params: {} 2023-09-29T22:59:54.949 DEBUG:boto:Token: None 2023-09-29T22:59:54.949 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.949 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w3dgGXCtKoHka2NMKRD/Hz2lIv4= 2023-09-29T22:59:54.949 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w3dgGXCtKoHka2NMKRD/Hz2lIv4=', 'Content-Length': '0'} 2023-09-29T22:59:54.952 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c82b3047f6461e0-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.952 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:54.953 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:54.953 DEBUG:boto:path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.953 DEBUG:boto:auth_path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.953 DEBUG:boto:Method: HEAD 2023-09-29T22:59:54.953 DEBUG:boto:Path: /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.953 DEBUG:boto:Data: 2023-09-29T22:59:54.953 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.953 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.954 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.954 DEBUG:boto:Params: {} 2023-09-29T22:59:54.954 DEBUG:boto:Token: None 2023-09-29T22:59:54.954 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.954 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w3dgGXCtKoHka2NMKRD/Hz2lIv4= 2023-09-29T22:59:54.954 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w3dgGXCtKoHka2NMKRD/Hz2lIv4=', 'Content-Length': '0'} 2023-09-29T22:59:54.957 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3e2cd1e6e8859f6-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:59:54.958 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:54.958 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:54.958 DEBUG:boto:path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.958 DEBUG:boto:auth_path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.958 DEBUG:boto:Method: GET 2023-09-29T22:59:54.958 DEBUG:boto:Path: /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.958 DEBUG:boto:Data: 2023-09-29T22:59:54.958 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.958 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.959 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.959 DEBUG:boto:Params: {} 2023-09-29T22:59:54.959 DEBUG:boto:Token: None 2023-09-29T22:59:54.959 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.959 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc= 2023-09-29T22:59:54.959 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc=', 'Content-Length': '0'} 2023-09-29T22:59:54.962 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000180ae183048b932d-00651756ea-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.962 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:54.962 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:54.962 DEBUG:boto:path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.962 DEBUG:boto:auth_path=/hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.963 DEBUG:boto:Method: GET 2023-09-29T22:59:54.963 DEBUG:boto:Path: /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.963 DEBUG:boto:Data: 2023-09-29T22:59:54.963 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.963 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.963 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.963 DEBUG:boto:Params: {} 2023-09-29T22:59:54.963 DEBUG:boto:Token: None 2023-09-29T22:59:54.964 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/obj31?versionId=null 2023-09-29T22:59:54.964 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc= 2023-09-29T22:59:54.964 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mhTV+LgGjC8+tlEJPVTyxjvQMGc=', 'Content-Length': '0'} 2023-09-29T22:59:54.967 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000129c25b1486d0e8c-00651756ea-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.967 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.967 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.967 DEBUG:boto:path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:54.967 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:54.968 DEBUG:boto:Method: GET 2023-09-29T22:59:54.968 DEBUG:boto:Path: /hcnphz-61/?prefix=obj31 2023-09-29T22:59:54.968 DEBUG:boto:Data: 2023-09-29T22:59:54.968 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.968 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:54.968 DEBUG:boto:Port: 8001 2023-09-29T22:59:54.968 DEBUG:boto:Params: {} 2023-09-29T22:59:54.968 DEBUG:boto:Token: None 2023-09-29T22:59:54.968 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.969 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.969 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.973 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000044174f0a9c3084f9-00651756ea-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:54.973 DEBUG:boto:b'hcnphz-61obj311000falseobj312023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:54.973 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:54.974 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:54.974 DEBUG:boto:path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:54.974 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:54.974 DEBUG:boto:Method: GET 2023-09-29T22:59:54.974 DEBUG:boto:Path: /hcnphz-61/?prefix=obj31 2023-09-29T22:59:54.974 DEBUG:boto:Data: 2023-09-29T22:59:54.974 DEBUG:boto:Headers: {} 2023-09-29T22:59:54.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:54.974 DEBUG:boto:Port: 8000 2023-09-29T22:59:54.975 DEBUG:boto:Params: {} 2023-09-29T22:59:54.977 DEBUG:boto:Token: None 2023-09-29T22:59:54.977 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:54 GMT /hcnphz-61/ 2023-09-29T22:59:54.977 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw= 2023-09-29T22:59:54.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sRd/NSM86hzR5JO/p82t2c6tgaw=', 'Content-Length': '0'} 2023-09-29T22:59:54.984 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f219261d59c9e77e-00651756ea-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.026 DEBUG:boto:b'hcnphz-61obj311000falseobj312023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.027 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:55.027 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:55.027 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.028 DEBUG:boto:Path: /hcnphz-61/obj31 2023-09-29T22:59:55.028 DEBUG:boto:Data: 2023-09-29T22:59:55.028 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.028 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.028 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.029 DEBUG:boto:Params: {} 2023-09-29T22:59:55.029 DEBUG:boto:Token: None 2023-09-29T22:59:55.029 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj31 2023-09-29T22:59:55.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BN1Brdc4vuRtMs8ZqwbGIDbk+zg= 2023-09-29T22:59:55.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BN1Brdc4vuRtMs8ZqwbGIDbk+zg=', 'Content-Length': '0'} 2023-09-29T22:59:55.033 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006853b2e27cbd503a-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.033 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:55.033 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:55.034 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.034 DEBUG:boto:Path: /hcnphz-61/obj31 2023-09-29T22:59:55.034 DEBUG:boto:Data: 2023-09-29T22:59:55.034 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.034 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.034 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.034 DEBUG:boto:Params: {} 2023-09-29T22:59:55.034 DEBUG:boto:Token: None 2023-09-29T22:59:55.035 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj31 2023-09-29T22:59:55.035 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BN1Brdc4vuRtMs8ZqwbGIDbk+zg= 2023-09-29T22:59:55.035 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BN1Brdc4vuRtMs8ZqwbGIDbk+zg=', 'Content-Length': '0'} 2023-09-29T22:59:55.037 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bafb7ba0bf39f0-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj31 2023-09-29T22:59:55.038 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:55.038 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:55.038 DEBUG:boto:Method: GET 2023-09-29T22:59:55.038 DEBUG:boto:Path: /hcnphz-61/obj31 2023-09-29T22:59:55.038 DEBUG:boto:Data: 2023-09-29T22:59:55.039 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.039 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.039 DEBUG:boto:Params: {} 2023-09-29T22:59:55.039 DEBUG:boto:Token: None 2023-09-29T22:59:55.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj31 2023-09-29T22:59:55.039 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7CF8nh/upYtkJHRnsjKP5Rqf3SE= 2023-09-29T22:59:55.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7CF8nh/upYtkJHRnsjKP5Rqf3SE=', 'Content-Length': '0'} 2023-09-29T22:59:55.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033fefc6582dd9e78-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.042 DEBUG:boto:path=/hcnphz-61/obj31 2023-09-29T22:59:55.042 DEBUG:boto:auth_path=/hcnphz-61/obj31 2023-09-29T22:59:55.043 DEBUG:boto:Method: GET 2023-09-29T22:59:55.043 DEBUG:boto:Path: /hcnphz-61/obj31 2023-09-29T22:59:55.043 DEBUG:boto:Data: 2023-09-29T22:59:55.043 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.043 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.043 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.043 DEBUG:boto:Params: {} 2023-09-29T22:59:55.043 DEBUG:boto:Token: None 2023-09-29T22:59:55.044 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj31 2023-09-29T22:59:55.044 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7CF8nh/upYtkJHRnsjKP5Rqf3SE= 2023-09-29T22:59:55.044 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7CF8nh/upYtkJHRnsjKP5Rqf3SE=', 'Content-Length': '0'} 2023-09-29T22:59:55.047 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007e6bf9ba4c026954-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.047 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.047 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.047 DEBUG:boto:path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:55.047 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:55.048 DEBUG:boto:Method: GET 2023-09-29T22:59:55.048 DEBUG:boto:Path: /hcnphz-61/?prefix=obj31 2023-09-29T22:59:55.048 DEBUG:boto:Data: 2023-09-29T22:59:55.048 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.048 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.048 DEBUG:boto:Params: {} 2023-09-29T22:59:55.048 DEBUG:boto:Token: None 2023-09-29T22:59:55.049 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.053 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003e63958ffe52aeea-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.054 DEBUG:boto:b'hcnphz-61obj311000falseobj312023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.054 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.054 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.054 DEBUG:boto:path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:55.054 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj31 2023-09-29T22:59:55.055 DEBUG:boto:Method: GET 2023-09-29T22:59:55.055 DEBUG:boto:Path: /hcnphz-61/?prefix=obj31 2023-09-29T22:59:55.055 DEBUG:boto:Data: 2023-09-29T22:59:55.055 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.055 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.055 DEBUG:boto:Params: {} 2023-09-29T22:59:55.055 DEBUG:boto:Token: None 2023-09-29T22:59:55.056 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.063 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006216570348fd5cb8-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.063 DEBUG:boto:b'hcnphz-61obj311000falseobj312023-09-29T22:55:14.861Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:59:55.064 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.064 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.064 DEBUG:boto:path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.064 DEBUG:boto:auth_path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.065 DEBUG:boto:Method: GET 2023-09-29T22:59:55.065 DEBUG:boto:Path: /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.065 DEBUG:boto:Data: 2023-09-29T22:59:55.065 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.065 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.065 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.065 DEBUG:boto:Params: {} 2023-09-29T22:59:55.065 DEBUG:boto:Token: None 2023-09-29T22:59:55.066 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.066 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds= 2023-09-29T22:59:55.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds=', 'Content-Length': '0'} 2023-09-29T22:59:55.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d896e613d35da92-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.110 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.111 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.111 DEBUG:boto:path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.111 DEBUG:boto:auth_path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.111 DEBUG:boto:Method: GET 2023-09-29T22:59:55.111 DEBUG:boto:Path: /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.111 DEBUG:boto:Data: 2023-09-29T22:59:55.111 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.112 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.112 DEBUG:boto:Params: {} 2023-09-29T22:59:55.112 DEBUG:boto:Token: None 2023-09-29T22:59:55.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds= 2023-09-29T22:59:55.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds=', 'Content-Length': '0'} 2023-09-29T22:59:55.115 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000106af883fc17b4a5-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.115 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.116 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.116 DEBUG:boto:path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.116 DEBUG:boto:auth_path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.116 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.116 DEBUG:boto:Path: /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.116 DEBUG:boto:Data: 2023-09-29T22:59:55.116 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.117 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.117 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.117 DEBUG:boto:Params: {} 2023-09-29T22:59:55.117 DEBUG:boto:Token: None 2023-09-29T22:59:55.117 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5NvzpRMR8UVishUiDS+eEA4vmw8= 2023-09-29T22:59:55.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5NvzpRMR8UVishUiDS+eEA4vmw8=', 'Content-Length': '0'} 2023-09-29T22:59:55.120 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f3e53a3ecd281b2-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.120 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.121 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.121 DEBUG:boto:path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.121 DEBUG:boto:auth_path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.121 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.121 DEBUG:boto:Path: /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.121 DEBUG:boto:Data: 2023-09-29T22:59:55.121 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.122 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.122 DEBUG:boto:Params: {} 2023-09-29T22:59:55.122 DEBUG:boto:Token: None 2023-09-29T22:59:55.122 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.122 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5NvzpRMR8UVishUiDS+eEA4vmw8= 2023-09-29T22:59:55.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5NvzpRMR8UVishUiDS+eEA4vmw8=', 'Content-Length': '0'} 2023-09-29T22:59:55.125 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b56009ee92ff5d6-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:59:55.126 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.126 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.126 DEBUG:boto:path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.126 DEBUG:boto:auth_path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.126 DEBUG:boto:Method: GET 2023-09-29T22:59:55.126 DEBUG:boto:Path: /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.127 DEBUG:boto:Data: 2023-09-29T22:59:55.127 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.127 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.127 DEBUG:boto:Params: {} 2023-09-29T22:59:55.127 DEBUG:boto:Token: None 2023-09-29T22:59:55.127 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds= 2023-09-29T22:59:55.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds=', 'Content-Length': '0'} 2023-09-29T22:59:55.130 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000048b524bab6295e95-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.131 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.132 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.132 DEBUG:boto:path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.132 DEBUG:boto:auth_path=/hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.132 DEBUG:boto:Method: GET 2023-09-29T22:59:55.133 DEBUG:boto:Path: /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.133 DEBUG:boto:Data: 2023-09-29T22:59:55.133 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.133 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.133 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.134 DEBUG:boto:Params: {} 2023-09-29T22:59:55.134 DEBUG:boto:Token: None 2023-09-29T22:59:55.134 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33?versionId=null 2023-09-29T22:59:55.135 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds= 2023-09-29T22:59:55.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gGvEyiinCLBZ+GcRcY0/CULlsds=', 'Content-Length': '0'} 2023-09-29T22:59:55.139 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8f6ca1260c13f79-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.182 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.183 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.183 DEBUG:boto:path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.183 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.183 DEBUG:boto:Method: GET 2023-09-29T22:59:55.183 DEBUG:boto:Path: /hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.183 DEBUG:boto:Data: 2023-09-29T22:59:55.183 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.183 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.185 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.185 DEBUG:boto:Params: {} 2023-09-29T22:59:55.185 DEBUG:boto:Token: None 2023-09-29T22:59:55.185 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.186 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005664eb6d9da6755d-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.190 DEBUG:boto:b'hcnphz-61obj331000falseobj332023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.191 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.191 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.191 DEBUG:boto:path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.191 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.192 DEBUG:boto:Method: GET 2023-09-29T22:59:55.192 DEBUG:boto:Path: /hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.192 DEBUG:boto:Data: 2023-09-29T22:59:55.192 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.192 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.192 DEBUG:boto:Params: {} 2023-09-29T22:59:55.193 DEBUG:boto:Token: None 2023-09-29T22:59:55.193 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.200 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000119953348be63e60-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.200 DEBUG:boto:b'hcnphz-61obj331000falseobj332023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.201 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.201 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.201 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.201 DEBUG:boto:Path: /hcnphz-61/obj33 2023-09-29T22:59:55.202 DEBUG:boto:Data: 2023-09-29T22:59:55.202 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.202 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.202 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.202 DEBUG:boto:Params: {} 2023-09-29T22:59:55.202 DEBUG:boto:Token: None 2023-09-29T22:59:55.202 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33 2023-09-29T22:59:55.203 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:It5WHt/PJ5cnRUWK+beHFnr047A= 2023-09-29T22:59:55.203 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:It5WHt/PJ5cnRUWK+beHFnr047A=', 'Content-Length': '0'} 2023-09-29T22:59:55.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000859afa046cbee763-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.205 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.205 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.206 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.206 DEBUG:boto:Path: /hcnphz-61/obj33 2023-09-29T22:59:55.206 DEBUG:boto:Data: 2023-09-29T22:59:55.206 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.206 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.206 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.206 DEBUG:boto:Params: {} 2023-09-29T22:59:55.206 DEBUG:boto:Token: None 2023-09-29T22:59:55.207 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33 2023-09-29T22:59:55.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:It5WHt/PJ5cnRUWK+beHFnr047A= 2023-09-29T22:59:55.207 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:It5WHt/PJ5cnRUWK+beHFnr047A=', 'Content-Length': '0'} 2023-09-29T22:59:55.209 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da13e3e6b76285f4-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T22:59:55.209 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.209 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.210 DEBUG:boto:Method: GET 2023-09-29T22:59:55.210 DEBUG:boto:Path: /hcnphz-61/obj33 2023-09-29T22:59:55.210 DEBUG:boto:Data: 2023-09-29T22:59:55.210 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.210 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.210 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.210 DEBUG:boto:Params: {} 2023-09-29T22:59:55.211 DEBUG:boto:Token: None 2023-09-29T22:59:55.211 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33 2023-09-29T22:59:55.211 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lJXuNIqnBLlh04dK6FqguG8sjFw= 2023-09-29T22:59:55.211 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lJXuNIqnBLlh04dK6FqguG8sjFw=', 'Content-Length': '0'} 2023-09-29T22:59:55.213 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b01cd2cfa837e050-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.254 DEBUG:boto:path=/hcnphz-61/obj33 2023-09-29T22:59:55.255 DEBUG:boto:auth_path=/hcnphz-61/obj33 2023-09-29T22:59:55.255 DEBUG:boto:Method: GET 2023-09-29T22:59:55.255 DEBUG:boto:Path: /hcnphz-61/obj33 2023-09-29T22:59:55.255 DEBUG:boto:Data: 2023-09-29T22:59:55.255 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.255 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.255 DEBUG:boto:Params: {} 2023-09-29T22:59:55.256 DEBUG:boto:Token: None 2023-09-29T22:59:55.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj33 2023-09-29T22:59:55.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lJXuNIqnBLlh04dK6FqguG8sjFw= 2023-09-29T22:59:55.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lJXuNIqnBLlh04dK6FqguG8sjFw=', 'Content-Length': '0'} 2023-09-29T22:59:55.259 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c598da885608aef-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.259 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.259 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.259 DEBUG:boto:path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.260 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.260 DEBUG:boto:Method: GET 2023-09-29T22:59:55.260 DEBUG:boto:Path: /hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.260 DEBUG:boto:Data: 2023-09-29T22:59:55.260 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.260 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.260 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.261 DEBUG:boto:Params: {} 2023-09-29T22:59:55.261 DEBUG:boto:Token: None 2023-09-29T22:59:55.261 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.261 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.262 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.266 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001a98013e7893b47e-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.266 DEBUG:boto:b'hcnphz-61obj331000falseobj332023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.267 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.267 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.267 DEBUG:boto:path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.268 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.268 DEBUG:boto:Method: GET 2023-09-29T22:59:55.268 DEBUG:boto:Path: /hcnphz-61/?prefix=obj33 2023-09-29T22:59:55.268 DEBUG:boto:Data: 2023-09-29T22:59:55.268 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.269 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.269 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.269 DEBUG:boto:Params: {} 2023-09-29T22:59:55.269 DEBUG:boto:Token: None 2023-09-29T22:59:55.270 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.278 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cde9b75d9e673d3a-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.318 DEBUG:boto:b'hcnphz-61obj331000falseobj332023-09-29T22:55:12.793Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:59:55.319 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.319 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.319 DEBUG:boto:path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.319 DEBUG:boto:auth_path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.320 DEBUG:boto:Method: GET 2023-09-29T22:59:55.320 DEBUG:boto:Path: /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.320 DEBUG:boto:Data: 2023-09-29T22:59:55.320 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.320 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.320 DEBUG:boto:Params: {} 2023-09-29T22:59:55.320 DEBUG:boto:Token: None 2023-09-29T22:59:55.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc= 2023-09-29T22:59:55.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc=', 'Content-Length': '0'} 2023-09-29T22:59:55.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003ee6b68bbe1b2c49-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.324 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.325 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.325 DEBUG:boto:path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.325 DEBUG:boto:auth_path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.325 DEBUG:boto:Method: GET 2023-09-29T22:59:55.325 DEBUG:boto:Path: /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.325 DEBUG:boto:Data: 2023-09-29T22:59:55.325 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.326 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.326 DEBUG:boto:Params: {} 2023-09-29T22:59:55.326 DEBUG:boto:Token: None 2023-09-29T22:59:55.326 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.326 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc= 2023-09-29T22:59:55.326 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc=', 'Content-Length': '0'} 2023-09-29T22:59:55.329 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc546689a0e11042-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.330 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.330 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.330 DEBUG:boto:path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.330 DEBUG:boto:auth_path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.330 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.330 DEBUG:boto:Path: /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.330 DEBUG:boto:Data: 2023-09-29T22:59:55.330 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.331 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.331 DEBUG:boto:Params: {} 2023-09-29T22:59:55.331 DEBUG:boto:Token: None 2023-09-29T22:59:55.331 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.331 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u/LbDC6kYOz2eIjwzUoLcwByUsc= 2023-09-29T22:59:55.331 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u/LbDC6kYOz2eIjwzUoLcwByUsc=', 'Content-Length': '0'} 2023-09-29T22:59:55.334 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bd8f3ad416848e8e-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.334 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.334 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.334 DEBUG:boto:path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.335 DEBUG:boto:auth_path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.335 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.335 DEBUG:boto:Path: /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.335 DEBUG:boto:Data: 2023-09-29T22:59:55.335 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.335 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.335 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.335 DEBUG:boto:Params: {} 2023-09-29T22:59:55.336 DEBUG:boto:Token: None 2023-09-29T22:59:55.336 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.336 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:u/LbDC6kYOz2eIjwzUoLcwByUsc= 2023-09-29T22:59:55.336 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:u/LbDC6kYOz2eIjwzUoLcwByUsc=', 'Content-Length': '0'} 2023-09-29T22:59:55.339 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062142244275c6080-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:59:55.339 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.339 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.340 DEBUG:boto:path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.340 DEBUG:boto:auth_path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.340 DEBUG:boto:Method: GET 2023-09-29T22:59:55.340 DEBUG:boto:Path: /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.340 DEBUG:boto:Data: 2023-09-29T22:59:55.340 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.340 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.340 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.340 DEBUG:boto:Params: {} 2023-09-29T22:59:55.341 DEBUG:boto:Token: None 2023-09-29T22:59:55.341 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.341 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc= 2023-09-29T22:59:55.341 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc=', 'Content-Length': '0'} 2023-09-29T22:59:55.344 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4d8b12c894fb2a4-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.344 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.344 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.344 DEBUG:boto:path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.344 DEBUG:boto:auth_path=/hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.345 DEBUG:boto:Method: GET 2023-09-29T22:59:55.345 DEBUG:boto:Path: /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.345 DEBUG:boto:Data: 2023-09-29T22:59:55.345 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.345 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.345 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.345 DEBUG:boto:Params: {} 2023-09-29T22:59:55.345 DEBUG:boto:Token: None 2023-09-29T22:59:55.346 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34?versionId=null 2023-09-29T22:59:55.346 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc= 2023-09-29T22:59:55.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rlQt0oWCwXRcC01UGvwdlgQ9pjc=', 'Content-Length': '0'} 2023-09-29T22:59:55.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f5c6517d070a535e-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.349 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.349 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.349 DEBUG:boto:path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.349 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.349 DEBUG:boto:Method: GET 2023-09-29T22:59:55.350 DEBUG:boto:Path: /hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.350 DEBUG:boto:Data: 2023-09-29T22:59:55.350 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.350 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.350 DEBUG:boto:Params: {} 2023-09-29T22:59:55.350 DEBUG:boto:Token: None 2023-09-29T22:59:55.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.355 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9cf41ad4487906f-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.402 DEBUG:boto:b'hcnphz-61obj341000falseobj342023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.403 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.403 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.403 DEBUG:boto:path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.403 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.404 DEBUG:boto:Method: GET 2023-09-29T22:59:55.404 DEBUG:boto:Path: /hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.404 DEBUG:boto:Data: 2023-09-29T22:59:55.404 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.404 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.404 DEBUG:boto:Params: {} 2023-09-29T22:59:55.405 DEBUG:boto:Token: None 2023-09-29T22:59:55.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.405 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.416 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001d4bfb015968b2a2-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.416 DEBUG:boto:b'hcnphz-61obj341000falseobj342023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.417 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.417 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.417 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.417 DEBUG:boto:Path: /hcnphz-61/obj34 2023-09-29T22:59:55.417 DEBUG:boto:Data: 2023-09-29T22:59:55.417 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.418 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.418 DEBUG:boto:Params: {} 2023-09-29T22:59:55.418 DEBUG:boto:Token: None 2023-09-29T22:59:55.418 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34 2023-09-29T22:59:55.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IjL6KJbqgzcc/OMqux8bpkUSzI4= 2023-09-29T22:59:55.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IjL6KJbqgzcc/OMqux8bpkUSzI4=', 'Content-Length': '0'} 2023-09-29T22:59:55.420 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000056b6ac388af2e38-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.421 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.421 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.421 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.421 DEBUG:boto:Path: /hcnphz-61/obj34 2023-09-29T22:59:55.421 DEBUG:boto:Data: 2023-09-29T22:59:55.421 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.422 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.422 DEBUG:boto:Params: {} 2023-09-29T22:59:55.422 DEBUG:boto:Token: None 2023-09-29T22:59:55.422 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34 2023-09-29T22:59:55.422 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IjL6KJbqgzcc/OMqux8bpkUSzI4= 2023-09-29T22:59:55.422 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IjL6KJbqgzcc/OMqux8bpkUSzI4=', 'Content-Length': '0'} 2023-09-29T22:59:55.424 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000895c39c5663b1c07-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj34 2023-09-29T22:59:55.425 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.425 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.425 DEBUG:boto:Method: GET 2023-09-29T22:59:55.425 DEBUG:boto:Path: /hcnphz-61/obj34 2023-09-29T22:59:55.425 DEBUG:boto:Data: 2023-09-29T22:59:55.425 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.426 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.426 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.426 DEBUG:boto:Params: {} 2023-09-29T22:59:55.426 DEBUG:boto:Token: None 2023-09-29T22:59:55.426 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34 2023-09-29T22:59:55.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vi5IHPID05+AWtKqcqFUf98TmwE= 2023-09-29T22:59:55.426 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vi5IHPID05+AWtKqcqFUf98TmwE=', 'Content-Length': '0'} 2023-09-29T22:59:55.429 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad93519d8f5bef42-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.429 DEBUG:boto:path=/hcnphz-61/obj34 2023-09-29T22:59:55.429 DEBUG:boto:auth_path=/hcnphz-61/obj34 2023-09-29T22:59:55.429 DEBUG:boto:Method: GET 2023-09-29T22:59:55.429 DEBUG:boto:Path: /hcnphz-61/obj34 2023-09-29T22:59:55.429 DEBUG:boto:Data: 2023-09-29T22:59:55.430 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.430 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.430 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.430 DEBUG:boto:Params: {} 2023-09-29T22:59:55.430 DEBUG:boto:Token: None 2023-09-29T22:59:55.430 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj34 2023-09-29T22:59:55.430 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vi5IHPID05+AWtKqcqFUf98TmwE= 2023-09-29T22:59:55.431 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vi5IHPID05+AWtKqcqFUf98TmwE=', 'Content-Length': '0'} 2023-09-29T22:59:55.433 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fbc5ab87edee01b3-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.478 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.479 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.479 DEBUG:boto:path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.479 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.479 DEBUG:boto:Method: GET 2023-09-29T22:59:55.480 DEBUG:boto:Path: /hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.480 DEBUG:boto:Data: 2023-09-29T22:59:55.480 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.481 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.481 DEBUG:boto:Params: {} 2023-09-29T22:59:55.481 DEBUG:boto:Token: None 2023-09-29T22:59:55.481 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.482 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.486 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaace415ceb4b190-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.487 DEBUG:boto:b'hcnphz-61obj341000falseobj342023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.487 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.487 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.487 DEBUG:boto:path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.487 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.488 DEBUG:boto:Method: GET 2023-09-29T22:59:55.488 DEBUG:boto:Path: /hcnphz-61/?prefix=obj34 2023-09-29T22:59:55.488 DEBUG:boto:Data: 2023-09-29T22:59:55.488 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.488 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.488 DEBUG:boto:Params: {} 2023-09-29T22:59:55.488 DEBUG:boto:Token: None 2023-09-29T22:59:55.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.497 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fab887246682915c-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.497 DEBUG:boto:b'hcnphz-61obj341000falseobj342023-09-29T22:55:13.149Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:59:55.498 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.498 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.498 DEBUG:boto:path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.498 DEBUG:boto:auth_path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.498 DEBUG:boto:Method: GET 2023-09-29T22:59:55.498 DEBUG:boto:Path: /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.498 DEBUG:boto:Data: 2023-09-29T22:59:55.499 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.499 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.499 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.499 DEBUG:boto:Params: {} 2023-09-29T22:59:55.499 DEBUG:boto:Token: None 2023-09-29T22:59:55.499 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.499 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI= 2023-09-29T22:59:55.500 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI=', 'Content-Length': '0'} 2023-09-29T22:59:55.501 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002dcc8db8e9a754c0-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.502 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.502 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.502 DEBUG:boto:path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.502 DEBUG:boto:auth_path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.502 DEBUG:boto:Method: GET 2023-09-29T22:59:55.502 DEBUG:boto:Path: /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.503 DEBUG:boto:Data: 2023-09-29T22:59:55.503 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.503 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.503 DEBUG:boto:Params: {} 2023-09-29T22:59:55.503 DEBUG:boto:Token: None 2023-09-29T22:59:55.503 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.503 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI= 2023-09-29T22:59:55.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI=', 'Content-Length': '0'} 2023-09-29T22:59:55.506 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001fa340a2af447a38-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.506 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.507 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.507 DEBUG:boto:path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.507 DEBUG:boto:auth_path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.507 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.508 DEBUG:boto:Path: /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.508 DEBUG:boto:Data: 2023-09-29T22:59:55.508 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.508 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.509 DEBUG:boto:Params: {} 2023-09-29T22:59:55.509 DEBUG:boto:Token: None 2023-09-29T22:59:55.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.511 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gvJGf6WnZdlYZleHr7c2uF0Muqg= 2023-09-29T22:59:55.511 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gvJGf6WnZdlYZleHr7c2uF0Muqg=', 'Content-Length': '0'} 2023-09-29T22:59:55.514 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb3bf2fe4cb9883-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.514 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.514 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.514 DEBUG:boto:path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.514 DEBUG:boto:auth_path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.514 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.514 DEBUG:boto:Path: /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.515 DEBUG:boto:Data: 2023-09-29T22:59:55.515 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.515 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.515 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.515 DEBUG:boto:Params: {} 2023-09-29T22:59:55.515 DEBUG:boto:Token: None 2023-09-29T22:59:55.515 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.515 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gvJGf6WnZdlYZleHr7c2uF0Muqg= 2023-09-29T22:59:55.516 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gvJGf6WnZdlYZleHr7c2uF0Muqg=', 'Content-Length': '0'} 2023-09-29T22:59:55.518 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006e134d7032366b86-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:59:55.518 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.518 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.518 DEBUG:boto:path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.518 DEBUG:boto:auth_path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.519 DEBUG:boto:Method: GET 2023-09-29T22:59:55.519 DEBUG:boto:Path: /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.519 DEBUG:boto:Data: 2023-09-29T22:59:55.519 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.519 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.519 DEBUG:boto:Params: {} 2023-09-29T22:59:55.519 DEBUG:boto:Token: None 2023-09-29T22:59:55.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI= 2023-09-29T22:59:55.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI=', 'Content-Length': '0'} 2023-09-29T22:59:55.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008c5ca7db7cb03377-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.523 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.523 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.523 DEBUG:boto:path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.523 DEBUG:boto:auth_path=/hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.523 DEBUG:boto:Method: GET 2023-09-29T22:59:55.523 DEBUG:boto:Path: /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.523 DEBUG:boto:Data: 2023-09-29T22:59:55.523 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.523 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.524 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.524 DEBUG:boto:Params: {} 2023-09-29T22:59:55.524 DEBUG:boto:Token: None 2023-09-29T22:59:55.524 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37?versionId=null 2023-09-29T22:59:55.524 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI= 2023-09-29T22:59:55.524 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K991VLxwlasMDDzxMicqcIu9QSI=', 'Content-Length': '0'} 2023-09-29T22:59:55.526 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aab49f6045c6ee09-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.527 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.527 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.527 DEBUG:boto:path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.528 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.528 DEBUG:boto:Method: GET 2023-09-29T22:59:55.528 DEBUG:boto:Path: /hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.528 DEBUG:boto:Data: 2023-09-29T22:59:55.528 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.529 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.529 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.529 DEBUG:boto:Params: {} 2023-09-29T22:59:55.529 DEBUG:boto:Token: None 2023-09-29T22:59:55.530 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.530 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.530 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.534 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007484d82529032974-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.578 DEBUG:boto:b'hcnphz-61obj371000falseobj372023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.579 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.579 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.579 DEBUG:boto:path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.579 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.579 DEBUG:boto:Method: GET 2023-09-29T22:59:55.579 DEBUG:boto:Path: /hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.580 DEBUG:boto:Data: 2023-09-29T22:59:55.580 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.580 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.580 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.580 DEBUG:boto:Params: {} 2023-09-29T22:59:55.580 DEBUG:boto:Token: None 2023-09-29T22:59:55.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.581 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.581 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.590 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035d164bcb5149022-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.590 DEBUG:boto:b'hcnphz-61obj371000falseobj372023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.591 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.591 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.591 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.591 DEBUG:boto:Path: /hcnphz-61/obj37 2023-09-29T22:59:55.591 DEBUG:boto:Data: 2023-09-29T22:59:55.591 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.592 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.592 DEBUG:boto:Params: {} 2023-09-29T22:59:55.592 DEBUG:boto:Token: None 2023-09-29T22:59:55.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37 2023-09-29T22:59:55.592 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cQKO6vd4dqVwOipOUbRhQxKtzYU= 2023-09-29T22:59:55.592 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cQKO6vd4dqVwOipOUbRhQxKtzYU=', 'Content-Length': '0'} 2023-09-29T22:59:55.595 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000944333c93ef1024-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.598 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.598 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.599 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.599 DEBUG:boto:Path: /hcnphz-61/obj37 2023-09-29T22:59:55.599 DEBUG:boto:Data: 2023-09-29T22:59:55.600 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.600 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.600 DEBUG:boto:Params: {} 2023-09-29T22:59:55.600 DEBUG:boto:Token: None 2023-09-29T22:59:55.600 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37 2023-09-29T22:59:55.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cQKO6vd4dqVwOipOUbRhQxKtzYU= 2023-09-29T22:59:55.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cQKO6vd4dqVwOipOUbRhQxKtzYU=', 'Content-Length': '0'} 2023-09-29T22:59:55.603 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092fec68bdd416aca-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj37 2023-09-29T22:59:55.604 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.604 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.604 DEBUG:boto:Method: GET 2023-09-29T22:59:55.604 DEBUG:boto:Path: /hcnphz-61/obj37 2023-09-29T22:59:55.604 DEBUG:boto:Data: 2023-09-29T22:59:55.604 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.604 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.604 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.605 DEBUG:boto:Params: {} 2023-09-29T22:59:55.605 DEBUG:boto:Token: None 2023-09-29T22:59:55.605 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37 2023-09-29T22:59:55.605 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JUJRs4Kkt+9/9CvICN1JcsXaoHE= 2023-09-29T22:59:55.605 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JUJRs4Kkt+9/9CvICN1JcsXaoHE=', 'Content-Length': '0'} 2023-09-29T22:59:55.607 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a823a917d9d8a0f2-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.607 DEBUG:boto:path=/hcnphz-61/obj37 2023-09-29T22:59:55.607 DEBUG:boto:auth_path=/hcnphz-61/obj37 2023-09-29T22:59:55.608 DEBUG:boto:Method: GET 2023-09-29T22:59:55.608 DEBUG:boto:Path: /hcnphz-61/obj37 2023-09-29T22:59:55.608 DEBUG:boto:Data: 2023-09-29T22:59:55.608 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.608 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.608 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.608 DEBUG:boto:Params: {} 2023-09-29T22:59:55.608 DEBUG:boto:Token: None 2023-09-29T22:59:55.608 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj37 2023-09-29T22:59:55.609 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JUJRs4Kkt+9/9CvICN1JcsXaoHE= 2023-09-29T22:59:55.609 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JUJRs4Kkt+9/9CvICN1JcsXaoHE=', 'Content-Length': '0'} 2023-09-29T22:59:55.611 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e607e25289a8c9-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.654 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.655 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.655 DEBUG:boto:path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.655 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.655 DEBUG:boto:Method: GET 2023-09-29T22:59:55.655 DEBUG:boto:Path: /hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.655 DEBUG:boto:Data: 2023-09-29T22:59:55.655 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.655 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.655 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.656 DEBUG:boto:Params: {} 2023-09-29T22:59:55.656 DEBUG:boto:Token: None 2023-09-29T22:59:55.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.660 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008b33ecd7cf37171e-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.661 DEBUG:boto:b'hcnphz-61obj371000falseobj372023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.661 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.661 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.661 DEBUG:boto:path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.662 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.662 DEBUG:boto:Method: GET 2023-09-29T22:59:55.662 DEBUG:boto:Path: /hcnphz-61/?prefix=obj37 2023-09-29T22:59:55.662 DEBUG:boto:Data: 2023-09-29T22:59:55.662 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.662 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.662 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.662 DEBUG:boto:Params: {} 2023-09-29T22:59:55.663 DEBUG:boto:Token: None 2023-09-29T22:59:55.663 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.663 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.663 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.671 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bf960c4229a8646e-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.672 DEBUG:boto:b'hcnphz-61obj371000falseobj372023-09-29T22:55:15.308Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:59:55.672 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.672 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.672 DEBUG:boto:path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.673 DEBUG:boto:auth_path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.673 DEBUG:boto:Method: GET 2023-09-29T22:59:55.673 DEBUG:boto:Path: /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.673 DEBUG:boto:Data: 2023-09-29T22:59:55.673 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.673 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.673 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.673 DEBUG:boto:Params: {} 2023-09-29T22:59:55.674 DEBUG:boto:Token: None 2023-09-29T22:59:55.674 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.674 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g= 2023-09-29T22:59:55.674 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g=', 'Content-Length': '0'} 2023-09-29T22:59:55.676 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fb0ecb9024759ced-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.718 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.719 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.719 DEBUG:boto:path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.719 DEBUG:boto:auth_path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.719 DEBUG:boto:Method: GET 2023-09-29T22:59:55.719 DEBUG:boto:Path: /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.719 DEBUG:boto:Data: 2023-09-29T22:59:55.719 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.719 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.720 DEBUG:boto:Params: {} 2023-09-29T22:59:55.720 DEBUG:boto:Token: None 2023-09-29T22:59:55.720 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g= 2023-09-29T22:59:55.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g=', 'Content-Length': '0'} 2023-09-29T22:59:55.723 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c39b3206f5796427-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.723 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.724 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.724 DEBUG:boto:path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.724 DEBUG:boto:auth_path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.724 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.724 DEBUG:boto:Path: /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.724 DEBUG:boto:Data: 2023-09-29T22:59:55.724 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.724 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.724 DEBUG:boto:Params: {} 2023-09-29T22:59:55.725 DEBUG:boto:Token: None 2023-09-29T22:59:55.725 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wvfRfgKzyc77nUAhkUOn5mQMIds= 2023-09-29T22:59:55.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wvfRfgKzyc77nUAhkUOn5mQMIds=', 'Content-Length': '0'} 2023-09-29T22:59:55.728 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d64192b7fd0bfd72-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.728 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.728 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.728 DEBUG:boto:path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.728 DEBUG:boto:auth_path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.728 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.729 DEBUG:boto:Path: /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.729 DEBUG:boto:Data: 2023-09-29T22:59:55.729 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.729 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.729 DEBUG:boto:Params: {} 2023-09-29T22:59:55.729 DEBUG:boto:Token: None 2023-09-29T22:59:55.729 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wvfRfgKzyc77nUAhkUOn5mQMIds= 2023-09-29T22:59:55.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wvfRfgKzyc77nUAhkUOn5mQMIds=', 'Content-Length': '0'} 2023-09-29T22:59:55.733 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000500b1b8bd24ff32b-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:59:55.733 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.733 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.734 DEBUG:boto:path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.734 DEBUG:boto:auth_path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.734 DEBUG:boto:Method: GET 2023-09-29T22:59:55.734 DEBUG:boto:Path: /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.734 DEBUG:boto:Data: 2023-09-29T22:59:55.734 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.734 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.734 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.734 DEBUG:boto:Params: {} 2023-09-29T22:59:55.735 DEBUG:boto:Token: None 2023-09-29T22:59:55.735 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.735 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g= 2023-09-29T22:59:55.735 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g=', 'Content-Length': '0'} 2023-09-29T22:59:55.737 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2313cea946b556f-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.738 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.738 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.738 DEBUG:boto:path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.738 DEBUG:boto:auth_path=/hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.738 DEBUG:boto:Method: GET 2023-09-29T22:59:55.739 DEBUG:boto:Path: /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.739 DEBUG:boto:Data: 2023-09-29T22:59:55.739 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.739 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.739 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.739 DEBUG:boto:Params: {} 2023-09-29T22:59:55.739 DEBUG:boto:Token: None 2023-09-29T22:59:55.739 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38?versionId=null 2023-09-29T22:59:55.740 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g= 2023-09-29T22:59:55.740 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KpEyMEF5WS7IfOJ/oVcKavSwF8g=', 'Content-Length': '0'} 2023-09-29T22:59:55.742 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aeb4ed5b15c4edb2-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.743 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.743 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.743 DEBUG:boto:path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.743 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.743 DEBUG:boto:Method: GET 2023-09-29T22:59:55.743 DEBUG:boto:Path: /hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.743 DEBUG:boto:Data: 2023-09-29T22:59:55.743 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.744 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.744 DEBUG:boto:Params: {} 2023-09-29T22:59:55.744 DEBUG:boto:Token: None 2023-09-29T22:59:55.744 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.744 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.744 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.749 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fe92ce51b60770f7-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.749 DEBUG:boto:b'hcnphz-61obj381000falseobj382023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.749 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.749 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.749 DEBUG:boto:path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.750 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.750 DEBUG:boto:Method: GET 2023-09-29T22:59:55.750 DEBUG:boto:Path: /hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.750 DEBUG:boto:Data: 2023-09-29T22:59:55.750 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.750 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.750 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.750 DEBUG:boto:Params: {} 2023-09-29T22:59:55.751 DEBUG:boto:Token: None 2023-09-29T22:59:55.751 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.751 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.751 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.758 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c5554d42b134b79f-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.802 DEBUG:boto:b'hcnphz-61obj381000falseobj382023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.803 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.803 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.803 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.804 DEBUG:boto:Path: /hcnphz-61/obj38 2023-09-29T22:59:55.804 DEBUG:boto:Data: 2023-09-29T22:59:55.804 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.804 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.804 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.804 DEBUG:boto:Params: {} 2023-09-29T22:59:55.804 DEBUG:boto:Token: None 2023-09-29T22:59:55.805 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38 2023-09-29T22:59:55.805 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mPGdWTnDQLT/7JtXxsSHWa59tXc= 2023-09-29T22:59:55.805 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mPGdWTnDQLT/7JtXxsSHWa59tXc=', 'Content-Length': '0'} 2023-09-29T22:59:55.808 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e03b39e01fc012e3-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.808 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.808 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.808 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.808 DEBUG:boto:Path: /hcnphz-61/obj38 2023-09-29T22:59:55.808 DEBUG:boto:Data: 2023-09-29T22:59:55.809 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.809 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.809 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.809 DEBUG:boto:Params: {} 2023-09-29T22:59:55.809 DEBUG:boto:Token: None 2023-09-29T22:59:55.809 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38 2023-09-29T22:59:55.809 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mPGdWTnDQLT/7JtXxsSHWa59tXc= 2023-09-29T22:59:55.810 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mPGdWTnDQLT/7JtXxsSHWa59tXc=', 'Content-Length': '0'} 2023-09-29T22:59:55.812 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003fb9711c507d0c66-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj38 2023-09-29T22:59:55.813 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.813 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.813 DEBUG:boto:Method: GET 2023-09-29T22:59:55.813 DEBUG:boto:Path: /hcnphz-61/obj38 2023-09-29T22:59:55.813 DEBUG:boto:Data: 2023-09-29T22:59:55.813 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.813 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.813 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.813 DEBUG:boto:Params: {} 2023-09-29T22:59:55.814 DEBUG:boto:Token: None 2023-09-29T22:59:55.814 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38 2023-09-29T22:59:55.814 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S26FhHWGc95GFCKu5M5GI1L0cVQ= 2023-09-29T22:59:55.814 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S26FhHWGc95GFCKu5M5GI1L0cVQ=', 'Content-Length': '0'} 2023-09-29T22:59:55.817 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000769174b2b4c5169b-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.858 DEBUG:boto:path=/hcnphz-61/obj38 2023-09-29T22:59:55.858 DEBUG:boto:auth_path=/hcnphz-61/obj38 2023-09-29T22:59:55.859 DEBUG:boto:Method: GET 2023-09-29T22:59:55.859 DEBUG:boto:Path: /hcnphz-61/obj38 2023-09-29T22:59:55.859 DEBUG:boto:Data: 2023-09-29T22:59:55.859 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.859 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.859 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.859 DEBUG:boto:Params: {} 2023-09-29T22:59:55.859 DEBUG:boto:Token: None 2023-09-29T22:59:55.860 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj38 2023-09-29T22:59:55.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:S26FhHWGc95GFCKu5M5GI1L0cVQ= 2023-09-29T22:59:55.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:S26FhHWGc95GFCKu5M5GI1L0cVQ=', 'Content-Length': '0'} 2023-09-29T22:59:55.862 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000789311d89ca6b600-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.863 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.863 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.863 DEBUG:boto:path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.863 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.863 DEBUG:boto:Method: GET 2023-09-29T22:59:55.863 DEBUG:boto:Path: /hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.863 DEBUG:boto:Data: 2023-09-29T22:59:55.864 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.864 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.864 DEBUG:boto:Params: {} 2023-09-29T22:59:55.864 DEBUG:boto:Token: None 2023-09-29T22:59:55.864 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.864 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.868 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee00c14b2f2506f1-00651756eb-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.869 DEBUG:boto:b'hcnphz-61obj381000falseobj382023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.869 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.869 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.869 DEBUG:boto:path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.870 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.870 DEBUG:boto:Method: GET 2023-09-29T22:59:55.870 DEBUG:boto:Path: /hcnphz-61/?prefix=obj38 2023-09-29T22:59:55.870 DEBUG:boto:Data: 2023-09-29T22:59:55.870 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.870 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.870 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.870 DEBUG:boto:Params: {} 2023-09-29T22:59:55.870 DEBUG:boto:Token: None 2023-09-29T22:59:55.871 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.871 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.871 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:55.879 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004344b3461999efd4-00651756eb-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.879 DEBUG:boto:b'hcnphz-61obj381000falseobj382023-09-29T22:55:13.587Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:55.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:59:55.880 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:55.880 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:55.880 DEBUG:boto:path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.880 DEBUG:boto:auth_path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.880 DEBUG:boto:Method: GET 2023-09-29T22:59:55.880 DEBUG:boto:Path: /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.880 DEBUG:boto:Data: 2023-09-29T22:59:55.880 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.881 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.881 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.881 DEBUG:boto:Params: {} 2023-09-29T22:59:55.881 DEBUG:boto:Token: None 2023-09-29T22:59:55.881 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A= 2023-09-29T22:59:55.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A=', 'Content-Length': '0'} 2023-09-29T22:59:55.884 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ac4e0925bd3f9344-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.884 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:55.884 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:55.884 DEBUG:boto:path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.884 DEBUG:boto:auth_path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.884 DEBUG:boto:Method: GET 2023-09-29T22:59:55.885 DEBUG:boto:Path: /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.885 DEBUG:boto:Data: 2023-09-29T22:59:55.885 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.885 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.885 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.885 DEBUG:boto:Params: {} 2023-09-29T22:59:55.885 DEBUG:boto:Token: None 2023-09-29T22:59:55.885 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.886 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A= 2023-09-29T22:59:55.886 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A=', 'Content-Length': '0'} 2023-09-29T22:59:55.888 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d6fff2a49f29be12-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.930 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:55.931 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:55.931 DEBUG:boto:path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.931 DEBUG:boto:auth_path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.931 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.931 DEBUG:boto:Path: /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.931 DEBUG:boto:Data: 2023-09-29T22:59:55.931 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.931 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.932 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.932 DEBUG:boto:Params: {} 2023-09-29T22:59:55.932 DEBUG:boto:Token: None 2023-09-29T22:59:55.932 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.932 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jpRMNOeaEt+jSxlh89HGmdxkd/w= 2023-09-29T22:59:55.932 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jpRMNOeaEt+jSxlh89HGmdxkd/w=', 'Content-Length': '0'} 2023-09-29T22:59:55.935 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000646d78ef8c8932fc-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.935 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:55.935 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:55.936 DEBUG:boto:path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.936 DEBUG:boto:auth_path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.936 DEBUG:boto:Method: HEAD 2023-09-29T22:59:55.936 DEBUG:boto:Path: /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.936 DEBUG:boto:Data: 2023-09-29T22:59:55.936 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.936 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.936 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.936 DEBUG:boto:Params: {} 2023-09-29T22:59:55.936 DEBUG:boto:Token: None 2023-09-29T22:59:55.937 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.937 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jpRMNOeaEt+jSxlh89HGmdxkd/w= 2023-09-29T22:59:55.937 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jpRMNOeaEt+jSxlh89HGmdxkd/w=', 'Content-Length': '0'} 2023-09-29T22:59:55.940 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b484d64b28111908-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:59:55.940 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:55.940 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:55.940 DEBUG:boto:path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.940 DEBUG:boto:auth_path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.941 DEBUG:boto:Method: GET 2023-09-29T22:59:55.941 DEBUG:boto:Path: /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.941 DEBUG:boto:Data: 2023-09-29T22:59:55.941 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.941 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.941 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.941 DEBUG:boto:Params: {} 2023-09-29T22:59:55.942 DEBUG:boto:Token: None 2023-09-29T22:59:55.942 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.942 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A= 2023-09-29T22:59:55.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A=', 'Content-Length': '0'} 2023-09-29T22:59:55.945 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081f4130bf64eb4a9-00651756eb-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.990 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:55.991 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:55.991 DEBUG:boto:path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.991 DEBUG:boto:auth_path=/hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.991 DEBUG:boto:Method: GET 2023-09-29T22:59:55.991 DEBUG:boto:Path: /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.991 DEBUG:boto:Data: 2023-09-29T22:59:55.991 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:55.991 DEBUG:boto:Port: 8000 2023-09-29T22:59:55.992 DEBUG:boto:Params: {} 2023-09-29T22:59:55.992 DEBUG:boto:Token: None 2023-09-29T22:59:55.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/obj44?versionId=null 2023-09-29T22:59:55.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A= 2023-09-29T22:59:55.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6QPeScs43Qe4WVeXKcT+drbz11A=', 'Content-Length': '0'} 2023-09-29T22:59:55.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e794e797a39137f8-00651756eb-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:55 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:55.996 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:55.996 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:55.996 DEBUG:boto:path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:55.996 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:55.996 DEBUG:boto:Method: GET 2023-09-29T22:59:55.996 DEBUG:boto:Path: /hcnphz-61/?prefix=obj44 2023-09-29T22:59:55.996 DEBUG:boto:Data: 2023-09-29T22:59:55.996 DEBUG:boto:Headers: {} 2023-09-29T22:59:55.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:55.996 DEBUG:boto:Port: 8001 2023-09-29T22:59:55.997 DEBUG:boto:Params: {} 2023-09-29T22:59:55.997 DEBUG:boto:Token: None 2023-09-29T22:59:55.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:55 GMT /hcnphz-61/ 2023-09-29T22:59:55.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU= 2023-09-29T22:59:55.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:55 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eFsXtxfneEOalHRUgtB7Qnx4IOU=', 'Content-Length': '0'} 2023-09-29T22:59:56.002 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b259da6988c64c5-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.002 DEBUG:boto:b'hcnphz-61obj441000falseobj442023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.005 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.006 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.006 DEBUG:boto:path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.006 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.006 DEBUG:boto:Method: GET 2023-09-29T22:59:56.006 DEBUG:boto:Path: /hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.006 DEBUG:boto:Data: 2023-09-29T22:59:56.007 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.007 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.007 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.007 DEBUG:boto:Params: {} 2023-09-29T22:59:56.007 DEBUG:boto:Token: None 2023-09-29T22:59:56.007 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.007 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.008 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013ab768fed3c4b73-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.015 DEBUG:boto:b'hcnphz-61obj441000falseobj442023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.016 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:56.016 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:56.016 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.016 DEBUG:boto:Path: /hcnphz-61/obj44 2023-09-29T22:59:56.016 DEBUG:boto:Data: 2023-09-29T22:59:56.016 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.017 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.017 DEBUG:boto:Params: {} 2023-09-29T22:59:56.017 DEBUG:boto:Token: None 2023-09-29T22:59:56.017 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj44 2023-09-29T22:59:56.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xKvMOUoACpCqR+XyVn2+UGZPvVU= 2023-09-29T22:59:56.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xKvMOUoACpCqR+XyVn2+UGZPvVU=', 'Content-Length': '0'} 2023-09-29T22:59:56.020 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007d5bf27049a56ed9-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.020 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:56.020 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:56.020 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.020 DEBUG:boto:Path: /hcnphz-61/obj44 2023-09-29T22:59:56.021 DEBUG:boto:Data: 2023-09-29T22:59:56.021 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.021 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.021 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.021 DEBUG:boto:Params: {} 2023-09-29T22:59:56.021 DEBUG:boto:Token: None 2023-09-29T22:59:56.021 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj44 2023-09-29T22:59:56.021 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xKvMOUoACpCqR+XyVn2+UGZPvVU= 2023-09-29T22:59:56.022 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xKvMOUoACpCqR+XyVn2+UGZPvVU=', 'Content-Length': '0'} 2023-09-29T22:59:56.024 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c05ebe2296b1f348-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T22:59:56.024 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:56.024 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:56.024 DEBUG:boto:Method: GET 2023-09-29T22:59:56.024 DEBUG:boto:Path: /hcnphz-61/obj44 2023-09-29T22:59:56.024 DEBUG:boto:Data: 2023-09-29T22:59:56.025 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.025 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.025 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.025 DEBUG:boto:Params: {} 2023-09-29T22:59:56.025 DEBUG:boto:Token: None 2023-09-29T22:59:56.025 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj44 2023-09-29T22:59:56.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:baoh9x3+3cg7QIJF4mZmtQuRedw= 2023-09-29T22:59:56.026 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:baoh9x3+3cg7QIJF4mZmtQuRedw=', 'Content-Length': '0'} 2023-09-29T22:59:56.028 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000067af032607350d94-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.070 DEBUG:boto:path=/hcnphz-61/obj44 2023-09-29T22:59:56.071 DEBUG:boto:auth_path=/hcnphz-61/obj44 2023-09-29T22:59:56.071 DEBUG:boto:Method: GET 2023-09-29T22:59:56.071 DEBUG:boto:Path: /hcnphz-61/obj44 2023-09-29T22:59:56.071 DEBUG:boto:Data: 2023-09-29T22:59:56.071 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.071 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.071 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.071 DEBUG:boto:Params: {} 2023-09-29T22:59:56.072 DEBUG:boto:Token: None 2023-09-29T22:59:56.072 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj44 2023-09-29T22:59:56.072 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:baoh9x3+3cg7QIJF4mZmtQuRedw= 2023-09-29T22:59:56.072 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:baoh9x3+3cg7QIJF4mZmtQuRedw=', 'Content-Length': '0'} 2023-09-29T22:59:56.075 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d077b2a00304e892-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.076 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.076 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.076 DEBUG:boto:path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.076 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.076 DEBUG:boto:Method: GET 2023-09-29T22:59:56.076 DEBUG:boto:Path: /hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.076 DEBUG:boto:Data: 2023-09-29T22:59:56.076 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.076 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.077 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.077 DEBUG:boto:Params: {} 2023-09-29T22:59:56.077 DEBUG:boto:Token: None 2023-09-29T22:59:56.077 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.077 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.077 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.082 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008df6c3d34f8c32cc-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.082 DEBUG:boto:b'hcnphz-61obj441000falseobj442023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.083 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.083 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.083 DEBUG:boto:path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.083 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.083 DEBUG:boto:Method: GET 2023-09-29T22:59:56.083 DEBUG:boto:Path: /hcnphz-61/?prefix=obj44 2023-09-29T22:59:56.083 DEBUG:boto:Data: 2023-09-29T22:59:56.083 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.084 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.084 DEBUG:boto:Params: {} 2023-09-29T22:59:56.084 DEBUG:boto:Token: None 2023-09-29T22:59:56.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.091 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c2ee2a2bb86fcb82-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.092 DEBUG:boto:b'hcnphz-61obj441000falseobj442023-09-29T22:55:12.847Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:59:56.093 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.093 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.093 DEBUG:boto:path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.093 DEBUG:boto:auth_path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.093 DEBUG:boto:Method: GET 2023-09-29T22:59:56.093 DEBUG:boto:Path: /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.094 DEBUG:boto:Data: 2023-09-29T22:59:56.094 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.094 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.094 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.094 DEBUG:boto:Params: {} 2023-09-29T22:59:56.094 DEBUG:boto:Token: None 2023-09-29T22:59:56.095 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.095 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA= 2023-09-29T22:59:56.095 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA=', 'Content-Length': '0'} 2023-09-29T22:59:56.099 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000485c0ce385357514-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.101 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.101 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.101 DEBUG:boto:path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.101 DEBUG:boto:auth_path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.101 DEBUG:boto:Method: GET 2023-09-29T22:59:56.101 DEBUG:boto:Path: /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.102 DEBUG:boto:Data: 2023-09-29T22:59:56.102 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.102 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.102 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.102 DEBUG:boto:Params: {} 2023-09-29T22:59:56.102 DEBUG:boto:Token: None 2023-09-29T22:59:56.102 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.103 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA= 2023-09-29T22:59:56.103 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA=', 'Content-Length': '0'} 2023-09-29T22:59:56.105 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003b4b71323ebc5037-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.105 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.105 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.106 DEBUG:boto:path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.106 DEBUG:boto:auth_path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.106 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.106 DEBUG:boto:Path: /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.106 DEBUG:boto:Data: 2023-09-29T22:59:56.106 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.106 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.106 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.107 DEBUG:boto:Params: {} 2023-09-29T22:59:56.107 DEBUG:boto:Token: None 2023-09-29T22:59:56.107 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.107 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+krqm3UVBFFnvaQxpke9Wm1iLbw= 2023-09-29T22:59:56.107 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+krqm3UVBFFnvaQxpke9Wm1iLbw=', 'Content-Length': '0'} 2023-09-29T22:59:56.109 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a89ee07b0ace8eb4-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.110 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.110 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.110 DEBUG:boto:path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.110 DEBUG:boto:auth_path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.110 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.110 DEBUG:boto:Path: /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.110 DEBUG:boto:Data: 2023-09-29T22:59:56.110 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.111 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.111 DEBUG:boto:Params: {} 2023-09-29T22:59:56.111 DEBUG:boto:Token: None 2023-09-29T22:59:56.111 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.111 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+krqm3UVBFFnvaQxpke9Wm1iLbw= 2023-09-29T22:59:56.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+krqm3UVBFFnvaQxpke9Wm1iLbw=', 'Content-Length': '0'} 2023-09-29T22:59:56.114 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031e9db0b98b0ea51-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:59:56.114 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.115 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.115 DEBUG:boto:path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.115 DEBUG:boto:auth_path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.115 DEBUG:boto:Method: GET 2023-09-29T22:59:56.115 DEBUG:boto:Path: /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.115 DEBUG:boto:Data: 2023-09-29T22:59:56.115 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.115 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.115 DEBUG:boto:Params: {} 2023-09-29T22:59:56.116 DEBUG:boto:Token: None 2023-09-29T22:59:56.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA= 2023-09-29T22:59:56.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA=', 'Content-Length': '0'} 2023-09-29T22:59:56.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000042b4bcaf44cd629a-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.119 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.119 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.119 DEBUG:boto:path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.119 DEBUG:boto:auth_path=/hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.119 DEBUG:boto:Method: GET 2023-09-29T22:59:56.120 DEBUG:boto:Path: /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.120 DEBUG:boto:Data: 2023-09-29T22:59:56.120 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.120 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.120 DEBUG:boto:Params: {} 2023-09-29T22:59:56.120 DEBUG:boto:Token: None 2023-09-29T22:59:56.120 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46?versionId=null 2023-09-29T22:59:56.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA= 2023-09-29T22:59:56.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1N6AE1ZxB055O3G0gBXQv08iWHA=', 'Content-Length': '0'} 2023-09-29T22:59:56.123 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004332b38a5e88cf69-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.124 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.124 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.124 DEBUG:boto:path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.124 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.124 DEBUG:boto:Method: GET 2023-09-29T22:59:56.125 DEBUG:boto:Path: /hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.125 DEBUG:boto:Data: 2023-09-29T22:59:56.125 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.125 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.125 DEBUG:boto:Params: {} 2023-09-29T22:59:56.126 DEBUG:boto:Token: None 2023-09-29T22:59:56.126 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.131 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000096c235b5f5a20eca-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.178 DEBUG:boto:b'hcnphz-61obj461000falseobj462023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.179 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.179 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.179 DEBUG:boto:path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.179 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.179 DEBUG:boto:Method: GET 2023-09-29T22:59:56.179 DEBUG:boto:Path: /hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.180 DEBUG:boto:Data: 2023-09-29T22:59:56.180 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.180 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.180 DEBUG:boto:Params: {} 2023-09-29T22:59:56.180 DEBUG:boto:Token: None 2023-09-29T22:59:56.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.190 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c746c2d050281d04-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.190 DEBUG:boto:b'hcnphz-61obj461000falseobj462023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.191 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.191 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.191 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.191 DEBUG:boto:Path: /hcnphz-61/obj46 2023-09-29T22:59:56.191 DEBUG:boto:Data: 2023-09-29T22:59:56.192 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.192 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.192 DEBUG:boto:Params: {} 2023-09-29T22:59:56.192 DEBUG:boto:Token: None 2023-09-29T22:59:56.192 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46 2023-09-29T22:59:56.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:16S/wFM333jn5FTLVef2HeKxWU8= 2023-09-29T22:59:56.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:16S/wFM333jn5FTLVef2HeKxWU8=', 'Content-Length': '0'} 2023-09-29T22:59:56.194 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f726c3684736c55b-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.195 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.195 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.195 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.195 DEBUG:boto:Path: /hcnphz-61/obj46 2023-09-29T22:59:56.196 DEBUG:boto:Data: 2023-09-29T22:59:56.196 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.196 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.196 DEBUG:boto:Params: {} 2023-09-29T22:59:56.196 DEBUG:boto:Token: None 2023-09-29T22:59:56.196 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46 2023-09-29T22:59:56.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:16S/wFM333jn5FTLVef2HeKxWU8= 2023-09-29T22:59:56.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:16S/wFM333jn5FTLVef2HeKxWU8=', 'Content-Length': '0'} 2023-09-29T22:59:56.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f22d86514ce8529b-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.199 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj46 2023-09-29T22:59:56.199 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.199 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.200 DEBUG:boto:Method: GET 2023-09-29T22:59:56.200 DEBUG:boto:Path: /hcnphz-61/obj46 2023-09-29T22:59:56.200 DEBUG:boto:Data: 2023-09-29T22:59:56.200 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.200 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.200 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.200 DEBUG:boto:Params: {} 2023-09-29T22:59:56.201 DEBUG:boto:Token: None 2023-09-29T22:59:56.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46 2023-09-29T22:59:56.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JYlDZN7zJY1UT/5mxo5QCe61mR4= 2023-09-29T22:59:56.201 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JYlDZN7zJY1UT/5mxo5QCe61mR4=', 'Content-Length': '0'} 2023-09-29T22:59:56.205 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed5fed0ab40aaa27-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.206 DEBUG:boto:path=/hcnphz-61/obj46 2023-09-29T22:59:56.206 DEBUG:boto:auth_path=/hcnphz-61/obj46 2023-09-29T22:59:56.206 DEBUG:boto:Method: GET 2023-09-29T22:59:56.206 DEBUG:boto:Path: /hcnphz-61/obj46 2023-09-29T22:59:56.206 DEBUG:boto:Data: 2023-09-29T22:59:56.206 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.207 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.207 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.207 DEBUG:boto:Params: {} 2023-09-29T22:59:56.207 DEBUG:boto:Token: None 2023-09-29T22:59:56.207 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj46 2023-09-29T22:59:56.207 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JYlDZN7zJY1UT/5mxo5QCe61mR4= 2023-09-29T22:59:56.208 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JYlDZN7zJY1UT/5mxo5QCe61mR4=', 'Content-Length': '0'} 2023-09-29T22:59:56.210 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cbcb1673d534b7c-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.250 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.251 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.251 DEBUG:boto:path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.251 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.251 DEBUG:boto:Method: GET 2023-09-29T22:59:56.251 DEBUG:boto:Path: /hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.251 DEBUG:boto:Data: 2023-09-29T22:59:56.251 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.251 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.251 DEBUG:boto:Params: {} 2023-09-29T22:59:56.252 DEBUG:boto:Token: None 2023-09-29T22:59:56.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.256 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bfef8f261182e6b2-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.256 DEBUG:boto:b'hcnphz-61obj461000falseobj462023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.257 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.257 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.257 DEBUG:boto:path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.257 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.257 DEBUG:boto:Method: GET 2023-09-29T22:59:56.257 DEBUG:boto:Path: /hcnphz-61/?prefix=obj46 2023-09-29T22:59:56.258 DEBUG:boto:Data: 2023-09-29T22:59:56.258 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.258 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.258 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.258 DEBUG:boto:Params: {} 2023-09-29T22:59:56.258 DEBUG:boto:Token: None 2023-09-29T22:59:56.258 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.259 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.259 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.267 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001c85344568410b7e-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.267 DEBUG:boto:b'hcnphz-61obj461000falseobj462023-09-29T22:55:13.993Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:59:56.267 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.268 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.268 DEBUG:boto:path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.268 DEBUG:boto:auth_path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.268 DEBUG:boto:Method: GET 2023-09-29T22:59:56.268 DEBUG:boto:Path: /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.268 DEBUG:boto:Data: 2023-09-29T22:59:56.268 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.268 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.269 DEBUG:boto:Params: {} 2023-09-29T22:59:56.269 DEBUG:boto:Token: None 2023-09-29T22:59:56.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps= 2023-09-29T22:59:56.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps=', 'Content-Length': '0'} 2023-09-29T22:59:56.271 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000043908c97ffbbb7d6-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.314 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.314 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.315 DEBUG:boto:path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.315 DEBUG:boto:auth_path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.315 DEBUG:boto:Method: GET 2023-09-29T22:59:56.315 DEBUG:boto:Path: /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.315 DEBUG:boto:Data: 2023-09-29T22:59:56.315 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.315 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.315 DEBUG:boto:Params: {} 2023-09-29T22:59:56.316 DEBUG:boto:Token: None 2023-09-29T22:59:56.316 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps= 2023-09-29T22:59:56.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps=', 'Content-Length': '0'} 2023-09-29T22:59:56.319 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed6d2b8c7edb41c9-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.319 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.320 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.320 DEBUG:boto:path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.320 DEBUG:boto:auth_path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.320 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.320 DEBUG:boto:Path: /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.320 DEBUG:boto:Data: 2023-09-29T22:59:56.320 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.320 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.320 DEBUG:boto:Params: {} 2023-09-29T22:59:56.321 DEBUG:boto:Token: None 2023-09-29T22:59:56.321 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qdGzkW9WgNSZ9t/OqGM20D8/xVw= 2023-09-29T22:59:56.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qdGzkW9WgNSZ9t/OqGM20D8/xVw=', 'Content-Length': '0'} 2023-09-29T22:59:56.323 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008293d20b6338a9a8-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.324 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.324 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.324 DEBUG:boto:path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.324 DEBUG:boto:auth_path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.324 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.324 DEBUG:boto:Path: /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.324 DEBUG:boto:Data: 2023-09-29T22:59:56.324 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.325 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.325 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.325 DEBUG:boto:Params: {} 2023-09-29T22:59:56.325 DEBUG:boto:Token: None 2023-09-29T22:59:56.325 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:qdGzkW9WgNSZ9t/OqGM20D8/xVw= 2023-09-29T22:59:56.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qdGzkW9WgNSZ9t/OqGM20D8/xVw=', 'Content-Length': '0'} 2023-09-29T22:59:56.328 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082f0528853b5c529-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:59:56.328 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.329 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.329 DEBUG:boto:path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.329 DEBUG:boto:auth_path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.329 DEBUG:boto:Method: GET 2023-09-29T22:59:56.329 DEBUG:boto:Path: /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.329 DEBUG:boto:Data: 2023-09-29T22:59:56.329 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.329 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.329 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.329 DEBUG:boto:Params: {} 2023-09-29T22:59:56.330 DEBUG:boto:Token: None 2023-09-29T22:59:56.330 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.330 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps= 2023-09-29T22:59:56.330 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps=', 'Content-Length': '0'} 2023-09-29T22:59:56.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afa9da26b1bd8400-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.333 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.333 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.333 DEBUG:boto:path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.333 DEBUG:boto:auth_path=/hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.333 DEBUG:boto:Method: GET 2023-09-29T22:59:56.333 DEBUG:boto:Path: /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.333 DEBUG:boto:Data: 2023-09-29T22:59:56.334 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.334 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.334 DEBUG:boto:Params: {} 2023-09-29T22:59:56.334 DEBUG:boto:Token: None 2023-09-29T22:59:56.334 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51?versionId=null 2023-09-29T22:59:56.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps= 2023-09-29T22:59:56.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AOplhS5Sn3zmYJm9Zx0wvDp0Dps=', 'Content-Length': '0'} 2023-09-29T22:59:56.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cf81d55bebb925ee-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.378 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.378 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.379 DEBUG:boto:path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.379 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.379 DEBUG:boto:Method: GET 2023-09-29T22:59:56.379 DEBUG:boto:Path: /hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.379 DEBUG:boto:Data: 2023-09-29T22:59:56.379 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.379 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.379 DEBUG:boto:Params: {} 2023-09-29T22:59:56.380 DEBUG:boto:Token: None 2023-09-29T22:59:56.380 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.384 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005102aa9bc815856-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.384 DEBUG:boto:b'hcnphz-61obj511000falseobj512023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.385 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.385 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.385 DEBUG:boto:path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.385 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.385 DEBUG:boto:Method: GET 2023-09-29T22:59:56.385 DEBUG:boto:Path: /hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.386 DEBUG:boto:Data: 2023-09-29T22:59:56.386 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.386 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.386 DEBUG:boto:Params: {} 2023-09-29T22:59:56.386 DEBUG:boto:Token: None 2023-09-29T22:59:56.386 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.386 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.395 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a0d979b7c84cf02-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.395 DEBUG:boto:b'hcnphz-61obj511000falseobj512023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.395 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.395 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.395 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.396 DEBUG:boto:Path: /hcnphz-61/obj51 2023-09-29T22:59:56.396 DEBUG:boto:Data: 2023-09-29T22:59:56.396 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.396 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.396 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.396 DEBUG:boto:Params: {} 2023-09-29T22:59:56.396 DEBUG:boto:Token: None 2023-09-29T22:59:56.396 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51 2023-09-29T22:59:56.397 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OR8CI91YMGXJ9kxA86985uRyYLc= 2023-09-29T22:59:56.397 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OR8CI91YMGXJ9kxA86985uRyYLc=', 'Content-Length': '0'} 2023-09-29T22:59:56.398 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030f0c084d81c6af1-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.399 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.399 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.399 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.399 DEBUG:boto:Path: /hcnphz-61/obj51 2023-09-29T22:59:56.399 DEBUG:boto:Data: 2023-09-29T22:59:56.399 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.399 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.400 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.400 DEBUG:boto:Params: {} 2023-09-29T22:59:56.400 DEBUG:boto:Token: None 2023-09-29T22:59:56.400 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51 2023-09-29T22:59:56.400 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OR8CI91YMGXJ9kxA86985uRyYLc= 2023-09-29T22:59:56.400 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OR8CI91YMGXJ9kxA86985uRyYLc=', 'Content-Length': '0'} 2023-09-29T22:59:56.402 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000689234240af331e9-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj51 2023-09-29T22:59:56.403 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.403 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.403 DEBUG:boto:Method: GET 2023-09-29T22:59:56.403 DEBUG:boto:Path: /hcnphz-61/obj51 2023-09-29T22:59:56.403 DEBUG:boto:Data: 2023-09-29T22:59:56.403 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.404 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.405 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.405 DEBUG:boto:Params: {} 2023-09-29T22:59:56.405 DEBUG:boto:Token: None 2023-09-29T22:59:56.405 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51 2023-09-29T22:59:56.405 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KX73OvVd8Ztwc+vUf8gJxOmSouk= 2023-09-29T22:59:56.406 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KX73OvVd8Ztwc+vUf8gJxOmSouk=', 'Content-Length': '0'} 2023-09-29T22:59:56.409 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b057a4685279c1c7-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.450 DEBUG:boto:path=/hcnphz-61/obj51 2023-09-29T22:59:56.451 DEBUG:boto:auth_path=/hcnphz-61/obj51 2023-09-29T22:59:56.451 DEBUG:boto:Method: GET 2023-09-29T22:59:56.451 DEBUG:boto:Path: /hcnphz-61/obj51 2023-09-29T22:59:56.451 DEBUG:boto:Data: 2023-09-29T22:59:56.451 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.451 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.451 DEBUG:boto:Params: {} 2023-09-29T22:59:56.452 DEBUG:boto:Token: None 2023-09-29T22:59:56.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj51 2023-09-29T22:59:56.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KX73OvVd8Ztwc+vUf8gJxOmSouk= 2023-09-29T22:59:56.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KX73OvVd8Ztwc+vUf8gJxOmSouk=', 'Content-Length': '0'} 2023-09-29T22:59:56.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001343a9f9b18a2407-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.455 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.455 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.456 DEBUG:boto:path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.456 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.456 DEBUG:boto:Method: GET 2023-09-29T22:59:56.456 DEBUG:boto:Path: /hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.456 DEBUG:boto:Data: 2023-09-29T22:59:56.456 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.456 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.456 DEBUG:boto:Params: {} 2023-09-29T22:59:56.457 DEBUG:boto:Token: None 2023-09-29T22:59:56.457 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.457 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.457 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e118fd1cdac278c3-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.462 DEBUG:boto:b'hcnphz-61obj511000falseobj512023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.462 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.463 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.463 DEBUG:boto:path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.463 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.463 DEBUG:boto:Method: GET 2023-09-29T22:59:56.463 DEBUG:boto:Path: /hcnphz-61/?prefix=obj51 2023-09-29T22:59:56.463 DEBUG:boto:Data: 2023-09-29T22:59:56.463 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.463 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.464 DEBUG:boto:Params: {} 2023-09-29T22:59:56.464 DEBUG:boto:Token: None 2023-09-29T22:59:56.464 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.472 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000926fb97a00376d03-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.514 DEBUG:boto:b'hcnphz-61obj511000falseobj512023-09-29T22:55:13.200Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:59:56.515 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.515 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.515 DEBUG:boto:path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.515 DEBUG:boto:auth_path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.516 DEBUG:boto:Method: GET 2023-09-29T22:59:56.516 DEBUG:boto:Path: /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.516 DEBUG:boto:Data: 2023-09-29T22:59:56.516 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.516 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.516 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.516 DEBUG:boto:Params: {} 2023-09-29T22:59:56.516 DEBUG:boto:Token: None 2023-09-29T22:59:56.516 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.517 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc= 2023-09-29T22:59:56.517 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc=', 'Content-Length': '0'} 2023-09-29T22:59:56.520 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000049ce4f59e8415d8-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.520 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.520 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.520 DEBUG:boto:path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.520 DEBUG:boto:auth_path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.520 DEBUG:boto:Method: GET 2023-09-29T22:59:56.521 DEBUG:boto:Path: /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.521 DEBUG:boto:Data: 2023-09-29T22:59:56.521 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.521 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.521 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.521 DEBUG:boto:Params: {} 2023-09-29T22:59:56.521 DEBUG:boto:Token: None 2023-09-29T22:59:56.521 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.522 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc= 2023-09-29T22:59:56.522 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc=', 'Content-Length': '0'} 2023-09-29T22:59:56.525 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca7a9ce445bd6615-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.525 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.525 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.526 DEBUG:boto:path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.526 DEBUG:boto:auth_path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.526 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.526 DEBUG:boto:Path: /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.526 DEBUG:boto:Data: 2023-09-29T22:59:56.526 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.526 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.526 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.526 DEBUG:boto:Params: {} 2023-09-29T22:59:56.527 DEBUG:boto:Token: None 2023-09-29T22:59:56.527 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.527 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LzhfnBLN5AwXN4ZbIR6FBcEG9TI= 2023-09-29T22:59:56.527 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LzhfnBLN5AwXN4ZbIR6FBcEG9TI=', 'Content-Length': '0'} 2023-09-29T22:59:56.530 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007658a43311a75160-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.530 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.530 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.530 DEBUG:boto:path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.531 DEBUG:boto:auth_path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.531 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.531 DEBUG:boto:Path: /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.531 DEBUG:boto:Data: 2023-09-29T22:59:56.531 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.531 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.531 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.531 DEBUG:boto:Params: {} 2023-09-29T22:59:56.531 DEBUG:boto:Token: None 2023-09-29T22:59:56.532 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.532 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LzhfnBLN5AwXN4ZbIR6FBcEG9TI= 2023-09-29T22:59:56.532 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LzhfnBLN5AwXN4ZbIR6FBcEG9TI=', 'Content-Length': '0'} 2023-09-29T22:59:56.534 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f85f026869b5fc9b-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:59:56.535 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.535 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.536 DEBUG:boto:path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.536 DEBUG:boto:auth_path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.536 DEBUG:boto:Method: GET 2023-09-29T22:59:56.536 DEBUG:boto:Path: /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.536 DEBUG:boto:Data: 2023-09-29T22:59:56.536 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.536 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.536 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.536 DEBUG:boto:Params: {} 2023-09-29T22:59:56.537 DEBUG:boto:Token: None 2023-09-29T22:59:56.537 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.537 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc= 2023-09-29T22:59:56.537 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc=', 'Content-Length': '0'} 2023-09-29T22:59:56.540 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000208b444e2c364d39-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.540 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.540 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.540 DEBUG:boto:path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.540 DEBUG:boto:auth_path=/hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.540 DEBUG:boto:Method: GET 2023-09-29T22:59:56.540 DEBUG:boto:Path: /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.541 DEBUG:boto:Data: 2023-09-29T22:59:56.541 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.541 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.541 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.541 DEBUG:boto:Params: {} 2023-09-29T22:59:56.541 DEBUG:boto:Token: None 2023-09-29T22:59:56.541 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55?versionId=null 2023-09-29T22:59:56.541 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc= 2023-09-29T22:59:56.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dbf25XxOgnb4vWLQIWsvnRB+Fdc=', 'Content-Length': '0'} 2023-09-29T22:59:56.544 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b552e43754c125b9-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.544 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.544 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.545 DEBUG:boto:path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.545 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.545 DEBUG:boto:Method: GET 2023-09-29T22:59:56.545 DEBUG:boto:Path: /hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.545 DEBUG:boto:Data: 2023-09-29T22:59:56.545 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.545 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.545 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.545 DEBUG:boto:Params: {} 2023-09-29T22:59:56.546 DEBUG:boto:Token: None 2023-09-29T22:59:56.546 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.546 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.546 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.550 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b9c1469e990492b0-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.598 DEBUG:boto:b'hcnphz-61obj551000falseobj552023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.599 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.599 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.599 DEBUG:boto:path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.599 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.600 DEBUG:boto:Method: GET 2023-09-29T22:59:56.600 DEBUG:boto:Path: /hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.600 DEBUG:boto:Data: 2023-09-29T22:59:56.600 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.600 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.600 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.600 DEBUG:boto:Params: {} 2023-09-29T22:59:56.600 DEBUG:boto:Token: None 2023-09-29T22:59:56.600 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.601 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.601 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.611 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f5e8431a6f7dc388-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.611 DEBUG:boto:b'hcnphz-61obj551000falseobj552023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.611 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.611 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.612 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.612 DEBUG:boto:Path: /hcnphz-61/obj55 2023-09-29T22:59:56.612 DEBUG:boto:Data: 2023-09-29T22:59:56.612 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.612 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.612 DEBUG:boto:Params: {} 2023-09-29T22:59:56.612 DEBUG:boto:Token: None 2023-09-29T22:59:56.613 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55 2023-09-29T22:59:56.613 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+4/SnyrTo80LHNBRiiDASIiAGuY= 2023-09-29T22:59:56.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+4/SnyrTo80LHNBRiiDASIiAGuY=', 'Content-Length': '0'} 2023-09-29T22:59:56.615 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c633281ef854d1c0-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.617 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.618 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.618 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.618 DEBUG:boto:Path: /hcnphz-61/obj55 2023-09-29T22:59:56.618 DEBUG:boto:Data: 2023-09-29T22:59:56.618 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.618 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.618 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.618 DEBUG:boto:Params: {} 2023-09-29T22:59:56.619 DEBUG:boto:Token: None 2023-09-29T22:59:56.619 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55 2023-09-29T22:59:56.619 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+4/SnyrTo80LHNBRiiDASIiAGuY= 2023-09-29T22:59:56.619 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+4/SnyrTo80LHNBRiiDASIiAGuY=', 'Content-Length': '0'} 2023-09-29T22:59:56.621 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005cd5bfb636e53413-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T22:59:56.621 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.621 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.622 DEBUG:boto:Method: GET 2023-09-29T22:59:56.622 DEBUG:boto:Path: /hcnphz-61/obj55 2023-09-29T22:59:56.622 DEBUG:boto:Data: 2023-09-29T22:59:56.622 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.622 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.622 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.622 DEBUG:boto:Params: {} 2023-09-29T22:59:56.622 DEBUG:boto:Token: None 2023-09-29T22:59:56.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55 2023-09-29T22:59:56.623 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/czn5Rms9R4lI6IG0T2lJXxNWvk= 2023-09-29T22:59:56.623 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/czn5Rms9R4lI6IG0T2lJXxNWvk=', 'Content-Length': '0'} 2023-09-29T22:59:56.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000831d4b72e415b8a0-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.625 DEBUG:boto:path=/hcnphz-61/obj55 2023-09-29T22:59:56.625 DEBUG:boto:auth_path=/hcnphz-61/obj55 2023-09-29T22:59:56.626 DEBUG:boto:Method: GET 2023-09-29T22:59:56.626 DEBUG:boto:Path: /hcnphz-61/obj55 2023-09-29T22:59:56.626 DEBUG:boto:Data: 2023-09-29T22:59:56.626 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.626 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.626 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.626 DEBUG:boto:Params: {} 2023-09-29T22:59:56.626 DEBUG:boto:Token: None 2023-09-29T22:59:56.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj55 2023-09-29T22:59:56.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/czn5Rms9R4lI6IG0T2lJXxNWvk= 2023-09-29T22:59:56.627 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/czn5Rms9R4lI6IG0T2lJXxNWvk=', 'Content-Length': '0'} 2023-09-29T22:59:56.629 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:12 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b136b512389d29c-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.670 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.671 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.671 DEBUG:boto:path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.671 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.671 DEBUG:boto:Method: GET 2023-09-29T22:59:56.671 DEBUG:boto:Path: /hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.671 DEBUG:boto:Data: 2023-09-29T22:59:56.671 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.671 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.672 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.672 DEBUG:boto:Params: {} 2023-09-29T22:59:56.672 DEBUG:boto:Token: None 2023-09-29T22:59:56.672 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.672 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.672 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.677 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bfea2e804f98614-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.677 DEBUG:boto:b'hcnphz-61obj551000falseobj552023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.677 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.677 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.678 DEBUG:boto:path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.678 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.678 DEBUG:boto:Method: GET 2023-09-29T22:59:56.678 DEBUG:boto:Path: /hcnphz-61/?prefix=obj55 2023-09-29T22:59:56.678 DEBUG:boto:Data: 2023-09-29T22:59:56.678 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.678 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.678 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.678 DEBUG:boto:Params: {} 2023-09-29T22:59:56.679 DEBUG:boto:Token: None 2023-09-29T22:59:56.679 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.679 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.679 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.687 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000269e232315b1bc06-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.687 DEBUG:boto:b'hcnphz-61obj551000falseobj552023-09-29T22:55:12.866Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:59:56.688 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.688 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.688 DEBUG:boto:path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.688 DEBUG:boto:auth_path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.688 DEBUG:boto:Method: GET 2023-09-29T22:59:56.688 DEBUG:boto:Path: /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.689 DEBUG:boto:Data: 2023-09-29T22:59:56.689 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.689 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.689 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.689 DEBUG:boto:Params: {} 2023-09-29T22:59:56.689 DEBUG:boto:Token: None 2023-09-29T22:59:56.689 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.689 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM= 2023-09-29T22:59:56.690 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM=', 'Content-Length': '0'} 2023-09-29T22:59:56.692 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f09c71079b134b9b-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.734 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.734 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.735 DEBUG:boto:path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.735 DEBUG:boto:auth_path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.735 DEBUG:boto:Method: GET 2023-09-29T22:59:56.735 DEBUG:boto:Path: /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.735 DEBUG:boto:Data: 2023-09-29T22:59:56.735 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.735 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.735 DEBUG:boto:Params: {} 2023-09-29T22:59:56.736 DEBUG:boto:Token: None 2023-09-29T22:59:56.736 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM= 2023-09-29T22:59:56.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM=', 'Content-Length': '0'} 2023-09-29T22:59:56.739 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4e336f12ece8f1b-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.739 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.739 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.739 DEBUG:boto:path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.740 DEBUG:boto:auth_path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.740 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.740 DEBUG:boto:Path: /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.740 DEBUG:boto:Data: 2023-09-29T22:59:56.740 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.740 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.740 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.740 DEBUG:boto:Params: {} 2023-09-29T22:59:56.740 DEBUG:boto:Token: None 2023-09-29T22:59:56.741 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.741 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:woxu+dCXU64tsjd1LgoqOJjKn64= 2023-09-29T22:59:56.741 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:woxu+dCXU64tsjd1LgoqOJjKn64=', 'Content-Length': '0'} 2023-09-29T22:59:56.743 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000957f8f02fd1044c1-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.743 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.743 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.744 DEBUG:boto:path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.744 DEBUG:boto:auth_path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.744 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.744 DEBUG:boto:Path: /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.744 DEBUG:boto:Data: 2023-09-29T22:59:56.744 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.744 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.744 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.745 DEBUG:boto:Params: {} 2023-09-29T22:59:56.745 DEBUG:boto:Token: None 2023-09-29T22:59:56.745 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.745 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:woxu+dCXU64tsjd1LgoqOJjKn64= 2023-09-29T22:59:56.745 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:woxu+dCXU64tsjd1LgoqOJjKn64=', 'Content-Length': '0'} 2023-09-29T22:59:56.747 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da2b0d25982faf42-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:59:56.748 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.748 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.748 DEBUG:boto:path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.748 DEBUG:boto:auth_path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.748 DEBUG:boto:Method: GET 2023-09-29T22:59:56.748 DEBUG:boto:Path: /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.748 DEBUG:boto:Data: 2023-09-29T22:59:56.748 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.748 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.749 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.749 DEBUG:boto:Params: {} 2023-09-29T22:59:56.749 DEBUG:boto:Token: None 2023-09-29T22:59:56.749 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.749 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM= 2023-09-29T22:59:56.749 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM=', 'Content-Length': '0'} 2023-09-29T22:59:56.751 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f4f59be87decb1b-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.752 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.752 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.752 DEBUG:boto:path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.752 DEBUG:boto:auth_path=/hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.752 DEBUG:boto:Method: GET 2023-09-29T22:59:56.752 DEBUG:boto:Path: /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.752 DEBUG:boto:Data: 2023-09-29T22:59:56.753 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.753 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.753 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.753 DEBUG:boto:Params: {} 2023-09-29T22:59:56.753 DEBUG:boto:Token: None 2023-09-29T22:59:56.753 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57?versionId=null 2023-09-29T22:59:56.753 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM= 2023-09-29T22:59:56.754 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VAcCCMprg1iT6Aqy27W2fWdhKdM=', 'Content-Length': '0'} 2023-09-29T22:59:56.756 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f90caf807b7d4e3d-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.802 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.803 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.803 DEBUG:boto:path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.803 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.803 DEBUG:boto:Method: GET 2023-09-29T22:59:56.803 DEBUG:boto:Path: /hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.803 DEBUG:boto:Data: 2023-09-29T22:59:56.803 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.803 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.804 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.804 DEBUG:boto:Params: {} 2023-09-29T22:59:56.804 DEBUG:boto:Token: None 2023-09-29T22:59:56.804 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.804 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.804 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.808 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b260cf7c9a2b73a5-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.809 DEBUG:boto:b'hcnphz-61obj571000falseobj572023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.809 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.809 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.809 DEBUG:boto:path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.810 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.810 DEBUG:boto:Method: GET 2023-09-29T22:59:56.810 DEBUG:boto:Path: /hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.810 DEBUG:boto:Data: 2023-09-29T22:59:56.810 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.810 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.810 DEBUG:boto:Params: {} 2023-09-29T22:59:56.810 DEBUG:boto:Token: None 2023-09-29T22:59:56.811 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.818 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000413860fa33ed3fc2-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.818 DEBUG:boto:b'hcnphz-61obj571000falseobj572023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.819 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.819 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.819 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.819 DEBUG:boto:Path: /hcnphz-61/obj57 2023-09-29T22:59:56.819 DEBUG:boto:Data: 2023-09-29T22:59:56.819 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.820 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.820 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.820 DEBUG:boto:Params: {} 2023-09-29T22:59:56.820 DEBUG:boto:Token: None 2023-09-29T22:59:56.820 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57 2023-09-29T22:59:56.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lyRiqCfgsTLGpZndUPC03aOKj4I= 2023-09-29T22:59:56.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lyRiqCfgsTLGpZndUPC03aOKj4I=', 'Content-Length': '0'} 2023-09-29T22:59:56.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006eac1ff5921d973a-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.822 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.822 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.823 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.823 DEBUG:boto:Path: /hcnphz-61/obj57 2023-09-29T22:59:56.823 DEBUG:boto:Data: 2023-09-29T22:59:56.823 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.823 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.823 DEBUG:boto:Params: {} 2023-09-29T22:59:56.823 DEBUG:boto:Token: None 2023-09-29T22:59:56.824 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57 2023-09-29T22:59:56.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lyRiqCfgsTLGpZndUPC03aOKj4I= 2023-09-29T22:59:56.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lyRiqCfgsTLGpZndUPC03aOKj4I=', 'Content-Length': '0'} 2023-09-29T22:59:56.826 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c75cec2a2dbfc9f-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj57 2023-09-29T22:59:56.826 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.826 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.826 DEBUG:boto:Method: GET 2023-09-29T22:59:56.826 DEBUG:boto:Path: /hcnphz-61/obj57 2023-09-29T22:59:56.826 DEBUG:boto:Data: 2023-09-29T22:59:56.827 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.827 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.827 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.827 DEBUG:boto:Params: {} 2023-09-29T22:59:56.827 DEBUG:boto:Token: None 2023-09-29T22:59:56.827 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57 2023-09-29T22:59:56.827 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JfiKuGvPt4k/Vp84H/7TGDQnZik= 2023-09-29T22:59:56.827 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JfiKuGvPt4k/Vp84H/7TGDQnZik=', 'Content-Length': '0'} 2023-09-29T22:59:56.829 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000932f1471a3145fd2-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.870 DEBUG:boto:path=/hcnphz-61/obj57 2023-09-29T22:59:56.870 DEBUG:boto:auth_path=/hcnphz-61/obj57 2023-09-29T22:59:56.871 DEBUG:boto:Method: GET 2023-09-29T22:59:56.871 DEBUG:boto:Path: /hcnphz-61/obj57 2023-09-29T22:59:56.871 DEBUG:boto:Data: 2023-09-29T22:59:56.871 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.871 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.871 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.871 DEBUG:boto:Params: {} 2023-09-29T22:59:56.872 DEBUG:boto:Token: None 2023-09-29T22:59:56.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj57 2023-09-29T22:59:56.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JfiKuGvPt4k/Vp84H/7TGDQnZik= 2023-09-29T22:59:56.872 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JfiKuGvPt4k/Vp84H/7TGDQnZik=', 'Content-Length': '0'} 2023-09-29T22:59:56.875 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000146a8193e718875f-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.876 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.876 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.876 DEBUG:boto:path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.876 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.876 DEBUG:boto:Method: GET 2023-09-29T22:59:56.876 DEBUG:boto:Path: /hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.877 DEBUG:boto:Data: 2023-09-29T22:59:56.877 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.877 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.877 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.877 DEBUG:boto:Params: {} 2023-09-29T22:59:56.877 DEBUG:boto:Token: None 2023-09-29T22:59:56.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.878 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.878 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.882 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019c6ce61af142a6b-00651756ec-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.882 DEBUG:boto:b'hcnphz-61obj571000falseobj572023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.883 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:56.883 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:56.883 DEBUG:boto:path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.883 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.883 DEBUG:boto:Method: GET 2023-09-29T22:59:56.883 DEBUG:boto:Path: /hcnphz-61/?prefix=obj57 2023-09-29T22:59:56.884 DEBUG:boto:Data: 2023-09-29T22:59:56.884 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.884 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.884 DEBUG:boto:Params: {} 2023-09-29T22:59:56.884 DEBUG:boto:Token: None 2023-09-29T22:59:56.884 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/ 2023-09-29T22:59:56.885 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4= 2023-09-29T22:59:56.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c3MA7IOrqR0ekfzI72yQDAJupB4=', 'Content-Length': '0'} 2023-09-29T22:59:56.893 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ddf5232f19e0ee64-00651756ec-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.938 DEBUG:boto:b'hcnphz-61obj571000falseobj572023-09-29T22:55:13.602Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:56.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:59:56.939 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:56.939 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:56.939 DEBUG:boto:path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.939 DEBUG:boto:auth_path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.939 DEBUG:boto:Method: GET 2023-09-29T22:59:56.940 DEBUG:boto:Path: /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.940 DEBUG:boto:Data: 2023-09-29T22:59:56.940 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.940 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.940 DEBUG:boto:Params: {} 2023-09-29T22:59:56.940 DEBUG:boto:Token: None 2023-09-29T22:59:56.940 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4= 2023-09-29T22:59:56.941 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4=', 'Content-Length': '0'} 2023-09-29T22:59:56.943 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000badc443dba9a40c8-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.944 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:56.944 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:56.944 DEBUG:boto:path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.944 DEBUG:boto:auth_path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.944 DEBUG:boto:Method: GET 2023-09-29T22:59:56.944 DEBUG:boto:Path: /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.945 DEBUG:boto:Data: 2023-09-29T22:59:56.945 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.945 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.945 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.945 DEBUG:boto:Params: {} 2023-09-29T22:59:56.945 DEBUG:boto:Token: None 2023-09-29T22:59:56.945 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.945 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4= 2023-09-29T22:59:56.946 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4=', 'Content-Length': '0'} 2023-09-29T22:59:56.948 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007eae30674120be9a-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.948 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:56.949 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:56.949 DEBUG:boto:path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.949 DEBUG:boto:auth_path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.949 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.949 DEBUG:boto:Path: /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.949 DEBUG:boto:Data: 2023-09-29T22:59:56.949 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.949 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.949 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.949 DEBUG:boto:Params: {} 2023-09-29T22:59:56.950 DEBUG:boto:Token: None 2023-09-29T22:59:56.950 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.950 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0lwBbyhTZ1ATzAu+9wyjKLUbmPE= 2023-09-29T22:59:56.950 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0lwBbyhTZ1ATzAu+9wyjKLUbmPE=', 'Content-Length': '0'} 2023-09-29T22:59:56.953 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4896a2e81114350-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.953 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:56.954 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:56.954 DEBUG:boto:path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.954 DEBUG:boto:auth_path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.954 DEBUG:boto:Method: HEAD 2023-09-29T22:59:56.954 DEBUG:boto:Path: /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.954 DEBUG:boto:Data: 2023-09-29T22:59:56.954 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.954 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.955 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.955 DEBUG:boto:Params: {} 2023-09-29T22:59:56.955 DEBUG:boto:Token: None 2023-09-29T22:59:56.956 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0lwBbyhTZ1ATzAu+9wyjKLUbmPE= 2023-09-29T22:59:56.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0lwBbyhTZ1ATzAu+9wyjKLUbmPE=', 'Content-Length': '0'} 2023-09-29T22:59:56.959 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0a8c67dce5e4ac2-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:59:56.959 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:56.960 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:56.960 DEBUG:boto:path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.960 DEBUG:boto:auth_path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.960 DEBUG:boto:Method: GET 2023-09-29T22:59:56.960 DEBUG:boto:Path: /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.960 DEBUG:boto:Data: 2023-09-29T22:59:56.960 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:56.960 DEBUG:boto:Port: 8001 2023-09-29T22:59:56.961 DEBUG:boto:Params: {} 2023-09-29T22:59:56.961 DEBUG:boto:Token: None 2023-09-29T22:59:56.961 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4= 2023-09-29T22:59:56.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4=', 'Content-Length': '0'} 2023-09-29T22:59:56.964 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000093003b039c46bf2-00651756ec-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:56.964 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:56.965 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:56.965 DEBUG:boto:path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.965 DEBUG:boto:auth_path=/hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.965 DEBUG:boto:Method: GET 2023-09-29T22:59:56.965 DEBUG:boto:Path: /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.965 DEBUG:boto:Data: 2023-09-29T22:59:56.965 DEBUG:boto:Headers: {} 2023-09-29T22:59:56.965 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:56.966 DEBUG:boto:Port: 8000 2023-09-29T22:59:56.966 DEBUG:boto:Params: {} 2023-09-29T22:59:56.966 DEBUG:boto:Token: None 2023-09-29T22:59:56.966 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:56 GMT /hcnphz-61/obj58?versionId=null 2023-09-29T22:59:56.966 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4= 2023-09-29T22:59:56.966 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:56 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7VSq5NvKE0BQKh0/OI135a6Sub4=', 'Content-Length': '0'} 2023-09-29T22:59:56.969 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015b9c4705738b1f4-00651756ec-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:56 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.014 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.015 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.015 DEBUG:boto:path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.015 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.015 DEBUG:boto:Method: GET 2023-09-29T22:59:57.015 DEBUG:boto:Path: /hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.015 DEBUG:boto:Data: 2023-09-29T22:59:57.015 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.015 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.016 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.016 DEBUG:boto:Params: {} 2023-09-29T22:59:57.016 DEBUG:boto:Token: None 2023-09-29T22:59:57.016 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.016 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.016 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005e922c26a2da1fe0-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.021 DEBUG:boto:b'hcnphz-61obj581000falseobj582023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.022 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.022 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.022 DEBUG:boto:path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.022 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.022 DEBUG:boto:Method: GET 2023-09-29T22:59:57.022 DEBUG:boto:Path: /hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.022 DEBUG:boto:Data: 2023-09-29T22:59:57.022 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.023 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.023 DEBUG:boto:Params: {} 2023-09-29T22:59:57.023 DEBUG:boto:Token: None 2023-09-29T22:59:57.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.030 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba4652c5d4b20b7e-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.030 DEBUG:boto:b'hcnphz-61obj581000falseobj582023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.031 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:57.031 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:57.031 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.031 DEBUG:boto:Path: /hcnphz-61/obj58 2023-09-29T22:59:57.031 DEBUG:boto:Data: 2023-09-29T22:59:57.031 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.032 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.032 DEBUG:boto:Params: {} 2023-09-29T22:59:57.032 DEBUG:boto:Token: None 2023-09-29T22:59:57.032 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj58 2023-09-29T22:59:57.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xoyv64hxoCfIt4t3U1tdA0qNLmo= 2023-09-29T22:59:57.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xoyv64hxoCfIt4t3U1tdA0qNLmo=', 'Content-Length': '0'} 2023-09-29T22:59:57.034 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095d574600a4328fb-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.035 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:57.035 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:57.035 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.035 DEBUG:boto:Path: /hcnphz-61/obj58 2023-09-29T22:59:57.035 DEBUG:boto:Data: 2023-09-29T22:59:57.035 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.035 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.036 DEBUG:boto:Params: {} 2023-09-29T22:59:57.036 DEBUG:boto:Token: None 2023-09-29T22:59:57.036 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj58 2023-09-29T22:59:57.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xoyv64hxoCfIt4t3U1tdA0qNLmo= 2023-09-29T22:59:57.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xoyv64hxoCfIt4t3U1tdA0qNLmo=', 'Content-Length': '0'} 2023-09-29T22:59:57.038 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f86db9fd75e84cbc-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj58 2023-09-29T22:59:57.038 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:57.038 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:57.039 DEBUG:boto:Method: GET 2023-09-29T22:59:57.039 DEBUG:boto:Path: /hcnphz-61/obj58 2023-09-29T22:59:57.039 DEBUG:boto:Data: 2023-09-29T22:59:57.039 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.039 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.039 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.039 DEBUG:boto:Params: {} 2023-09-29T22:59:57.039 DEBUG:boto:Token: None 2023-09-29T22:59:57.039 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj58 2023-09-29T22:59:57.040 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wx8OkGN8NdWh8OnG4/oSMWA/cs0= 2023-09-29T22:59:57.040 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wx8OkGN8NdWh8OnG4/oSMWA/cs0=', 'Content-Length': '0'} 2023-09-29T22:59:57.042 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b996f469e4b7b032-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.082 DEBUG:boto:path=/hcnphz-61/obj58 2023-09-29T22:59:57.083 DEBUG:boto:auth_path=/hcnphz-61/obj58 2023-09-29T22:59:57.083 DEBUG:boto:Method: GET 2023-09-29T22:59:57.083 DEBUG:boto:Path: /hcnphz-61/obj58 2023-09-29T22:59:57.083 DEBUG:boto:Data: 2023-09-29T22:59:57.083 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.083 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.083 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.084 DEBUG:boto:Params: {} 2023-09-29T22:59:57.084 DEBUG:boto:Token: None 2023-09-29T22:59:57.084 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj58 2023-09-29T22:59:57.084 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Wx8OkGN8NdWh8OnG4/oSMWA/cs0= 2023-09-29T22:59:57.084 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Wx8OkGN8NdWh8OnG4/oSMWA/cs0=', 'Content-Length': '0'} 2023-09-29T22:59:57.087 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000468862202da40c31-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.088 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.088 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.088 DEBUG:boto:path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.088 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.088 DEBUG:boto:Method: GET 2023-09-29T22:59:57.088 DEBUG:boto:Path: /hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.088 DEBUG:boto:Data: 2023-09-29T22:59:57.089 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.089 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.089 DEBUG:boto:Params: {} 2023-09-29T22:59:57.089 DEBUG:boto:Token: None 2023-09-29T22:59:57.090 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.090 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.095 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ffdc2ec1a1bcd84-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.095 DEBUG:boto:b'hcnphz-61obj581000falseobj582023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.097 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.097 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.097 DEBUG:boto:path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.097 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.097 DEBUG:boto:Method: GET 2023-09-29T22:59:57.097 DEBUG:boto:Path: /hcnphz-61/?prefix=obj58 2023-09-29T22:59:57.097 DEBUG:boto:Data: 2023-09-29T22:59:57.098 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.098 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.098 DEBUG:boto:Params: {} 2023-09-29T22:59:57.098 DEBUG:boto:Token: None 2023-09-29T22:59:57.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.106 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000718577a71cd0c7f5-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.150 DEBUG:boto:b'hcnphz-61obj581000falseobj582023-09-29T22:55:14.439Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:59:57.151 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.151 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.151 DEBUG:boto:path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.151 DEBUG:boto:auth_path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.151 DEBUG:boto:Method: GET 2023-09-29T22:59:57.152 DEBUG:boto:Path: /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.152 DEBUG:boto:Data: 2023-09-29T22:59:57.152 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.152 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.152 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.152 DEBUG:boto:Params: {} 2023-09-29T22:59:57.152 DEBUG:boto:Token: None 2023-09-29T22:59:57.152 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4= 2023-09-29T22:59:57.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4=', 'Content-Length': '0'} 2023-09-29T22:59:57.156 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0c5fe83ea7be0ab-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.157 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.157 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.157 DEBUG:boto:path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.157 DEBUG:boto:auth_path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.157 DEBUG:boto:Method: GET 2023-09-29T22:59:57.157 DEBUG:boto:Path: /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.157 DEBUG:boto:Data: 2023-09-29T22:59:57.157 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.158 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.158 DEBUG:boto:Params: {} 2023-09-29T22:59:57.158 DEBUG:boto:Token: None 2023-09-29T22:59:57.158 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4= 2023-09-29T22:59:57.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4=', 'Content-Length': '0'} 2023-09-29T22:59:57.161 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003082532a363b3d31-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.161 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.161 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.162 DEBUG:boto:path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.162 DEBUG:boto:auth_path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.162 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.162 DEBUG:boto:Path: /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.162 DEBUG:boto:Data: 2023-09-29T22:59:57.162 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.162 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.162 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.162 DEBUG:boto:Params: {} 2023-09-29T22:59:57.163 DEBUG:boto:Token: None 2023-09-29T22:59:57.163 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.163 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W/Xs2f9pFS6Ne7uaMjIYF9pxSnc= 2023-09-29T22:59:57.163 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W/Xs2f9pFS6Ne7uaMjIYF9pxSnc=', 'Content-Length': '0'} 2023-09-29T22:59:57.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000abf26ec9c2c7e04e-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.166 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.166 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.166 DEBUG:boto:path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.166 DEBUG:boto:auth_path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.166 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.166 DEBUG:boto:Path: /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.167 DEBUG:boto:Data: 2023-09-29T22:59:57.167 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.167 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.167 DEBUG:boto:Params: {} 2023-09-29T22:59:57.167 DEBUG:boto:Token: None 2023-09-29T22:59:57.167 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.167 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W/Xs2f9pFS6Ne7uaMjIYF9pxSnc= 2023-09-29T22:59:57.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W/Xs2f9pFS6Ne7uaMjIYF9pxSnc=', 'Content-Length': '0'} 2023-09-29T22:59:57.169 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8e5186c2dbc664d-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:59:57.170 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.170 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.170 DEBUG:boto:path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.170 DEBUG:boto:auth_path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.170 DEBUG:boto:Method: GET 2023-09-29T22:59:57.170 DEBUG:boto:Path: /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.171 DEBUG:boto:Data: 2023-09-29T22:59:57.171 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.171 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.171 DEBUG:boto:Params: {} 2023-09-29T22:59:57.171 DEBUG:boto:Token: None 2023-09-29T22:59:57.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4= 2023-09-29T22:59:57.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4=', 'Content-Length': '0'} 2023-09-29T22:59:57.174 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000069f08a89b66e1d27-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.174 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.174 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.174 DEBUG:boto:path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.174 DEBUG:boto:auth_path=/hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.175 DEBUG:boto:Method: GET 2023-09-29T22:59:57.175 DEBUG:boto:Path: /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.175 DEBUG:boto:Data: 2023-09-29T22:59:57.175 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.175 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.175 DEBUG:boto:Params: {} 2023-09-29T22:59:57.175 DEBUG:boto:Token: None 2023-09-29T22:59:57.175 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62?versionId=null 2023-09-29T22:59:57.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4= 2023-09-29T22:59:57.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGeDem+grYYVj9KZ9IHp7BlG2/4=', 'Content-Length': '0'} 2023-09-29T22:59:57.178 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000095b0f56ed2c1242c-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.178 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.178 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.179 DEBUG:boto:path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.179 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.179 DEBUG:boto:Method: GET 2023-09-29T22:59:57.179 DEBUG:boto:Path: /hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.179 DEBUG:boto:Data: 2023-09-29T22:59:57.179 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.179 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.179 DEBUG:boto:Params: {} 2023-09-29T22:59:57.180 DEBUG:boto:Token: None 2023-09-29T22:59:57.180 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c0ded40ec6f00e03-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.184 DEBUG:boto:b'hcnphz-61obj621000falseobj622023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.184 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.184 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.184 DEBUG:boto:path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.185 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.185 DEBUG:boto:Method: GET 2023-09-29T22:59:57.185 DEBUG:boto:Path: /hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.185 DEBUG:boto:Data: 2023-09-29T22:59:57.185 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.185 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.185 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.185 DEBUG:boto:Params: {} 2023-09-29T22:59:57.185 DEBUG:boto:Token: None 2023-09-29T22:59:57.186 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.186 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.192 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36a243ccd3cc7eb-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.193 DEBUG:boto:b'hcnphz-61obj621000falseobj622023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.193 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.193 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.193 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.193 DEBUG:boto:Path: /hcnphz-61/obj62 2023-09-29T22:59:57.194 DEBUG:boto:Data: 2023-09-29T22:59:57.194 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.194 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.194 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.194 DEBUG:boto:Params: {} 2023-09-29T22:59:57.194 DEBUG:boto:Token: None 2023-09-29T22:59:57.194 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62 2023-09-29T22:59:57.194 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LqERUWsK99LAKQnjgCF9g79qKoo= 2023-09-29T22:59:57.195 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LqERUWsK99LAKQnjgCF9g79qKoo=', 'Content-Length': '0'} 2023-09-29T22:59:57.196 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a1c96560d33d011-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.197 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.197 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.197 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.197 DEBUG:boto:Path: /hcnphz-61/obj62 2023-09-29T22:59:57.197 DEBUG:boto:Data: 2023-09-29T22:59:57.197 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.197 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.197 DEBUG:boto:Params: {} 2023-09-29T22:59:57.198 DEBUG:boto:Token: None 2023-09-29T22:59:57.198 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62 2023-09-29T22:59:57.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LqERUWsK99LAKQnjgCF9g79qKoo= 2023-09-29T22:59:57.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LqERUWsK99LAKQnjgCF9g79qKoo=', 'Content-Length': '0'} 2023-09-29T22:59:57.200 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aaa455496b44fa3-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj62 2023-09-29T22:59:57.201 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.201 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.201 DEBUG:boto:Method: GET 2023-09-29T22:59:57.201 DEBUG:boto:Path: /hcnphz-61/obj62 2023-09-29T22:59:57.201 DEBUG:boto:Data: 2023-09-29T22:59:57.201 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.202 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.202 DEBUG:boto:Params: {} 2023-09-29T22:59:57.202 DEBUG:boto:Token: None 2023-09-29T22:59:57.202 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62 2023-09-29T22:59:57.202 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3hd4MWWxb3r8mLirEQWnjyZDMC4= 2023-09-29T22:59:57.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3hd4MWWxb3r8mLirEQWnjyZDMC4=', 'Content-Length': '0'} 2023-09-29T22:59:57.204 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08bdf623bc10e51-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.251 DEBUG:boto:path=/hcnphz-61/obj62 2023-09-29T22:59:57.251 DEBUG:boto:auth_path=/hcnphz-61/obj62 2023-09-29T22:59:57.251 DEBUG:boto:Method: GET 2023-09-29T22:59:57.251 DEBUG:boto:Path: /hcnphz-61/obj62 2023-09-29T22:59:57.251 DEBUG:boto:Data: 2023-09-29T22:59:57.252 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.252 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.252 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.252 DEBUG:boto:Params: {} 2023-09-29T22:59:57.252 DEBUG:boto:Token: None 2023-09-29T22:59:57.252 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj62 2023-09-29T22:59:57.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3hd4MWWxb3r8mLirEQWnjyZDMC4= 2023-09-29T22:59:57.253 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3hd4MWWxb3r8mLirEQWnjyZDMC4=', 'Content-Length': '0'} 2023-09-29T22:59:57.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002c5d7793a735bbf0-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.256 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.256 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.256 DEBUG:boto:path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.256 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.256 DEBUG:boto:Method: GET 2023-09-29T22:59:57.256 DEBUG:boto:Path: /hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.257 DEBUG:boto:Data: 2023-09-29T22:59:57.257 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.257 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.257 DEBUG:boto:Params: {} 2023-09-29T22:59:57.257 DEBUG:boto:Token: None 2023-09-29T22:59:57.257 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.257 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.262 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000024832de3ae675702-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.262 DEBUG:boto:b'hcnphz-61obj621000falseobj622023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.262 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.263 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.263 DEBUG:boto:path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.263 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.263 DEBUG:boto:Method: GET 2023-09-29T22:59:57.263 DEBUG:boto:Path: /hcnphz-61/?prefix=obj62 2023-09-29T22:59:57.263 DEBUG:boto:Data: 2023-09-29T22:59:57.263 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.263 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.263 DEBUG:boto:Params: {} 2023-09-29T22:59:57.264 DEBUG:boto:Token: None 2023-09-29T22:59:57.264 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.264 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000214bf45cfe00df55-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.318 DEBUG:boto:b'hcnphz-61obj621000falseobj622023-09-29T22:55:14.880Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:59:57.319 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.319 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.319 DEBUG:boto:path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.319 DEBUG:boto:auth_path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.320 DEBUG:boto:Method: GET 2023-09-29T22:59:57.320 DEBUG:boto:Path: /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.320 DEBUG:boto:Data: 2023-09-29T22:59:57.320 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.320 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.320 DEBUG:boto:Params: {} 2023-09-29T22:59:57.320 DEBUG:boto:Token: None 2023-09-29T22:59:57.321 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.321 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc= 2023-09-29T22:59:57.321 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc=', 'Content-Length': '0'} 2023-09-29T22:59:57.324 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006034c5dd5082f3ce-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.325 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.325 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.325 DEBUG:boto:path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.325 DEBUG:boto:auth_path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.325 DEBUG:boto:Method: GET 2023-09-29T22:59:57.326 DEBUG:boto:Path: /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.326 DEBUG:boto:Data: 2023-09-29T22:59:57.326 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.326 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.327 DEBUG:boto:Params: {} 2023-09-29T22:59:57.327 DEBUG:boto:Token: None 2023-09-29T22:59:57.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc= 2023-09-29T22:59:57.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc=', 'Content-Length': '0'} 2023-09-29T22:59:57.331 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df9d8de8432c7491-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.331 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.331 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.331 DEBUG:boto:path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.332 DEBUG:boto:auth_path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.332 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.332 DEBUG:boto:Path: /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.332 DEBUG:boto:Data: 2023-09-29T22:59:57.332 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.332 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.333 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.333 DEBUG:boto:Params: {} 2023-09-29T22:59:57.333 DEBUG:boto:Token: None 2023-09-29T22:59:57.334 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BC++YIyDoZXAnlrchB6YSM+3qyk= 2023-09-29T22:59:57.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BC++YIyDoZXAnlrchB6YSM+3qyk=', 'Content-Length': '0'} 2023-09-29T22:59:57.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec718bbc73579ea7-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.337 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.338 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.338 DEBUG:boto:path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.338 DEBUG:boto:auth_path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.338 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.338 DEBUG:boto:Path: /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.338 DEBUG:boto:Data: 2023-09-29T22:59:57.338 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.339 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.339 DEBUG:boto:Params: {} 2023-09-29T22:59:57.339 DEBUG:boto:Token: None 2023-09-29T22:59:57.339 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.339 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BC++YIyDoZXAnlrchB6YSM+3qyk= 2023-09-29T22:59:57.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BC++YIyDoZXAnlrchB6YSM+3qyk=', 'Content-Length': '0'} 2023-09-29T22:59:57.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000029d4548ebea60ae4-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:59:57.343 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.343 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.343 DEBUG:boto:path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.343 DEBUG:boto:auth_path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.344 DEBUG:boto:Method: GET 2023-09-29T22:59:57.344 DEBUG:boto:Path: /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.344 DEBUG:boto:Data: 2023-09-29T22:59:57.344 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.344 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.344 DEBUG:boto:Params: {} 2023-09-29T22:59:57.345 DEBUG:boto:Token: None 2023-09-29T22:59:57.345 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc= 2023-09-29T22:59:57.346 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc=', 'Content-Length': '0'} 2023-09-29T22:59:57.349 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7b3bed934a1930e-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.349 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.349 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.349 DEBUG:boto:path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.349 DEBUG:boto:auth_path=/hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.350 DEBUG:boto:Method: GET 2023-09-29T22:59:57.350 DEBUG:boto:Path: /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.350 DEBUG:boto:Data: 2023-09-29T22:59:57.350 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.350 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.350 DEBUG:boto:Params: {} 2023-09-29T22:59:57.350 DEBUG:boto:Token: None 2023-09-29T22:59:57.351 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68?versionId=null 2023-09-29T22:59:57.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc= 2023-09-29T22:59:57.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sdzedTnTryQLt+B4tqCekBp9wZc=', 'Content-Length': '0'} 2023-09-29T22:59:57.354 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004bc23beecad1b2be-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.354 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.354 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.354 DEBUG:boto:path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.354 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.355 DEBUG:boto:Method: GET 2023-09-29T22:59:57.355 DEBUG:boto:Path: /hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.355 DEBUG:boto:Data: 2023-09-29T22:59:57.356 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.356 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.356 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.356 DEBUG:boto:Params: {} 2023-09-29T22:59:57.356 DEBUG:boto:Token: None 2023-09-29T22:59:57.356 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.360 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045503d349884dc5e-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.361 DEBUG:boto:b'hcnphz-61obj681000falseobj682023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.361 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.361 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.361 DEBUG:boto:path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.362 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.362 DEBUG:boto:Method: GET 2023-09-29T22:59:57.362 DEBUG:boto:Path: /hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.362 DEBUG:boto:Data: 2023-09-29T22:59:57.362 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.362 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.362 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.362 DEBUG:boto:Params: {} 2023-09-29T22:59:57.362 DEBUG:boto:Token: None 2023-09-29T22:59:57.363 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.363 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.363 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.371 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c29f3cf9be3ea4bd-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.371 DEBUG:boto:b'hcnphz-61obj681000falseobj682023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.371 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.372 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.372 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.372 DEBUG:boto:Path: /hcnphz-61/obj68 2023-09-29T22:59:57.372 DEBUG:boto:Data: 2023-09-29T22:59:57.372 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.372 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.372 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.372 DEBUG:boto:Params: {} 2023-09-29T22:59:57.373 DEBUG:boto:Token: None 2023-09-29T22:59:57.373 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68 2023-09-29T22:59:57.373 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/ltgJGCKD3i5w2oVjNdTff4BQeA= 2023-09-29T22:59:57.373 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/ltgJGCKD3i5w2oVjNdTff4BQeA=', 'Content-Length': '0'} 2023-09-29T22:59:57.375 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004fe187d0ffb28df2-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.375 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.375 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.375 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.376 DEBUG:boto:Path: /hcnphz-61/obj68 2023-09-29T22:59:57.376 DEBUG:boto:Data: 2023-09-29T22:59:57.376 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.376 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.377 DEBUG:boto:Params: {} 2023-09-29T22:59:57.377 DEBUG:boto:Token: None 2023-09-29T22:59:57.377 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68 2023-09-29T22:59:57.377 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/ltgJGCKD3i5w2oVjNdTff4BQeA= 2023-09-29T22:59:57.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/ltgJGCKD3i5w2oVjNdTff4BQeA=', 'Content-Length': '0'} 2023-09-29T22:59:57.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002028e91b3b1cd716-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj68 2023-09-29T22:59:57.380 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.380 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.380 DEBUG:boto:Method: GET 2023-09-29T22:59:57.380 DEBUG:boto:Path: /hcnphz-61/obj68 2023-09-29T22:59:57.380 DEBUG:boto:Data: 2023-09-29T22:59:57.380 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.380 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.380 DEBUG:boto:Params: {} 2023-09-29T22:59:57.381 DEBUG:boto:Token: None 2023-09-29T22:59:57.381 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68 2023-09-29T22:59:57.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SshdX4xPC8IogKBXtX39TM0TOOg= 2023-09-29T22:59:57.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SshdX4xPC8IogKBXtX39TM0TOOg=', 'Content-Length': '0'} 2023-09-29T22:59:57.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf153a4698f7feb7-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.430 DEBUG:boto:path=/hcnphz-61/obj68 2023-09-29T22:59:57.431 DEBUG:boto:auth_path=/hcnphz-61/obj68 2023-09-29T22:59:57.433 DEBUG:boto:Method: GET 2023-09-29T22:59:57.434 DEBUG:boto:Path: /hcnphz-61/obj68 2023-09-29T22:59:57.434 DEBUG:boto:Data: 2023-09-29T22:59:57.434 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.434 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.435 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.435 DEBUG:boto:Params: {} 2023-09-29T22:59:57.435 DEBUG:boto:Token: None 2023-09-29T22:59:57.435 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj68 2023-09-29T22:59:57.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SshdX4xPC8IogKBXtX39TM0TOOg= 2023-09-29T22:59:57.436 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SshdX4xPC8IogKBXtX39TM0TOOg=', 'Content-Length': '0'} 2023-09-29T22:59:57.439 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092f53478f4bd7842-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.440 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.440 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.440 DEBUG:boto:path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.441 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.441 DEBUG:boto:Method: GET 2023-09-29T22:59:57.441 DEBUG:boto:Path: /hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.441 DEBUG:boto:Data: 2023-09-29T22:59:57.442 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.442 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.442 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.442 DEBUG:boto:Params: {} 2023-09-29T22:59:57.443 DEBUG:boto:Token: None 2023-09-29T22:59:57.443 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.443 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000369e4d8ccb724d14-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.449 DEBUG:boto:b'hcnphz-61obj681000falseobj682023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.450 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.450 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.450 DEBUG:boto:path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.451 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.451 DEBUG:boto:Method: GET 2023-09-29T22:59:57.451 DEBUG:boto:Path: /hcnphz-61/?prefix=obj68 2023-09-29T22:59:57.451 DEBUG:boto:Data: 2023-09-29T22:59:57.452 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.452 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.452 DEBUG:boto:Params: {} 2023-09-29T22:59:57.453 DEBUG:boto:Token: None 2023-09-29T22:59:57.453 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.454 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.462 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000eaf8eb644c6bfa0f-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.462 DEBUG:boto:b'hcnphz-61obj681000falseobj682023-09-29T22:55:13.218Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:59:57.463 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.463 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.463 DEBUG:boto:path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.463 DEBUG:boto:auth_path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.464 DEBUG:boto:Method: GET 2023-09-29T22:59:57.464 DEBUG:boto:Path: /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.464 DEBUG:boto:Data: 2023-09-29T22:59:57.464 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.465 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.465 DEBUG:boto:Params: {} 2023-09-29T22:59:57.465 DEBUG:boto:Token: None 2023-09-29T22:59:57.465 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.466 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk= 2023-09-29T22:59:57.466 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk=', 'Content-Length': '0'} 2023-09-29T22:59:57.469 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005525a0e75e38575a-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.470 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.470 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.471 DEBUG:boto:path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.471 DEBUG:boto:auth_path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.471 DEBUG:boto:Method: GET 2023-09-29T22:59:57.471 DEBUG:boto:Path: /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.471 DEBUG:boto:Data: 2023-09-29T22:59:57.471 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.472 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.472 DEBUG:boto:Params: {} 2023-09-29T22:59:57.472 DEBUG:boto:Token: None 2023-09-29T22:59:57.472 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk= 2023-09-29T22:59:57.473 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk=', 'Content-Length': '0'} 2023-09-29T22:59:57.475 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea095dcd71b90819-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.475 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.475 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.475 DEBUG:boto:path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.476 DEBUG:boto:auth_path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.476 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.476 DEBUG:boto:Path: /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.476 DEBUG:boto:Data: 2023-09-29T22:59:57.477 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.477 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.477 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.477 DEBUG:boto:Params: {} 2023-09-29T22:59:57.477 DEBUG:boto:Token: None 2023-09-29T22:59:57.478 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6f+50rmnWxOxrbUiyFu7BLHIGnM= 2023-09-29T22:59:57.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6f+50rmnWxOxrbUiyFu7BLHIGnM=', 'Content-Length': '0'} 2023-09-29T22:59:57.483 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dd9b22fff1c491cc-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.483 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.483 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.483 DEBUG:boto:path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.483 DEBUG:boto:auth_path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.484 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.484 DEBUG:boto:Path: /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.484 DEBUG:boto:Data: 2023-09-29T22:59:57.484 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.484 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.484 DEBUG:boto:Params: {} 2023-09-29T22:59:57.484 DEBUG:boto:Token: None 2023-09-29T22:59:57.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.485 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6f+50rmnWxOxrbUiyFu7BLHIGnM= 2023-09-29T22:59:57.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6f+50rmnWxOxrbUiyFu7BLHIGnM=', 'Content-Length': '0'} 2023-09-29T22:59:57.487 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f738a26c71804895-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:59:57.488 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.488 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.488 DEBUG:boto:path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.488 DEBUG:boto:auth_path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.488 DEBUG:boto:Method: GET 2023-09-29T22:59:57.488 DEBUG:boto:Path: /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.488 DEBUG:boto:Data: 2023-09-29T22:59:57.488 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.489 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.489 DEBUG:boto:Params: {} 2023-09-29T22:59:57.489 DEBUG:boto:Token: None 2023-09-29T22:59:57.489 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk= 2023-09-29T22:59:57.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk=', 'Content-Length': '0'} 2023-09-29T22:59:57.492 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a96c3a9e5b4ac84a-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.493 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.493 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.493 DEBUG:boto:path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.493 DEBUG:boto:auth_path=/hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.493 DEBUG:boto:Method: GET 2023-09-29T22:59:57.493 DEBUG:boto:Path: /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.493 DEBUG:boto:Data: 2023-09-29T22:59:57.494 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.494 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.494 DEBUG:boto:Params: {} 2023-09-29T22:59:57.494 DEBUG:boto:Token: None 2023-09-29T22:59:57.494 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69?versionId=null 2023-09-29T22:59:57.494 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk= 2023-09-29T22:59:57.494 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tU6MCqtAQLnDl9luxe+cQJ0gmnk=', 'Content-Length': '0'} 2023-09-29T22:59:57.497 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009276b58b95393e99-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.542 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.543 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.543 DEBUG:boto:path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.543 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.543 DEBUG:boto:Method: GET 2023-09-29T22:59:57.543 DEBUG:boto:Path: /hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.543 DEBUG:boto:Data: 2023-09-29T22:59:57.543 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.543 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.544 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.544 DEBUG:boto:Params: {} 2023-09-29T22:59:57.544 DEBUG:boto:Token: None 2023-09-29T22:59:57.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.544 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.544 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.549 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000d061d5264c6005b-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.549 DEBUG:boto:b'hcnphz-61obj691000falseobj692023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.550 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.550 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.550 DEBUG:boto:path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.550 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.550 DEBUG:boto:Method: GET 2023-09-29T22:59:57.551 DEBUG:boto:Path: /hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.551 DEBUG:boto:Data: 2023-09-29T22:59:57.551 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.551 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.551 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.551 DEBUG:boto:Params: {} 2023-09-29T22:59:57.551 DEBUG:boto:Token: None 2023-09-29T22:59:57.551 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.552 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.552 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.559 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008ae763fe8f107d16-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.559 DEBUG:boto:b'hcnphz-61obj691000falseobj692023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.560 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.560 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.560 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.560 DEBUG:boto:Path: /hcnphz-61/obj69 2023-09-29T22:59:57.560 DEBUG:boto:Data: 2023-09-29T22:59:57.560 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.560 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.561 DEBUG:boto:Params: {} 2023-09-29T22:59:57.561 DEBUG:boto:Token: None 2023-09-29T22:59:57.561 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69 2023-09-29T22:59:57.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LR5+ZUqYBpBI8+VDVYKHGJsQPrY= 2023-09-29T22:59:57.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LR5+ZUqYBpBI8+VDVYKHGJsQPrY=', 'Content-Length': '0'} 2023-09-29T22:59:57.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d515ea4fcf62731-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.563 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.564 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.564 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.564 DEBUG:boto:Path: /hcnphz-61/obj69 2023-09-29T22:59:57.564 DEBUG:boto:Data: 2023-09-29T22:59:57.564 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.564 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.564 DEBUG:boto:Params: {} 2023-09-29T22:59:57.565 DEBUG:boto:Token: None 2023-09-29T22:59:57.565 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69 2023-09-29T22:59:57.565 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LR5+ZUqYBpBI8+VDVYKHGJsQPrY= 2023-09-29T22:59:57.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LR5+ZUqYBpBI8+VDVYKHGJsQPrY=', 'Content-Length': '0'} 2023-09-29T22:59:57.567 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002f8f8a3eb33ca5a5-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj69 2023-09-29T22:59:57.568 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.568 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.568 DEBUG:boto:Method: GET 2023-09-29T22:59:57.568 DEBUG:boto:Path: /hcnphz-61/obj69 2023-09-29T22:59:57.569 DEBUG:boto:Data: 2023-09-29T22:59:57.569 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.569 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.569 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.569 DEBUG:boto:Params: {} 2023-09-29T22:59:57.570 DEBUG:boto:Token: None 2023-09-29T22:59:57.570 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69 2023-09-29T22:59:57.570 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X+13Q6HZ4tZ/4vwIhRAEJiw5odw= 2023-09-29T22:59:57.570 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X+13Q6HZ4tZ/4vwIhRAEJiw5odw=', 'Content-Length': '0'} 2023-09-29T22:59:57.572 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025ba1f60a5dc1d63-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.614 DEBUG:boto:path=/hcnphz-61/obj69 2023-09-29T22:59:57.615 DEBUG:boto:auth_path=/hcnphz-61/obj69 2023-09-29T22:59:57.615 DEBUG:boto:Method: GET 2023-09-29T22:59:57.615 DEBUG:boto:Path: /hcnphz-61/obj69 2023-09-29T22:59:57.615 DEBUG:boto:Data: 2023-09-29T22:59:57.615 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.616 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.616 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.616 DEBUG:boto:Params: {} 2023-09-29T22:59:57.616 DEBUG:boto:Token: None 2023-09-29T22:59:57.616 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj69 2023-09-29T22:59:57.616 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X+13Q6HZ4tZ/4vwIhRAEJiw5odw= 2023-09-29T22:59:57.616 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X+13Q6HZ4tZ/4vwIhRAEJiw5odw=', 'Content-Length': '0'} 2023-09-29T22:59:57.619 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000f8452d6b2a40711-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.619 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.619 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.620 DEBUG:boto:path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.620 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.620 DEBUG:boto:Method: GET 2023-09-29T22:59:57.620 DEBUG:boto:Path: /hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.620 DEBUG:boto:Data: 2023-09-29T22:59:57.620 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.620 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.620 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.620 DEBUG:boto:Params: {} 2023-09-29T22:59:57.621 DEBUG:boto:Token: None 2023-09-29T22:59:57.621 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.621 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.621 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.625 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fea7a7df676a747-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.625 DEBUG:boto:b'hcnphz-61obj691000falseobj692023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.626 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.626 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.626 DEBUG:boto:path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.626 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.626 DEBUG:boto:Method: GET 2023-09-29T22:59:57.627 DEBUG:boto:Path: /hcnphz-61/?prefix=obj69 2023-09-29T22:59:57.627 DEBUG:boto:Data: 2023-09-29T22:59:57.627 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.627 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.627 DEBUG:boto:Params: {} 2023-09-29T22:59:57.627 DEBUG:boto:Token: None 2023-09-29T22:59:57.627 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.627 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.628 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.637 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000088b0f9f593133937-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.678 DEBUG:boto:b'hcnphz-61obj691000falseobj692023-09-29T22:55:14.046Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:59:57.679 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.679 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.679 DEBUG:boto:path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.679 DEBUG:boto:auth_path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.679 DEBUG:boto:Method: GET 2023-09-29T22:59:57.680 DEBUG:boto:Path: /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.680 DEBUG:boto:Data: 2023-09-29T22:59:57.680 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.680 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.680 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.680 DEBUG:boto:Params: {} 2023-09-29T22:59:57.680 DEBUG:boto:Token: None 2023-09-29T22:59:57.680 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.681 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E= 2023-09-29T22:59:57.681 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E=', 'Content-Length': '0'} 2023-09-29T22:59:57.684 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001f73608121bc8bd8-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.684 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.684 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.684 DEBUG:boto:path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.685 DEBUG:boto:auth_path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.685 DEBUG:boto:Method: GET 2023-09-29T22:59:57.685 DEBUG:boto:Path: /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.685 DEBUG:boto:Data: 2023-09-29T22:59:57.685 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.685 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.685 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.685 DEBUG:boto:Params: {} 2023-09-29T22:59:57.685 DEBUG:boto:Token: None 2023-09-29T22:59:57.686 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.686 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E= 2023-09-29T22:59:57.686 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E=', 'Content-Length': '0'} 2023-09-29T22:59:57.688 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009491a3b8f8e36269-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.689 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.689 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.689 DEBUG:boto:path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.689 DEBUG:boto:auth_path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.689 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.689 DEBUG:boto:Path: /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.690 DEBUG:boto:Data: 2023-09-29T22:59:57.690 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.690 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.690 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.690 DEBUG:boto:Params: {} 2023-09-29T22:59:57.690 DEBUG:boto:Token: None 2023-09-29T22:59:57.690 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.690 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O1DnZ+2YlKQZ2lK4/PWXQKdpNio= 2023-09-29T22:59:57.691 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O1DnZ+2YlKQZ2lK4/PWXQKdpNio=', 'Content-Length': '0'} 2023-09-29T22:59:57.693 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a3cbbdcb1b0672e1-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.693 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.693 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.693 DEBUG:boto:path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.694 DEBUG:boto:auth_path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.694 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.694 DEBUG:boto:Path: /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.694 DEBUG:boto:Data: 2023-09-29T22:59:57.694 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.694 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.694 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.694 DEBUG:boto:Params: {} 2023-09-29T22:59:57.694 DEBUG:boto:Token: None 2023-09-29T22:59:57.695 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.695 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:O1DnZ+2YlKQZ2lK4/PWXQKdpNio= 2023-09-29T22:59:57.695 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:O1DnZ+2YlKQZ2lK4/PWXQKdpNio=', 'Content-Length': '0'} 2023-09-29T22:59:57.697 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c4b83617a0f10390-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:59:57.697 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.698 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.698 DEBUG:boto:path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.698 DEBUG:boto:auth_path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.698 DEBUG:boto:Method: GET 2023-09-29T22:59:57.698 DEBUG:boto:Path: /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.698 DEBUG:boto:Data: 2023-09-29T22:59:57.698 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.698 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.698 DEBUG:boto:Params: {} 2023-09-29T22:59:57.699 DEBUG:boto:Token: None 2023-09-29T22:59:57.699 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E= 2023-09-29T22:59:57.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E=', 'Content-Length': '0'} 2023-09-29T22:59:57.702 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044cd84db0e65c579-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.702 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.702 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.702 DEBUG:boto:path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.702 DEBUG:boto:auth_path=/hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.703 DEBUG:boto:Method: GET 2023-09-29T22:59:57.703 DEBUG:boto:Path: /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.703 DEBUG:boto:Data: 2023-09-29T22:59:57.703 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.703 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.703 DEBUG:boto:Params: {} 2023-09-29T22:59:57.703 DEBUG:boto:Token: None 2023-09-29T22:59:57.703 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74?versionId=null 2023-09-29T22:59:57.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E= 2023-09-29T22:59:57.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zyCpTlMOSD8oXycsRwmAyRpLW9E=', 'Content-Length': '0'} 2023-09-29T22:59:57.706 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003412766ffff24515-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.706 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.707 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.707 DEBUG:boto:path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.707 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.707 DEBUG:boto:Method: GET 2023-09-29T22:59:57.707 DEBUG:boto:Path: /hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.707 DEBUG:boto:Data: 2023-09-29T22:59:57.707 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.708 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.708 DEBUG:boto:Params: {} 2023-09-29T22:59:57.708 DEBUG:boto:Token: None 2023-09-29T22:59:57.708 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.713 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c3c9d85962116e0a-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.713 DEBUG:boto:b'hcnphz-61obj741000falseobj742023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.713 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.714 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.714 DEBUG:boto:path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.714 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.714 DEBUG:boto:Method: GET 2023-09-29T22:59:57.714 DEBUG:boto:Path: /hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.714 DEBUG:boto:Data: 2023-09-29T22:59:57.714 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.714 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.715 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.715 DEBUG:boto:Params: {} 2023-09-29T22:59:57.715 DEBUG:boto:Token: None 2023-09-29T22:59:57.715 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.715 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.715 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.723 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f3949a1af3d428ca-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.723 DEBUG:boto:b'hcnphz-61obj741000falseobj742023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.723 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.724 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.724 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.724 DEBUG:boto:Path: /hcnphz-61/obj74 2023-09-29T22:59:57.724 DEBUG:boto:Data: 2023-09-29T22:59:57.724 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.724 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.724 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.724 DEBUG:boto:Params: {} 2023-09-29T22:59:57.724 DEBUG:boto:Token: None 2023-09-29T22:59:57.725 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74 2023-09-29T22:59:57.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yF+lzOXIwV5PDfqF9eKjXfqxwYs= 2023-09-29T22:59:57.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yF+lzOXIwV5PDfqF9eKjXfqxwYs=', 'Content-Length': '0'} 2023-09-29T22:59:57.727 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092b3f605feec91c1-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.727 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.727 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.727 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.727 DEBUG:boto:Path: /hcnphz-61/obj74 2023-09-29T22:59:57.727 DEBUG:boto:Data: 2023-09-29T22:59:57.727 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.728 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.728 DEBUG:boto:Params: {} 2023-09-29T22:59:57.728 DEBUG:boto:Token: None 2023-09-29T22:59:57.728 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74 2023-09-29T22:59:57.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yF+lzOXIwV5PDfqF9eKjXfqxwYs= 2023-09-29T22:59:57.728 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yF+lzOXIwV5PDfqF9eKjXfqxwYs=', 'Content-Length': '0'} 2023-09-29T22:59:57.730 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055a66bf9eb5d82b7-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj74 2023-09-29T22:59:57.731 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.731 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.731 DEBUG:boto:Method: GET 2023-09-29T22:59:57.731 DEBUG:boto:Path: /hcnphz-61/obj74 2023-09-29T22:59:57.731 DEBUG:boto:Data: 2023-09-29T22:59:57.731 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.731 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.731 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.731 DEBUG:boto:Params: {} 2023-09-29T22:59:57.732 DEBUG:boto:Token: None 2023-09-29T22:59:57.732 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74 2023-09-29T22:59:57.732 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jfPixVWafjhB3j78Slk+mkNy82g= 2023-09-29T22:59:57.732 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jfPixVWafjhB3j78Slk+mkNy82g=', 'Content-Length': '0'} 2023-09-29T22:59:57.734 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eb21f10556293f69-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.778 DEBUG:boto:path=/hcnphz-61/obj74 2023-09-29T22:59:57.779 DEBUG:boto:auth_path=/hcnphz-61/obj74 2023-09-29T22:59:57.779 DEBUG:boto:Method: GET 2023-09-29T22:59:57.779 DEBUG:boto:Path: /hcnphz-61/obj74 2023-09-29T22:59:57.779 DEBUG:boto:Data: 2023-09-29T22:59:57.779 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.779 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.779 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.779 DEBUG:boto:Params: {} 2023-09-29T22:59:57.780 DEBUG:boto:Token: None 2023-09-29T22:59:57.780 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj74 2023-09-29T22:59:57.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jfPixVWafjhB3j78Slk+mkNy82g= 2023-09-29T22:59:57.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jfPixVWafjhB3j78Slk+mkNy82g=', 'Content-Length': '0'} 2023-09-29T22:59:57.783 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:15 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002a70eaf40ce0ff4-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.783 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.783 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.783 DEBUG:boto:path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.784 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.784 DEBUG:boto:Method: GET 2023-09-29T22:59:57.784 DEBUG:boto:Path: /hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.784 DEBUG:boto:Data: 2023-09-29T22:59:57.784 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.784 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.784 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.784 DEBUG:boto:Params: {} 2023-09-29T22:59:57.784 DEBUG:boto:Token: None 2023-09-29T22:59:57.785 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.785 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.785 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000093110a4ebb35ca3b-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.790 DEBUG:boto:b'hcnphz-61obj741000falseobj742023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.790 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.790 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.790 DEBUG:boto:path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.791 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.791 DEBUG:boto:Method: GET 2023-09-29T22:59:57.791 DEBUG:boto:Path: /hcnphz-61/?prefix=obj74 2023-09-29T22:59:57.791 DEBUG:boto:Data: 2023-09-29T22:59:57.791 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.791 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.791 DEBUG:boto:Params: {} 2023-09-29T22:59:57.791 DEBUG:boto:Token: None 2023-09-29T22:59:57.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.799 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019b4262f1d9db43f-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.799 DEBUG:boto:b'hcnphz-61obj741000falseobj742023-09-29T22:55:15.363Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:59:57.800 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.800 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.800 DEBUG:boto:path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.800 DEBUG:boto:auth_path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.800 DEBUG:boto:Method: GET 2023-09-29T22:59:57.801 DEBUG:boto:Path: /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.801 DEBUG:boto:Data: 2023-09-29T22:59:57.801 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.801 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.801 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.801 DEBUG:boto:Params: {} 2023-09-29T22:59:57.801 DEBUG:boto:Token: None 2023-09-29T22:59:57.802 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0= 2023-09-29T22:59:57.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0=', 'Content-Length': '0'} 2023-09-29T22:59:57.804 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc74dbabe633e5ed-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.805 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.805 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.805 DEBUG:boto:path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.805 DEBUG:boto:auth_path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.805 DEBUG:boto:Method: GET 2023-09-29T22:59:57.805 DEBUG:boto:Path: /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.805 DEBUG:boto:Data: 2023-09-29T22:59:57.805 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.806 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.806 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.806 DEBUG:boto:Params: {} 2023-09-29T22:59:57.806 DEBUG:boto:Token: None 2023-09-29T22:59:57.806 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.807 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0= 2023-09-29T22:59:57.807 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0=', 'Content-Length': '0'} 2023-09-29T22:59:57.809 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a0597dd3254409a8-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.809 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.809 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.809 DEBUG:boto:path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.809 DEBUG:boto:auth_path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.810 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.810 DEBUG:boto:Path: /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.810 DEBUG:boto:Data: 2023-09-29T22:59:57.810 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.810 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.810 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.810 DEBUG:boto:Params: {} 2023-09-29T22:59:57.810 DEBUG:boto:Token: None 2023-09-29T22:59:57.811 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.811 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wx49JgXRQE5QFAryIrLQ8kjp2O0= 2023-09-29T22:59:57.811 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wx49JgXRQE5QFAryIrLQ8kjp2O0=', 'Content-Length': '0'} 2023-09-29T22:59:57.813 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001eed46e25bdcc658-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.813 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.814 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.814 DEBUG:boto:path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.814 DEBUG:boto:auth_path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.814 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.814 DEBUG:boto:Path: /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.814 DEBUG:boto:Data: 2023-09-29T22:59:57.814 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.814 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.814 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.815 DEBUG:boto:Params: {} 2023-09-29T22:59:57.815 DEBUG:boto:Token: None 2023-09-29T22:59:57.815 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.815 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wx49JgXRQE5QFAryIrLQ8kjp2O0= 2023-09-29T22:59:57.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wx49JgXRQE5QFAryIrLQ8kjp2O0=', 'Content-Length': '0'} 2023-09-29T22:59:57.818 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009bb73fbd4898d08b-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:59:57.818 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.818 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.818 DEBUG:boto:path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.818 DEBUG:boto:auth_path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.819 DEBUG:boto:Method: GET 2023-09-29T22:59:57.819 DEBUG:boto:Path: /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.819 DEBUG:boto:Data: 2023-09-29T22:59:57.819 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.819 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.819 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.819 DEBUG:boto:Params: {} 2023-09-29T22:59:57.819 DEBUG:boto:Token: None 2023-09-29T22:59:57.819 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.820 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0= 2023-09-29T22:59:57.820 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0=', 'Content-Length': '0'} 2023-09-29T22:59:57.822 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f2a3a16fdefe772-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.822 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.823 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.823 DEBUG:boto:path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.823 DEBUG:boto:auth_path=/hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.823 DEBUG:boto:Method: GET 2023-09-29T22:59:57.823 DEBUG:boto:Path: /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.823 DEBUG:boto:Data: 2023-09-29T22:59:57.823 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.823 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.823 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.824 DEBUG:boto:Params: {} 2023-09-29T22:59:57.824 DEBUG:boto:Token: None 2023-09-29T22:59:57.824 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76?versionId=null 2023-09-29T22:59:57.824 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0= 2023-09-29T22:59:57.824 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tOhe9xOZhAeHaU56p5ahJ79N5V0=', 'Content-Length': '0'} 2023-09-29T22:59:57.827 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc794e0a8e02bd0c-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.828 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.828 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.828 DEBUG:boto:path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.828 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.828 DEBUG:boto:Method: GET 2023-09-29T22:59:57.828 DEBUG:boto:Path: /hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.828 DEBUG:boto:Data: 2023-09-29T22:59:57.828 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.828 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.828 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.829 DEBUG:boto:Params: {} 2023-09-29T22:59:57.829 DEBUG:boto:Token: None 2023-09-29T22:59:57.829 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.829 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.829 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.833 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000326b49c279c290bc-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.874 DEBUG:boto:b'hcnphz-61obj761000falseobj762023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.875 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.875 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.875 DEBUG:boto:path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.875 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.875 DEBUG:boto:Method: GET 2023-09-29T22:59:57.876 DEBUG:boto:Path: /hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.876 DEBUG:boto:Data: 2023-09-29T22:59:57.876 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.876 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.876 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.876 DEBUG:boto:Params: {} 2023-09-29T22:59:57.876 DEBUG:boto:Token: None 2023-09-29T22:59:57.877 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.877 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.877 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.887 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cbf185da8b52e257-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.888 DEBUG:boto:b'hcnphz-61obj761000falseobj762023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.888 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.888 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.888 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.888 DEBUG:boto:Path: /hcnphz-61/obj76 2023-09-29T22:59:57.888 DEBUG:boto:Data: 2023-09-29T22:59:57.889 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.889 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.889 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.889 DEBUG:boto:Params: {} 2023-09-29T22:59:57.889 DEBUG:boto:Token: None 2023-09-29T22:59:57.889 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76 2023-09-29T22:59:57.889 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+H3Kaeh3O0ziKgDxMGs8vvz2ULY= 2023-09-29T22:59:57.889 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+H3Kaeh3O0ziKgDxMGs8vvz2ULY=', 'Content-Length': '0'} 2023-09-29T22:59:57.892 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d7a97ff2937b814c-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.892 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.892 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.892 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.892 DEBUG:boto:Path: /hcnphz-61/obj76 2023-09-29T22:59:57.893 DEBUG:boto:Data: 2023-09-29T22:59:57.893 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.893 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.893 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.893 DEBUG:boto:Params: {} 2023-09-29T22:59:57.893 DEBUG:boto:Token: None 2023-09-29T22:59:57.893 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76 2023-09-29T22:59:57.893 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+H3Kaeh3O0ziKgDxMGs8vvz2ULY= 2023-09-29T22:59:57.894 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+H3Kaeh3O0ziKgDxMGs8vvz2ULY=', 'Content-Length': '0'} 2023-09-29T22:59:57.895 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010a918bcd88a305f-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj76 2023-09-29T22:59:57.896 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.896 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.897 DEBUG:boto:Method: GET 2023-09-29T22:59:57.897 DEBUG:boto:Path: /hcnphz-61/obj76 2023-09-29T22:59:57.897 DEBUG:boto:Data: 2023-09-29T22:59:57.897 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.897 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.897 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.897 DEBUG:boto:Params: {} 2023-09-29T22:59:57.897 DEBUG:boto:Token: None 2023-09-29T22:59:57.898 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76 2023-09-29T22:59:57.898 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TaaLmtFCommDgyV40iL4nZFF1HY= 2023-09-29T22:59:57.898 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TaaLmtFCommDgyV40iL4nZFF1HY=', 'Content-Length': '0'} 2023-09-29T22:59:57.900 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dca1a84affdc797a-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.901 DEBUG:boto:path=/hcnphz-61/obj76 2023-09-29T22:59:57.901 DEBUG:boto:auth_path=/hcnphz-61/obj76 2023-09-29T22:59:57.901 DEBUG:boto:Method: GET 2023-09-29T22:59:57.901 DEBUG:boto:Path: /hcnphz-61/obj76 2023-09-29T22:59:57.901 DEBUG:boto:Data: 2023-09-29T22:59:57.901 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.901 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.902 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.902 DEBUG:boto:Params: {} 2023-09-29T22:59:57.902 DEBUG:boto:Token: None 2023-09-29T22:59:57.902 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj76 2023-09-29T22:59:57.902 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TaaLmtFCommDgyV40iL4nZFF1HY= 2023-09-29T22:59:57.902 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TaaLmtFCommDgyV40iL4nZFF1HY=', 'Content-Length': '0'} 2023-09-29T22:59:57.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005db2f65a29ccec5e-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.950 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.951 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.951 DEBUG:boto:path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.951 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.951 DEBUG:boto:Method: GET 2023-09-29T22:59:57.951 DEBUG:boto:Path: /hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.951 DEBUG:boto:Data: 2023-09-29T22:59:57.951 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.952 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.952 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.952 DEBUG:boto:Params: {} 2023-09-29T22:59:57.952 DEBUG:boto:Token: None 2023-09-29T22:59:57.953 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.953 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.953 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.957 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002f6977f1acddf43c-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.958 DEBUG:boto:b'hcnphz-61obj761000falseobj762023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.958 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.959 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.959 DEBUG:boto:path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.959 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.959 DEBUG:boto:Method: GET 2023-09-29T22:59:57.959 DEBUG:boto:Path: /hcnphz-61/?prefix=obj76 2023-09-29T22:59:57.959 DEBUG:boto:Data: 2023-09-29T22:59:57.959 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.959 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.959 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.960 DEBUG:boto:Params: {} 2023-09-29T22:59:57.960 DEBUG:boto:Token: None 2023-09-29T22:59:57.960 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.960 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.960 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:57.968 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004c9360b48d17a6d1-00651756ed-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.968 DEBUG:boto:b'hcnphz-61obj761000falseobj762023-09-29T22:55:13.659Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:57.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:59:57.969 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:57.969 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:57.969 DEBUG:boto:path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.969 DEBUG:boto:auth_path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.969 DEBUG:boto:Method: GET 2023-09-29T22:59:57.970 DEBUG:boto:Path: /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.970 DEBUG:boto:Data: 2023-09-29T22:59:57.970 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.970 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.970 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.970 DEBUG:boto:Params: {} 2023-09-29T22:59:57.970 DEBUG:boto:Token: None 2023-09-29T22:59:57.970 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.971 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc= 2023-09-29T22:59:57.971 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc=', 'Content-Length': '0'} 2023-09-29T22:59:57.973 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a2d21f4116bdfffd-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.973 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:57.973 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:57.973 DEBUG:boto:path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.974 DEBUG:boto:auth_path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.974 DEBUG:boto:Method: GET 2023-09-29T22:59:57.974 DEBUG:boto:Path: /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.974 DEBUG:boto:Data: 2023-09-29T22:59:57.974 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.974 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.974 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.974 DEBUG:boto:Params: {} 2023-09-29T22:59:57.975 DEBUG:boto:Token: None 2023-09-29T22:59:57.975 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.975 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc= 2023-09-29T22:59:57.975 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc=', 'Content-Length': '0'} 2023-09-29T22:59:57.977 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e0de41c1bbe8bf39-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.977 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:57.978 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:57.978 DEBUG:boto:path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.978 DEBUG:boto:auth_path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.978 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.978 DEBUG:boto:Path: /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.978 DEBUG:boto:Data: 2023-09-29T22:59:57.978 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.978 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.979 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.979 DEBUG:boto:Params: {} 2023-09-29T22:59:57.979 DEBUG:boto:Token: None 2023-09-29T22:59:57.979 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.980 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jyiUhLnwKrUD1rt34jfZZVv3ZSw= 2023-09-29T22:59:57.980 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jyiUhLnwKrUD1rt34jfZZVv3ZSw=', 'Content-Length': '0'} 2023-09-29T22:59:57.982 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000025d28b472c22419d-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.982 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:57.982 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:57.982 DEBUG:boto:path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.983 DEBUG:boto:auth_path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.983 DEBUG:boto:Method: HEAD 2023-09-29T22:59:57.983 DEBUG:boto:Path: /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.983 DEBUG:boto:Data: 2023-09-29T22:59:57.983 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.983 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.983 DEBUG:boto:Params: {} 2023-09-29T22:59:57.983 DEBUG:boto:Token: None 2023-09-29T22:59:57.984 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jyiUhLnwKrUD1rt34jfZZVv3ZSw= 2023-09-29T22:59:57.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jyiUhLnwKrUD1rt34jfZZVv3ZSw=', 'Content-Length': '0'} 2023-09-29T22:59:57.986 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d1e30aedda3ff1a-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:59:57.986 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:57.986 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:57.986 DEBUG:boto:path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.987 DEBUG:boto:auth_path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.987 DEBUG:boto:Method: GET 2023-09-29T22:59:57.987 DEBUG:boto:Path: /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.987 DEBUG:boto:Data: 2023-09-29T22:59:57.987 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.987 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.987 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.987 DEBUG:boto:Params: {} 2023-09-29T22:59:57.987 DEBUG:boto:Token: None 2023-09-29T22:59:57.988 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.988 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc= 2023-09-29T22:59:57.988 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc=', 'Content-Length': '0'} 2023-09-29T22:59:57.990 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009098a6d9e98428e0-00651756ed-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.991 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:57.991 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:57.991 DEBUG:boto:path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.991 DEBUG:boto:auth_path=/hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.991 DEBUG:boto:Method: GET 2023-09-29T22:59:57.991 DEBUG:boto:Path: /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.991 DEBUG:boto:Data: 2023-09-29T22:59:57.991 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:57.992 DEBUG:boto:Port: 8000 2023-09-29T22:59:57.992 DEBUG:boto:Params: {} 2023-09-29T22:59:57.992 DEBUG:boto:Token: None 2023-09-29T22:59:57.992 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/obj85?versionId=null 2023-09-29T22:59:57.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc= 2023-09-29T22:59:57.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NzCRrblBgiEjGfA+FfTe8WAI2Gc=', 'Content-Length': '0'} 2023-09-29T22:59:57.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002cc79c1116908124-00651756ed-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:57 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:57.995 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:57.995 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:57.995 DEBUG:boto:path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:57.995 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:57.996 DEBUG:boto:Method: GET 2023-09-29T22:59:57.996 DEBUG:boto:Path: /hcnphz-61/?prefix=obj85 2023-09-29T22:59:57.996 DEBUG:boto:Data: 2023-09-29T22:59:57.996 DEBUG:boto:Headers: {} 2023-09-29T22:59:57.996 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:57.996 DEBUG:boto:Port: 8001 2023-09-29T22:59:57.996 DEBUG:boto:Params: {} 2023-09-29T22:59:57.996 DEBUG:boto:Token: None 2023-09-29T22:59:57.996 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:57 GMT /hcnphz-61/ 2023-09-29T22:59:57.997 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY= 2023-09-29T22:59:57.997 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:57 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2qUBleedrtJ431+r3JUMBye0QPY=', 'Content-Length': '0'} 2023-09-29T22:59:58.000 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003b6c90b63b6ff881-00651756ed-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.000 DEBUG:boto:b'hcnphz-61obj851000falseobj852023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.001 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.001 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.001 DEBUG:boto:path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.001 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.001 DEBUG:boto:Method: GET 2023-09-29T22:59:58.002 DEBUG:boto:Path: /hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.002 DEBUG:boto:Data: 2023-09-29T22:59:58.002 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.002 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.002 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.002 DEBUG:boto:Params: {} 2023-09-29T22:59:58.002 DEBUG:boto:Token: None 2023-09-29T22:59:58.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.003 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.010 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000129418c1277a3f7e-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.058 DEBUG:boto:b'hcnphz-61obj851000falseobj852023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.059 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:58.059 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:58.059 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.059 DEBUG:boto:Path: /hcnphz-61/obj85 2023-09-29T22:59:58.059 DEBUG:boto:Data: 2023-09-29T22:59:58.059 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.060 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.060 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.060 DEBUG:boto:Params: {} 2023-09-29T22:59:58.060 DEBUG:boto:Token: None 2023-09-29T22:59:58.060 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj85 2023-09-29T22:59:58.060 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I4poEmbcUrtG0KF55MjhflFMDlk= 2023-09-29T22:59:58.060 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I4poEmbcUrtG0KF55MjhflFMDlk=', 'Content-Length': '0'} 2023-09-29T22:59:58.063 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a77eee845ecc7fa-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.064 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:58.064 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:58.064 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.064 DEBUG:boto:Path: /hcnphz-61/obj85 2023-09-29T22:59:58.064 DEBUG:boto:Data: 2023-09-29T22:59:58.064 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.064 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.064 DEBUG:boto:Params: {} 2023-09-29T22:59:58.065 DEBUG:boto:Token: None 2023-09-29T22:59:58.065 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj85 2023-09-29T22:59:58.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:I4poEmbcUrtG0KF55MjhflFMDlk= 2023-09-29T22:59:58.065 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I4poEmbcUrtG0KF55MjhflFMDlk=', 'Content-Length': '0'} 2023-09-29T22:59:58.068 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cafb1d462c94a8ed-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj85 2023-09-29T22:59:58.068 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:58.069 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:58.069 DEBUG:boto:Method: GET 2023-09-29T22:59:58.069 DEBUG:boto:Path: /hcnphz-61/obj85 2023-09-29T22:59:58.069 DEBUG:boto:Data: 2023-09-29T22:59:58.069 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.069 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.069 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.069 DEBUG:boto:Params: {} 2023-09-29T22:59:58.069 DEBUG:boto:Token: None 2023-09-29T22:59:58.070 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj85 2023-09-29T22:59:58.070 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/B9i7Inmx9uOayuii60uII8VBek= 2023-09-29T22:59:58.070 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/B9i7Inmx9uOayuii60uII8VBek=', 'Content-Length': '0'} 2023-09-29T22:59:58.072 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000afed675e5687fe76-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.114 DEBUG:boto:path=/hcnphz-61/obj85 2023-09-29T22:59:58.115 DEBUG:boto:auth_path=/hcnphz-61/obj85 2023-09-29T22:59:58.115 DEBUG:boto:Method: GET 2023-09-29T22:59:58.115 DEBUG:boto:Path: /hcnphz-61/obj85 2023-09-29T22:59:58.115 DEBUG:boto:Data: 2023-09-29T22:59:58.115 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.115 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.115 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.115 DEBUG:boto:Params: {} 2023-09-29T22:59:58.116 DEBUG:boto:Token: None 2023-09-29T22:59:58.116 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj85 2023-09-29T22:59:58.116 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/B9i7Inmx9uOayuii60uII8VBek= 2023-09-29T22:59:58.116 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/B9i7Inmx9uOayuii60uII8VBek=', 'Content-Length': '0'} 2023-09-29T22:59:58.119 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001b1e8942f71153d5-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.119 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.119 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.119 DEBUG:boto:path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.120 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.120 DEBUG:boto:Method: GET 2023-09-29T22:59:58.120 DEBUG:boto:Path: /hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.120 DEBUG:boto:Data: 2023-09-29T22:59:58.120 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.120 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.120 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.120 DEBUG:boto:Params: {} 2023-09-29T22:59:58.120 DEBUG:boto:Token: None 2023-09-29T22:59:58.121 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.121 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.121 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.125 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a926583f84430a6-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.125 DEBUG:boto:b'hcnphz-61obj851000falseobj852023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.126 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.126 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.126 DEBUG:boto:path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.126 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.127 DEBUG:boto:Method: GET 2023-09-29T22:59:58.127 DEBUG:boto:Path: /hcnphz-61/?prefix=obj85 2023-09-29T22:59:58.127 DEBUG:boto:Data: 2023-09-29T22:59:58.127 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.127 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.127 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.127 DEBUG:boto:Params: {} 2023-09-29T22:59:58.127 DEBUG:boto:Token: None 2023-09-29T22:59:58.128 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.128 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.128 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.135 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074e685ecc12cbda9-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.136 DEBUG:boto:b'hcnphz-61obj851000falseobj852023-09-29T22:55:13.272Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:59:58.136 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.136 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.136 DEBUG:boto:path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.136 DEBUG:boto:auth_path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.137 DEBUG:boto:Method: GET 2023-09-29T22:59:58.137 DEBUG:boto:Path: /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.137 DEBUG:boto:Data: 2023-09-29T22:59:58.137 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.137 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.137 DEBUG:boto:Params: {} 2023-09-29T22:59:58.137 DEBUG:boto:Token: None 2023-09-29T22:59:58.138 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw= 2023-09-29T22:59:58.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw=', 'Content-Length': '0'} 2023-09-29T22:59:58.140 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038cc29dce470ee70-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.140 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.141 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.141 DEBUG:boto:path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.141 DEBUG:boto:auth_path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.141 DEBUG:boto:Method: GET 2023-09-29T22:59:58.141 DEBUG:boto:Path: /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.141 DEBUG:boto:Data: 2023-09-29T22:59:58.141 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.141 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.142 DEBUG:boto:Params: {} 2023-09-29T22:59:58.142 DEBUG:boto:Token: None 2023-09-29T22:59:58.142 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.142 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw= 2023-09-29T22:59:58.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw=', 'Content-Length': '0'} 2023-09-29T22:59:58.144 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002caef8f1f3f87d15-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.186 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.187 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.187 DEBUG:boto:path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.187 DEBUG:boto:auth_path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.187 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.187 DEBUG:boto:Path: /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.187 DEBUG:boto:Data: 2023-09-29T22:59:58.187 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.187 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.187 DEBUG:boto:Params: {} 2023-09-29T22:59:58.188 DEBUG:boto:Token: None 2023-09-29T22:59:58.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i7EFdpIfYgVBTeddRDYgFKNrIW4= 2023-09-29T22:59:58.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i7EFdpIfYgVBTeddRDYgFKNrIW4=', 'Content-Length': '0'} 2023-09-29T22:59:58.191 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1cb5be534d226e0-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.191 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.191 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.191 DEBUG:boto:path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.191 DEBUG:boto:auth_path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.192 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.192 DEBUG:boto:Path: /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.192 DEBUG:boto:Data: 2023-09-29T22:59:58.192 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.192 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.192 DEBUG:boto:Params: {} 2023-09-29T22:59:58.192 DEBUG:boto:Token: None 2023-09-29T22:59:58.192 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.193 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:i7EFdpIfYgVBTeddRDYgFKNrIW4= 2023-09-29T22:59:58.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i7EFdpIfYgVBTeddRDYgFKNrIW4=', 'Content-Length': '0'} 2023-09-29T22:59:58.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009acc4ebb4b5b1f06-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:59:58.195 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.195 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.196 DEBUG:boto:path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.196 DEBUG:boto:auth_path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.196 DEBUG:boto:Method: GET 2023-09-29T22:59:58.196 DEBUG:boto:Path: /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.196 DEBUG:boto:Data: 2023-09-29T22:59:58.196 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.196 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.196 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.196 DEBUG:boto:Params: {} 2023-09-29T22:59:58.196 DEBUG:boto:Token: None 2023-09-29T22:59:58.197 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw= 2023-09-29T22:59:58.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw=', 'Content-Length': '0'} 2023-09-29T22:59:58.199 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000153874f9862dd409-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.246 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.247 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.247 DEBUG:boto:path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.247 DEBUG:boto:auth_path=/hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.247 DEBUG:boto:Method: GET 2023-09-29T22:59:58.247 DEBUG:boto:Path: /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.247 DEBUG:boto:Data: 2023-09-29T22:59:58.247 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.247 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.248 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.248 DEBUG:boto:Params: {} 2023-09-29T22:59:58.248 DEBUG:boto:Token: None 2023-09-29T22:59:58.248 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87?versionId=null 2023-09-29T22:59:58.248 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw= 2023-09-29T22:59:58.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aDi5JPFVW0UJP+p4wNiBJN3+jRw=', 'Content-Length': '0'} 2023-09-29T22:59:58.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c02ac1c6ed87ec7a-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.254 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.254 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.254 DEBUG:boto:path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.254 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.254 DEBUG:boto:Method: GET 2023-09-29T22:59:58.255 DEBUG:boto:Path: /hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.255 DEBUG:boto:Data: 2023-09-29T22:59:58.255 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.255 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.255 DEBUG:boto:Params: {} 2023-09-29T22:59:58.255 DEBUG:boto:Token: None 2023-09-29T22:59:58.255 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.260 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cdd8131b1244db6-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.260 DEBUG:boto:b'hcnphz-61obj871000falseobj872023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.261 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.261 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.261 DEBUG:boto:path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.261 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.261 DEBUG:boto:Method: GET 2023-09-29T22:59:58.261 DEBUG:boto:Path: /hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.262 DEBUG:boto:Data: 2023-09-29T22:59:58.262 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.262 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.262 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.262 DEBUG:boto:Params: {} 2023-09-29T22:59:58.262 DEBUG:boto:Token: None 2023-09-29T22:59:58.262 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.263 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.263 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.271 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da7760013ca5f583-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.271 DEBUG:boto:b'hcnphz-61obj871000falseobj872023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.271 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.272 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.272 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.272 DEBUG:boto:Path: /hcnphz-61/obj87 2023-09-29T22:59:58.272 DEBUG:boto:Data: 2023-09-29T22:59:58.272 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.272 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.273 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.273 DEBUG:boto:Params: {} 2023-09-29T22:59:58.273 DEBUG:boto:Token: None 2023-09-29T22:59:58.273 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87 2023-09-29T22:59:58.273 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fZay+XQRGBc5mGzwRFGsL5Fi1ZY= 2023-09-29T22:59:58.273 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fZay+XQRGBc5mGzwRFGsL5Fi1ZY=', 'Content-Length': '0'} 2023-09-29T22:59:58.275 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009a7839802c31d7d-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.276 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.276 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.276 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.276 DEBUG:boto:Path: /hcnphz-61/obj87 2023-09-29T22:59:58.276 DEBUG:boto:Data: 2023-09-29T22:59:58.276 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.277 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.277 DEBUG:boto:Params: {} 2023-09-29T22:59:58.277 DEBUG:boto:Token: None 2023-09-29T22:59:58.277 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87 2023-09-29T22:59:58.277 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fZay+XQRGBc5mGzwRFGsL5Fi1ZY= 2023-09-29T22:59:58.277 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fZay+XQRGBc5mGzwRFGsL5Fi1ZY=', 'Content-Length': '0'} 2023-09-29T22:59:58.279 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000630606cb3539857e-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj87 2023-09-29T22:59:58.279 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.279 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.280 DEBUG:boto:Method: GET 2023-09-29T22:59:58.280 DEBUG:boto:Path: /hcnphz-61/obj87 2023-09-29T22:59:58.280 DEBUG:boto:Data: 2023-09-29T22:59:58.280 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.280 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.280 DEBUG:boto:Params: {} 2023-09-29T22:59:58.280 DEBUG:boto:Token: None 2023-09-29T22:59:58.281 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87 2023-09-29T22:59:58.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IqYSdpCzPuTK85iHTlZkeRqQr1A= 2023-09-29T22:59:58.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IqYSdpCzPuTK85iHTlZkeRqQr1A=', 'Content-Length': '0'} 2023-09-29T22:59:58.283 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003683d8aa39fc9c00-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.330 DEBUG:boto:path=/hcnphz-61/obj87 2023-09-29T22:59:58.331 DEBUG:boto:auth_path=/hcnphz-61/obj87 2023-09-29T22:59:58.331 DEBUG:boto:Method: GET 2023-09-29T22:59:58.331 DEBUG:boto:Path: /hcnphz-61/obj87 2023-09-29T22:59:58.331 DEBUG:boto:Data: 2023-09-29T22:59:58.331 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.331 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.331 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.331 DEBUG:boto:Params: {} 2023-09-29T22:59:58.332 DEBUG:boto:Token: None 2023-09-29T22:59:58.332 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj87 2023-09-29T22:59:58.332 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IqYSdpCzPuTK85iHTlZkeRqQr1A= 2023-09-29T22:59:58.332 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IqYSdpCzPuTK85iHTlZkeRqQr1A=', 'Content-Length': '0'} 2023-09-29T22:59:58.335 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f3638cc5cbd3cf06-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.335 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.335 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.335 DEBUG:boto:path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.335 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.336 DEBUG:boto:Method: GET 2023-09-29T22:59:58.336 DEBUG:boto:Path: /hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.336 DEBUG:boto:Data: 2023-09-29T22:59:58.336 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.336 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.336 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.336 DEBUG:boto:Params: {} 2023-09-29T22:59:58.336 DEBUG:boto:Token: None 2023-09-29T22:59:58.336 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.337 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.337 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.341 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000022d985b1e24b20d9-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.341 DEBUG:boto:b'hcnphz-61obj871000falseobj872023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.342 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.342 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.342 DEBUG:boto:path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.342 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.343 DEBUG:boto:Method: GET 2023-09-29T22:59:58.343 DEBUG:boto:Path: /hcnphz-61/?prefix=obj87 2023-09-29T22:59:58.343 DEBUG:boto:Data: 2023-09-29T22:59:58.343 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.343 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.343 DEBUG:boto:Params: {} 2023-09-29T22:59:58.343 DEBUG:boto:Token: None 2023-09-29T22:59:58.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060131ea30e35eee8-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.352 DEBUG:boto:b'hcnphz-61obj871000falseobj872023-09-29T22:55:14.459Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:59:58.352 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.352 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.353 DEBUG:boto:path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.353 DEBUG:boto:auth_path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.353 DEBUG:boto:Method: GET 2023-09-29T22:59:58.353 DEBUG:boto:Path: /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.353 DEBUG:boto:Data: 2023-09-29T22:59:58.353 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.353 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.353 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.353 DEBUG:boto:Params: {} 2023-09-29T22:59:58.354 DEBUG:boto:Token: None 2023-09-29T22:59:58.354 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.354 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk= 2023-09-29T22:59:58.354 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk=', 'Content-Length': '0'} 2023-09-29T22:59:58.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b416cc15b9090646-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.357 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.357 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.357 DEBUG:boto:path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.357 DEBUG:boto:auth_path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.357 DEBUG:boto:Method: GET 2023-09-29T22:59:58.357 DEBUG:boto:Path: /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.357 DEBUG:boto:Data: 2023-09-29T22:59:58.357 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.357 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.358 DEBUG:boto:Params: {} 2023-09-29T22:59:58.358 DEBUG:boto:Token: None 2023-09-29T22:59:58.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk= 2023-09-29T22:59:58.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk=', 'Content-Length': '0'} 2023-09-29T22:59:58.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0cc152029d0d17f-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.360 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.361 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.361 DEBUG:boto:path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.361 DEBUG:boto:auth_path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.361 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.361 DEBUG:boto:Path: /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.361 DEBUG:boto:Data: 2023-09-29T22:59:58.361 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.361 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.361 DEBUG:boto:Params: {} 2023-09-29T22:59:58.362 DEBUG:boto:Token: None 2023-09-29T22:59:58.362 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jr3x6xy7MHMUZAnisNqlVyfxcAc= 2023-09-29T22:59:58.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jr3x6xy7MHMUZAnisNqlVyfxcAc=', 'Content-Length': '0'} 2023-09-29T22:59:58.364 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6625f526f397d34-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.364 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.364 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.364 DEBUG:boto:path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.365 DEBUG:boto:auth_path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.365 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.365 DEBUG:boto:Path: /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.365 DEBUG:boto:Data: 2023-09-29T22:59:58.365 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.365 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.365 DEBUG:boto:Params: {} 2023-09-29T22:59:58.365 DEBUG:boto:Token: None 2023-09-29T22:59:58.366 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.366 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jr3x6xy7MHMUZAnisNqlVyfxcAc= 2023-09-29T22:59:58.366 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jr3x6xy7MHMUZAnisNqlVyfxcAc=', 'Content-Length': '0'} 2023-09-29T22:59:58.368 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f24e32020e5138e-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:59:58.368 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.369 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.369 DEBUG:boto:path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.369 DEBUG:boto:auth_path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.369 DEBUG:boto:Method: GET 2023-09-29T22:59:58.369 DEBUG:boto:Path: /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.369 DEBUG:boto:Data: 2023-09-29T22:59:58.369 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.369 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.369 DEBUG:boto:Params: {} 2023-09-29T22:59:58.370 DEBUG:boto:Token: None 2023-09-29T22:59:58.370 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.370 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk= 2023-09-29T22:59:58.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk=', 'Content-Length': '0'} 2023-09-29T22:59:58.372 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cd2bf960cf561bfb-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.372 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.373 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.373 DEBUG:boto:path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.373 DEBUG:boto:auth_path=/hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.373 DEBUG:boto:Method: GET 2023-09-29T22:59:58.373 DEBUG:boto:Path: /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.373 DEBUG:boto:Data: 2023-09-29T22:59:58.373 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.373 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.374 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.374 DEBUG:boto:Params: {} 2023-09-29T22:59:58.374 DEBUG:boto:Token: None 2023-09-29T22:59:58.374 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92?versionId=null 2023-09-29T22:59:58.374 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk= 2023-09-29T22:59:58.374 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nRRnnPvTKLpyYqcOjg+Vh2K+Pfk=', 'Content-Length': '0'} 2023-09-29T22:59:58.377 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a74284c7632f30ec-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.377 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.377 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.377 DEBUG:boto:path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.377 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.377 DEBUG:boto:Method: GET 2023-09-29T22:59:58.378 DEBUG:boto:Path: /hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.378 DEBUG:boto:Data: 2023-09-29T22:59:58.378 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.378 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.378 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.378 DEBUG:boto:Params: {} 2023-09-29T22:59:58.378 DEBUG:boto:Token: None 2023-09-29T22:59:58.378 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.379 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.379 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.382 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e383d1c13874204b-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.383 DEBUG:boto:b'hcnphz-61obj921000falseobj922023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.383 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.383 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.383 DEBUG:boto:path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.383 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.383 DEBUG:boto:Method: GET 2023-09-29T22:59:58.384 DEBUG:boto:Path: /hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.384 DEBUG:boto:Data: 2023-09-29T22:59:58.384 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.384 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.384 DEBUG:boto:Params: {} 2023-09-29T22:59:58.384 DEBUG:boto:Token: None 2023-09-29T22:59:58.384 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.384 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.393 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dd0fdb3083c2a0c2-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.393 DEBUG:boto:b'hcnphz-61obj921000falseobj922023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.393 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.393 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.393 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.394 DEBUG:boto:Path: /hcnphz-61/obj92 2023-09-29T22:59:58.394 DEBUG:boto:Data: 2023-09-29T22:59:58.394 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.394 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.394 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.394 DEBUG:boto:Params: {} 2023-09-29T22:59:58.394 DEBUG:boto:Token: None 2023-09-29T22:59:58.394 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92 2023-09-29T22:59:58.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U0HB2DIRj3bo5HV40+ZdC/7U0UM= 2023-09-29T22:59:58.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0HB2DIRj3bo5HV40+ZdC/7U0UM=', 'Content-Length': '0'} 2023-09-29T22:59:58.396 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004cc9d9f6fce3bbc2-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.397 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.397 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.397 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.397 DEBUG:boto:Path: /hcnphz-61/obj92 2023-09-29T22:59:58.397 DEBUG:boto:Data: 2023-09-29T22:59:58.397 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.397 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.397 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.398 DEBUG:boto:Params: {} 2023-09-29T22:59:58.398 DEBUG:boto:Token: None 2023-09-29T22:59:58.398 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92 2023-09-29T22:59:58.398 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:U0HB2DIRj3bo5HV40+ZdC/7U0UM= 2023-09-29T22:59:58.398 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0HB2DIRj3bo5HV40+ZdC/7U0UM=', 'Content-Length': '0'} 2023-09-29T22:59:58.400 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000060c2721e789c0229-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj92 2023-09-29T22:59:58.401 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.401 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.401 DEBUG:boto:Method: GET 2023-09-29T22:59:58.401 DEBUG:boto:Path: /hcnphz-61/obj92 2023-09-29T22:59:58.401 DEBUG:boto:Data: 2023-09-29T22:59:58.401 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.401 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.401 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.401 DEBUG:boto:Params: {} 2023-09-29T22:59:58.402 DEBUG:boto:Token: None 2023-09-29T22:59:58.402 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92 2023-09-29T22:59:58.402 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GExVPYX/zpyrdXfPstuAP2STYn0= 2023-09-29T22:59:58.402 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GExVPYX/zpyrdXfPstuAP2STYn0=', 'Content-Length': '0'} 2023-09-29T22:59:58.404 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4a2e6df133ea687-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.446 DEBUG:boto:path=/hcnphz-61/obj92 2023-09-29T22:59:58.447 DEBUG:boto:auth_path=/hcnphz-61/obj92 2023-09-29T22:59:58.447 DEBUG:boto:Method: GET 2023-09-29T22:59:58.447 DEBUG:boto:Path: /hcnphz-61/obj92 2023-09-29T22:59:58.447 DEBUG:boto:Data: 2023-09-29T22:59:58.447 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.447 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.447 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.447 DEBUG:boto:Params: {} 2023-09-29T22:59:58.448 DEBUG:boto:Token: None 2023-09-29T22:59:58.448 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj92 2023-09-29T22:59:58.448 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GExVPYX/zpyrdXfPstuAP2STYn0= 2023-09-29T22:59:58.448 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GExVPYX/zpyrdXfPstuAP2STYn0=', 'Content-Length': '0'} 2023-09-29T22:59:58.450 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000408cad1ca540e88b-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.451 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.451 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.451 DEBUG:boto:path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.451 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.451 DEBUG:boto:Method: GET 2023-09-29T22:59:58.451 DEBUG:boto:Path: /hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.451 DEBUG:boto:Data: 2023-09-29T22:59:58.452 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.452 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.452 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.452 DEBUG:boto:Params: {} 2023-09-29T22:59:58.452 DEBUG:boto:Token: None 2023-09-29T22:59:58.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.452 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.452 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.457 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000907c41df59a2c8da-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.458 DEBUG:boto:b'hcnphz-61obj921000falseobj922023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.458 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.458 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.459 DEBUG:boto:path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.459 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.459 DEBUG:boto:Method: GET 2023-09-29T22:59:58.459 DEBUG:boto:Path: /hcnphz-61/?prefix=obj92 2023-09-29T22:59:58.459 DEBUG:boto:Data: 2023-09-29T22:59:58.459 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.460 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.460 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.460 DEBUG:boto:Params: {} 2023-09-29T22:59:58.460 DEBUG:boto:Token: None 2023-09-29T22:59:58.460 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.461 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.461 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.468 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000012f1e5f2c2982653-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.469 DEBUG:boto:b'hcnphz-61obj921000falseobj922023-09-29T22:55:14.065Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:59:58.469 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.470 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.470 DEBUG:boto:path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.470 DEBUG:boto:auth_path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.470 DEBUG:boto:Method: GET 2023-09-29T22:59:58.470 DEBUG:boto:Path: /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.470 DEBUG:boto:Data: 2023-09-29T22:59:58.471 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.471 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.471 DEBUG:boto:Params: {} 2023-09-29T22:59:58.471 DEBUG:boto:Token: None 2023-09-29T22:59:58.471 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04= 2023-09-29T22:59:58.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04=', 'Content-Length': '0'} 2023-09-29T22:59:58.474 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007fc63f14e197c660-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.474 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.474 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.475 DEBUG:boto:path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.475 DEBUG:boto:auth_path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.475 DEBUG:boto:Method: GET 2023-09-29T22:59:58.475 DEBUG:boto:Path: /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.475 DEBUG:boto:Data: 2023-09-29T22:59:58.475 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.475 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.475 DEBUG:boto:Params: {} 2023-09-29T22:59:58.475 DEBUG:boto:Token: None 2023-09-29T22:59:58.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04= 2023-09-29T22:59:58.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04=', 'Content-Length': '0'} 2023-09-29T22:59:58.478 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c594c65e3e86cc53-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.478 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.479 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.479 DEBUG:boto:path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.479 DEBUG:boto:auth_path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.479 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.479 DEBUG:boto:Path: /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.479 DEBUG:boto:Data: 2023-09-29T22:59:58.479 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.479 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.479 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.480 DEBUG:boto:Params: {} 2023-09-29T22:59:58.480 DEBUG:boto:Token: None 2023-09-29T22:59:58.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dl6ufgXeyTySX4gy9zN0nCADGhc= 2023-09-29T22:59:58.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dl6ufgXeyTySX4gy9zN0nCADGhc=', 'Content-Length': '0'} 2023-09-29T22:59:58.482 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000303ca9c57c69d85d-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.482 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.483 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.483 DEBUG:boto:path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.483 DEBUG:boto:auth_path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.483 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.483 DEBUG:boto:Path: /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.483 DEBUG:boto:Data: 2023-09-29T22:59:58.483 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.483 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.484 DEBUG:boto:Params: {} 2023-09-29T22:59:58.484 DEBUG:boto:Token: None 2023-09-29T22:59:58.484 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Dl6ufgXeyTySX4gy9zN0nCADGhc= 2023-09-29T22:59:58.484 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Dl6ufgXeyTySX4gy9zN0nCADGhc=', 'Content-Length': '0'} 2023-09-29T22:59:58.486 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006892e090e360d06c-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:59:58.486 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.487 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.487 DEBUG:boto:path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.487 DEBUG:boto:auth_path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.487 DEBUG:boto:Method: GET 2023-09-29T22:59:58.487 DEBUG:boto:Path: /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.487 DEBUG:boto:Data: 2023-09-29T22:59:58.487 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.487 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.487 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.487 DEBUG:boto:Params: {} 2023-09-29T22:59:58.488 DEBUG:boto:Token: None 2023-09-29T22:59:58.488 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.488 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04= 2023-09-29T22:59:58.488 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04=', 'Content-Length': '0'} 2023-09-29T22:59:58.491 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000affac20f2fcc8708-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.491 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.491 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.491 DEBUG:boto:path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.491 DEBUG:boto:auth_path=/hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.491 DEBUG:boto:Method: GET 2023-09-29T22:59:58.492 DEBUG:boto:Path: /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.492 DEBUG:boto:Data: 2023-09-29T22:59:58.492 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.492 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.492 DEBUG:boto:Params: {} 2023-09-29T22:59:58.492 DEBUG:boto:Token: None 2023-09-29T22:59:58.493 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93?versionId=null 2023-09-29T22:59:58.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04= 2023-09-29T22:59:58.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s/iXWZDEWZ5a7Q71ytdajI+Ju04=', 'Content-Length': '0'} 2023-09-29T22:59:58.495 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003cd43476ac30d117-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.495 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.495 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.496 DEBUG:boto:path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.496 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.496 DEBUG:boto:Method: GET 2023-09-29T22:59:58.496 DEBUG:boto:Path: /hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.496 DEBUG:boto:Data: 2023-09-29T22:59:58.496 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.496 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.496 DEBUG:boto:Params: {} 2023-09-29T22:59:58.496 DEBUG:boto:Token: None 2023-09-29T22:59:58.497 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.501 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002b83fef61fa0450a-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.542 DEBUG:boto:b'hcnphz-61obj931000falseobj932023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.543 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.543 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.543 DEBUG:boto:path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.543 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.543 DEBUG:boto:Method: GET 2023-09-29T22:59:58.543 DEBUG:boto:Path: /hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.544 DEBUG:boto:Data: 2023-09-29T22:59:58.544 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.544 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.544 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.544 DEBUG:boto:Params: {} 2023-09-29T22:59:58.544 DEBUG:boto:Token: None 2023-09-29T22:59:58.544 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.545 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.545 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.554 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005162a4a900c76ded-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.554 DEBUG:boto:b'hcnphz-61obj931000falseobj932023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.555 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.555 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.555 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.555 DEBUG:boto:Path: /hcnphz-61/obj93 2023-09-29T22:59:58.555 DEBUG:boto:Data: 2023-09-29T22:59:58.556 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.556 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.556 DEBUG:boto:Params: {} 2023-09-29T22:59:58.556 DEBUG:boto:Token: None 2023-09-29T22:59:58.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93 2023-09-29T22:59:58.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6ElrRRNHhu3F/PpChYxDy7a35as= 2023-09-29T22:59:58.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6ElrRRNHhu3F/PpChYxDy7a35as=', 'Content-Length': '0'} 2023-09-29T22:59:58.559 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bbb3fb14db8520cb-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.559 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.559 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.559 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.559 DEBUG:boto:Path: /hcnphz-61/obj93 2023-09-29T22:59:58.560 DEBUG:boto:Data: 2023-09-29T22:59:58.560 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.560 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.560 DEBUG:boto:Params: {} 2023-09-29T22:59:58.560 DEBUG:boto:Token: None 2023-09-29T22:59:58.560 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93 2023-09-29T22:59:58.560 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6ElrRRNHhu3F/PpChYxDy7a35as= 2023-09-29T22:59:58.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6ElrRRNHhu3F/PpChYxDy7a35as=', 'Content-Length': '0'} 2023-09-29T22:59:58.563 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f2b848e9f6989cbe-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj93 2023-09-29T22:59:58.563 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.563 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.563 DEBUG:boto:Method: GET 2023-09-29T22:59:58.563 DEBUG:boto:Path: /hcnphz-61/obj93 2023-09-29T22:59:58.564 DEBUG:boto:Data: 2023-09-29T22:59:58.564 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.564 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.564 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.564 DEBUG:boto:Params: {} 2023-09-29T22:59:58.564 DEBUG:boto:Token: None 2023-09-29T22:59:58.564 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93 2023-09-29T22:59:58.564 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zgkjIxp2uJD8uAIxwG5pLbS70KU= 2023-09-29T22:59:58.565 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zgkjIxp2uJD8uAIxwG5pLbS70KU=', 'Content-Length': '0'} 2023-09-29T22:59:58.566 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ecd5d93663fb7252-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.567 DEBUG:boto:path=/hcnphz-61/obj93 2023-09-29T22:59:58.567 DEBUG:boto:auth_path=/hcnphz-61/obj93 2023-09-29T22:59:58.567 DEBUG:boto:Method: GET 2023-09-29T22:59:58.567 DEBUG:boto:Path: /hcnphz-61/obj93 2023-09-29T22:59:58.567 DEBUG:boto:Data: 2023-09-29T22:59:58.567 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.567 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.568 DEBUG:boto:Params: {} 2023-09-29T22:59:58.568 DEBUG:boto:Token: None 2023-09-29T22:59:58.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj93 2023-09-29T22:59:58.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zgkjIxp2uJD8uAIxwG5pLbS70KU= 2023-09-29T22:59:58.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zgkjIxp2uJD8uAIxwG5pLbS70KU=', 'Content-Length': '0'} 2023-09-29T22:59:58.570 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:14 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000879f0faf19036c3d-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.610 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.611 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.611 DEBUG:boto:path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.611 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.611 DEBUG:boto:Method: GET 2023-09-29T22:59:58.611 DEBUG:boto:Path: /hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.611 DEBUG:boto:Data: 2023-09-29T22:59:58.611 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.612 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.612 DEBUG:boto:Params: {} 2023-09-29T22:59:58.612 DEBUG:boto:Token: None 2023-09-29T22:59:58.612 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.612 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.617 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000192203e2abd2a40c-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.617 DEBUG:boto:b'hcnphz-61obj931000falseobj932023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.618 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.618 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.618 DEBUG:boto:path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.618 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.618 DEBUG:boto:Method: GET 2023-09-29T22:59:58.619 DEBUG:boto:Path: /hcnphz-61/?prefix=obj93 2023-09-29T22:59:58.619 DEBUG:boto:Data: 2023-09-29T22:59:58.619 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.619 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.619 DEBUG:boto:Params: {} 2023-09-29T22:59:58.620 DEBUG:boto:Token: None 2023-09-29T22:59:58.620 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.628 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000035c8c1a8c7223051-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.628 DEBUG:boto:b'hcnphz-61obj931000falseobj932023-09-29T22:55:14.933Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:59:58.629 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.629 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.629 DEBUG:boto:path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.630 DEBUG:boto:auth_path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.630 DEBUG:boto:Method: GET 2023-09-29T22:59:58.630 DEBUG:boto:Path: /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.630 DEBUG:boto:Data: 2023-09-29T22:59:58.630 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.630 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.630 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.630 DEBUG:boto:Params: {} 2023-09-29T22:59:58.631 DEBUG:boto:Token: None 2023-09-29T22:59:58.631 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.631 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc= 2023-09-29T22:59:58.631 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc=', 'Content-Length': '0'} 2023-09-29T22:59:58.633 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ef21608fb4131f66-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.674 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.675 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.675 DEBUG:boto:path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.675 DEBUG:boto:auth_path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.675 DEBUG:boto:Method: GET 2023-09-29T22:59:58.675 DEBUG:boto:Path: /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.675 DEBUG:boto:Data: 2023-09-29T22:59:58.675 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.676 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.676 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.676 DEBUG:boto:Params: {} 2023-09-29T22:59:58.676 DEBUG:boto:Token: None 2023-09-29T22:59:58.676 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.676 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc= 2023-09-29T22:59:58.677 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc=', 'Content-Length': '0'} 2023-09-29T22:59:58.680 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f4ae8f8a8ded2aa6-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.680 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.680 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.680 DEBUG:boto:path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.680 DEBUG:boto:auth_path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.681 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.681 DEBUG:boto:Path: /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.681 DEBUG:boto:Data: 2023-09-29T22:59:58.681 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.681 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.681 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.681 DEBUG:boto:Params: {} 2023-09-29T22:59:58.681 DEBUG:boto:Token: None 2023-09-29T22:59:58.682 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.682 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pZQJf+bNzOFPpt3O5clKuzP40RQ= 2023-09-29T22:59:58.682 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pZQJf+bNzOFPpt3O5clKuzP40RQ=', 'Content-Length': '0'} 2023-09-29T22:59:58.685 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d34af74e704fd05-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.685 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.685 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.685 DEBUG:boto:path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.685 DEBUG:boto:auth_path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.686 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.686 DEBUG:boto:Path: /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.686 DEBUG:boto:Data: 2023-09-29T22:59:58.686 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.686 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.686 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.686 DEBUG:boto:Params: {} 2023-09-29T22:59:58.687 DEBUG:boto:Token: None 2023-09-29T22:59:58.687 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.687 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pZQJf+bNzOFPpt3O5clKuzP40RQ= 2023-09-29T22:59:58.687 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pZQJf+bNzOFPpt3O5clKuzP40RQ=', 'Content-Length': '0'} 2023-09-29T22:59:58.690 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebb0a83d072b9c86-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:59:58.690 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.691 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.691 DEBUG:boto:path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.691 DEBUG:boto:auth_path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.691 DEBUG:boto:Method: GET 2023-09-29T22:59:58.691 DEBUG:boto:Path: /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.691 DEBUG:boto:Data: 2023-09-29T22:59:58.691 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.691 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.692 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.692 DEBUG:boto:Params: {} 2023-09-29T22:59:58.692 DEBUG:boto:Token: None 2023-09-29T22:59:58.692 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc= 2023-09-29T22:59:58.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc=', 'Content-Length': '0'} 2023-09-29T22:59:58.695 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d3548d5f5ad51f0-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.695 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.696 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.696 DEBUG:boto:path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.696 DEBUG:boto:auth_path=/hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.696 DEBUG:boto:Method: GET 2023-09-29T22:59:58.696 DEBUG:boto:Path: /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.696 DEBUG:boto:Data: 2023-09-29T22:59:58.697 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.697 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.697 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.697 DEBUG:boto:Params: {} 2023-09-29T22:59:58.697 DEBUG:boto:Token: None 2023-09-29T22:59:58.697 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95?versionId=null 2023-09-29T22:59:58.698 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc= 2023-09-29T22:59:58.698 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/2EIIOfra76mEnRHWnyD6qznZDc=', 'Content-Length': '0'} 2023-09-29T22:59:58.700 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b8cde632c86a82d-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.701 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.701 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.701 DEBUG:boto:path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.701 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.701 DEBUG:boto:Method: GET 2023-09-29T22:59:58.701 DEBUG:boto:Path: /hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.702 DEBUG:boto:Data: 2023-09-29T22:59:58.702 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.702 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.702 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.702 DEBUG:boto:Params: {} 2023-09-29T22:59:58.702 DEBUG:boto:Token: None 2023-09-29T22:59:58.702 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.703 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.703 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.708 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d5a8f51db848a052-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.708 DEBUG:boto:b'hcnphz-61obj951000falseobj952023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.708 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.708 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.709 DEBUG:boto:path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.709 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.709 DEBUG:boto:Method: GET 2023-09-29T22:59:58.709 DEBUG:boto:Path: /hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.709 DEBUG:boto:Data: 2023-09-29T22:59:58.709 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.709 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.710 DEBUG:boto:Params: {} 2023-09-29T22:59:58.710 DEBUG:boto:Token: None 2023-09-29T22:59:58.710 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.717 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000503da935bf6ae483-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.717 DEBUG:boto:b'hcnphz-61obj951000falseobj952023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.718 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.718 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.718 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.718 DEBUG:boto:Path: /hcnphz-61/obj95 2023-09-29T22:59:58.718 DEBUG:boto:Data: 2023-09-29T22:59:58.718 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.718 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.718 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.719 DEBUG:boto:Params: {} 2023-09-29T22:59:58.719 DEBUG:boto:Token: None 2023-09-29T22:59:58.719 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95 2023-09-29T22:59:58.719 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wywBeliyPgaUFwl5eA+oFCZFodA= 2023-09-29T22:59:58.719 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wywBeliyPgaUFwl5eA+oFCZFodA=', 'Content-Length': '0'} 2023-09-29T22:59:58.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036fde9f954210a53-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.721 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.722 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.722 DEBUG:boto:Method: HEAD 2023-09-29T22:59:58.722 DEBUG:boto:Path: /hcnphz-61/obj95 2023-09-29T22:59:58.722 DEBUG:boto:Data: 2023-09-29T22:59:58.722 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.722 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.722 DEBUG:boto:Params: {} 2023-09-29T22:59:58.722 DEBUG:boto:Token: None 2023-09-29T22:59:58.723 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95 2023-09-29T22:59:58.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wywBeliyPgaUFwl5eA+oFCZFodA= 2023-09-29T22:59:58.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wywBeliyPgaUFwl5eA+oFCZFodA=', 'Content-Length': '0'} 2023-09-29T22:59:58.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004f529642b0c4ceae-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj95 2023-09-29T22:59:58.725 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.725 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.725 DEBUG:boto:Method: GET 2023-09-29T22:59:58.726 DEBUG:boto:Path: /hcnphz-61/obj95 2023-09-29T22:59:58.726 DEBUG:boto:Data: 2023-09-29T22:59:58.726 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.726 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.726 DEBUG:boto:Params: {} 2023-09-29T22:59:58.726 DEBUG:boto:Token: None 2023-09-29T22:59:58.726 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95 2023-09-29T22:59:58.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nt5UrrjvbrbEZPb1//40gCEwgio= 2023-09-29T22:59:58.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nt5UrrjvbrbEZPb1//40gCEwgio=', 'Content-Length': '0'} 2023-09-29T22:59:58.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-61:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.26'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000793b60d66db98619-00651756ee-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.774 DEBUG:boto:path=/hcnphz-61/obj95 2023-09-29T22:59:58.775 DEBUG:boto:auth_path=/hcnphz-61/obj95 2023-09-29T22:59:58.775 DEBUG:boto:Method: GET 2023-09-29T22:59:58.775 DEBUG:boto:Path: /hcnphz-61/obj95 2023-09-29T22:59:58.775 DEBUG:boto:Data: 2023-09-29T22:59:58.775 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.775 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.775 DEBUG:boto:Params: {} 2023-09-29T22:59:58.776 DEBUG:boto:Token: None 2023-09-29T22:59:58.776 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/obj95 2023-09-29T22:59:58.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nt5UrrjvbrbEZPb1//40gCEwgio= 2023-09-29T22:59:58.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nt5UrrjvbrbEZPb1//40gCEwgio=', 'Content-Length': '0'} 2023-09-29T22:59:58.779 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:55:13 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d108801d460787ee-00651756ee-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.779 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.779 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.779 DEBUG:boto:path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.779 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.780 DEBUG:boto:Method: GET 2023-09-29T22:59:58.780 DEBUG:boto:Path: /hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.780 DEBUG:boto:Data: 2023-09-29T22:59:58.780 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T22:59:58.780 DEBUG:boto:Port: 8001 2023-09-29T22:59:58.781 DEBUG:boto:Params: {} 2023-09-29T22:59:58.781 DEBUG:boto:Token: None 2023-09-29T22:59:58.781 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.781 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.781 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.785 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f14ebff10cff9851-00651756ee-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.785 DEBUG:boto:b'hcnphz-61obj951000falseobj952023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.786 DEBUG:boto:path=/hcnphz-61/ 2023-09-29T22:59:58.786 DEBUG:boto:auth_path=/hcnphz-61/ 2023-09-29T22:59:58.786 DEBUG:boto:path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.786 DEBUG:boto:auth_path=/hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.786 DEBUG:boto:Method: GET 2023-09-29T22:59:58.787 DEBUG:boto:Path: /hcnphz-61/?prefix=obj95 2023-09-29T22:59:58.787 DEBUG:boto:Data: 2023-09-29T22:59:58.787 DEBUG:boto:Headers: {} 2023-09-29T22:59:58.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:58.787 DEBUG:boto:Port: 8000 2023-09-29T22:59:58.788 DEBUG:boto:Params: {} 2023-09-29T22:59:58.788 DEBUG:boto:Token: None 2023-09-29T22:59:58.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 22:59:58 GMT /hcnphz-61/ 2023-09-29T22:59:58.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A= 2023-09-29T22:59:58.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mjVOEAyzJqGHb5RYZkdIxvvb4/A=', 'Content-Length': '0'} 2023-09-29T22:59:58.796 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000005ca98add8b1323d-00651756ee-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:59:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:58.796 DEBUG:boto:b'hcnphz-61obj951000falseobj952023-09-29T22:55:13.676Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T22:59:58.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-61 zones={a2, a1} 2023-09-29T22:59:58.797 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T22:59:58.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:58.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:58.798 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:58.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:58.839 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:58.934 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:59:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037386", 2023-09-29T22:59:58.934 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:55:15.462466Z" 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000033286", 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:55:21.645843Z" 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032347", 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:55:21.649129Z" 2023-09-29T22:59:58.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000022720", 2023-09-29T22:59:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:55:21.652745Z" 2023-09-29T22:59:58.936 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:58.936 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:59:58.937 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000037386', 1: '00000000000000000000:00000000000000033286', 2: '00000000000000000000:00000000000000032347', 3: '00000000000000000000:00000000000000022720'} 2023-09-29T22:59:58.937 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:59:58.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:58.938 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:58.938 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:58.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:58.985 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 18095951992126222295 2023-09-29T22:59:59.068 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037386", 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.069 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 52, 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:55:15.462466Z" 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000033286", 2023-09-29T22:59:59.070 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:55:21.645843Z" 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:59:59.071 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.073 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032347", 2023-09-29T22:59:59.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:55:21.649129Z" 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000022720", 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:55:21.652745Z" 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 61, 2023-09-29T22:59:59.075 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 61 2023-09-29T22:59:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.076 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:59:59Z" 2023-09-29T22:59:59.076 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:59:59.077 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 18095951992126222295 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037386", "next_step_marker": "", "total_entries": 52, "pos": 0, "timestamp": "2023-09-29T22:55:15.462466Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033286", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T22:55:21.645843Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032347", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:21.649129Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022720", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:21.652745Z" } } ] }, "full_sync": { "total": 61, "complete": 61 }, "current_time": "2023-09-29T22:59:59Z" } 2023-09-29T22:59:59.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037386', 'next_step_marker': '', 'total_entries': 52, 'pos': 0, 'timestamp': '2023-09-29T22:55:15.462466Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033286', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T22:55:21.645843Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032347', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:21.649129Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022720', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:21.652745Z'}}] 2023-09-29T22:59:59.077 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000037386', 1: '00000000000000000000:00000000000000033286', 2: '00000000000000000000:00000000000000032347', 3: '00000000000000000000:00000000000000022720'} 2023-09-29T22:59:59.077 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000037386', 1: '00000000000000000000:00000000000000033286', 2: '00000000000000000000:00000000000000032347', 3: '00000000000000000000:00000000000000022720'} 2023-09-29T22:59:59.077 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T22:59:59.078 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T22:59:59.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.078 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:59.119 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:59.120 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:59.215 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:59:59.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.216 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015447", 2023-09-29T22:59:59.216 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:57:05.820217Z" 2023-09-29T22:59:59.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014181", 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:57:47.528367Z" 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011394", 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:57:05.827114Z" 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.217 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.218 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T22:59:59.218 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:57:05.830505Z" 2023-09-29T22:59:59.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.218 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:59:59.219 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015447', 1: '00000000000000000000:00000000000000014181', 2: '00000000000000000000:00000000000000011394', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T22:59:59.219 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:59:59.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.220 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:59.265 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:59.265 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:59.348 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:59:59.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015447", 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.350 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.820217Z" 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014181", 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.351 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:47.528367Z" 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011394", 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.352 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.827114Z" 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T22:59:59.353 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.830505Z" 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.354 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:59:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T22:59:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T22:59:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.355 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:59:59Z" 2023-09-29T22:59:59.355 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:59:59.356 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015447", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.820217Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014181", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:47.528367Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011394", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.827114Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.830505Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T22:59:59Z" } 2023-09-29T22:59:59.356 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015447', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.820217Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014181', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:47.528367Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011394', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.827114Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.830505Z'}}] 2023-09-29T22:59:59.357 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015447', 1: '00000000000000000000:00000000000000014181', 2: '00000000000000000000:00000000000000011394', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T22:59:59.357 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015447', 1: '00000000000000000000:00000000000000014181', 2: '00000000000000000000:00000000000000011394', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T22:59:59.357 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T22:59:59.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_full_sync_after_data_sync_init ... ok 2023-09-29T22:59:59.358 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T22:59:59.359 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T22:59:59.359 DEBUG:boto:Method: PUT 2023-09-29T22:59:59.359 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T22:59:59.359 DEBUG:boto:Data: 2023-09-29T22:59:59.359 DEBUG:boto:Headers: {} 2023-09-29T22:59:59.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:59.359 DEBUG:boto:Port: 8000 2023-09-29T22:59:59.359 DEBUG:boto:Params: {} 2023-09-29T22:59:59.360 DEBUG:boto:Token: None 2023-09-29T22:59:59.360 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 22:59:59 GMT /hcnphz-62/ 2023-09-29T22:59:59.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:049/h7P8F/Z8UtO+GzkJc4MjuQw= 2023-09-29T22:59:59.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:59 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:049/h7P8F/Z8UtO+GzkJc4MjuQw=', 'Content-Length': '0'} 2023-09-29T22:59:59.390 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000467a6e7085e8cce8-00651756ef-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:59.390 DEBUG:rgw_multi.tests:created bucket=hcnphz-62 2023-09-29T22:59:59.391 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.391 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.391 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:59.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:59.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:59.485 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T22:59:59.512 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T22:59:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:59:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T22:59:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028399.390107_595.1", 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:59:59.390107Z" 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:59:59.513 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T22:59:59.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.514 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T22:59:59.515 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028399.390107_595.1', 2: '', 3: ''} 2023-09-29T22:59:59.515 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T22:59:59.515 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T22:59:59.515 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T22:59:59.601 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:59.602 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:59.657 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:59.657+0000 7fa69a4d8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:59.659 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:59.661+0000 7fa69a4d8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T22:59:59.684 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T22:59:59.684 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T22:59:59.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.686 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:59:59.687 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028399.390107_595.1", 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:59:59.390107Z", 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.688 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:59:59.689 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T22:59:59.690 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.691 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T22:59:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T22:59:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T22:59:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T22:59:59.692 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T22:59:59Z" 2023-09-29T22:59:59.692 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T22:59:59.693 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028399.390107_595.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:59:59.390107Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T22:59:59Z" } 2023-09-29T22:59:59.693 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028399.390107_595.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:59:59.390107Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T22:59:59.694 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028399.390107_595.1', 2: '', 3: ''} 2023-09-29T22:59:59.694 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028399.390107_595.1', 2: '', 3: ''} 2023-09-29T22:59:59.694 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T22:59:59.694 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T22:59:59.694 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T22:59:59.694 DEBUG:boto:Method: HEAD 2023-09-29T22:59:59.695 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T22:59:59.695 DEBUG:boto:Data: 2023-09-29T22:59:59.695 DEBUG:boto:Headers: {} 2023-09-29T22:59:59.695 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:59.695 DEBUG:boto:Port: 8000 2023-09-29T22:59:59.695 DEBUG:boto:Params: {} 2023-09-29T22:59:59.695 DEBUG:boto:Token: None 2023-09-29T22:59:59.696 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 22:59:59 GMT /hcnphz-62/ 2023-09-29T22:59:59.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y8uwH5erqSkZCTwghNUJZhlSLmc= 2023-09-29T22:59:59.696 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:59:59 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y8uwH5erqSkZCTwghNUJZhlSLmc=', 'Content-Length': '0'} 2023-09-29T22:59:59.700 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ec73c101955988e-00651756ef-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:59.700 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T22:59:59.700 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T22:59:59.700 DEBUG:boto:Method: PUT 2023-09-29T22:59:59.700 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T22:59:59.701 DEBUG:boto:Data: 2023-09-29T22:59:59.701 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T22:59:59.701 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T22:59:59.701 DEBUG:boto:Port: 8000 2023-09-29T22:59:59.701 DEBUG:boto:Params: {} 2023-09-29T22:59:59.701 DEBUG:boto:Token: None 2023-09-29T22:59:59.701 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 22:59:59 GMT /hcnphz-62/dummy 2023-09-29T22:59:59.702 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1qhwdBTLi3BDDIDoFf4OnE01EwQ= 2023-09-29T22:59:59.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:59:59 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1qhwdBTLi3BDDIDoFf4OnE01EwQ='} 2023-09-29T22:59:59.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000da15ea93c162faf8-00651756ef-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T22:59:59.715 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-62 2023-09-29T22:59:59.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:59:59.716 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T22:59:59.716 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T22:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T22:59:59.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T22:59:59.834 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T22:59:59.833+0000 7f50953fef40 1 waiting to reach incremental sync.. 2023-09-29T23:00:04.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:04.836+0000 7f50953fef40 1 waiting to reach incremental sync.. 2023-09-29T23:00:09.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:09.840+0000 7f50953fef40 1 bucket sync caught up with source: 2023-09-29T23:00:09.839 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:09.839 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:09.839 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:09.840+0000 7f50953fef40 0 bucket checkpoint complete 2023-09-29T23:00:09.869 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-62 2023-09-29T23:00:09.869 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:00:09.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:00:09.870 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:00:09.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:09.918 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:09.984 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:09.984+0000 7f40f9e85f40 1 waiting to reach incremental sync.. 2023-09-29T23:00:14.985 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:14.984+0000 7f40f9e85f40 1 waiting to reach incremental sync.. 2023-09-29T23:00:19.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:19.988+0000 7f40f9e85f40 1 bucket sync caught up with source: 2023-09-29T23:00:19.989 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.155.6, , , , , ] 2023-09-29T23:00:19.989 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.155.6, , , , , ] 2023-09-29T23:00:19.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:19.988+0000 7f40f9e85f40 0 bucket checkpoint complete 2023-09-29T23:00:20.018 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-62 zones={a2, a1} 2023-09-29T23:00:20.018 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.018 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.018 DEBUG:boto:Method: HEAD 2023-09-29T23:00:20.018 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:20.018 DEBUG:boto:Data: 2023-09-29T23:00:20.018 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.019 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.019 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.019 DEBUG:boto:Params: {} 2023-09-29T23:00:20.019 DEBUG:boto:Token: None 2023-09-29T23:00:20.019 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/ 2023-09-29T23:00:20.019 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8Ftf8gZdsk3Q5NxsYYCd2dSFmFI= 2023-09-29T23:00:20.019 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8Ftf8gZdsk3Q5NxsYYCd2dSFmFI=', 'Content-Length': '0'} 2023-09-29T23:00:20.023 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cb1e4cd0b814469a-0065175704-1e9a-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.023 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.023 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.023 DEBUG:boto:Method: HEAD 2023-09-29T23:00:20.023 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:20.024 DEBUG:boto:Data: 2023-09-29T23:00:20.024 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.024 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.024 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.024 DEBUG:boto:Params: {} 2023-09-29T23:00:20.024 DEBUG:boto:Token: None 2023-09-29T23:00:20.024 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/ 2023-09-29T23:00:20.024 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8Ftf8gZdsk3Q5NxsYYCd2dSFmFI= 2023-09-29T23:00:20.025 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8Ftf8gZdsk3Q5NxsYYCd2dSFmFI=', 'Content-Length': '0'} 2023-09-29T23:00:20.029 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000efc4bfd6fd100cc-0065175704-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.029 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T23:00:20.029 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.029 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.029 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.029 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.029 DEBUG:boto:Method: GET 2023-09-29T23:00:20.030 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.030 DEBUG:boto:Data: 2023-09-29T23:00:20.030 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.030 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.030 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.030 DEBUG:boto:Params: {} 2023-09-29T23:00:20.030 DEBUG:boto:Token: None 2023-09-29T23:00:20.030 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/?versions 2023-09-29T23:00:20.031 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE= 2023-09-29T23:00:20.031 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE=', 'Content-Length': '0'} 2023-09-29T23:00:20.035 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000074d7dd23bc911880-0065175704-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.035 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.035 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T23:00:20.036 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T23:00:20.036 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.036 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.036 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.036 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.036 DEBUG:boto:Method: GET 2023-09-29T23:00:20.036 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.036 DEBUG:boto:Data: 2023-09-29T23:00:20.037 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.037 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.037 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.037 DEBUG:boto:Params: {} 2023-09-29T23:00:20.037 DEBUG:boto:Token: None 2023-09-29T23:00:20.037 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/?versions 2023-09-29T23:00:20.037 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE= 2023-09-29T23:00:20.038 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE=', 'Content-Length': '0'} 2023-09-29T23:00:20.041 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000dff212e99bab2f2-0065175704-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.082 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.083 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T23:00:20.083 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.083 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.083 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.083 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.084 DEBUG:boto:Method: GET 2023-09-29T23:00:20.084 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.084 DEBUG:boto:Data: 2023-09-29T23:00:20.084 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.084 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.084 DEBUG:boto:Params: {} 2023-09-29T23:00:20.084 DEBUG:boto:Token: None 2023-09-29T23:00:20.085 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/?versions 2023-09-29T23:00:20.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE= 2023-09-29T23:00:20.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE=', 'Content-Length': '0'} 2023-09-29T23:00:20.089 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005dd2330d2ee4113c-0065175704-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.090 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.090 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.090 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.090 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.090 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.090 DEBUG:boto:Method: GET 2023-09-29T23:00:20.091 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:00:20.091 DEBUG:boto:Data: 2023-09-29T23:00:20.091 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.091 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.091 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.091 DEBUG:boto:Params: {} 2023-09-29T23:00:20.091 DEBUG:boto:Token: None 2023-09-29T23:00:20.091 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/?versions 2023-09-29T23:00:20.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE= 2023-09-29T23:00:20.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DXLs+OYwsYakge/AZiGjoyBMTWE=', 'Content-Length': '0'} 2023-09-29T23:00:20.096 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea57a24d95665569-0065175704-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.096 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T23:00:20.097 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.097 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.097 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.097 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.097 DEBUG:boto:Method: GET 2023-09-29T23:00:20.097 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.098 DEBUG:boto:Data: 2023-09-29T23:00:20.098 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.098 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.098 DEBUG:boto:Params: {} 2023-09-29T23:00:20.098 DEBUG:boto:Token: None 2023-09-29T23:00:20.098 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.098 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8= 2023-09-29T23:00:20.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8=', 'Content-Length': '0'} 2023-09-29T23:00:20.101 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e87d71766556305d-0065175704-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.142 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.143 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.143 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.143 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.143 DEBUG:boto:Method: GET 2023-09-29T23:00:20.143 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.143 DEBUG:boto:Data: 2023-09-29T23:00:20.143 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.143 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.143 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.144 DEBUG:boto:Params: {} 2023-09-29T23:00:20.144 DEBUG:boto:Token: None 2023-09-29T23:00:20.144 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.144 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8= 2023-09-29T23:00:20.144 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8=', 'Content-Length': '0'} 2023-09-29T23:00:20.147 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001097ae2c69f97108-0065175704-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.148 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.148 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.148 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.148 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.148 DEBUG:boto:Method: HEAD 2023-09-29T23:00:20.148 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.148 DEBUG:boto:Data: 2023-09-29T23:00:20.148 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.149 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.149 DEBUG:boto:Params: {} 2023-09-29T23:00:20.149 DEBUG:boto:Token: None 2023-09-29T23:00:20.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2zcbnqLpvIKMRlPCieqyGMqZZPs= 2023-09-29T23:00:20.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2zcbnqLpvIKMRlPCieqyGMqZZPs=', 'Content-Length': '0'} 2023-09-29T23:00:20.152 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e68e5c1d7e6071a-0065175704-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.152 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.152 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.152 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.152 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.153 DEBUG:boto:Method: HEAD 2023-09-29T23:00:20.153 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.153 DEBUG:boto:Data: 2023-09-29T23:00:20.153 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.153 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.153 DEBUG:boto:Params: {} 2023-09-29T23:00:20.153 DEBUG:boto:Token: None 2023-09-29T23:00:20.153 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.154 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2zcbnqLpvIKMRlPCieqyGMqZZPs= 2023-09-29T23:00:20.154 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2zcbnqLpvIKMRlPCieqyGMqZZPs=', 'Content-Length': '0'} 2023-09-29T23:00:20.157 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ace475f8626b01dd-0065175704-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T23:00:20.158 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.158 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.158 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.158 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.158 DEBUG:boto:Method: GET 2023-09-29T23:00:20.158 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.158 DEBUG:boto:Data: 2023-09-29T23:00:20.158 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.159 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.159 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.159 DEBUG:boto:Params: {} 2023-09-29T23:00:20.159 DEBUG:boto:Token: None 2023-09-29T23:00:20.159 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8= 2023-09-29T23:00:20.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8=', 'Content-Length': '0'} 2023-09-29T23:00:20.162 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006560a04dbff7af2-0065175704-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.162 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.162 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.162 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.162 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.163 DEBUG:boto:Method: GET 2023-09-29T23:00:20.163 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.163 DEBUG:boto:Data: 2023-09-29T23:00:20.163 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.163 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.163 DEBUG:boto:Params: {} 2023-09-29T23:00:20.163 DEBUG:boto:Token: None 2023-09-29T23:00:20.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:00:20.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8= 2023-09-29T23:00:20.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lo+dQffuDmzeK5RVuKA6co1SPj8=', 'Content-Length': '0'} 2023-09-29T23:00:20.166 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c08edfab45c47e6e-0065175704-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.167 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.167 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.167 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.167 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.167 DEBUG:boto:Method: GET 2023-09-29T23:00:20.167 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.167 DEBUG:boto:Data: 2023-09-29T23:00:20.168 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.168 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.168 DEBUG:boto:Params: {} 2023-09-29T23:00:20.168 DEBUG:boto:Token: None 2023-09-29T23:00:20.168 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/ 2023-09-29T23:00:20.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ= 2023-09-29T23:00:20.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ=', 'Content-Length': '0'} 2023-09-29T23:00:20.174 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009ad2004836db98ce-0065175704-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.174 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.174 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.174 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.175 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.175 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.175 DEBUG:boto:Method: GET 2023-09-29T23:00:20.175 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.175 DEBUG:boto:Data: 2023-09-29T23:00:20.175 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.175 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.175 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.175 DEBUG:boto:Params: {} 2023-09-29T23:00:20.175 DEBUG:boto:Token: None 2023-09-29T23:00:20.176 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/ 2023-09-29T23:00:20.176 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ= 2023-09-29T23:00:20.176 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ=', 'Content-Length': '0'} 2023-09-29T23:00:20.183 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dcc4fbbc9dc5ee93-0065175704-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.226 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.227 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.227 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.227 DEBUG:boto:Method: HEAD 2023-09-29T23:00:20.227 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:00:20.227 DEBUG:boto:Data: 2023-09-29T23:00:20.227 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.228 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.228 DEBUG:boto:Params: {} 2023-09-29T23:00:20.228 DEBUG:boto:Token: None 2023-09-29T23:00:20.228 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy 2023-09-29T23:00:20.228 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gEWtVUeGpAkenk+/sCu/3zTGsSw= 2023-09-29T23:00:20.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gEWtVUeGpAkenk+/sCu/3zTGsSw=', 'Content-Length': '0'} 2023-09-29T23:00:20.231 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000be5892b058ad88f4-0065175704-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.231 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.231 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.232 DEBUG:boto:Method: HEAD 2023-09-29T23:00:20.232 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:00:20.232 DEBUG:boto:Data: 2023-09-29T23:00:20.232 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.232 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.232 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.232 DEBUG:boto:Params: {} 2023-09-29T23:00:20.233 DEBUG:boto:Token: None 2023-09-29T23:00:20.233 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy 2023-09-29T23:00:20.233 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gEWtVUeGpAkenk+/sCu/3zTGsSw= 2023-09-29T23:00:20.233 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gEWtVUeGpAkenk+/sCu/3zTGsSw=', 'Content-Length': '0'} 2023-09-29T23:00:20.235 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e3b6ae10cde47dfa-0065175704-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T23:00:20.236 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.236 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.236 DEBUG:boto:Method: GET 2023-09-29T23:00:20.236 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:00:20.236 DEBUG:boto:Data: 2023-09-29T23:00:20.237 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.237 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.237 DEBUG:boto:Params: {} 2023-09-29T23:00:20.237 DEBUG:boto:Token: None 2023-09-29T23:00:20.237 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy 2023-09-29T23:00:20.237 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Sl/EtJtzYHvh4aODE+Vut3pXSg= 2023-09-29T23:00:20.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Sl/EtJtzYHvh4aODE+Vut3pXSg=', 'Content-Length': '0'} 2023-09-29T23:00:20.240 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d43914908e094d71-0065175704-1e9a-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.286 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:00:20.287 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:00:20.287 DEBUG:boto:Method: GET 2023-09-29T23:00:20.287 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:00:20.287 DEBUG:boto:Data: 2023-09-29T23:00:20.287 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.287 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.287 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.287 DEBUG:boto:Params: {} 2023-09-29T23:00:20.287 DEBUG:boto:Token: None 2023-09-29T23:00:20.288 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/dummy 2023-09-29T23:00:20.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+Sl/EtJtzYHvh4aODE+Vut3pXSg= 2023-09-29T23:00:20.288 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+Sl/EtJtzYHvh4aODE+Vut3pXSg=', 'Content-Length': '0'} 2023-09-29T23:00:20.291 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c8fc632bbfb74bb8-0065175704-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.291 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.291 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.291 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.291 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.292 DEBUG:boto:Method: GET 2023-09-29T23:00:20.292 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.292 DEBUG:boto:Data: 2023-09-29T23:00:20.292 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.292 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:20.292 DEBUG:boto:Port: 8001 2023-09-29T23:00:20.292 DEBUG:boto:Params: {} 2023-09-29T23:00:20.292 DEBUG:boto:Token: None 2023-09-29T23:00:20.292 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/ 2023-09-29T23:00:20.293 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ= 2023-09-29T23:00:20.293 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ=', 'Content-Length': '0'} 2023-09-29T23:00:20.297 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060ab4996182cb2c2-0065175704-1e9a-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.297 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.298 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:20.298 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:20.298 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.298 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.298 DEBUG:boto:Method: GET 2023-09-29T23:00:20.298 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:00:20.299 DEBUG:boto:Data: 2023-09-29T23:00:20.299 DEBUG:boto:Headers: {} 2023-09-29T23:00:20.299 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:20.299 DEBUG:boto:Port: 8000 2023-09-29T23:00:20.299 DEBUG:boto:Params: {} 2023-09-29T23:00:20.299 DEBUG:boto:Token: None 2023-09-29T23:00:20.299 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:00:20 GMT /hcnphz-62/ 2023-09-29T23:00:20.299 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ= 2023-09-29T23:00:20.299 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MJ4fO1FssLA96XqSgLK5R2Ff6TQ=', 'Content-Length': '0'} 2023-09-29T23:00:20.304 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da3bbd0ba70539cf-0065175704-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:00:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:20.304 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:00:20.304 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-62 zones={a2, a1} 2023-09-29T23:00:20.305 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T23:00:20.305 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:00:20.306 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:00:20.304+0000 7fe850ec4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 77676) UID: 0 2023-09-29T23:00:20.306 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:00:20.304+0000 7fe854723f80 -1 shutting down 2023-09-29T23:00:24.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T23:00:26.406 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T23:00:26.406 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:26.407 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:26.407 DEBUG:boto:Method: HEAD 2023-09-29T23:00:26.407 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:26.407 DEBUG:boto:Data: 2023-09-29T23:00:26.407 DEBUG:boto:Headers: {} 2023-09-29T23:00:26.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.407 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.407 DEBUG:boto:Params: {} 2023-09-29T23:00:26.408 DEBUG:boto:Token: None 2023-09-29T23:00:26.408 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/ 2023-09-29T23:00:26.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0= 2023-09-29T23:00:26.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0=', 'Content-Length': '0'} 2023-09-29T23:00:26.412 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000000f9d0901df4150-006517570a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.413 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:00:26.413 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:00:26.413 DEBUG:boto:Method: PUT 2023-09-29T23:00:26.413 DEBUG:boto:Path: /hcnphz-62/obj11 2023-09-29T23:00:26.413 DEBUG:boto:Data: 2023-09-29T23:00:26.413 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:00:26.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.413 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.413 DEBUG:boto:Params: {} 2023-09-29T23:00:26.414 DEBUG:boto:Token: None 2023-09-29T23:00:26.414 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/obj11 2023-09-29T23:00:26.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TphSl2UmtQNAUQLK1JxTmKn30nw= 2023-09-29T23:00:26.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TphSl2UmtQNAUQLK1JxTmKn30nw='} 2023-09-29T23:00:26.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a1505e29416ce581-006517570a-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.463 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:26.463 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:26.463 DEBUG:boto:Method: HEAD 2023-09-29T23:00:26.463 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:26.463 DEBUG:boto:Data: 2023-09-29T23:00:26.463 DEBUG:boto:Headers: {} 2023-09-29T23:00:26.464 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.464 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.464 DEBUG:boto:Params: {} 2023-09-29T23:00:26.464 DEBUG:boto:Token: None 2023-09-29T23:00:26.464 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/ 2023-09-29T23:00:26.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0= 2023-09-29T23:00:26.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0=', 'Content-Length': '0'} 2023-09-29T23:00:26.468 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000924b89b1714e0369-006517570a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.468 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:00:26.469 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:00:26.469 DEBUG:boto:Method: PUT 2023-09-29T23:00:26.469 DEBUG:boto:Path: /hcnphz-62/obj22 2023-09-29T23:00:26.469 DEBUG:boto:Data: 2023-09-29T23:00:26.469 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:00:26.469 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.469 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.469 DEBUG:boto:Params: {} 2023-09-29T23:00:26.470 DEBUG:boto:Token: None 2023-09-29T23:00:26.470 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/obj22 2023-09-29T23:00:26.470 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BoyLzVUDho5SZ5Z9U4IKlivUR4c= 2023-09-29T23:00:26.470 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BoyLzVUDho5SZ5Z9U4IKlivUR4c='} 2023-09-29T23:00:26.479 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c3a0812f2e83bdbc-006517570a-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.479 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:26.479 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:26.479 DEBUG:boto:Method: HEAD 2023-09-29T23:00:26.479 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:26.479 DEBUG:boto:Data: 2023-09-29T23:00:26.479 DEBUG:boto:Headers: {} 2023-09-29T23:00:26.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.480 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.480 DEBUG:boto:Params: {} 2023-09-29T23:00:26.480 DEBUG:boto:Token: None 2023-09-29T23:00:26.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/ 2023-09-29T23:00:26.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0= 2023-09-29T23:00:26.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0=', 'Content-Length': '0'} 2023-09-29T23:00:26.484 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004e9d246c23112273-006517570a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.484 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:00:26.484 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:00:26.484 DEBUG:boto:Method: PUT 2023-09-29T23:00:26.485 DEBUG:boto:Path: /hcnphz-62/obj33 2023-09-29T23:00:26.485 DEBUG:boto:Data: 2023-09-29T23:00:26.485 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:00:26.485 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.485 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.485 DEBUG:boto:Params: {} 2023-09-29T23:00:26.485 DEBUG:boto:Token: None 2023-09-29T23:00:26.485 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/obj33 2023-09-29T23:00:26.486 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nY1rWTUx4biB09FtaGP0P2zUXb4= 2023-09-29T23:00:26.486 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nY1rWTUx4biB09FtaGP0P2zUXb4='} 2023-09-29T23:00:26.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6c098ea41efb077-006517570a-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.535 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:26.535 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:26.535 DEBUG:boto:Method: HEAD 2023-09-29T23:00:26.535 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:26.535 DEBUG:boto:Data: 2023-09-29T23:00:26.535 DEBUG:boto:Headers: {} 2023-09-29T23:00:26.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.536 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.536 DEBUG:boto:Params: {} 2023-09-29T23:00:26.536 DEBUG:boto:Token: None 2023-09-29T23:00:26.536 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/ 2023-09-29T23:00:26.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0= 2023-09-29T23:00:26.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0=', 'Content-Length': '0'} 2023-09-29T23:00:26.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b3b6046616e9785-006517570a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.541 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:00:26.541 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:00:26.541 DEBUG:boto:Method: PUT 2023-09-29T23:00:26.541 DEBUG:boto:Path: /hcnphz-62/obj44 2023-09-29T23:00:26.542 DEBUG:boto:Data: 2023-09-29T23:00:26.542 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:00:26.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.542 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.542 DEBUG:boto:Params: {} 2023-09-29T23:00:26.542 DEBUG:boto:Token: None 2023-09-29T23:00:26.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/obj44 2023-09-29T23:00:26.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5n58bPbRddiXkQpfwiWjSg/aSUs= 2023-09-29T23:00:26.543 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5n58bPbRddiXkQpfwiWjSg/aSUs='} 2023-09-29T23:00:26.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005f04e52ef8b2b4a5-006517570a-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.554 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:00:26.554 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:00:26.555 DEBUG:boto:Method: HEAD 2023-09-29T23:00:26.555 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:00:26.555 DEBUG:boto:Data: 2023-09-29T23:00:26.555 DEBUG:boto:Headers: {} 2023-09-29T23:00:26.555 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.555 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.555 DEBUG:boto:Params: {} 2023-09-29T23:00:26.555 DEBUG:boto:Token: None 2023-09-29T23:00:26.556 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/ 2023-09-29T23:00:26.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0= 2023-09-29T23:00:26.556 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SVJzA587lIbZeeIItOoUW+YubL0=', 'Content-Length': '0'} 2023-09-29T23:00:26.560 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000059da574433c6aa10-006517570a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.560 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:00:26.560 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:00:26.560 DEBUG:boto:Method: PUT 2023-09-29T23:00:26.560 DEBUG:boto:Path: /hcnphz-62/obj55 2023-09-29T23:00:26.560 DEBUG:boto:Data: 2023-09-29T23:00:26.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:00:26.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:00:26.561 DEBUG:boto:Port: 8000 2023-09-29T23:00:26.561 DEBUG:boto:Params: {} 2023-09-29T23:00:26.561 DEBUG:boto:Token: None 2023-09-29T23:00:26.561 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:00:26 GMT /hcnphz-62/obj55 2023-09-29T23:00:26.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:r7OhTSJ5LLT2Rz383+8hkA1GRTQ= 2023-09-29T23:00:26.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:00:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r7OhTSJ5LLT2Rz383+8hkA1GRTQ='} 2023-09-29T23:00:26.607 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003f33324c778a73bc-006517570a-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:00:26.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:00:26.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:00:26.608 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:00:26.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:26.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:26.715 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:26.716+0000 7fb2cbb13f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-29T23:00:26.718 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:26.720+0000 7fb2cbb13f40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-29T23:00:26.758 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T23:00:26.759 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T23:00:26.761 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T23:00:26.761 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:26.777 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:00:26.777 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:00:26.777 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T23:00:26.777 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T23:00:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T23:00:27.779 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T23:00:27.795 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:00:27.795 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:00:27.796 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-09-29T23:00:27.797 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T23:00:27.797 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-62 2023-09-29T23:00:27.797 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:00:27.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:00:27.798 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-62 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:00:27.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:00:27.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:00:27.910 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:27.912+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:27.911 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:27.911 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:00:30.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T23:00:32.913 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:32.912+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:32.913 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:32.913 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:00:37.915 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:37.916+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:37.915 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:37.915 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:00:42.918 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:42.915+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:42.918 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:42.918 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:00:47.920 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:47.919+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:47.920 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:47.920 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:00:52.922 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:52.923+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:52.923 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:52.923 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:00:57.925 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:00:57.923+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:00:57.925 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:00:57.925 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:02.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:02.927+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:02.927 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:02.927 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:07.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:07.927+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:07.930 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:07.930 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:12.933 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:12.931+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:12.934 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:12.934 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:17.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:17.935+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:17.934 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:17.935 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:22.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:22.934+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:22.937 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:22.937 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:27.939 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:27.938+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:27.940 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:27.940 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:32.942 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:32.938+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:32.942 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:32.942 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:37.944 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:37.942+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:37.944 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:37.944 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:42.946 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:42.946+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:42.947 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:42.947 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:47.948 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:47.946+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:47.949 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:47.949 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:52.951 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:52.950+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:52.952 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , , , ] 2023-09-29T23:01:52.952 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:01:57.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:01:57.949+0000 7f994a161f40 1 waiting for incremental sync to catch up: 2023-09-29T23:01:57.953 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000001.59.6, 00000000002.57.6] 2023-09-29T23:01:57.953 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:02:02.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:02.953+0000 7f994a161f40 1 bucket sync caught up with source: 2023-09-29T23:02:02.956 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:02:02.956 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.58.6, , , 00000000001.137.6, 00000000002.61.6, 00000000002.57.6] 2023-09-29T23:02:02.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:02.953+0000 7f994a161f40 0 bucket checkpoint complete 2023-09-29T23:02:02.987 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-62 2023-09-29T23:02:02.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:02:02.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-62', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:02:02.988 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-62 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:02:03.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:03.041 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:03.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:03.105+0000 7f877970bf40 1 waiting for incremental sync to catch up: 2023-09-29T23:02:03.107 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.155.6, , , , , ] 2023-09-29T23:02:03.107 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.155.6, , , 00000000001.635.6, 00000000002.161.6, 00000000002.159.6] 2023-09-29T23:02:08.109 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:08.105+0000 7f877970bf40 1 waiting for incremental sync to catch up: 2023-09-29T23:02:08.109 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.155.6, , , 00000000001.635.6, , ] 2023-09-29T23:02:08.109 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.155.6, , , 00000000001.635.6, 00000000002.161.6, 00000000002.159.6] 2023-09-29T23:02:13.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:13.113+0000 7f877970bf40 1 waiting for incremental sync to catch up: 2023-09-29T23:02:13.111 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.155.6, , , 00000000001.635.6, , ] 2023-09-29T23:02:13.111 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.155.6, , , 00000000001.635.6, 00000000002.161.6, 00000000002.159.6] 2023-09-29T23:02:18.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:18.113+0000 7f877970bf40 1 bucket sync caught up with source: 2023-09-29T23:02:18.114 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , 00000000001.155.6, , , 00000000001.635.6, 00000000002.161.6, 00000000002.159.6] 2023-09-29T23:02:18.114 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , 00000000001.155.6, , , 00000000001.635.6, 00000000002.161.6, 00000000002.159.6] 2023-09-29T23:02:18.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:18.113+0000 7f877970bf40 0 bucket checkpoint complete 2023-09-29T23:02:18.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-62 zones={a2, a1} 2023-09-29T23:02:18.143 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.143 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.143 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.143 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:02:18.144 DEBUG:boto:Data: 2023-09-29T23:02:18.144 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.144 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.144 DEBUG:boto:Params: {} 2023-09-29T23:02:18.144 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:02:18.144 DEBUG:boto:Token: None 2023-09-29T23:02:18.145 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NN+TioTFRYqOnlHBzG+BjVW9+JE= 2023-09-29T23:02:18.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NN+TioTFRYqOnlHBzG+BjVW9+JE=', 'Content-Length': '0'} 2023-09-29T23:02:18.154 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001d71c8903c864b1f-006517577a-1ef7-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.154 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.155 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.155 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.155 DEBUG:boto:Path: /hcnphz-62/ 2023-09-29T23:02:18.155 DEBUG:boto:Data: 2023-09-29T23:02:18.155 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.155 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.155 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.155 DEBUG:boto:Params: {} 2023-09-29T23:02:18.156 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:02:18.156 DEBUG:boto:Token: None 2023-09-29T23:02:18.156 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.156 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NN+TioTFRYqOnlHBzG+BjVW9+JE= 2023-09-29T23:02:18.156 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NN+TioTFRYqOnlHBzG+BjVW9+JE=', 'Content-Length': '0'} 2023-09-29T23:02:18.162 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000011020d6d77773d3f-006517577a-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.162 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T23:02:18.162 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.162 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.162 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.163 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.163 DEBUG:boto:Method: GET 2023-09-29T23:02:18.163 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.163 DEBUG:boto:Data: 2023-09-29T23:02:18.163 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.163 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.163 DEBUG:boto:Params: {} 2023-09-29T23:02:18.163 DEBUG:boto:Token: None 2023-09-29T23:02:18.164 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/?versions 2023-09-29T23:02:18.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4= 2023-09-29T23:02:18.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4=', 'Content-Length': '0'} 2023-09-29T23:02:18.168 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7b2e9c8075ba504-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.168 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.169 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T23:02:18.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T23:02:18.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T23:02:18.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T23:02:18.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T23:02:18.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T23:02:18.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T23:02:18.170 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.170 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.170 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.170 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.171 DEBUG:boto:Method: GET 2023-09-29T23:02:18.171 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.171 DEBUG:boto:Data: 2023-09-29T23:02:18.171 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.171 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.171 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.171 DEBUG:boto:Params: {} 2023-09-29T23:02:18.171 DEBUG:boto:Token: None 2023-09-29T23:02:18.171 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/?versions 2023-09-29T23:02:18.171 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4= 2023-09-29T23:02:18.172 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4=', 'Content-Length': '0'} 2023-09-29T23:02:18.176 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b010fec3974a70cc-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.177 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.177 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2023-09-29T23:02:18.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T23:02:18.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T23:02:18.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T23:02:18.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T23:02:18.178 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T23:02:18.178 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.178 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.178 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.178 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.178 DEBUG:boto:Method: GET 2023-09-29T23:02:18.178 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.179 DEBUG:boto:Data: 2023-09-29T23:02:18.179 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.179 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.179 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.179 DEBUG:boto:Params: {} 2023-09-29T23:02:18.179 DEBUG:boto:Token: None 2023-09-29T23:02:18.179 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/?versions 2023-09-29T23:02:18.180 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4= 2023-09-29T23:02:18.180 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4=', 'Content-Length': '0'} 2023-09-29T23:02:18.184 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000016c13a8274cc0a36-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.184 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.184 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.185 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.185 DEBUG:boto:path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.185 DEBUG:boto:auth_path=/hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.185 DEBUG:boto:Method: GET 2023-09-29T23:02:18.185 DEBUG:boto:Path: /hcnphz-62/?versions&max-keys=999 2023-09-29T23:02:18.185 DEBUG:boto:Data: 2023-09-29T23:02:18.185 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.186 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.186 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.186 DEBUG:boto:Params: {} 2023-09-29T23:02:18.186 DEBUG:boto:Token: None 2023-09-29T23:02:18.186 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/?versions 2023-09-29T23:02:18.186 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4= 2023-09-29T23:02:18.186 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w8gxeu6qjxTbs8XJbGk31B5t8i4=', 'Content-Length': '0'} 2023-09-29T23:02:18.189 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000daa75303d3dc7bed-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.189 DEBUG:boto:b'hcnphz-62999falsedummynulltrue2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj11nulltrue2023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T23:02:18.190 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.190 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.190 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.190 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.190 DEBUG:boto:Method: GET 2023-09-29T23:02:18.190 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.190 DEBUG:boto:Data: 2023-09-29T23:02:18.190 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.191 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.191 DEBUG:boto:Params: {} 2023-09-29T23:02:18.191 DEBUG:boto:Token: None 2023-09-29T23:02:18.191 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.191 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU= 2023-09-29T23:02:18.191 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU=', 'Content-Length': '0'} 2023-09-29T23:02:18.193 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e8609a189ad2033-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.238 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.239 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.239 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.239 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.239 DEBUG:boto:Method: GET 2023-09-29T23:02:18.240 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.240 DEBUG:boto:Data: 2023-09-29T23:02:18.240 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.240 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.240 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.240 DEBUG:boto:Params: {} 2023-09-29T23:02:18.241 DEBUG:boto:Token: None 2023-09-29T23:02:18.241 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.241 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU= 2023-09-29T23:02:18.242 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU=', 'Content-Length': '0'} 2023-09-29T23:02:18.245 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f15dbe6ebd989c7-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.245 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.245 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.246 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.246 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.246 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.246 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.246 DEBUG:boto:Data: 2023-09-29T23:02:18.246 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.246 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.247 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.247 DEBUG:boto:Params: {} 2023-09-29T23:02:18.247 DEBUG:boto:Token: None 2023-09-29T23:02:18.247 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.247 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kvTXR3iLPx0YbZ1xexvtn2auhcA= 2023-09-29T23:02:18.247 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kvTXR3iLPx0YbZ1xexvtn2auhcA=', 'Content-Length': '0'} 2023-09-29T23:02:18.250 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f499e75178b61e-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.250 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.250 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.250 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.250 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.251 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.251 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.251 DEBUG:boto:Data: 2023-09-29T23:02:18.251 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.251 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.251 DEBUG:boto:Params: {} 2023-09-29T23:02:18.251 DEBUG:boto:Token: None 2023-09-29T23:02:18.251 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:kvTXR3iLPx0YbZ1xexvtn2auhcA= 2023-09-29T23:02:18.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kvTXR3iLPx0YbZ1xexvtn2auhcA=', 'Content-Length': '0'} 2023-09-29T23:02:18.256 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000aac8059fb8b4faf8-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T23:02:18.256 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.256 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.257 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.257 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.257 DEBUG:boto:Method: GET 2023-09-29T23:02:18.257 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.257 DEBUG:boto:Data: 2023-09-29T23:02:18.257 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.257 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.257 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.257 DEBUG:boto:Params: {} 2023-09-29T23:02:18.258 DEBUG:boto:Token: None 2023-09-29T23:02:18.258 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.258 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU= 2023-09-29T23:02:18.258 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU=', 'Content-Length': '0'} 2023-09-29T23:02:18.261 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b81718e7886e736f-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.261 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.262 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.262 DEBUG:boto:path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.262 DEBUG:boto:auth_path=/hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.262 DEBUG:boto:Method: GET 2023-09-29T23:02:18.262 DEBUG:boto:Path: /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.263 DEBUG:boto:Data: 2023-09-29T23:02:18.263 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.263 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.263 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.263 DEBUG:boto:Params: {} 2023-09-29T23:02:18.263 DEBUG:boto:Token: None 2023-09-29T23:02:18.263 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy?versionId=null 2023-09-29T23:02:18.263 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU= 2023-09-29T23:02:18.264 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jHVNl0nVHVmvIwrI31Z9kn5/5VU=', 'Content-Length': '0'} 2023-09-29T23:02:18.266 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a4e39787a056d9fd-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.267 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.267 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.267 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.267 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.267 DEBUG:boto:Method: GET 2023-09-29T23:02:18.267 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.268 DEBUG:boto:Data: 2023-09-29T23:02:18.268 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.268 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.268 DEBUG:boto:Params: {} 2023-09-29T23:02:18.269 DEBUG:boto:Token: None 2023-09-29T23:02:18.269 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.269 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.269 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.274 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000696554dd51b9533b-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.274 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.275 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.275 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.275 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.275 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.275 DEBUG:boto:Method: GET 2023-09-29T23:02:18.275 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.275 DEBUG:boto:Data: 2023-09-29T23:02:18.275 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.275 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.276 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.276 DEBUG:boto:Params: {} 2023-09-29T23:02:18.276 DEBUG:boto:Token: None 2023-09-29T23:02:18.276 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.276 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.276 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.281 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b19bbd870903a164-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.322 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.323 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.323 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.323 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.323 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:02:18.323 DEBUG:boto:Data: 2023-09-29T23:02:18.324 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.324 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.324 DEBUG:boto:Params: {} 2023-09-29T23:02:18.324 DEBUG:boto:Token: None 2023-09-29T23:02:18.324 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy 2023-09-29T23:02:18.324 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ANpAzh3YdM64LubIPljiuc1+lc8= 2023-09-29T23:02:18.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ANpAzh3YdM64LubIPljiuc1+lc8=', 'Content-Length': '0'} 2023-09-29T23:02:18.327 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b78de2c02a0ed69d-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.327 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.328 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.328 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.328 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:02:18.328 DEBUG:boto:Data: 2023-09-29T23:02:18.328 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.328 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.328 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.328 DEBUG:boto:Params: {} 2023-09-29T23:02:18.329 DEBUG:boto:Token: None 2023-09-29T23:02:18.329 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy 2023-09-29T23:02:18.329 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ANpAzh3YdM64LubIPljiuc1+lc8= 2023-09-29T23:02:18.329 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ANpAzh3YdM64LubIPljiuc1+lc8=', 'Content-Length': '0'} 2023-09-29T23:02:18.332 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000002e192d243562b4e-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2023-09-29T23:02:18.333 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.333 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.333 DEBUG:boto:Method: GET 2023-09-29T23:02:18.333 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:02:18.333 DEBUG:boto:Data: 2023-09-29T23:02:18.333 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.334 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.334 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.334 DEBUG:boto:Params: {} 2023-09-29T23:02:18.334 DEBUG:boto:Token: None 2023-09-29T23:02:18.334 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy 2023-09-29T23:02:18.334 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a6WCEULiIh+MvD977omSE6VmTMw= 2023-09-29T23:02:18.334 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a6WCEULiIh+MvD977omSE6VmTMw=', 'Content-Length': '0'} 2023-09-29T23:02:18.337 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bf9587b965aff9d3-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.337 DEBUG:boto:path=/hcnphz-62/dummy 2023-09-29T23:02:18.338 DEBUG:boto:auth_path=/hcnphz-62/dummy 2023-09-29T23:02:18.338 DEBUG:boto:Method: GET 2023-09-29T23:02:18.338 DEBUG:boto:Path: /hcnphz-62/dummy 2023-09-29T23:02:18.338 DEBUG:boto:Data: 2023-09-29T23:02:18.338 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.338 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.338 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.338 DEBUG:boto:Params: {} 2023-09-29T23:02:18.339 DEBUG:boto:Token: None 2023-09-29T23:02:18.339 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/dummy 2023-09-29T23:02:18.339 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a6WCEULiIh+MvD977omSE6VmTMw= 2023-09-29T23:02:18.339 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a6WCEULiIh+MvD977omSE6VmTMw=', 'Content-Length': '0'} 2023-09-29T23:02:18.342 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:59:59 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040363a9d4fac90d7-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.342 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.342 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.342 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.342 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.343 DEBUG:boto:Method: GET 2023-09-29T23:02:18.343 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.343 DEBUG:boto:Data: 2023-09-29T23:02:18.343 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.343 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.343 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.343 DEBUG:boto:Params: {} 2023-09-29T23:02:18.343 DEBUG:boto:Token: None 2023-09-29T23:02:18.343 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.344 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.344 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.348 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000450ce1258cf092c3-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.348 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.349 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.349 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.349 DEBUG:boto:path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.349 DEBUG:boto:auth_path=/hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.349 DEBUG:boto:Method: GET 2023-09-29T23:02:18.350 DEBUG:boto:Path: /hcnphz-62/?prefix=dummy 2023-09-29T23:02:18.350 DEBUG:boto:Data: 2023-09-29T23:02:18.350 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.350 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.350 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.350 DEBUG:boto:Params: {} 2023-09-29T23:02:18.350 DEBUG:boto:Token: None 2023-09-29T23:02:18.350 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.351 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.351 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.356 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fc489b9a94a120b8-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.356 DEBUG:boto:b'hcnphz-62dummy1000falsedummy2023-09-29T22:59:59.708Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T23:02:18.356 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.357 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.357 DEBUG:boto:path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.357 DEBUG:boto:auth_path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.357 DEBUG:boto:Method: GET 2023-09-29T23:02:18.357 DEBUG:boto:Path: /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.357 DEBUG:boto:Data: 2023-09-29T23:02:18.357 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.357 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.358 DEBUG:boto:Params: {} 2023-09-29T23:02:18.358 DEBUG:boto:Token: None 2023-09-29T23:02:18.358 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.358 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU= 2023-09-29T23:02:18.358 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU=', 'Content-Length': '0'} 2023-09-29T23:02:18.361 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000543ce64983738abf-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.406 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.406 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.407 DEBUG:boto:path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.407 DEBUG:boto:auth_path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.407 DEBUG:boto:Method: GET 2023-09-29T23:02:18.407 DEBUG:boto:Path: /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.407 DEBUG:boto:Data: 2023-09-29T23:02:18.407 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.407 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.407 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.408 DEBUG:boto:Params: {} 2023-09-29T23:02:18.408 DEBUG:boto:Token: None 2023-09-29T23:02:18.408 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.408 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU= 2023-09-29T23:02:18.408 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU=', 'Content-Length': '0'} 2023-09-29T23:02:18.411 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fffcabfccb1006d8-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.412 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.412 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.412 DEBUG:boto:path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.412 DEBUG:boto:auth_path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.412 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.412 DEBUG:boto:Path: /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.412 DEBUG:boto:Data: 2023-09-29T23:02:18.413 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.413 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.413 DEBUG:boto:Params: {} 2023-09-29T23:02:18.413 DEBUG:boto:Token: None 2023-09-29T23:02:18.413 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lOGmXkCztoDxUy1HOHC9jvgVJzU= 2023-09-29T23:02:18.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lOGmXkCztoDxUy1HOHC9jvgVJzU=', 'Content-Length': '0'} 2023-09-29T23:02:18.416 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000da57ff1524fb2d2b-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.416 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.417 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.417 DEBUG:boto:path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.417 DEBUG:boto:auth_path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.417 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.417 DEBUG:boto:Path: /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.417 DEBUG:boto:Data: 2023-09-29T23:02:18.417 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.418 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.418 DEBUG:boto:Params: {} 2023-09-29T23:02:18.418 DEBUG:boto:Token: None 2023-09-29T23:02:18.418 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.418 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lOGmXkCztoDxUy1HOHC9jvgVJzU= 2023-09-29T23:02:18.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lOGmXkCztoDxUy1HOHC9jvgVJzU=', 'Content-Length': '0'} 2023-09-29T23:02:18.421 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ccdb607af9cc372c-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T23:02:18.421 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.421 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.421 DEBUG:boto:path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.422 DEBUG:boto:auth_path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.422 DEBUG:boto:Method: GET 2023-09-29T23:02:18.422 DEBUG:boto:Path: /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.422 DEBUG:boto:Data: 2023-09-29T23:02:18.422 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.422 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.422 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.422 DEBUG:boto:Params: {} 2023-09-29T23:02:18.423 DEBUG:boto:Token: None 2023-09-29T23:02:18.423 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.423 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU= 2023-09-29T23:02:18.423 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU=', 'Content-Length': '0'} 2023-09-29T23:02:18.426 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d9cf3c6601d6ae7-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.426 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.427 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.427 DEBUG:boto:path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.427 DEBUG:boto:auth_path=/hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.427 DEBUG:boto:Method: GET 2023-09-29T23:02:18.427 DEBUG:boto:Path: /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.427 DEBUG:boto:Data: 2023-09-29T23:02:18.427 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.427 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.428 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.428 DEBUG:boto:Params: {} 2023-09-29T23:02:18.428 DEBUG:boto:Token: None 2023-09-29T23:02:18.428 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11?versionId=null 2023-09-29T23:02:18.428 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU= 2023-09-29T23:02:18.428 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:juYHv3ZJCPjDYfPNwa6CLcLGGkU=', 'Content-Length': '0'} 2023-09-29T23:02:18.431 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008be8fd162addc569-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.474 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.475 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.475 DEBUG:boto:path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.475 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.475 DEBUG:boto:Method: GET 2023-09-29T23:02:18.476 DEBUG:boto:Path: /hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.476 DEBUG:boto:Data: 2023-09-29T23:02:18.476 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.476 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.477 DEBUG:boto:Params: {} 2023-09-29T23:02:18.477 DEBUG:boto:Token: None 2023-09-29T23:02:18.477 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.478 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.478 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.483 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000045461269e1ae9ca5-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.483 DEBUG:boto:b'hcnphz-62obj111000falseobj112023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.484 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.484 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.484 DEBUG:boto:path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.485 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.485 DEBUG:boto:Method: GET 2023-09-29T23:02:18.485 DEBUG:boto:Path: /hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.485 DEBUG:boto:Data: 2023-09-29T23:02:18.485 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.486 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.486 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.486 DEBUG:boto:Params: {} 2023-09-29T23:02:18.486 DEBUG:boto:Token: None 2023-09-29T23:02:18.487 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.487 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.487 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.493 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028a995448867e04e-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.493 DEBUG:boto:b'hcnphz-62obj111000falseobj112023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.493 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.494 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.494 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.494 DEBUG:boto:Path: /hcnphz-62/obj11 2023-09-29T23:02:18.494 DEBUG:boto:Data: 2023-09-29T23:02:18.494 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.495 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.495 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.495 DEBUG:boto:Params: {} 2023-09-29T23:02:18.495 DEBUG:boto:Token: None 2023-09-29T23:02:18.495 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11 2023-09-29T23:02:18.496 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6VPG1F/hKXaj7IGbqaaYUxRxe/E= 2023-09-29T23:02:18.496 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6VPG1F/hKXaj7IGbqaaYUxRxe/E=', 'Content-Length': '0'} 2023-09-29T23:02:18.499 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002a554c4362af44bb-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.499 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.500 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.500 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.500 DEBUG:boto:Path: /hcnphz-62/obj11 2023-09-29T23:02:18.500 DEBUG:boto:Data: 2023-09-29T23:02:18.501 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.501 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.501 DEBUG:boto:Params: {} 2023-09-29T23:02:18.502 DEBUG:boto:Token: None 2023-09-29T23:02:18.502 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11 2023-09-29T23:02:18.502 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6VPG1F/hKXaj7IGbqaaYUxRxe/E= 2023-09-29T23:02:18.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6VPG1F/hKXaj7IGbqaaYUxRxe/E=', 'Content-Length': '0'} 2023-09-29T23:02:18.505 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009518406681f9759a-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T23:02:18.506 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.506 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.506 DEBUG:boto:Method: GET 2023-09-29T23:02:18.506 DEBUG:boto:Path: /hcnphz-62/obj11 2023-09-29T23:02:18.507 DEBUG:boto:Data: 2023-09-29T23:02:18.507 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.507 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.507 DEBUG:boto:Params: {} 2023-09-29T23:02:18.507 DEBUG:boto:Token: None 2023-09-29T23:02:18.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11 2023-09-29T23:02:18.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g6kfqXWFXMLQcxoS1YQyMmOSl9c= 2023-09-29T23:02:18.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g6kfqXWFXMLQcxoS1YQyMmOSl9c=', 'Content-Length': '0'} 2023-09-29T23:02:18.511 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000d816aa420fb0ee8-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.554 DEBUG:boto:path=/hcnphz-62/obj11 2023-09-29T23:02:18.555 DEBUG:boto:auth_path=/hcnphz-62/obj11 2023-09-29T23:02:18.555 DEBUG:boto:Method: GET 2023-09-29T23:02:18.555 DEBUG:boto:Path: /hcnphz-62/obj11 2023-09-29T23:02:18.555 DEBUG:boto:Data: 2023-09-29T23:02:18.555 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.556 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.556 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.556 DEBUG:boto:Params: {} 2023-09-29T23:02:18.556 DEBUG:boto:Token: None 2023-09-29T23:02:18.556 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj11 2023-09-29T23:02:18.556 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:g6kfqXWFXMLQcxoS1YQyMmOSl9c= 2023-09-29T23:02:18.557 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:g6kfqXWFXMLQcxoS1YQyMmOSl9c=', 'Content-Length': '0'} 2023-09-29T23:02:18.560 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dfe4717d9e1fbe91-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.560 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.560 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.560 DEBUG:boto:path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.561 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.561 DEBUG:boto:Method: GET 2023-09-29T23:02:18.561 DEBUG:boto:Path: /hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.561 DEBUG:boto:Data: 2023-09-29T23:02:18.561 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.561 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.561 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.562 DEBUG:boto:Params: {} 2023-09-29T23:02:18.562 DEBUG:boto:Token: None 2023-09-29T23:02:18.562 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.562 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.562 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.568 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d252426763c590a4-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.568 DEBUG:boto:b'hcnphz-62obj111000falseobj112023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.569 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.569 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.569 DEBUG:boto:path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.569 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.569 DEBUG:boto:Method: GET 2023-09-29T23:02:18.569 DEBUG:boto:Path: /hcnphz-62/?prefix=obj11 2023-09-29T23:02:18.570 DEBUG:boto:Data: 2023-09-29T23:02:18.570 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.570 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.570 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.570 DEBUG:boto:Params: {} 2023-09-29T23:02:18.571 DEBUG:boto:Token: None 2023-09-29T23:02:18.571 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.571 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.571 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.575 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000681fe7a96cc6668b-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.618 DEBUG:boto:b'hcnphz-62obj111000falseobj112023-09-29T23:00:26.419Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T23:02:18.619 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.619 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.620 DEBUG:boto:path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.620 DEBUG:boto:auth_path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.620 DEBUG:boto:Method: GET 2023-09-29T23:02:18.620 DEBUG:boto:Path: /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.620 DEBUG:boto:Data: 2023-09-29T23:02:18.621 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.621 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.621 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.621 DEBUG:boto:Params: {} 2023-09-29T23:02:18.621 DEBUG:boto:Token: None 2023-09-29T23:02:18.622 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.622 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM= 2023-09-29T23:02:18.622 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM=', 'Content-Length': '0'} 2023-09-29T23:02:18.625 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089e8040517221171-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.625 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.626 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.626 DEBUG:boto:path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.626 DEBUG:boto:auth_path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.626 DEBUG:boto:Method: GET 2023-09-29T23:02:18.626 DEBUG:boto:Path: /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.626 DEBUG:boto:Data: 2023-09-29T23:02:18.627 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.627 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.627 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.627 DEBUG:boto:Params: {} 2023-09-29T23:02:18.628 DEBUG:boto:Token: None 2023-09-29T23:02:18.628 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.628 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM= 2023-09-29T23:02:18.629 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM=', 'Content-Length': '0'} 2023-09-29T23:02:18.631 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f9372aa144c6c8f3-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.632 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.632 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.632 DEBUG:boto:path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.632 DEBUG:boto:auth_path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.632 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.632 DEBUG:boto:Path: /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.632 DEBUG:boto:Data: 2023-09-29T23:02:18.632 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.633 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.633 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.633 DEBUG:boto:Params: {} 2023-09-29T23:02:18.633 DEBUG:boto:Token: None 2023-09-29T23:02:18.633 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.633 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a0MpOpKHRp0yGCpnKcK2sPggjoo= 2023-09-29T23:02:18.633 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a0MpOpKHRp0yGCpnKcK2sPggjoo=', 'Content-Length': '0'} 2023-09-29T23:02:18.636 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000460df03be1f73226-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.636 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.636 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.636 DEBUG:boto:path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.636 DEBUG:boto:auth_path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.637 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.637 DEBUG:boto:Path: /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.637 DEBUG:boto:Data: 2023-09-29T23:02:18.637 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.637 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.637 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.637 DEBUG:boto:Params: {} 2023-09-29T23:02:18.638 DEBUG:boto:Token: None 2023-09-29T23:02:18.638 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.638 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:a0MpOpKHRp0yGCpnKcK2sPggjoo= 2023-09-29T23:02:18.638 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:a0MpOpKHRp0yGCpnKcK2sPggjoo=', 'Content-Length': '0'} 2023-09-29T23:02:18.641 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cba16c2d7707ef03-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T23:02:18.642 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.642 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.642 DEBUG:boto:path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.642 DEBUG:boto:auth_path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.642 DEBUG:boto:Method: GET 2023-09-29T23:02:18.642 DEBUG:boto:Path: /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.643 DEBUG:boto:Data: 2023-09-29T23:02:18.643 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.643 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.643 DEBUG:boto:Params: {} 2023-09-29T23:02:18.643 DEBUG:boto:Token: None 2023-09-29T23:02:18.643 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM= 2023-09-29T23:02:18.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM=', 'Content-Length': '0'} 2023-09-29T23:02:18.647 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007025f6aad83489b5-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.647 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.648 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.648 DEBUG:boto:path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.648 DEBUG:boto:auth_path=/hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.648 DEBUG:boto:Method: GET 2023-09-29T23:02:18.649 DEBUG:boto:Path: /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.649 DEBUG:boto:Data: 2023-09-29T23:02:18.649 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.649 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.649 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.649 DEBUG:boto:Params: {} 2023-09-29T23:02:18.650 DEBUG:boto:Token: None 2023-09-29T23:02:18.650 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22?versionId=null 2023-09-29T23:02:18.650 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM= 2023-09-29T23:02:18.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AS1Tt8SJGgLifLzHa56swLJmToM=', 'Content-Length': '0'} 2023-09-29T23:02:18.654 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006929e32c555ce2d4-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.655 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.655 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.655 DEBUG:boto:path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.655 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.655 DEBUG:boto:Method: GET 2023-09-29T23:02:18.655 DEBUG:boto:Path: /hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.655 DEBUG:boto:Data: 2023-09-29T23:02:18.655 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.656 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.656 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.656 DEBUG:boto:Params: {} 2023-09-29T23:02:18.656 DEBUG:boto:Token: None 2023-09-29T23:02:18.656 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.656 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.656 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.661 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013a368874bb9e376-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.702 DEBUG:boto:b'hcnphz-62obj221000falseobj222023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.703 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.703 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.703 DEBUG:boto:path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.703 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.703 DEBUG:boto:Method: GET 2023-09-29T23:02:18.704 DEBUG:boto:Path: /hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.704 DEBUG:boto:Data: 2023-09-29T23:02:18.704 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.704 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.704 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.704 DEBUG:boto:Params: {} 2023-09-29T23:02:18.704 DEBUG:boto:Token: None 2023-09-29T23:02:18.705 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.705 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.705 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.711 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000668e79b28d773689-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.712 DEBUG:boto:b'hcnphz-62obj221000falseobj222023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.712 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.712 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.712 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.712 DEBUG:boto:Path: /hcnphz-62/obj22 2023-09-29T23:02:18.713 DEBUG:boto:Data: 2023-09-29T23:02:18.713 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.713 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.713 DEBUG:boto:Params: {} 2023-09-29T23:02:18.713 DEBUG:boto:Token: None 2023-09-29T23:02:18.713 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22 2023-09-29T23:02:18.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5UrYUG/c1PyRGfxwXVTIpCRMhug= 2023-09-29T23:02:18.714 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5UrYUG/c1PyRGfxwXVTIpCRMhug=', 'Content-Length': '0'} 2023-09-29T23:02:18.716 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f13a2ea400b330fa-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.717 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.717 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.717 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.717 DEBUG:boto:Path: /hcnphz-62/obj22 2023-09-29T23:02:18.717 DEBUG:boto:Data: 2023-09-29T23:02:18.717 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.718 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.718 DEBUG:boto:Params: {} 2023-09-29T23:02:18.718 DEBUG:boto:Token: None 2023-09-29T23:02:18.718 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22 2023-09-29T23:02:18.718 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5UrYUG/c1PyRGfxwXVTIpCRMhug= 2023-09-29T23:02:18.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5UrYUG/c1PyRGfxwXVTIpCRMhug=', 'Content-Length': '0'} 2023-09-29T23:02:18.721 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001085cf123fa4a572-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.721 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T23:02:18.721 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.721 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.721 DEBUG:boto:Method: GET 2023-09-29T23:02:18.722 DEBUG:boto:Path: /hcnphz-62/obj22 2023-09-29T23:02:18.722 DEBUG:boto:Data: 2023-09-29T23:02:18.722 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.722 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.722 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.722 DEBUG:boto:Params: {} 2023-09-29T23:02:18.722 DEBUG:boto:Token: None 2023-09-29T23:02:18.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22 2023-09-29T23:02:18.723 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bbW+X84h6aSaOmkobUtl72YxUqQ= 2023-09-29T23:02:18.723 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bbW+X84h6aSaOmkobUtl72YxUqQ=', 'Content-Length': '0'} 2023-09-29T23:02:18.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e2dc95177411987b-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.725 DEBUG:boto:path=/hcnphz-62/obj22 2023-09-29T23:02:18.725 DEBUG:boto:auth_path=/hcnphz-62/obj22 2023-09-29T23:02:18.725 DEBUG:boto:Method: GET 2023-09-29T23:02:18.726 DEBUG:boto:Path: /hcnphz-62/obj22 2023-09-29T23:02:18.726 DEBUG:boto:Data: 2023-09-29T23:02:18.726 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.726 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.726 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.726 DEBUG:boto:Params: {} 2023-09-29T23:02:18.726 DEBUG:boto:Token: None 2023-09-29T23:02:18.726 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj22 2023-09-29T23:02:18.727 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bbW+X84h6aSaOmkobUtl72YxUqQ= 2023-09-29T23:02:18.727 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bbW+X84h6aSaOmkobUtl72YxUqQ=', 'Content-Length': '0'} 2023-09-29T23:02:18.729 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005298bd0d88e5e9b8-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.770 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.771 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.771 DEBUG:boto:path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.771 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.771 DEBUG:boto:Method: GET 2023-09-29T23:02:18.772 DEBUG:boto:Path: /hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.772 DEBUG:boto:Data: 2023-09-29T23:02:18.772 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.772 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.773 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.773 DEBUG:boto:Params: {} 2023-09-29T23:02:18.773 DEBUG:boto:Token: None 2023-09-29T23:02:18.773 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.780 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a056d5934866a725-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.780 DEBUG:boto:b'hcnphz-62obj221000falseobj222023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.781 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.781 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.781 DEBUG:boto:path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.782 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.782 DEBUG:boto:Method: GET 2023-09-29T23:02:18.782 DEBUG:boto:Path: /hcnphz-62/?prefix=obj22 2023-09-29T23:02:18.782 DEBUG:boto:Data: 2023-09-29T23:02:18.782 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.783 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.783 DEBUG:boto:Params: {} 2023-09-29T23:02:18.783 DEBUG:boto:Token: None 2023-09-29T23:02:18.783 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.789 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f4ce642860ed7a73-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.789 DEBUG:boto:b'hcnphz-62obj221000falseobj222023-09-29T23:00:26.475Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.789 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T23:02:18.790 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.790 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.790 DEBUG:boto:path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.790 DEBUG:boto:auth_path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.791 DEBUG:boto:Method: GET 2023-09-29T23:02:18.791 DEBUG:boto:Path: /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.791 DEBUG:boto:Data: 2023-09-29T23:02:18.791 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.791 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.792 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.792 DEBUG:boto:Params: {} 2023-09-29T23:02:18.792 DEBUG:boto:Token: None 2023-09-29T23:02:18.792 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.793 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU= 2023-09-29T23:02:18.793 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU=', 'Content-Length': '0'} 2023-09-29T23:02:18.795 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000044696becb8b484a1-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.838 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.839 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.839 DEBUG:boto:path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.839 DEBUG:boto:auth_path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.839 DEBUG:boto:Method: GET 2023-09-29T23:02:18.839 DEBUG:boto:Path: /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.840 DEBUG:boto:Data: 2023-09-29T23:02:18.840 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.840 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.840 DEBUG:boto:Params: {} 2023-09-29T23:02:18.841 DEBUG:boto:Token: None 2023-09-29T23:02:18.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.841 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU= 2023-09-29T23:02:18.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU=', 'Content-Length': '0'} 2023-09-29T23:02:18.844 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c7680387dde2b8b1-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.844 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.845 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.845 DEBUG:boto:path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.845 DEBUG:boto:auth_path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.845 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.845 DEBUG:boto:Path: /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.846 DEBUG:boto:Data: 2023-09-29T23:02:18.846 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.846 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.846 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.846 DEBUG:boto:Params: {} 2023-09-29T23:02:18.846 DEBUG:boto:Token: None 2023-09-29T23:02:18.847 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.847 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pNlCyMUT2TOieT0mq8XHDqQkTzM= 2023-09-29T23:02:18.847 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pNlCyMUT2TOieT0mq8XHDqQkTzM=', 'Content-Length': '0'} 2023-09-29T23:02:18.849 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e75b9f67c311c9dd-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.850 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.850 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.850 DEBUG:boto:path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.850 DEBUG:boto:auth_path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.850 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.850 DEBUG:boto:Path: /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.850 DEBUG:boto:Data: 2023-09-29T23:02:18.851 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.851 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.851 DEBUG:boto:Params: {} 2023-09-29T23:02:18.851 DEBUG:boto:Token: None 2023-09-29T23:02:18.851 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pNlCyMUT2TOieT0mq8XHDqQkTzM= 2023-09-29T23:02:18.851 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pNlCyMUT2TOieT0mq8XHDqQkTzM=', 'Content-Length': '0'} 2023-09-29T23:02:18.854 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006b157b23f23aba7d-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T23:02:18.854 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.854 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.854 DEBUG:boto:path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.854 DEBUG:boto:auth_path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.855 DEBUG:boto:Method: GET 2023-09-29T23:02:18.855 DEBUG:boto:Path: /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.855 DEBUG:boto:Data: 2023-09-29T23:02:18.855 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.855 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.855 DEBUG:boto:Params: {} 2023-09-29T23:02:18.855 DEBUG:boto:Token: None 2023-09-29T23:02:18.856 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU= 2023-09-29T23:02:18.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU=', 'Content-Length': '0'} 2023-09-29T23:02:18.858 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031f29de516ee89b9-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.859 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.859 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.859 DEBUG:boto:path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.859 DEBUG:boto:auth_path=/hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.859 DEBUG:boto:Method: GET 2023-09-29T23:02:18.859 DEBUG:boto:Path: /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.859 DEBUG:boto:Data: 2023-09-29T23:02:18.859 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.860 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.860 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.860 DEBUG:boto:Params: {} 2023-09-29T23:02:18.860 DEBUG:boto:Token: None 2023-09-29T23:02:18.860 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33?versionId=null 2023-09-29T23:02:18.860 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU= 2023-09-29T23:02:18.860 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:c4HCd0u7b/B/5BWAVVVve4s6YBU=', 'Content-Length': '0'} 2023-09-29T23:02:18.863 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1b2a91dca320022-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.863 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.863 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.864 DEBUG:boto:path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.864 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.864 DEBUG:boto:Method: GET 2023-09-29T23:02:18.864 DEBUG:boto:Path: /hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.864 DEBUG:boto:Data: 2023-09-29T23:02:18.864 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.864 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.864 DEBUG:boto:Params: {} 2023-09-29T23:02:18.864 DEBUG:boto:Token: None 2023-09-29T23:02:18.865 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.865 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.870 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013f1cddb003ce869-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.871 DEBUG:boto:b'hcnphz-62obj331000falseobj332023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.871 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.871 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.871 DEBUG:boto:path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.871 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.871 DEBUG:boto:Method: GET 2023-09-29T23:02:18.872 DEBUG:boto:Path: /hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.872 DEBUG:boto:Data: 2023-09-29T23:02:18.872 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.872 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.872 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.872 DEBUG:boto:Params: {} 2023-09-29T23:02:18.872 DEBUG:boto:Token: None 2023-09-29T23:02:18.872 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.872 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.873 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.877 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007ab18e2b5315212b-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.918 DEBUG:boto:b'hcnphz-62obj331000falseobj332023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.919 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.919 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.919 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.919 DEBUG:boto:Path: /hcnphz-62/obj33 2023-09-29T23:02:18.919 DEBUG:boto:Data: 2023-09-29T23:02:18.919 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.920 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.920 DEBUG:boto:Params: {} 2023-09-29T23:02:18.920 DEBUG:boto:Token: None 2023-09-29T23:02:18.920 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33 2023-09-29T23:02:18.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8z4Y7Yo/zcpo+ZH7Kb3yNCImGK8= 2023-09-29T23:02:18.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8z4Y7Yo/zcpo+ZH7Kb3yNCImGK8=', 'Content-Length': '0'} 2023-09-29T23:02:18.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f6c55c9e5375b093-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.924 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.924 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.924 DEBUG:boto:Method: HEAD 2023-09-29T23:02:18.924 DEBUG:boto:Path: /hcnphz-62/obj33 2023-09-29T23:02:18.924 DEBUG:boto:Data: 2023-09-29T23:02:18.924 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.925 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.925 DEBUG:boto:Params: {} 2023-09-29T23:02:18.925 DEBUG:boto:Token: None 2023-09-29T23:02:18.925 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33 2023-09-29T23:02:18.925 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8z4Y7Yo/zcpo+ZH7Kb3yNCImGK8= 2023-09-29T23:02:18.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8z4Y7Yo/zcpo+ZH7Kb3yNCImGK8=', 'Content-Length': '0'} 2023-09-29T23:02:18.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8cb1bf93a093b4d-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T23:02:18.928 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.928 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.929 DEBUG:boto:Method: GET 2023-09-29T23:02:18.929 DEBUG:boto:Path: /hcnphz-62/obj33 2023-09-29T23:02:18.929 DEBUG:boto:Data: 2023-09-29T23:02:18.929 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.929 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.929 DEBUG:boto:Params: {} 2023-09-29T23:02:18.930 DEBUG:boto:Token: None 2023-09-29T23:02:18.930 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33 2023-09-29T23:02:18.930 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j6+zue8jEdx4kpdbc+//P2RuIq4= 2023-09-29T23:02:18.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j6+zue8jEdx4kpdbc+//P2RuIq4=', 'Content-Length': '0'} 2023-09-29T23:02:18.933 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000023c325b2b2b3e513-006517577a-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.974 DEBUG:boto:path=/hcnphz-62/obj33 2023-09-29T23:02:18.975 DEBUG:boto:auth_path=/hcnphz-62/obj33 2023-09-29T23:02:18.975 DEBUG:boto:Method: GET 2023-09-29T23:02:18.975 DEBUG:boto:Path: /hcnphz-62/obj33 2023-09-29T23:02:18.975 DEBUG:boto:Data: 2023-09-29T23:02:18.975 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.975 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.975 DEBUG:boto:Params: {} 2023-09-29T23:02:18.976 DEBUG:boto:Token: None 2023-09-29T23:02:18.976 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj33 2023-09-29T23:02:18.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j6+zue8jEdx4kpdbc+//P2RuIq4= 2023-09-29T23:02:18.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j6+zue8jEdx4kpdbc+//P2RuIq4=', 'Content-Length': '0'} 2023-09-29T23:02:18.979 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000038769d7784de6841-006517577a-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.980 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.980 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.980 DEBUG:boto:path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.980 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.980 DEBUG:boto:Method: GET 2023-09-29T23:02:18.980 DEBUG:boto:Path: /hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.980 DEBUG:boto:Data: 2023-09-29T23:02:18.981 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.981 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.981 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.981 DEBUG:boto:Params: {} 2023-09-29T23:02:18.981 DEBUG:boto:Token: None 2023-09-29T23:02:18.981 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.981 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.982 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.987 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001afb0008bd92faed-006517577a-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.987 DEBUG:boto:b'hcnphz-62obj331000falseobj332023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.988 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:18.988 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:18.988 DEBUG:boto:path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.988 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.989 DEBUG:boto:Method: GET 2023-09-29T23:02:18.989 DEBUG:boto:Path: /hcnphz-62/?prefix=obj33 2023-09-29T23:02:18.989 DEBUG:boto:Data: 2023-09-29T23:02:18.989 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.989 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:18.989 DEBUG:boto:Port: 8000 2023-09-29T23:02:18.989 DEBUG:boto:Params: {} 2023-09-29T23:02:18.990 DEBUG:boto:Token: None 2023-09-29T23:02:18.990 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/ 2023-09-29T23:02:18.990 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA= 2023-09-29T23:02:18.990 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y0h1xVv9qrIqP71cXb6b3NIgguA=', 'Content-Length': '0'} 2023-09-29T23:02:18.995 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fb529911ea43d34-006517577a-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:18 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:18.995 DEBUG:boto:b'hcnphz-62obj331000falseobj332023-09-29T23:00:26.491Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:18.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T23:02:18.996 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:18.996 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:18.996 DEBUG:boto:path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:18.996 DEBUG:boto:auth_path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:18.996 DEBUG:boto:Method: GET 2023-09-29T23:02:18.996 DEBUG:boto:Path: /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:18.997 DEBUG:boto:Data: 2023-09-29T23:02:18.997 DEBUG:boto:Headers: {} 2023-09-29T23:02:18.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:18.997 DEBUG:boto:Port: 8001 2023-09-29T23:02:18.997 DEBUG:boto:Params: {} 2023-09-29T23:02:18.997 DEBUG:boto:Token: None 2023-09-29T23:02:18.997 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:18 GMT /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:18.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:B9J0iVCRPu6p+XJNaSNIRvDEqLA= 2023-09-29T23:02:18.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:18 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:B9J0iVCRPu6p+XJNaSNIRvDEqLA=', 'Content-Length': '0'} 2023-09-29T23:02:19.000 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002ba7ca6c7b16b8f0-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.001 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.001 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.001 DEBUG:boto:path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.001 DEBUG:boto:auth_path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.001 DEBUG:boto:Method: GET 2023-09-29T23:02:19.001 DEBUG:boto:Path: /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.001 DEBUG:boto:Data: 2023-09-29T23:02:19.001 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.002 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.002 DEBUG:boto:Params: {} 2023-09-29T23:02:19.002 DEBUG:boto:Token: None 2023-09-29T23:02:19.002 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMrUXOmQdQMaWJGeGPqFPnZV7Ug= 2023-09-29T23:02:19.002 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMrUXOmQdQMaWJGeGPqFPnZV7Ug=', 'Content-Length': '0'} 2023-09-29T23:02:19.005 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e9862b7129072355-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.046 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.047 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.047 DEBUG:boto:path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.047 DEBUG:boto:auth_path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.047 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.048 DEBUG:boto:Path: /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.048 DEBUG:boto:Data: 2023-09-29T23:02:19.048 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.048 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.048 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.048 DEBUG:boto:Params: {} 2023-09-29T23:02:19.049 DEBUG:boto:Token: None 2023-09-29T23:02:19.049 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.049 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2QUrEDscA301zer5cW05bmfdkg= 2023-09-29T23:02:19.049 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2QUrEDscA301zer5cW05bmfdkg=', 'Content-Length': '0'} 2023-09-29T23:02:19.052 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000888a9422e1df12b7-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.053 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.053 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.053 DEBUG:boto:path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.053 DEBUG:boto:auth_path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.054 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.054 DEBUG:boto:Path: /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.054 DEBUG:boto:Data: 2023-09-29T23:02:19.054 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.055 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.055 DEBUG:boto:Params: {} 2023-09-29T23:02:19.055 DEBUG:boto:Token: None 2023-09-29T23:02:19.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:A2QUrEDscA301zer5cW05bmfdkg= 2023-09-29T23:02:19.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2QUrEDscA301zer5cW05bmfdkg=', 'Content-Length': '0'} 2023-09-29T23:02:19.059 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cc9e91844a4ffe87-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T23:02:19.059 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.060 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.060 DEBUG:boto:path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.060 DEBUG:boto:auth_path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.060 DEBUG:boto:Method: GET 2023-09-29T23:02:19.060 DEBUG:boto:Path: /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.060 DEBUG:boto:Data: 2023-09-29T23:02:19.060 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.061 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.061 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.061 DEBUG:boto:Params: {} 2023-09-29T23:02:19.061 DEBUG:boto:Token: None 2023-09-29T23:02:19.061 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.061 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMrUXOmQdQMaWJGeGPqFPnZV7Ug= 2023-09-29T23:02:19.061 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMrUXOmQdQMaWJGeGPqFPnZV7Ug=', 'Content-Length': '0'} 2023-09-29T23:02:19.064 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007f79fbb3c511a908-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.106 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.107 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.107 DEBUG:boto:path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.107 DEBUG:boto:auth_path=/hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.107 DEBUG:boto:Method: GET 2023-09-29T23:02:19.108 DEBUG:boto:Path: /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.108 DEBUG:boto:Data: 2023-09-29T23:02:19.108 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.108 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.108 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.109 DEBUG:boto:Params: {} 2023-09-29T23:02:19.109 DEBUG:boto:Token: None 2023-09-29T23:02:19.109 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44?versionId=null 2023-09-29T23:02:19.109 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yMrUXOmQdQMaWJGeGPqFPnZV7Ug= 2023-09-29T23:02:19.110 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yMrUXOmQdQMaWJGeGPqFPnZV7Ug=', 'Content-Length': '0'} 2023-09-29T23:02:19.112 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000950b04a5b892ae68-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.113 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.113 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.114 DEBUG:boto:path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.114 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.114 DEBUG:boto:Method: GET 2023-09-29T23:02:19.115 DEBUG:boto:Path: /hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.115 DEBUG:boto:Data: 2023-09-29T23:02:19.115 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.116 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.116 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.116 DEBUG:boto:Params: {} 2023-09-29T23:02:19.117 DEBUG:boto:Token: None 2023-09-29T23:02:19.117 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.117 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.118 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.123 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e45a9389dbb1be9c-006517577b-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.124 DEBUG:boto:b'hcnphz-62obj441000falseobj442023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.124 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.124 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.124 DEBUG:boto:path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.124 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.124 DEBUG:boto:Method: GET 2023-09-29T23:02:19.125 DEBUG:boto:Path: /hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.125 DEBUG:boto:Data: 2023-09-29T23:02:19.125 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.125 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.125 DEBUG:boto:Params: {} 2023-09-29T23:02:19.125 DEBUG:boto:Token: None 2023-09-29T23:02:19.125 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.125 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.130 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7e9ae165da4f1e7-006517577b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.130 DEBUG:boto:b'hcnphz-62obj441000falseobj442023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.131 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.131 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.131 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.131 DEBUG:boto:Path: /hcnphz-62/obj44 2023-09-29T23:02:19.131 DEBUG:boto:Data: 2023-09-29T23:02:19.131 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.131 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.131 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.131 DEBUG:boto:Params: {} 2023-09-29T23:02:19.132 DEBUG:boto:Token: None 2023-09-29T23:02:19.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44 2023-09-29T23:02:19.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RhcAxRHTe216Veb7u3N4IaA7dAw= 2023-09-29T23:02:19.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RhcAxRHTe216Veb7u3N4IaA7dAw=', 'Content-Length': '0'} 2023-09-29T23:02:19.134 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a72a8d9d18c6ff0a-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.135 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.135 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.135 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.135 DEBUG:boto:Path: /hcnphz-62/obj44 2023-09-29T23:02:19.135 DEBUG:boto:Data: 2023-09-29T23:02:19.135 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.135 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.135 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.135 DEBUG:boto:Params: {} 2023-09-29T23:02:19.136 DEBUG:boto:Token: None 2023-09-29T23:02:19.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44 2023-09-29T23:02:19.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RhcAxRHTe216Veb7u3N4IaA7dAw= 2023-09-29T23:02:19.136 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RhcAxRHTe216Veb7u3N4IaA7dAw=', 'Content-Length': '0'} 2023-09-29T23:02:19.138 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c3f41af98e60d21b-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T23:02:19.139 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.139 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.139 DEBUG:boto:Method: GET 2023-09-29T23:02:19.139 DEBUG:boto:Path: /hcnphz-62/obj44 2023-09-29T23:02:19.139 DEBUG:boto:Data: 2023-09-29T23:02:19.139 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.140 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.140 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.140 DEBUG:boto:Params: {} 2023-09-29T23:02:19.140 DEBUG:boto:Token: None 2023-09-29T23:02:19.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44 2023-09-29T23:02:19.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gpdFoTahUKco5he0nFG49LR9tII= 2023-09-29T23:02:19.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gpdFoTahUKco5he0nFG49LR9tII=', 'Content-Length': '0'} 2023-09-29T23:02:19.143 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000031c41328387bb7e7-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.190 DEBUG:boto:path=/hcnphz-62/obj44 2023-09-29T23:02:19.191 DEBUG:boto:auth_path=/hcnphz-62/obj44 2023-09-29T23:02:19.191 DEBUG:boto:Method: GET 2023-09-29T23:02:19.191 DEBUG:boto:Path: /hcnphz-62/obj44 2023-09-29T23:02:19.191 DEBUG:boto:Data: 2023-09-29T23:02:19.191 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.192 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.192 DEBUG:boto:Params: {} 2023-09-29T23:02:19.192 DEBUG:boto:Token: None 2023-09-29T23:02:19.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj44 2023-09-29T23:02:19.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gpdFoTahUKco5he0nFG49LR9tII= 2023-09-29T23:02:19.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gpdFoTahUKco5he0nFG49LR9tII=', 'Content-Length': '0'} 2023-09-29T23:02:19.195 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d74177b4b94523e-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.196 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.196 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.196 DEBUG:boto:path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.196 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.197 DEBUG:boto:Method: GET 2023-09-29T23:02:19.197 DEBUG:boto:Path: /hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.197 DEBUG:boto:Data: 2023-09-29T23:02:19.197 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.197 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.197 DEBUG:boto:Params: {} 2023-09-29T23:02:19.197 DEBUG:boto:Token: None 2023-09-29T23:02:19.198 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.202 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e009b5e1bae4af9f-006517577b-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.203 DEBUG:boto:b'hcnphz-62obj441000falseobj442023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.203 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.203 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.203 DEBUG:boto:path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.203 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.204 DEBUG:boto:Method: GET 2023-09-29T23:02:19.204 DEBUG:boto:Path: /hcnphz-62/?prefix=obj44 2023-09-29T23:02:19.204 DEBUG:boto:Data: 2023-09-29T23:02:19.204 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.204 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.204 DEBUG:boto:Params: {} 2023-09-29T23:02:19.204 DEBUG:boto:Token: None 2023-09-29T23:02:19.205 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.209 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000767754444e9f2b0f-006517577b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.210 DEBUG:boto:b'hcnphz-62obj441000falseobj442023-09-29T23:00:26.548Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T23:02:19.211 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.211 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.211 DEBUG:boto:path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.211 DEBUG:boto:auth_path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.212 DEBUG:boto:Method: GET 2023-09-29T23:02:19.212 DEBUG:boto:Path: /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.212 DEBUG:boto:Data: 2023-09-29T23:02:19.212 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.212 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.213 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.213 DEBUG:boto:Params: {} 2023-09-29T23:02:19.213 DEBUG:boto:Token: None 2023-09-29T23:02:19.213 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.214 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4= 2023-09-29T23:02:19.214 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4=', 'Content-Length': '0'} 2023-09-29T23:02:19.217 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b391e99f12bcaa57-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.217 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.218 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.218 DEBUG:boto:path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.218 DEBUG:boto:auth_path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.218 DEBUG:boto:Method: GET 2023-09-29T23:02:19.218 DEBUG:boto:Path: /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.219 DEBUG:boto:Data: 2023-09-29T23:02:19.219 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.219 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.219 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.219 DEBUG:boto:Params: {} 2023-09-29T23:02:19.220 DEBUG:boto:Token: None 2023-09-29T23:02:19.220 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.220 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4= 2023-09-29T23:02:19.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4=', 'Content-Length': '0'} 2023-09-29T23:02:19.223 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f84371ffc6d8c4ca-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.266 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.267 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.267 DEBUG:boto:path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.267 DEBUG:boto:auth_path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.267 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.268 DEBUG:boto:Path: /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.268 DEBUG:boto:Data: 2023-09-29T23:02:19.268 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.268 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.269 DEBUG:boto:Params: {} 2023-09-29T23:02:19.269 DEBUG:boto:Token: None 2023-09-29T23:02:19.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G7R/Xyl0E1HCL+9dTdi8xwtAuNo= 2023-09-29T23:02:19.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G7R/Xyl0E1HCL+9dTdi8xwtAuNo=', 'Content-Length': '0'} 2023-09-29T23:02:19.273 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d29d73b773bdc211-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.273 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.273 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.274 DEBUG:boto:path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.274 DEBUG:boto:auth_path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.274 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.274 DEBUG:boto:Path: /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.274 DEBUG:boto:Data: 2023-09-29T23:02:19.274 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.274 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.274 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.275 DEBUG:boto:Params: {} 2023-09-29T23:02:19.275 DEBUG:boto:Token: None 2023-09-29T23:02:19.275 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.275 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:G7R/Xyl0E1HCL+9dTdi8xwtAuNo= 2023-09-29T23:02:19.275 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:G7R/Xyl0E1HCL+9dTdi8xwtAuNo=', 'Content-Length': '0'} 2023-09-29T23:02:19.278 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000078f7461080567697-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T23:02:19.278 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.278 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.279 DEBUG:boto:path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.279 DEBUG:boto:auth_path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.279 DEBUG:boto:Method: GET 2023-09-29T23:02:19.279 DEBUG:boto:Path: /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.279 DEBUG:boto:Data: 2023-09-29T23:02:19.279 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.279 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.279 DEBUG:boto:Params: {} 2023-09-29T23:02:19.280 DEBUG:boto:Token: None 2023-09-29T23:02:19.280 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4= 2023-09-29T23:02:19.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4=', 'Content-Length': '0'} 2023-09-29T23:02:19.282 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bdc0c26f7d3a128e-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.283 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.283 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.283 DEBUG:boto:path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.283 DEBUG:boto:auth_path=/hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.283 DEBUG:boto:Method: GET 2023-09-29T23:02:19.283 DEBUG:boto:Path: /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.284 DEBUG:boto:Data: 2023-09-29T23:02:19.284 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.284 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.284 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.284 DEBUG:boto:Params: {} 2023-09-29T23:02:19.284 DEBUG:boto:Token: None 2023-09-29T23:02:19.284 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55?versionId=null 2023-09-29T23:02:19.284 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4= 2023-09-29T23:02:19.285 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ggFTU0DVdmEJprHWE4j1DCjFiR4=', 'Content-Length': '0'} 2023-09-29T23:02:19.287 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004e6145c06ed10223-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.288 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.288 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.288 DEBUG:boto:path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.288 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.288 DEBUG:boto:Method: GET 2023-09-29T23:02:19.288 DEBUG:boto:Path: /hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.288 DEBUG:boto:Data: 2023-09-29T23:02:19.288 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.289 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.289 DEBUG:boto:Params: {} 2023-09-29T23:02:19.289 DEBUG:boto:Token: None 2023-09-29T23:02:19.289 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.289 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.294 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a7d940f8b10eaab0-006517577b-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.343 DEBUG:boto:b'hcnphz-62obj551000falseobj552023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.344 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.344 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.344 DEBUG:boto:path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.344 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.344 DEBUG:boto:Method: GET 2023-09-29T23:02:19.344 DEBUG:boto:Path: /hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.344 DEBUG:boto:Data: 2023-09-29T23:02:19.344 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.344 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.345 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.345 DEBUG:boto:Params: {} 2023-09-29T23:02:19.345 DEBUG:boto:Token: None 2023-09-29T23:02:19.345 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.345 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.345 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.351 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008912cd973ea60aca-006517577b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.351 DEBUG:boto:b'hcnphz-62obj551000falseobj552023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.351 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.352 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.352 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.352 DEBUG:boto:Path: /hcnphz-62/obj55 2023-09-29T23:02:19.352 DEBUG:boto:Data: 2023-09-29T23:02:19.352 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.352 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.352 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.352 DEBUG:boto:Params: {} 2023-09-29T23:02:19.353 DEBUG:boto:Token: None 2023-09-29T23:02:19.353 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55 2023-09-29T23:02:19.353 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k7FOyEz211uJD7XuJvoYJyl63N4= 2023-09-29T23:02:19.353 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k7FOyEz211uJD7XuJvoYJyl63N4=', 'Content-Length': '0'} 2023-09-29T23:02:19.356 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008a73af3eb81ffd1f-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.356 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.356 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.356 DEBUG:boto:Method: HEAD 2023-09-29T23:02:19.356 DEBUG:boto:Path: /hcnphz-62/obj55 2023-09-29T23:02:19.356 DEBUG:boto:Data: 2023-09-29T23:02:19.356 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.357 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.357 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.357 DEBUG:boto:Params: {} 2023-09-29T23:02:19.357 DEBUG:boto:Token: None 2023-09-29T23:02:19.357 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55 2023-09-29T23:02:19.357 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k7FOyEz211uJD7XuJvoYJyl63N4= 2023-09-29T23:02:19.357 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k7FOyEz211uJD7XuJvoYJyl63N4=', 'Content-Length': '0'} 2023-09-29T23:02:19.360 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000033c9d808c1e5c605-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T23:02:19.361 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.361 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.361 DEBUG:boto:Method: GET 2023-09-29T23:02:19.361 DEBUG:boto:Path: /hcnphz-62/obj55 2023-09-29T23:02:19.361 DEBUG:boto:Data: 2023-09-29T23:02:19.361 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.361 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.362 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.362 DEBUG:boto:Params: {} 2023-09-29T23:02:19.362 DEBUG:boto:Token: None 2023-09-29T23:02:19.362 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55 2023-09-29T23:02:19.362 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:17QNPWBbmWTam7206XaSA/QYN1c= 2023-09-29T23:02:19.362 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:17QNPWBbmWTam7206XaSA/QYN1c=', 'Content-Length': '0'} 2023-09-29T23:02:19.365 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-62:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.27'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d31e7be8203af21a-006517577b-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.365 DEBUG:boto:path=/hcnphz-62/obj55 2023-09-29T23:02:19.365 DEBUG:boto:auth_path=/hcnphz-62/obj55 2023-09-29T23:02:19.366 DEBUG:boto:Method: GET 2023-09-29T23:02:19.366 DEBUG:boto:Path: /hcnphz-62/obj55 2023-09-29T23:02:19.366 DEBUG:boto:Data: 2023-09-29T23:02:19.366 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.366 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.366 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.366 DEBUG:boto:Params: {} 2023-09-29T23:02:19.366 DEBUG:boto:Token: None 2023-09-29T23:02:19.366 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/obj55 2023-09-29T23:02:19.367 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:17QNPWBbmWTam7206XaSA/QYN1c= 2023-09-29T23:02:19.367 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:17QNPWBbmWTam7206XaSA/QYN1c=', 'Content-Length': '0'} 2023-09-29T23:02:19.369 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:00:26 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abf142d628f6a53-006517577b-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.411 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.411 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.411 DEBUG:boto:path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.411 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.411 DEBUG:boto:Method: GET 2023-09-29T23:02:19.411 DEBUG:boto:Path: /hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.412 DEBUG:boto:Data: 2023-09-29T23:02:19.412 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.412 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:02:19.412 DEBUG:boto:Port: 8001 2023-09-29T23:02:19.412 DEBUG:boto:Params: {} 2023-09-29T23:02:19.412 DEBUG:boto:Token: None 2023-09-29T23:02:19.412 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.413 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.417 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009be3c6619e5de9c6-006517577b-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.418 DEBUG:boto:b'hcnphz-62obj551000falseobj552023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.418 DEBUG:boto:path=/hcnphz-62/ 2023-09-29T23:02:19.419 DEBUG:boto:auth_path=/hcnphz-62/ 2023-09-29T23:02:19.419 DEBUG:boto:path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.419 DEBUG:boto:auth_path=/hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.419 DEBUG:boto:Method: GET 2023-09-29T23:02:19.419 DEBUG:boto:Path: /hcnphz-62/?prefix=obj55 2023-09-29T23:02:19.419 DEBUG:boto:Data: 2023-09-29T23:02:19.419 DEBUG:boto:Headers: {} 2023-09-29T23:02:19.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:19.419 DEBUG:boto:Port: 8000 2023-09-29T23:02:19.419 DEBUG:boto:Params: {} 2023-09-29T23:02:19.420 DEBUG:boto:Token: None 2023-09-29T23:02:19.420 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:02:19 GMT /hcnphz-62/ 2023-09-29T23:02:19.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk= 2023-09-29T23:02:19.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D/jIq3abAcTo14RMoUqh6l3a8kk=', 'Content-Length': '0'} 2023-09-29T23:02:19.425 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002cd2f431e7f0ddba-006517577b-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:02:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:19.425 DEBUG:boto:b'hcnphz-62obj551000falseobj552023-09-29T23:00:26.566Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:02:19.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-62 zones={a2, a1} 2023-09-29T23:02:19.426 DEBUG:rgw_multi.tests:data checkpoint: source=a1 target=a2 2023-09-29T23:02:19.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.426 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:19.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:19.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:19.562 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:02:19.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037528", 2023-09-29T23:02:19.562 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:00:26.555273Z" 2023-09-29T23:02:19.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000033709", 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:00:29.658284Z" 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032631", 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:00:29.662019Z" 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000022720", 2023-09-29T23:02:19.564 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:55:21.652745Z" 2023-09-29T23:02:19.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.564 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:02:19.565 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000037528', 1: '00000000000000000000:00000000000000033709', 2: '00000000000000000000:00000000000000032631', 3: '00000000000000000000:00000000000000022720'} 2023-09-29T23:02:19.565 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:02:19.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.566 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:19.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:19.654 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:19.736 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:19.736 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:19.736 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:19.736 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:19.736 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:19.736 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 17337405482519760571 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000037528", 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:19.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 53, 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:00:26.555273Z" 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:19.738 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000033709", 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:00:29.658284Z" 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:19.739 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032631", 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:00:29.662019Z" 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.740 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000022720", 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:55:21.652745Z" 2023-09-29T23:02:19.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 62, 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 62 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.742 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:19Z" 2023-09-29T23:02:19.743 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:19.743 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 17337405482519760571 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000037528", "next_step_marker": "", "total_entries": 53, "pos": 0, "timestamp": "2023-09-29T23:00:26.555273Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000033709", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:00:29.658284Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032631", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:00:29.662019Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000022720", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:55:21.652745Z" } } ] }, "full_sync": { "total": 62, "complete": 62 }, "current_time": "2023-09-29T23:02:19Z" } 2023-09-29T23:02:19.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000037528', 'next_step_marker': '', 'total_entries': 53, 'pos': 0, 'timestamp': '2023-09-29T23:00:26.555273Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000033709', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:00:29.658284Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032631', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:00:29.662019Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000022720', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:55:21.652745Z'}}] 2023-09-29T23:02:19.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000037528', 1: '00000000000000000000:00000000000000033709', 2: '00000000000000000000:00000000000000032631', 3: '00000000000000000000:00000000000000022720'} 2023-09-29T23:02:19.744 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000037528', 1: '00000000000000000000:00000000000000033709', 2: '00000000000000000000:00000000000000032631', 3: '00000000000000000000:00000000000000022720'} 2023-09-29T23:02:19.744 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:02:19.744 DEBUG:rgw_multi.tests:data checkpoint: source=a2 target=a1 2023-09-29T23:02:19.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.745 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:19.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:19.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:19.894 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:02:19.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-09-29T23:02:19.894 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:01:57.516049Z" 2023-09-29T23:02:19.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014604", 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:02:16.893622Z" 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011678", 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:02:16.897031Z" 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:19.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T23:02:19.896 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:57:05.830505Z" 2023-09-29T23:02:19.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:19.896 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:02:19.897 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:19.897 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T23:02:19.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:19.897 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:20.030 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:20.031 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:20.032 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.516049Z" 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:20.033 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.481916Z" 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011536", 2023-09-29T23:02:20.034 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.482181Z" 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:20.035 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.830505Z" 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:20.036 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T23:02:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T23:02:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:20Z" 2023-09-29T23:02:20.037 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:20.038 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.516049Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014463", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.481916Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011536", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.482181Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.830505Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:20Z" } 2023-09-29T23:02:20.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.516049Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014463', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.481916Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011536', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.482181Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.830505Z'}}] 2023-09-29T23:02:20.038 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:20.038 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011536', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:20.039 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000014604 target=00000000000000000000:00000000000000014463, shard=2 master=00000000000000000000:00000000000000011678 target=00000000000000000000:00000000000000011536 2023-09-29T23:02:25.039 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:25.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:25.040 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:25.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:25.088 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:25.171 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:25.171 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:25.171 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:25.171 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.172 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.516049Z" 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:25.173 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.481916Z" 2023-09-29T23:02:25.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011536", 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:25.175 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.482181Z" 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:25.176 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.830505Z" 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:25.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:25.178 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:25.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T23:02:25.178 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T23:02:25.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:25.178 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:25Z" 2023-09-29T23:02:25.178 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:25.179 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.516049Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014463", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.481916Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011536", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.482181Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.830505Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:25Z" } 2023-09-29T23:02:25.180 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.516049Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014463', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.481916Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011536', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.482181Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.830505Z'}}] 2023-09-29T23:02:25.180 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:25.180 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011536', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:25.180 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000014604 target=00000000000000000000:00000000000000014463, shard=2 master=00000000000000000000:00000000000000011678 target=00000000000000000000:00000000000000011536 2023-09-29T23:02:30.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:30.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:30.182 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:30.231 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:30.231 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:30.237 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:02:30.233+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:02:30.311 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-09-29T23:02:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:30.313 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.516049Z" 2023-09-29T23:02:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:30.314 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-09-29T23:02:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:30.315 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.481916Z" 2023-09-29T23:02:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:30.316 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011536", 2023-09-29T23:02:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:30.317 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.482181Z" 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:30.318 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.830505Z" 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:30.319 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T23:02:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T23:02:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:30.320 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:30Z" 2023-09-29T23:02:30.320 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:30.321 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.516049Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014463", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.481916Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011536", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.482181Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.830505Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:30Z" } 2023-09-29T23:02:30.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.516049Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014463', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.481916Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011536', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.482181Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.830505Z'}}] 2023-09-29T23:02:30.321 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:30.321 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011536', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:30.322 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000014604 target=00000000000000000000:00000000000000014463, shard=2 master=00000000000000000000:00000000000000011678 target=00000000000000000000:00000000000000011536 2023-09-29T23:02:35.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:35.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:35.323 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:35.373 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:35.458 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:35.458 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:35.458 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:35.459 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.516049Z" 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:35.460 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014463", 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.481916Z" 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:35.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011536", 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.482181Z" 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:35.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:35.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.830505Z" 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:35Z" 2023-09-29T23:02:35.464 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:35.465 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.516049Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014463", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.481916Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011536", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.482181Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.830505Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:35Z" } 2023-09-29T23:02:35.466 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.516049Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014463', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.481916Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011536', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.482181Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.830505Z'}}] 2023-09-29T23:02:35.466 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:35.466 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014463', 2: '00000000000000000000:00000000000000011536', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:35.466 WARNING:rgw_multi.tests:data of zone a1 behind zone a2: shard=1 master=00000000000000000000:00000000000000014604 target=00000000000000000000:00000000000000014463, shard=2 master=00000000000000000000:00000000000000011678 target=00000000000000000000:00000000000000011536 2023-09-29T23:02:40.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.468 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:40.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:40.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:40.602 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:40.602 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.603 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000015588", 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:01:57.516049Z" 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000014604", 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:02:16.893622Z" 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000011678", 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:02:16.897031Z" 2023-09-29T23:02:40.606 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000005445", 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.607 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:57:05.830505Z" 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:40Z" 2023-09-29T23:02:40.608 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:40.609 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000015588", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:01:57.516049Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000014604", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:02:16.893622Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000011678", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:02:16.897031Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000005445", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T22:57:05.830505Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:02:40Z" } 2023-09-29T23:02:40.610 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000015588', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:01:57.516049Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000014604', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:02:16.893622Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000011678', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:02:16.897031Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000005445', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:57:05.830505Z'}}] 2023-09-29T23:02:40.610 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:40.610 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000015588', 1: '00000000000000000000:00000000000000014604', 2: '00000000000000000000:00000000000000011678', 3: '00000000000000000000:00000000000000005445'} 2023-09-29T23:02:40.610 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T23:02:40.610 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_incremental_sync_after_data_sync_init ... ok 2023-09-29T23:02:40.611 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:40.612 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:40.612 DEBUG:boto:Method: PUT 2023-09-29T23:02:40.612 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:40.612 DEBUG:boto:Data: 2023-09-29T23:02:40.612 DEBUG:boto:Headers: {} 2023-09-29T23:02:40.612 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:40.612 DEBUG:boto:Port: 8000 2023-09-29T23:02:40.612 DEBUG:boto:Params: {} 2023-09-29T23:02:40.613 DEBUG:boto:Token: None 2023-09-29T23:02:40.613 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:02:40 GMT /hcnphz-63/ 2023-09-29T23:02:40.613 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0njbPnIqthjxda0rs1Pyi/mKpCg= 2023-09-29T23:02:40.613 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0njbPnIqthjxda0rs1Pyi/mKpCg=', 'Content-Length': '0'} 2023-09-29T23:02:40.645 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cdc3c071a61318db-0065175790-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:40.645 DEBUG:rgw_multi.tests:created bucket=hcnphz-63 2023-09-29T23:02:40.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.646 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:40.685 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:40.686 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:40.741 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:02:40.770 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:02:40.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.770 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:02:40.770 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028560.644899_607.1", 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:02:40.644899Z" 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.772 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:02:40.772 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:02:40.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.772 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:02:40.773 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:02:40.773 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:02:40.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:40.773 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:40.817 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:40.817 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:40.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:40.876+0000 7ff913fb5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:02:40.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:40.876+0000 7ff913fb5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:02:40.904 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:40.904 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:40.905 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028560.633902_604.1", 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:02:40.633902Z", 2023-09-29T23:02:40.907 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.908 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:02:40.909 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.910 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:02:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:02:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:40.911 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:40Z" 2023-09-29T23:02:40.911 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:40.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028560.633902_604.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:02:40.633902Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:02:40Z" } 2023-09-29T23:02:40.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028560.633902_604.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:02:40.633902Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:02:40.913 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:02:40.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.633902_604.1', 2: '', 3: ''} 2023-09-29T23:02:40.913 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696028560.644899_607.1 target=1_1696028560.633902_604.1 2023-09-29T23:02:45.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:45.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:02:45.915 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:02:45.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:45.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:46.017 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:46.016+0000 7fcdb62ccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:02:46.019 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:46.020+0000 7fcdb62ccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:02:46.046 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:02:46.046 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:02:46.046 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:02:46.046 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:02:46.046 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:46.047 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:02:46.048 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028560.644899_607.1", 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:02:40.644899Z", 2023-09-29T23:02:46.049 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:46.050 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:02:46.051 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:46.052 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:02:46.053 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:02:46.053 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:02:46.053 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:02:46.053 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:02:46Z" 2023-09-29T23:02:46.053 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:02:46.054 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028560.644899_607.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:02:40.644899Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:02:46Z" } 2023-09-29T23:02:46.055 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028560.644899_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:02:40.644899Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:02:46.055 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:02:46.055 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:02:46.055 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:02:46.055 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.055 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.055 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.055 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.056 DEBUG:boto:Data: 2023-09-29T23:02:46.056 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.056 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.056 DEBUG:boto:Params: {} 2023-09-29T23:02:46.056 DEBUG:boto:Token: None 2023-09-29T23:02:46.056 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.061 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008db91138ab5140be-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.061 DEBUG:boto:path=/hcnphz-63/obj11 2023-09-29T23:02:46.062 DEBUG:boto:auth_path=/hcnphz-63/obj11 2023-09-29T23:02:46.062 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.062 DEBUG:boto:Path: /hcnphz-63/obj11 2023-09-29T23:02:46.062 DEBUG:boto:Data: 2023-09-29T23:02:46.062 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.063 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.063 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.063 DEBUG:boto:Params: {} 2023-09-29T23:02:46.063 DEBUG:boto:Token: None 2023-09-29T23:02:46.063 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj11 2023-09-29T23:02:46.064 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3VsyJq7RoXQdBcFwx1Bp+lIsl9w= 2023-09-29T23:02:46.064 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3VsyJq7RoXQdBcFwx1Bp+lIsl9w='} 2023-09-29T23:02:46.078 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049fe953529f292d0-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.079 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.079 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.079 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.079 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.079 DEBUG:boto:Data: 2023-09-29T23:02:46.079 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.079 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.079 DEBUG:boto:Params: {} 2023-09-29T23:02:46.080 DEBUG:boto:Token: None 2023-09-29T23:02:46.080 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.083 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f3be63aae232cee-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.084 DEBUG:boto:path=/hcnphz-63/obj22 2023-09-29T23:02:46.084 DEBUG:boto:auth_path=/hcnphz-63/obj22 2023-09-29T23:02:46.084 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.084 DEBUG:boto:Path: /hcnphz-63/obj22 2023-09-29T23:02:46.084 DEBUG:boto:Data: 2023-09-29T23:02:46.084 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.084 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.084 DEBUG:boto:Params: {} 2023-09-29T23:02:46.085 DEBUG:boto:Token: None 2023-09-29T23:02:46.085 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj22 2023-09-29T23:02:46.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Hrt2jr85Go8VfWHaw45mbebQ23I= 2023-09-29T23:02:46.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Hrt2jr85Go8VfWHaw45mbebQ23I='} 2023-09-29T23:02:46.130 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000073e66f286b822ad-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.131 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.131 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.131 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.131 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.131 DEBUG:boto:Data: 2023-09-29T23:02:46.131 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.132 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.132 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.132 DEBUG:boto:Params: {} 2023-09-29T23:02:46.132 DEBUG:boto:Token: None 2023-09-29T23:02:46.132 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.132 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.132 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.136 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d6d495b2e21c941-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.137 DEBUG:boto:path=/hcnphz-63/obj33 2023-09-29T23:02:46.137 DEBUG:boto:auth_path=/hcnphz-63/obj33 2023-09-29T23:02:46.137 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.137 DEBUG:boto:Path: /hcnphz-63/obj33 2023-09-29T23:02:46.137 DEBUG:boto:Data: 2023-09-29T23:02:46.137 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.137 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.138 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.138 DEBUG:boto:Params: {} 2023-09-29T23:02:46.138 DEBUG:boto:Token: None 2023-09-29T23:02:46.138 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj33 2023-09-29T23:02:46.138 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GrMFlL0/J+dh7SDXMURGoqHMymU= 2023-09-29T23:02:46.138 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GrMFlL0/J+dh7SDXMURGoqHMymU='} 2023-09-29T23:02:46.147 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e4559ee9d1367ea-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.148 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.148 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.148 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.148 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.148 DEBUG:boto:Data: 2023-09-29T23:02:46.148 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.148 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.148 DEBUG:boto:Params: {} 2023-09-29T23:02:46.149 DEBUG:boto:Token: None 2023-09-29T23:02:46.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.152 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005557cf311a69554c-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.152 DEBUG:boto:path=/hcnphz-63/obj44 2023-09-29T23:02:46.152 DEBUG:boto:auth_path=/hcnphz-63/obj44 2023-09-29T23:02:46.152 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.152 DEBUG:boto:Path: /hcnphz-63/obj44 2023-09-29T23:02:46.152 DEBUG:boto:Data: 2023-09-29T23:02:46.153 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.153 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.153 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.153 DEBUG:boto:Params: {} 2023-09-29T23:02:46.153 DEBUG:boto:Token: None 2023-09-29T23:02:46.153 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj44 2023-09-29T23:02:46.153 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YYorWEm0PsTcrvNRmM1CHBmIeqs= 2023-09-29T23:02:46.153 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YYorWEm0PsTcrvNRmM1CHBmIeqs='} 2023-09-29T23:02:46.198 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000657f1479de37c7f8-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.199 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.199 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.199 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.199 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.199 DEBUG:boto:Data: 2023-09-29T23:02:46.199 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.199 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.200 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.200 DEBUG:boto:Params: {} 2023-09-29T23:02:46.200 DEBUG:boto:Token: None 2023-09-29T23:02:46.200 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.200 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.200 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7ea29185cb9e25c-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.204 DEBUG:boto:path=/hcnphz-63/obj55 2023-09-29T23:02:46.204 DEBUG:boto:auth_path=/hcnphz-63/obj55 2023-09-29T23:02:46.204 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.204 DEBUG:boto:Path: /hcnphz-63/obj55 2023-09-29T23:02:46.204 DEBUG:boto:Data: 2023-09-29T23:02:46.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.204 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.205 DEBUG:boto:Params: {} 2023-09-29T23:02:46.205 DEBUG:boto:Token: None 2023-09-29T23:02:46.205 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj55 2023-09-29T23:02:46.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ZXBg8+Cg1AauOratjNMqhqQYIPc= 2023-09-29T23:02:46.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZXBg8+Cg1AauOratjNMqhqQYIPc='} 2023-09-29T23:02:46.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8022be96d1c8ba4-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:46.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:46.210 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:02:46.253 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:46.253 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:46.329 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:02:46.329 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-63 2023-09-29T23:02:46.337 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-09-29T23:02:46.413 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.413 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.413 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.413 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.413 DEBUG:boto:Data: 2023-09-29T23:02:46.413 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.413 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.414 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.414 DEBUG:boto:Params: {} 2023-09-29T23:02:46.414 DEBUG:boto:Token: None 2023-09-29T23:02:46.414 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.414 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.414 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.419 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f241fe4fceb46a-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.419 DEBUG:boto:path=/hcnphz-63/obj17 2023-09-29T23:02:46.419 DEBUG:boto:auth_path=/hcnphz-63/obj17 2023-09-29T23:02:46.419 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.420 DEBUG:boto:Path: /hcnphz-63/obj17 2023-09-29T23:02:46.420 DEBUG:boto:Data: 2023-09-29T23:02:46.420 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.420 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.420 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.420 DEBUG:boto:Params: {} 2023-09-29T23:02:46.420 DEBUG:boto:Token: None 2023-09-29T23:02:46.421 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj17 2023-09-29T23:02:46.421 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0zKe4BeCSVEnP1LSlP/HhUZxjjA= 2023-09-29T23:02:46.421 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0zKe4BeCSVEnP1LSlP/HhUZxjjA='} 2023-09-29T23:02:46.462 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b66d44bb25c4575d-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.463 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.463 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.463 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.463 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.463 DEBUG:boto:Data: 2023-09-29T23:02:46.463 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.463 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.464 DEBUG:boto:Params: {} 2023-09-29T23:02:46.464 DEBUG:boto:Token: None 2023-09-29T23:02:46.464 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.464 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.469 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe5bf73a964d79f-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.469 DEBUG:boto:path=/hcnphz-63/obj34 2023-09-29T23:02:46.469 DEBUG:boto:auth_path=/hcnphz-63/obj34 2023-09-29T23:02:46.470 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.470 DEBUG:boto:Path: /hcnphz-63/obj34 2023-09-29T23:02:46.470 DEBUG:boto:Data: 2023-09-29T23:02:46.470 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.470 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.470 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.470 DEBUG:boto:Params: {} 2023-09-29T23:02:46.470 DEBUG:boto:Token: None 2023-09-29T23:02:46.471 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj34 2023-09-29T23:02:46.471 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ODDMH7IInvwpT5cyz2YtAf6SDkI= 2023-09-29T23:02:46.471 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ODDMH7IInvwpT5cyz2YtAf6SDkI='} 2023-09-29T23:02:46.482 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026c1b553d3c8cfaf-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.482 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.482 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.482 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.482 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.482 DEBUG:boto:Data: 2023-09-29T23:02:46.483 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.483 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.483 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.483 DEBUG:boto:Params: {} 2023-09-29T23:02:46.483 DEBUG:boto:Token: None 2023-09-29T23:02:46.483 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.483 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.483 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000611594259b0daf2a-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.488 DEBUG:boto:path=/hcnphz-63/obj51 2023-09-29T23:02:46.488 DEBUG:boto:auth_path=/hcnphz-63/obj51 2023-09-29T23:02:46.488 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.488 DEBUG:boto:Path: /hcnphz-63/obj51 2023-09-29T23:02:46.488 DEBUG:boto:Data: 2023-09-29T23:02:46.488 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.489 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.489 DEBUG:boto:Params: {} 2023-09-29T23:02:46.489 DEBUG:boto:Token: None 2023-09-29T23:02:46.489 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj51 2023-09-29T23:02:46.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iPfiH3ImjZDxHOUhEys8rS7aaXc= 2023-09-29T23:02:46.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iPfiH3ImjZDxHOUhEys8rS7aaXc='} 2023-09-29T23:02:46.534 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052731fbdbe86c217-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.535 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.535 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.535 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.535 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.535 DEBUG:boto:Data: 2023-09-29T23:02:46.535 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.535 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.535 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.535 DEBUG:boto:Params: {} 2023-09-29T23:02:46.536 DEBUG:boto:Token: None 2023-09-29T23:02:46.536 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.536 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.536 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.541 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065c3e862aa8d4499-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.541 DEBUG:boto:path=/hcnphz-63/obj68 2023-09-29T23:02:46.541 DEBUG:boto:auth_path=/hcnphz-63/obj68 2023-09-29T23:02:46.541 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.541 DEBUG:boto:Path: /hcnphz-63/obj68 2023-09-29T23:02:46.541 DEBUG:boto:Data: 2023-09-29T23:02:46.541 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.542 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.542 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.542 DEBUG:boto:Params: {} 2023-09-29T23:02:46.542 DEBUG:boto:Token: None 2023-09-29T23:02:46.542 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj68 2023-09-29T23:02:46.542 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wPunL7oDPNxFjYU1g50Uwl752O0= 2023-09-29T23:02:46.542 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wPunL7oDPNxFjYU1g50Uwl752O0='} 2023-09-29T23:02:46.554 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e15e93de8b0866e-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.554 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.554 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.554 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.554 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.554 DEBUG:boto:Data: 2023-09-29T23:02:46.554 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.554 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.555 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.555 DEBUG:boto:Params: {} 2023-09-29T23:02:46.555 DEBUG:boto:Token: None 2023-09-29T23:02:46.555 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.555 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.555 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.559 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c02f48fdc01227a-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.559 DEBUG:boto:path=/hcnphz-63/obj85 2023-09-29T23:02:46.560 DEBUG:boto:auth_path=/hcnphz-63/obj85 2023-09-29T23:02:46.560 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.560 DEBUG:boto:Path: /hcnphz-63/obj85 2023-09-29T23:02:46.560 DEBUG:boto:Data: 2023-09-29T23:02:46.560 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.560 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.560 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.560 DEBUG:boto:Params: {} 2023-09-29T23:02:46.560 DEBUG:boto:Token: None 2023-09-29T23:02:46.561 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj85 2023-09-29T23:02:46.561 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6s0MFTX73hHVwLWXkhP7Nz9q8RU= 2023-09-29T23:02:46.561 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6s0MFTX73hHVwLWXkhP7Nz9q8RU='} 2023-09-29T23:02:46.602 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba926529d81888b4-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:46.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:46.603 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:02:46.651 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:46.651 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:46.728 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:02:46.728 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-63 2023-09-29T23:02:46.741 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-09-29T23:02:46.830 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.830 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.831 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.831 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.831 DEBUG:boto:Data: 2023-09-29T23:02:46.831 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.831 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.831 DEBUG:boto:Params: {} 2023-09-29T23:02:46.831 DEBUG:boto:Token: None 2023-09-29T23:02:46.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.837 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e09dc26bab91b2-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.837 DEBUG:boto:path=/hcnphz-63/obj19 2023-09-29T23:02:46.837 DEBUG:boto:auth_path=/hcnphz-63/obj19 2023-09-29T23:02:46.837 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.838 DEBUG:boto:Path: /hcnphz-63/obj19 2023-09-29T23:02:46.838 DEBUG:boto:Data: 2023-09-29T23:02:46.838 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.838 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.838 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.838 DEBUG:boto:Params: {} 2023-09-29T23:02:46.838 DEBUG:boto:Token: None 2023-09-29T23:02:46.838 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj19 2023-09-29T23:02:46.838 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ps5YyJST8EgnFDUDmU04vH55QPw= 2023-09-29T23:02:46.839 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ps5YyJST8EgnFDUDmU04vH55QPw='} 2023-09-29T23:02:46.849 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2b1b54557245f32-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.850 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.850 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.850 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.850 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.850 DEBUG:boto:Data: 2023-09-29T23:02:46.850 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.851 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.851 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.851 DEBUG:boto:Params: {} 2023-09-29T23:02:46.851 DEBUG:boto:Token: None 2023-09-29T23:02:46.851 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.851 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.852 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.855 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046f689522b89f716-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.855 DEBUG:boto:path=/hcnphz-63/obj38 2023-09-29T23:02:46.856 DEBUG:boto:auth_path=/hcnphz-63/obj38 2023-09-29T23:02:46.856 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.856 DEBUG:boto:Path: /hcnphz-63/obj38 2023-09-29T23:02:46.856 DEBUG:boto:Data: 2023-09-29T23:02:46.856 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.856 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.856 DEBUG:boto:Params: {} 2023-09-29T23:02:46.856 DEBUG:boto:Token: None 2023-09-29T23:02:46.857 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj38 2023-09-29T23:02:46.857 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wwznUoV+YviHjspkK27h4TMeTNU= 2023-09-29T23:02:46.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wwznUoV+YviHjspkK27h4TMeTNU='} 2023-09-29T23:02:46.902 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c4ed7870716cde5-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.903 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.903 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.903 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.903 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.903 DEBUG:boto:Data: 2023-09-29T23:02:46.903 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.904 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.904 DEBUG:boto:Params: {} 2023-09-29T23:02:46.904 DEBUG:boto:Token: None 2023-09-29T23:02:46.904 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.904 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.910 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002719abea70655cb0-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.910 DEBUG:boto:path=/hcnphz-63/obj57 2023-09-29T23:02:46.910 DEBUG:boto:auth_path=/hcnphz-63/obj57 2023-09-29T23:02:46.911 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.911 DEBUG:boto:Path: /hcnphz-63/obj57 2023-09-29T23:02:46.911 DEBUG:boto:Data: 2023-09-29T23:02:46.911 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.911 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.911 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.911 DEBUG:boto:Params: {} 2023-09-29T23:02:46.911 DEBUG:boto:Token: None 2023-09-29T23:02:46.912 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj57 2023-09-29T23:02:46.912 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JyFZNiQ68ra64ZZYmqTYUULpRe0= 2023-09-29T23:02:46.912 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JyFZNiQ68ra64ZZYmqTYUULpRe0='} 2023-09-29T23:02:46.920 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a072f0513237261-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.920 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.920 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.920 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.920 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.920 DEBUG:boto:Data: 2023-09-29T23:02:46.921 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.921 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.921 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.921 DEBUG:boto:Params: {} 2023-09-29T23:02:46.922 DEBUG:boto:Token: None 2023-09-29T23:02:46.922 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.922 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.922 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.927 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037d3784059aa80b4-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.927 DEBUG:boto:path=/hcnphz-63/obj76 2023-09-29T23:02:46.927 DEBUG:boto:auth_path=/hcnphz-63/obj76 2023-09-29T23:02:46.928 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.928 DEBUG:boto:Path: /hcnphz-63/obj76 2023-09-29T23:02:46.928 DEBUG:boto:Data: 2023-09-29T23:02:46.928 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.928 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.928 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.928 DEBUG:boto:Params: {} 2023-09-29T23:02:46.929 DEBUG:boto:Token: None 2023-09-29T23:02:46.929 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj76 2023-09-29T23:02:46.929 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RBHcvAog+V1s/K2OXdpH+KBSgck= 2023-09-29T23:02:46.929 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBHcvAog+V1s/K2OXdpH+KBSgck='} 2023-09-29T23:02:46.974 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b10a05d1cde2e77d-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.975 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:46.975 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:46.975 DEBUG:boto:Method: HEAD 2023-09-29T23:02:46.975 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:46.976 DEBUG:boto:Data: 2023-09-29T23:02:46.976 DEBUG:boto:Headers: {} 2023-09-29T23:02:46.976 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.976 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.976 DEBUG:boto:Params: {} 2023-09-29T23:02:46.976 DEBUG:boto:Token: None 2023-09-29T23:02:46.976 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/ 2023-09-29T23:02:46.977 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:02:46.977 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:02:46.982 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3fbb75fb59b957f-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.982 DEBUG:boto:path=/hcnphz-63/obj95 2023-09-29T23:02:46.982 DEBUG:boto:auth_path=/hcnphz-63/obj95 2023-09-29T23:02:46.982 DEBUG:boto:Method: PUT 2023-09-29T23:02:46.983 DEBUG:boto:Path: /hcnphz-63/obj95 2023-09-29T23:02:46.983 DEBUG:boto:Data: 2023-09-29T23:02:46.983 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:46.983 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:46.983 DEBUG:boto:Port: 8000 2023-09-29T23:02:46.983 DEBUG:boto:Params: {} 2023-09-29T23:02:46.983 DEBUG:boto:Token: None 2023-09-29T23:02:46.983 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:46 GMT /hcnphz-63/obj95 2023-09-29T23:02:46.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v+1bcK2g0QQCbFB3D8ThV97J5Gs= 2023-09-29T23:02:46.984 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v+1bcK2g0QQCbFB3D8ThV97J5Gs='} 2023-09-29T23:02:46.991 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000817563f5fe181771-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:46.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:46.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:46.992 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:02:47.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:47.038 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:47.118 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:02:47.118 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-63 2023-09-29T23:02:47.131 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-09-29T23:02:47.229 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.229 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.230 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.230 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.230 DEBUG:boto:Data: 2023-09-29T23:02:47.230 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.230 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.230 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.230 DEBUG:boto:Params: {} 2023-09-29T23:02:47.230 DEBUG:boto:Token: None 2023-09-29T23:02:47.231 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.231 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.231 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.236 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca98c4153c82cd5e-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.236 DEBUG:boto:path=/hcnphz-63/obj23 2023-09-29T23:02:47.236 DEBUG:boto:auth_path=/hcnphz-63/obj23 2023-09-29T23:02:47.237 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.237 DEBUG:boto:Path: /hcnphz-63/obj23 2023-09-29T23:02:47.237 DEBUG:boto:Data: 2023-09-29T23:02:47.237 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.237 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.237 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.237 DEBUG:boto:Params: {} 2023-09-29T23:02:47.237 DEBUG:boto:Token: None 2023-09-29T23:02:47.237 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj23 2023-09-29T23:02:47.238 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MVLCoVzTF3UsFvyn52H9Jjm1VBU= 2023-09-29T23:02:47.238 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MVLCoVzTF3UsFvyn52H9Jjm1VBU='} 2023-09-29T23:02:47.287 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dd149ceb746b042-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.287 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.287 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.287 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.287 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.287 DEBUG:boto:Data: 2023-09-29T23:02:47.288 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.288 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.288 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.288 DEBUG:boto:Params: {} 2023-09-29T23:02:47.288 DEBUG:boto:Token: None 2023-09-29T23:02:47.288 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.288 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.289 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.294 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaecc0358eceb312-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.294 DEBUG:boto:path=/hcnphz-63/obj46 2023-09-29T23:02:47.295 DEBUG:boto:auth_path=/hcnphz-63/obj46 2023-09-29T23:02:47.295 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.295 DEBUG:boto:Path: /hcnphz-63/obj46 2023-09-29T23:02:47.295 DEBUG:boto:Data: 2023-09-29T23:02:47.295 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.295 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.295 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.296 DEBUG:boto:Params: {} 2023-09-29T23:02:47.296 DEBUG:boto:Token: None 2023-09-29T23:02:47.296 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj46 2023-09-29T23:02:47.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ctMCAtc22knIfa5uwCLqvZbqCuI= 2023-09-29T23:02:47.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ctMCAtc22knIfa5uwCLqvZbqCuI='} 2023-09-29T23:02:47.307 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c7cc7f496659149-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.307 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.307 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.307 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.307 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.307 DEBUG:boto:Data: 2023-09-29T23:02:47.308 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.308 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.308 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.308 DEBUG:boto:Params: {} 2023-09-29T23:02:47.308 DEBUG:boto:Token: None 2023-09-29T23:02:47.308 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.308 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.313 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000640776d3bbf8427b-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.313 DEBUG:boto:path=/hcnphz-63/obj69 2023-09-29T23:02:47.313 DEBUG:boto:auth_path=/hcnphz-63/obj69 2023-09-29T23:02:47.314 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.314 DEBUG:boto:Path: /hcnphz-63/obj69 2023-09-29T23:02:47.314 DEBUG:boto:Data: 2023-09-29T23:02:47.314 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.314 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.314 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.314 DEBUG:boto:Params: {} 2023-09-29T23:02:47.314 DEBUG:boto:Token: None 2023-09-29T23:02:47.315 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj69 2023-09-29T23:02:47.315 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:8M20CrRUTC/suQZInGbrNpaMRpc= 2023-09-29T23:02:47.315 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8M20CrRUTC/suQZInGbrNpaMRpc='} 2023-09-29T23:02:47.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b8fb71a423e3d8d-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.359 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.359 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.359 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.359 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.359 DEBUG:boto:Data: 2023-09-29T23:02:47.359 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.360 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.360 DEBUG:boto:Params: {} 2023-09-29T23:02:47.360 DEBUG:boto:Token: None 2023-09-29T23:02:47.360 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.366 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002428654c9f8415ff-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.366 DEBUG:boto:path=/hcnphz-63/obj92 2023-09-29T23:02:47.366 DEBUG:boto:auth_path=/hcnphz-63/obj92 2023-09-29T23:02:47.366 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.367 DEBUG:boto:Path: /hcnphz-63/obj92 2023-09-29T23:02:47.367 DEBUG:boto:Data: 2023-09-29T23:02:47.367 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.367 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.367 DEBUG:boto:Params: {} 2023-09-29T23:02:47.367 DEBUG:boto:Token: None 2023-09-29T23:02:47.367 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj92 2023-09-29T23:02:47.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0etSwld5W7yrU4u9m2iA41CPv6o= 2023-09-29T23:02:47.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0etSwld5W7yrU4u9m2iA41CPv6o='} 2023-09-29T23:02:47.378 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c797b20651f5e4f3-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.378 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.379 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.379 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.379 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.379 DEBUG:boto:Data: 2023-09-29T23:02:47.379 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.379 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.379 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.379 DEBUG:boto:Params: {} 2023-09-29T23:02:47.379 DEBUG:boto:Token: None 2023-09-29T23:02:47.379 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.380 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.380 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.383 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfe54fa6100eafc7-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.383 DEBUG:boto:path=/hcnphz-63/obj115 2023-09-29T23:02:47.384 DEBUG:boto:auth_path=/hcnphz-63/obj115 2023-09-29T23:02:47.384 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.384 DEBUG:boto:Path: /hcnphz-63/obj115 2023-09-29T23:02:47.384 DEBUG:boto:Data: 2023-09-29T23:02:47.384 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.384 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.384 DEBUG:boto:Params: {} 2023-09-29T23:02:47.385 DEBUG:boto:Token: None 2023-09-29T23:02:47.385 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj115 2023-09-29T23:02:47.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fEuYBInLTwKQgQFQTEkBh3bjY40= 2023-09-29T23:02:47.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fEuYBInLTwKQgQFQTEkBh3bjY40='} 2023-09-29T23:02:47.434 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efeb4afdc1676fa7-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:47.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:47.435 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:02:47.489 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:47.489 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:47.573 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:02:47.573 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-63 2023-09-29T23:02:47.590 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-09-29T23:02:47.699 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.700 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.700 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.700 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.700 DEBUG:boto:Data: 2023-09-29T23:02:47.700 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.700 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.700 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.700 DEBUG:boto:Params: {} 2023-09-29T23:02:47.701 DEBUG:boto:Token: None 2023-09-29T23:02:47.701 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.701 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.702 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.707 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006808f738291ad124-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.708 DEBUG:boto:path=/hcnphz-63/obj29 2023-09-29T23:02:47.708 DEBUG:boto:auth_path=/hcnphz-63/obj29 2023-09-29T23:02:47.709 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.709 DEBUG:boto:Path: /hcnphz-63/obj29 2023-09-29T23:02:47.709 DEBUG:boto:Data: 2023-09-29T23:02:47.709 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.709 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.709 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.710 DEBUG:boto:Params: {} 2023-09-29T23:02:47.710 DEBUG:boto:Token: None 2023-09-29T23:02:47.710 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj29 2023-09-29T23:02:47.710 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7YdFd52UB0HpOMadJN+s7RreNs0= 2023-09-29T23:02:47.710 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7YdFd52UB0HpOMadJN+s7RreNs0='} 2023-09-29T23:02:47.720 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d13fa03a8b65a788-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.720 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.720 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.721 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.721 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.721 DEBUG:boto:Data: 2023-09-29T23:02:47.721 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.721 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.721 DEBUG:boto:Params: {} 2023-09-29T23:02:47.721 DEBUG:boto:Token: None 2023-09-29T23:02:47.722 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.727 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097e27d3e4d9196e7-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.727 DEBUG:boto:path=/hcnphz-63/obj58 2023-09-29T23:02:47.727 DEBUG:boto:auth_path=/hcnphz-63/obj58 2023-09-29T23:02:47.727 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.727 DEBUG:boto:Path: /hcnphz-63/obj58 2023-09-29T23:02:47.727 DEBUG:boto:Data: 2023-09-29T23:02:47.728 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.728 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.728 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.728 DEBUG:boto:Params: {} 2023-09-29T23:02:47.728 DEBUG:boto:Token: None 2023-09-29T23:02:47.728 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj58 2023-09-29T23:02:47.728 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xbawLMafL0BXIIvQBT3xqLnxR0o= 2023-09-29T23:02:47.729 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xbawLMafL0BXIIvQBT3xqLnxR0o='} 2023-09-29T23:02:47.775 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035e60c164cafbc96-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.775 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.775 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.775 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.775 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.776 DEBUG:boto:Data: 2023-09-29T23:02:47.776 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.776 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.776 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.777 DEBUG:boto:Params: {} 2023-09-29T23:02:47.777 DEBUG:boto:Token: None 2023-09-29T23:02:47.777 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.777 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.777 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.784 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d28d65eb380a6109-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.784 DEBUG:boto:path=/hcnphz-63/obj87 2023-09-29T23:02:47.785 DEBUG:boto:auth_path=/hcnphz-63/obj87 2023-09-29T23:02:47.785 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.785 DEBUG:boto:Path: /hcnphz-63/obj87 2023-09-29T23:02:47.785 DEBUG:boto:Data: 2023-09-29T23:02:47.785 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.785 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.785 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.786 DEBUG:boto:Params: {} 2023-09-29T23:02:47.786 DEBUG:boto:Token: None 2023-09-29T23:02:47.786 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj87 2023-09-29T23:02:47.786 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gySpIxGqMwVe3aseqv38+WWXyAY= 2023-09-29T23:02:47.786 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gySpIxGqMwVe3aseqv38+WWXyAY='} 2023-09-29T23:02:47.796 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d8354ef3c6f88d7-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.796 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.796 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.796 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.797 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.797 DEBUG:boto:Data: 2023-09-29T23:02:47.797 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.797 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.797 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.798 DEBUG:boto:Params: {} 2023-09-29T23:02:47.798 DEBUG:boto:Token: None 2023-09-29T23:02:47.799 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.799 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.799 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.804 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032ed4e00a783852d-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.804 DEBUG:boto:path=/hcnphz-63/obj116 2023-09-29T23:02:47.804 DEBUG:boto:auth_path=/hcnphz-63/obj116 2023-09-29T23:02:47.804 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.804 DEBUG:boto:Path: /hcnphz-63/obj116 2023-09-29T23:02:47.805 DEBUG:boto:Data: 2023-09-29T23:02:47.805 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.805 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.805 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.805 DEBUG:boto:Params: {} 2023-09-29T23:02:47.805 DEBUG:boto:Token: None 2023-09-29T23:02:47.806 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj116 2023-09-29T23:02:47.806 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KNRLhzYaTC8DDpDm4jkOXTwT2dg= 2023-09-29T23:02:47.806 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNRLhzYaTC8DDpDm4jkOXTwT2dg='} 2023-09-29T23:02:47.854 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033c941bb74d0ab1d-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.855 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:47.855 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:47.855 DEBUG:boto:Method: HEAD 2023-09-29T23:02:47.855 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:47.855 DEBUG:boto:Data: 2023-09-29T23:02:47.855 DEBUG:boto:Headers: {} 2023-09-29T23:02:47.856 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.856 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.856 DEBUG:boto:Params: {} 2023-09-29T23:02:47.856 DEBUG:boto:Token: None 2023-09-29T23:02:47.856 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/ 2023-09-29T23:02:47.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:02:47.857 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:02:47.863 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000336333591df41abb-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.863 DEBUG:boto:path=/hcnphz-63/obj145 2023-09-29T23:02:47.863 DEBUG:boto:auth_path=/hcnphz-63/obj145 2023-09-29T23:02:47.863 DEBUG:boto:Method: PUT 2023-09-29T23:02:47.863 DEBUG:boto:Path: /hcnphz-63/obj145 2023-09-29T23:02:47.863 DEBUG:boto:Data: 2023-09-29T23:02:47.864 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:47.864 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:47.864 DEBUG:boto:Port: 8000 2023-09-29T23:02:47.864 DEBUG:boto:Params: {} 2023-09-29T23:02:47.864 DEBUG:boto:Token: None 2023-09-29T23:02:47.864 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:47 GMT /hcnphz-63/obj145 2023-09-29T23:02:47.864 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PG6RNzC3aDTjvsPohNwmvlVQB70= 2023-09-29T23:02:47.865 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PG6RNzC3aDTjvsPohNwmvlVQB70='} 2023-09-29T23:02:47.874 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021857fd48101569b-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:47.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:47.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:47.875 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:02:47.922 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:47.922 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:48.006 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:02:48.006 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-63 2023-09-29T23:02:48.027 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-09-29T23:02:48.146 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.147 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.147 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.147 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.148 DEBUG:boto:Data: 2023-09-29T23:02:48.148 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.148 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.148 DEBUG:boto:Params: {} 2023-09-29T23:02:48.149 DEBUG:boto:Token: None 2023-09-29T23:02:48.149 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.156 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005766a85acbd49b8d-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.157 DEBUG:boto:path=/hcnphz-63/obj31 2023-09-29T23:02:48.157 DEBUG:boto:auth_path=/hcnphz-63/obj31 2023-09-29T23:02:48.157 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.157 DEBUG:boto:Path: /hcnphz-63/obj31 2023-09-29T23:02:48.157 DEBUG:boto:Data: 2023-09-29T23:02:48.157 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.157 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.157 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.158 DEBUG:boto:Params: {} 2023-09-29T23:02:48.158 DEBUG:boto:Token: None 2023-09-29T23:02:48.158 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj31 2023-09-29T23:02:48.158 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fKrKd+s7mIjwM/T3q57SC4R3h7M= 2023-09-29T23:02:48.158 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fKrKd+s7mIjwM/T3q57SC4R3h7M='} 2023-09-29T23:02:48.206 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000914d35a982c9a014-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.207 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.207 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.207 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.207 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.207 DEBUG:boto:Data: 2023-09-29T23:02:48.208 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.208 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.208 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.208 DEBUG:boto:Params: {} 2023-09-29T23:02:48.208 DEBUG:boto:Token: None 2023-09-29T23:02:48.208 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.208 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.209 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.214 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045f974f3e0f5b741-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.214 DEBUG:boto:path=/hcnphz-63/obj62 2023-09-29T23:02:48.214 DEBUG:boto:auth_path=/hcnphz-63/obj62 2023-09-29T23:02:48.214 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.214 DEBUG:boto:Path: /hcnphz-63/obj62 2023-09-29T23:02:48.215 DEBUG:boto:Data: 2023-09-29T23:02:48.215 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.215 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.215 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.215 DEBUG:boto:Params: {} 2023-09-29T23:02:48.215 DEBUG:boto:Token: None 2023-09-29T23:02:48.215 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj62 2023-09-29T23:02:48.216 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gL7iKGuacMWNP8bAnaANSrPUl1Y= 2023-09-29T23:02:48.216 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gL7iKGuacMWNP8bAnaANSrPUl1Y='} 2023-09-29T23:02:48.225 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000784d55ef5956295f-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.226 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.226 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.226 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.226 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.226 DEBUG:boto:Data: 2023-09-29T23:02:48.227 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.227 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.227 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.227 DEBUG:boto:Params: {} 2023-09-29T23:02:48.227 DEBUG:boto:Token: None 2023-09-29T23:02:48.227 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.227 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.232 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047a481ace3ea5317-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.233 DEBUG:boto:path=/hcnphz-63/obj93 2023-09-29T23:02:48.233 DEBUG:boto:auth_path=/hcnphz-63/obj93 2023-09-29T23:02:48.233 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.233 DEBUG:boto:Path: /hcnphz-63/obj93 2023-09-29T23:02:48.233 DEBUG:boto:Data: 2023-09-29T23:02:48.233 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.233 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.234 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.234 DEBUG:boto:Params: {} 2023-09-29T23:02:48.234 DEBUG:boto:Token: None 2023-09-29T23:02:48.234 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj93 2023-09-29T23:02:48.234 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:BN2tl3FwzzXMU+XE1JJDr0C2ozc= 2023-09-29T23:02:48.234 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BN2tl3FwzzXMU+XE1JJDr0C2ozc='} 2023-09-29T23:02:48.278 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4c96c74d4bbff73-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.279 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.279 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.279 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.279 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.279 DEBUG:boto:Data: 2023-09-29T23:02:48.279 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.280 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.280 DEBUG:boto:Params: {} 2023-09-29T23:02:48.280 DEBUG:boto:Token: None 2023-09-29T23:02:48.280 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.280 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.288 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087d64ba36ca8dea3-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.288 DEBUG:boto:path=/hcnphz-63/obj124 2023-09-29T23:02:48.288 DEBUG:boto:auth_path=/hcnphz-63/obj124 2023-09-29T23:02:48.288 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.289 DEBUG:boto:Path: /hcnphz-63/obj124 2023-09-29T23:02:48.289 DEBUG:boto:Data: 2023-09-29T23:02:48.289 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.289 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.289 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.289 DEBUG:boto:Params: {} 2023-09-29T23:02:48.289 DEBUG:boto:Token: None 2023-09-29T23:02:48.289 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj124 2023-09-29T23:02:48.290 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MCPIIkV2d1x6wgMjtqudDN0nYf8= 2023-09-29T23:02:48.290 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MCPIIkV2d1x6wgMjtqudDN0nYf8='} 2023-09-29T23:02:48.299 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aad2951f2349d5e-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.299 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.299 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.299 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.299 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.299 DEBUG:boto:Data: 2023-09-29T23:02:48.299 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.300 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.300 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.300 DEBUG:boto:Params: {} 2023-09-29T23:02:48.300 DEBUG:boto:Token: None 2023-09-29T23:02:48.300 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.300 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.300 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.305 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4963863d24ae374-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.305 DEBUG:boto:path=/hcnphz-63/obj155 2023-09-29T23:02:48.305 DEBUG:boto:auth_path=/hcnphz-63/obj155 2023-09-29T23:02:48.305 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.305 DEBUG:boto:Path: /hcnphz-63/obj155 2023-09-29T23:02:48.305 DEBUG:boto:Data: 2023-09-29T23:02:48.305 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.306 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.306 DEBUG:boto:Params: {} 2023-09-29T23:02:48.306 DEBUG:boto:Token: None 2023-09-29T23:02:48.306 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj155 2023-09-29T23:02:48.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Nc2UZt1lezJ0Yq8Z1RV59NsuJ9g= 2023-09-29T23:02:48.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nc2UZt1lezJ0Yq8Z1RV59NsuJ9g='} 2023-09-29T23:02:48.350 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001392567c8a8f0b36-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:48.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:02:48.351 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:02:48.401 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:48.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:48.489 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:02:48.489 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-63 2023-09-29T23:02:48.514 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-09-29T23:02:48.639 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.639 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.639 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.639 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.639 DEBUG:boto:Data: 2023-09-29T23:02:48.639 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.640 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.640 DEBUG:boto:Params: {} 2023-09-29T23:02:48.640 DEBUG:boto:Token: None 2023-09-29T23:02:48.640 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.647 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025b6e1f7fc7fe3ab-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.648 DEBUG:boto:path=/hcnphz-63/obj37 2023-09-29T23:02:48.648 DEBUG:boto:auth_path=/hcnphz-63/obj37 2023-09-29T23:02:48.648 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.648 DEBUG:boto:Path: /hcnphz-63/obj37 2023-09-29T23:02:48.648 DEBUG:boto:Data: 2023-09-29T23:02:48.648 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.648 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.648 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.648 DEBUG:boto:Params: {} 2023-09-29T23:02:48.649 DEBUG:boto:Token: None 2023-09-29T23:02:48.649 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj37 2023-09-29T23:02:48.649 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:f4AsM2zQVwu+sN1YkgYrfYIMOqs= 2023-09-29T23:02:48.649 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f4AsM2zQVwu+sN1YkgYrfYIMOqs='} 2023-09-29T23:02:48.659 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b48b884e97185892-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.659 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.659 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.660 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.660 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.660 DEBUG:boto:Data: 2023-09-29T23:02:48.660 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.660 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.660 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.661 DEBUG:boto:Params: {} 2023-09-29T23:02:48.661 DEBUG:boto:Token: None 2023-09-29T23:02:48.661 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.661 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.662 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.667 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8cefdbb17af77ee-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.668 DEBUG:boto:path=/hcnphz-63/obj74 2023-09-29T23:02:48.668 DEBUG:boto:auth_path=/hcnphz-63/obj74 2023-09-29T23:02:48.668 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.669 DEBUG:boto:Path: /hcnphz-63/obj74 2023-09-29T23:02:48.669 DEBUG:boto:Data: 2023-09-29T23:02:48.669 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.669 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.669 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.670 DEBUG:boto:Params: {} 2023-09-29T23:02:48.670 DEBUG:boto:Token: None 2023-09-29T23:02:48.670 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj74 2023-09-29T23:02:48.670 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YwLjbXbtDdz+2yH1D6DbRjtU5ec= 2023-09-29T23:02:48.671 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YwLjbXbtDdz+2yH1D6DbRjtU5ec='} 2023-09-29T23:02:48.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ecbe35e3c49760e-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.715 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.715 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.715 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.715 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.715 DEBUG:boto:Data: 2023-09-29T23:02:48.716 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.716 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.716 DEBUG:boto:Params: {} 2023-09-29T23:02:48.716 DEBUG:boto:Token: None 2023-09-29T23:02:48.716 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.724 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f7e2d324b9ba2b4-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.724 DEBUG:boto:path=/hcnphz-63/obj111 2023-09-29T23:02:48.724 DEBUG:boto:auth_path=/hcnphz-63/obj111 2023-09-29T23:02:48.724 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.724 DEBUG:boto:Path: /hcnphz-63/obj111 2023-09-29T23:02:48.724 DEBUG:boto:Data: 2023-09-29T23:02:48.725 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.725 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.725 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.725 DEBUG:boto:Params: {} 2023-09-29T23:02:48.725 DEBUG:boto:Token: None 2023-09-29T23:02:48.725 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj111 2023-09-29T23:02:48.725 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bgKW9zhmkCIky+Yh7aTIerigVCU= 2023-09-29T23:02:48.725 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bgKW9zhmkCIky+Yh7aTIerigVCU='} 2023-09-29T23:02:48.734 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093960dc7f26fb6bd-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.734 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.735 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.735 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.735 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.735 DEBUG:boto:Data: 2023-09-29T23:02:48.735 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.735 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.735 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.735 DEBUG:boto:Params: {} 2023-09-29T23:02:48.735 DEBUG:boto:Token: None 2023-09-29T23:02:48.736 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.736 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.736 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.741 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075ddacd481867411-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.742 DEBUG:boto:path=/hcnphz-63/obj148 2023-09-29T23:02:48.742 DEBUG:boto:auth_path=/hcnphz-63/obj148 2023-09-29T23:02:48.742 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.742 DEBUG:boto:Path: /hcnphz-63/obj148 2023-09-29T23:02:48.742 DEBUG:boto:Data: 2023-09-29T23:02:48.742 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.742 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.742 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.743 DEBUG:boto:Params: {} 2023-09-29T23:02:48.743 DEBUG:boto:Token: None 2023-09-29T23:02:48.743 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj148 2023-09-29T23:02:48.743 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VXI1Qq0yn4YICiMdB0QFLF8bwGs= 2023-09-29T23:02:48.743 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VXI1Qq0yn4YICiMdB0QFLF8bwGs='} 2023-09-29T23:02:48.791 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008742674c941057e5-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.791 DEBUG:boto:path=/hcnphz-63/ 2023-09-29T23:02:48.791 DEBUG:boto:auth_path=/hcnphz-63/ 2023-09-29T23:02:48.791 DEBUG:boto:Method: HEAD 2023-09-29T23:02:48.791 DEBUG:boto:Path: /hcnphz-63/ 2023-09-29T23:02:48.791 DEBUG:boto:Data: 2023-09-29T23:02:48.791 DEBUG:boto:Headers: {} 2023-09-29T23:02:48.792 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.792 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.792 DEBUG:boto:Params: {} 2023-09-29T23:02:48.792 DEBUG:boto:Token: None 2023-09-29T23:02:48.792 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/ 2023-09-29T23:02:48.792 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:02:48.792 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:02:48.801 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032b62e1be2425f68-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.801 DEBUG:boto:path=/hcnphz-63/obj185 2023-09-29T23:02:48.801 DEBUG:boto:auth_path=/hcnphz-63/obj185 2023-09-29T23:02:48.802 DEBUG:boto:Method: PUT 2023-09-29T23:02:48.802 DEBUG:boto:Path: /hcnphz-63/obj185 2023-09-29T23:02:48.802 DEBUG:boto:Data: 2023-09-29T23:02:48.802 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:02:48.802 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:02:48.802 DEBUG:boto:Port: 8000 2023-09-29T23:02:48.802 DEBUG:boto:Params: {} 2023-09-29T23:02:48.802 DEBUG:boto:Token: None 2023-09-29T23:02:48.803 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:02:48 GMT /hcnphz-63/obj185 2023-09-29T23:02:48.803 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EWroXpbMFg9Ex/gnfuKnAw3jGDk= 2023-09-29T23:02:48.803 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EWroXpbMFg9Ex/gnfuKnAw3jGDk='} 2023-09-29T23:02:48.812 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000006f1cfffbf56272-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:02:48.812 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-63 2023-09-29T23:02:48.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:02:48.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:02:48.813 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:02:48.861 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:48.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:48.929 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:48.928+0000 7fbb8821cf40 1 waiting to reach incremental sync.. 2023-09-29T23:02:53.930 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:53.928+0000 7fbb8821cf40 1 waiting to reach incremental sync.. 2023-09-29T23:02:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:58.936+0000 7fbb8821cf40 1 bucket sync caught up with source: 2023-09-29T23:02:58.936 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , 00000000002.118.6, , , , , , 00000000002.115.6, , , , , 00000000002.105.6, , , , , , , , , , , , , , , 00000000001.131.6, , , , 00000000001.107.6] 2023-09-29T23:02:58.936 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , 00000000002.118.6, , , , , , 00000000002.115.6, , , , , 00000000002.105.6, , , , , , , , , , , , , , , 00000000001.131.6, , , , 00000000001.107.6] 2023-09-29T23:02:58.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:58.936+0000 7fbb8821cf40 0 bucket checkpoint complete 2023-09-29T23:02:58.961 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-63 2023-09-29T23:02:58.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:02:58.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:02:58.962 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:02:59.012 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:02:59.012 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:02:59.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:02:59.080+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:04.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:04.080+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:09.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:09.080+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:14.083 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:14.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:19.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:19.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:24.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:24.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:29.086 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:29.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:34.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:34.087+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:39.088 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:39.087+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:44.089 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:44.091+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:49.090 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:49.091+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:54.091 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:54.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:03:59.092 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:03:59.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:04.093 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:04.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:09.094 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:09.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:14.095 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:14.094+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:19.096 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:19.094+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:24.097 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:24.094+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:29.098 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:29.093+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:34.099 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:34.097+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:39.100 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:39.101+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:44.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:44.101+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:49.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:49.101+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:54.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:54.105+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:04:59.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:04:59.105+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:04.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:04.105+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:09.107 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:09.108+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:14.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:14.108+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:19.110 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:19.108+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:24.111 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:24.112+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:29.112 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:29.112+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:34.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:34.112+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:39.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:39.116+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:44.116 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:44.115+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:49.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:49.115+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:54.118 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:54.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:05:59.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:05:59.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:04.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:04.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:09.122 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:09.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:14.123 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:14.123+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:19.124 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:19.123+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:24.125 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:24.122+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:29.126 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:29.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:34.127 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:34.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:39.128 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:39.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:44.129 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:44.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:49.130 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:49.130+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:54.132 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:54.130+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:06:59.133 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:06:59.130+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:04.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:04.133+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:09.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:09.133+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:14.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:14.137+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:19.138 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:19.137+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:24.139 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:24.141+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:29.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:29.141+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:30.253 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:07:30.251+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:07:34.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:34.141+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:39.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:39.144+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:44.143 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:44.144+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:49.144 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:49.144+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:54.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:54.144+0000 7fd4d3a25f40 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:07:54.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:54.144+0000 7fd4d3a25f40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:07:54.146 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:54.144+0000 7fd4d3a25f40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:07:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:07:54.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init ... FAIL 2023-09-29T23:07:54.181 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:07:54.182 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:07:54.182 DEBUG:boto:Method: PUT 2023-09-29T23:07:54.182 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:07:54.182 DEBUG:boto:Data: 2023-09-29T23:07:54.182 DEBUG:boto:Headers: {} 2023-09-29T23:07:54.182 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.182 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.182 DEBUG:boto:Params: {} 2023-09-29T23:07:54.183 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:07:54.183 DEBUG:boto:Token: None 2023-09-29T23:07:54.183 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/ 2023-09-29T23:07:54.183 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gHVhHSDGwpvcGz+geqan135uQKs= 2023-09-29T23:07:54.184 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gHVhHSDGwpvcGz+geqan135uQKs=', 'Content-Length': '0'} 2023-09-29T23:07:54.217 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000363e896aae6634de-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.218 DEBUG:rgw_multi.tests:created bucket=hcnphz-64 2023-09-29T23:07:54.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:07:54.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:07:54.219 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:07:54.258 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:07:54.259 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:07:54.314 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:07:54.337 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:07:54.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.337 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028874.218108_635.1", 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:07:54.218108Z" 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:07:54.338 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:07:54.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.339 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:07:54.339 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:07:54.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:07:54.339 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:07:54.340 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028874.218108_635.1', 2: '', 3: ''} 2023-09-29T23:07:54.340 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:07:54.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:07:54.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:07:54.341 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:07:54.384 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:07:54.384 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:07:54.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:54.440+0000 7f018c4e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:07:54.440 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:54.440+0000 7f018c4e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:07:54.466 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:07:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:07:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:07:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:07:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:07:54.466 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:07:54.467 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:07:54.468 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028874.218108_635.1", 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:07:54.218108Z", 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.469 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:07:54.470 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:07:54.471 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:07:54.472 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:07:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:07:54.473 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:07:54Z" 2023-09-29T23:07:54.473 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:07:54.474 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028874.218108_635.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:07:54.218108Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:07:54Z" } 2023-09-29T23:07:54.474 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028874.218108_635.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:07:54.218108Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:07:54.474 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028874.218108_635.1', 2: '', 3: ''} 2023-09-29T23:07:54.474 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028874.218108_635.1', 2: '', 3: ''} 2023-09-29T23:07:54.475 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:07:54.475 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:07:54.475 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:07:54.475 DEBUG:boto:Method: HEAD 2023-09-29T23:07:54.475 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:07:54.475 DEBUG:boto:Data: 2023-09-29T23:07:54.475 DEBUG:boto:Headers: {} 2023-09-29T23:07:54.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.476 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.476 DEBUG:boto:Params: {} 2023-09-29T23:07:54.476 DEBUG:boto:Token: None 2023-09-29T23:07:54.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/ 2023-09-29T23:07:54.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:07:54.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:07:54.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b3eb7c7582c835a-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.480 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:07:54.480 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:07:54.480 DEBUG:boto:Method: PUT 2023-09-29T23:07:54.480 DEBUG:boto:Path: /hcnphz-64/obj11 2023-09-29T23:07:54.481 DEBUG:boto:Data: 2023-09-29T23:07:54.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:07:54.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.481 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.481 DEBUG:boto:Params: {} 2023-09-29T23:07:54.481 DEBUG:boto:Token: None 2023-09-29T23:07:54.481 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/obj11 2023-09-29T23:07:54.481 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:oeLb5xZ/52xdRBkcafX5wDJG0EA= 2023-09-29T23:07:54.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oeLb5xZ/52xdRBkcafX5wDJG0EA='} 2023-09-29T23:07:54.495 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c5448adb18e18f2-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.496 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:07:54.496 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:07:54.496 DEBUG:boto:Method: HEAD 2023-09-29T23:07:54.496 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:07:54.496 DEBUG:boto:Data: 2023-09-29T23:07:54.496 DEBUG:boto:Headers: {} 2023-09-29T23:07:54.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.496 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.496 DEBUG:boto:Params: {} 2023-09-29T23:07:54.497 DEBUG:boto:Token: None 2023-09-29T23:07:54.497 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/ 2023-09-29T23:07:54.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:07:54.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:07:54.500 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f163f178eb7881b7-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.500 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:07:54.500 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:07:54.500 DEBUG:boto:Method: PUT 2023-09-29T23:07:54.500 DEBUG:boto:Path: /hcnphz-64/obj22 2023-09-29T23:07:54.501 DEBUG:boto:Data: 2023-09-29T23:07:54.501 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:07:54.501 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.501 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.501 DEBUG:boto:Params: {} 2023-09-29T23:07:54.501 DEBUG:boto:Token: None 2023-09-29T23:07:54.501 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/obj22 2023-09-29T23:07:54.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uyye1TxlBZ9FKkRsHP4j4swNt5M= 2023-09-29T23:07:54.502 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uyye1TxlBZ9FKkRsHP4j4swNt5M='} 2023-09-29T23:07:54.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ead9b5914aeca06-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.547 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:07:54.547 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:07:54.547 DEBUG:boto:Method: HEAD 2023-09-29T23:07:54.547 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:07:54.547 DEBUG:boto:Data: 2023-09-29T23:07:54.547 DEBUG:boto:Headers: {} 2023-09-29T23:07:54.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.547 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.548 DEBUG:boto:Params: {} 2023-09-29T23:07:54.548 DEBUG:boto:Token: None 2023-09-29T23:07:54.548 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/ 2023-09-29T23:07:54.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:07:54.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:07:54.552 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a589ad1c4d641514-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.553 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:07:54.553 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:07:54.553 DEBUG:boto:Method: PUT 2023-09-29T23:07:54.553 DEBUG:boto:Path: /hcnphz-64/obj33 2023-09-29T23:07:54.553 DEBUG:boto:Data: 2023-09-29T23:07:54.553 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:07:54.553 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.553 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.554 DEBUG:boto:Params: {} 2023-09-29T23:07:54.554 DEBUG:boto:Token: None 2023-09-29T23:07:54.554 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/obj33 2023-09-29T23:07:54.554 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hvAKVPFQaAom5Qr+MUQFgAeo5n4= 2023-09-29T23:07:54.554 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hvAKVPFQaAom5Qr+MUQFgAeo5n4='} 2023-09-29T23:07:54.566 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c892055b32f67b86-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.566 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:07:54.566 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:07:54.567 DEBUG:boto:Method: HEAD 2023-09-29T23:07:54.567 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:07:54.567 DEBUG:boto:Data: 2023-09-29T23:07:54.567 DEBUG:boto:Headers: {} 2023-09-29T23:07:54.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.567 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.567 DEBUG:boto:Params: {} 2023-09-29T23:07:54.567 DEBUG:boto:Token: None 2023-09-29T23:07:54.567 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/ 2023-09-29T23:07:54.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:07:54.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:07:54.571 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc552f8587f3db75-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.571 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:07:54.572 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:07:54.572 DEBUG:boto:Method: PUT 2023-09-29T23:07:54.572 DEBUG:boto:Path: /hcnphz-64/obj44 2023-09-29T23:07:54.572 DEBUG:boto:Data: 2023-09-29T23:07:54.572 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:07:54.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.572 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.572 DEBUG:boto:Params: {} 2023-09-29T23:07:54.573 DEBUG:boto:Token: None 2023-09-29T23:07:54.573 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/obj44 2023-09-29T23:07:54.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HmfdsPwFbA17aMgeLKssyU/8BtE= 2023-09-29T23:07:54.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HmfdsPwFbA17aMgeLKssyU/8BtE='} 2023-09-29T23:07:54.618 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000966a5a23e6e121e-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.619 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:07:54.619 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:07:54.619 DEBUG:boto:Method: HEAD 2023-09-29T23:07:54.619 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:07:54.619 DEBUG:boto:Data: 2023-09-29T23:07:54.619 DEBUG:boto:Headers: {} 2023-09-29T23:07:54.619 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.619 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.620 DEBUG:boto:Params: {} 2023-09-29T23:07:54.620 DEBUG:boto:Token: None 2023-09-29T23:07:54.620 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/ 2023-09-29T23:07:54.620 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:07:54.620 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:07:54.624 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e507b653268aea0d-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.625 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:07:54.625 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:07:54.625 DEBUG:boto:Method: PUT 2023-09-29T23:07:54.625 DEBUG:boto:Path: /hcnphz-64/obj55 2023-09-29T23:07:54.625 DEBUG:boto:Data: 2023-09-29T23:07:54.625 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:07:54.625 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:07:54.625 DEBUG:boto:Port: 8000 2023-09-29T23:07:54.626 DEBUG:boto:Params: {} 2023-09-29T23:07:54.626 DEBUG:boto:Token: None 2023-09-29T23:07:54.626 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:07:54 GMT /hcnphz-64/obj55 2023-09-29T23:07:54.626 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eb161C01IjoPMiIYcqjHiAhdvyU= 2023-09-29T23:07:54.626 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eb161C01IjoPMiIYcqjHiAhdvyU='} 2023-09-29T23:07:54.635 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082958461c1f87a2b-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:07:54.635 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-64 2023-09-29T23:07:54.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:07:54.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:07:54.635 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:07:54.686 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:07:54.686 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:07:54.756 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:54.756+0000 7f211af40f40 1 waiting to reach incremental sync.. 2023-09-29T23:07:59.760 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:59.760+0000 7f211af40f40 1 bucket sync caught up with source: 2023-09-29T23:07:59.761 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.70.6, 00000000002.112.6, 00000000002.137.6] 2023-09-29T23:07:59.761 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.70.6, 00000000002.112.6, 00000000002.137.6] 2023-09-29T23:07:59.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:59.760+0000 7f211af40f40 0 bucket checkpoint complete 2023-09-29T23:07:59.790 DEBUG:rgw_multi.tests:bucket checkpoint: source=a2 target=a1 bucket=hcnphz-64 2023-09-29T23:07:59.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:07:59.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:07:59.791 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:07:59.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:07:59.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:07:59.953 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:07:59.952+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:08:04.954 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:04.952+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:08:09.955 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:09.956+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:08:14.956 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:14.956+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:08:19.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:19.959+0000 7ff052e21f40 1 bucket sync caught up with source: 2023-09-29T23:08:19.960 INFO:teuthology.orchestra.run.smithi112.stderr: local status: [, , , , , , , , 00000000001.159.6, 00000000002.316.6, 00000000002.196.6] 2023-09-29T23:08:19.960 INFO:teuthology.orchestra.run.smithi112.stderr: remote markers: [, , , , , , , , 00000000001.159.6, 00000000002.316.6, 00000000002.196.6] 2023-09-29T23:08:19.961 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:19.959+0000 7ff052e21f40 0 bucket checkpoint complete 2023-09-29T23:08:19.990 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hcnphz-64 zones={a2, a1} 2023-09-29T23:08:19.990 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:19.990 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:19.990 DEBUG:boto:Method: HEAD 2023-09-29T23:08:19.990 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:19.990 DEBUG:boto:Data: 2023-09-29T23:08:19.991 DEBUG:boto:Headers: {} 2023-09-29T23:08:19.991 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:19.991 DEBUG:boto:Port: 8001 2023-09-29T23:08:19.991 DEBUG:boto:Params: {} 2023-09-29T23:08:19.991 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:08:19.991 DEBUG:boto:Token: None 2023-09-29T23:08:19.992 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:19 GMT /hcnphz-64/ 2023-09-29T23:08:19.992 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo= 2023-09-29T23:08:19.992 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo=', 'Content-Length': '0'} 2023-09-29T23:08:19.997 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5c767151567594f-00651758e3-1ef7-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:19.997 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:19.997 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:19.997 DEBUG:boto:Method: HEAD 2023-09-29T23:08:19.998 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:19.998 DEBUG:boto:Data: 2023-09-29T23:08:19.998 DEBUG:boto:Headers: {} 2023-09-29T23:08:19.998 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:19.998 DEBUG:boto:Port: 8000 2023-09-29T23:08:19.998 DEBUG:boto:Params: {} 2023-09-29T23:08:19.998 DEBUG:boto:Token: None 2023-09-29T23:08:19.998 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:19 GMT /hcnphz-64/ 2023-09-29T23:08:19.999 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo= 2023-09-29T23:08:19.999 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo=', 'Content-Length': '0'} 2023-09-29T23:08:20.003 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000287908df22875a45-00651758e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.003 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2023-09-29T23:08:20.003 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.003 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.003 DEBUG:boto:path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.003 DEBUG:boto:auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.003 DEBUG:boto:Method: GET 2023-09-29T23:08:20.004 DEBUG:boto:Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.004 DEBUG:boto:Data: 2023-09-29T23:08:20.004 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.004 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.004 DEBUG:boto:Params: {} 2023-09-29T23:08:20.004 DEBUG:boto:Token: None 2023-09-29T23:08:20.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/?versions 2023-09-29T23:08:20.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:08:20.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:08:20.008 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50596144f6cf85c-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.008 DEBUG:boto:b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T23:08:20.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T23:08:20.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T23:08:20.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T23:08:20.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T23:08:20.010 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2023-09-29T23:08:20.010 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.010 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.010 DEBUG:boto:path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.010 DEBUG:boto:auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.010 DEBUG:boto:Method: GET 2023-09-29T23:08:20.010 DEBUG:boto:Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.010 DEBUG:boto:Data: 2023-09-29T23:08:20.010 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.010 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.011 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.011 DEBUG:boto:Params: {} 2023-09-29T23:08:20.011 DEBUG:boto:Token: None 2023-09-29T23:08:20.011 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/?versions 2023-09-29T23:08:20.011 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:08:20.011 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:08:20.015 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d2dd516037c8d7-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.015 DEBUG:boto:b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj11 2023-09-29T23:08:20.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj22 2023-09-29T23:08:20.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj33 2023-09-29T23:08:20.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj44 2023-09-29T23:08:20.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj55 2023-09-29T23:08:20.016 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.016 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.016 DEBUG:boto:path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.016 DEBUG:boto:auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.016 DEBUG:boto:Method: GET 2023-09-29T23:08:20.016 DEBUG:boto:Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.017 DEBUG:boto:Data: 2023-09-29T23:08:20.017 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.017 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.017 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.017 DEBUG:boto:Params: {} 2023-09-29T23:08:20.017 DEBUG:boto:Token: None 2023-09-29T23:08:20.017 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/?versions 2023-09-29T23:08:20.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:08:20.018 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:08:20.021 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d11ed34f398f27-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.021 DEBUG:boto:b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.022 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.022 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.022 DEBUG:boto:path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.022 DEBUG:boto:auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.022 DEBUG:boto:Method: GET 2023-09-29T23:08:20.022 DEBUG:boto:Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:08:20.022 DEBUG:boto:Data: 2023-09-29T23:08:20.023 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.023 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.023 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.023 DEBUG:boto:Params: {} 2023-09-29T23:08:20.023 DEBUG:boto:Token: None 2023-09-29T23:08:20.023 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/?versions 2023-09-29T23:08:20.023 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:08:20.023 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:08:20.027 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4e5e35a2dce7287-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.027 DEBUG:boto:b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T23:08:20.028 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.028 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.028 DEBUG:boto:path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.028 DEBUG:boto:auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.028 DEBUG:boto:Method: GET 2023-09-29T23:08:20.028 DEBUG:boto:Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.029 DEBUG:boto:Data: 2023-09-29T23:08:20.029 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.029 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.029 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.029 DEBUG:boto:Params: {} 2023-09-29T23:08:20.029 DEBUG:boto:Token: None 2023-09-29T23:08:20.029 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.029 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:08:20.030 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:08:20.032 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1843eec3c709d43-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.078 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.079 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.079 DEBUG:boto:path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.079 DEBUG:boto:auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.079 DEBUG:boto:Method: GET 2023-09-29T23:08:20.079 DEBUG:boto:Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.079 DEBUG:boto:Data: 2023-09-29T23:08:20.079 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.079 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.079 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.080 DEBUG:boto:Params: {} 2023-09-29T23:08:20.080 DEBUG:boto:Token: None 2023-09-29T23:08:20.080 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.080 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:08:20.080 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:08:20.083 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e8e65707312a77b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.083 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.083 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.083 DEBUG:boto:path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.083 DEBUG:boto:auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.084 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.084 DEBUG:boto:Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.084 DEBUG:boto:Data: 2023-09-29T23:08:20.084 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.084 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.084 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.084 DEBUG:boto:Params: {} 2023-09-29T23:08:20.085 DEBUG:boto:Token: None 2023-09-29T23:08:20.085 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.085 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY= 2023-09-29T23:08:20.085 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY=', 'Content-Length': '0'} 2023-09-29T23:08:20.088 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246718f0c790efa7-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.088 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.088 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.088 DEBUG:boto:path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.088 DEBUG:boto:auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.088 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.089 DEBUG:boto:Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.089 DEBUG:boto:Data: 2023-09-29T23:08:20.089 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.089 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.089 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.089 DEBUG:boto:Params: {} 2023-09-29T23:08:20.089 DEBUG:boto:Token: None 2023-09-29T23:08:20.089 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.089 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY= 2023-09-29T23:08:20.090 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY=', 'Content-Length': '0'} 2023-09-29T23:08:20.092 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cf7cd370dc0fb2-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T23:08:20.092 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.092 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.093 DEBUG:boto:path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.093 DEBUG:boto:auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.093 DEBUG:boto:Method: GET 2023-09-29T23:08:20.093 DEBUG:boto:Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.093 DEBUG:boto:Data: 2023-09-29T23:08:20.093 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.093 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.093 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.093 DEBUG:boto:Params: {} 2023-09-29T23:08:20.093 DEBUG:boto:Token: None 2023-09-29T23:08:20.094 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.094 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:08:20.094 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:08:20.097 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d8b7900e9620c4-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.097 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.098 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.098 DEBUG:boto:path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.098 DEBUG:boto:auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.098 DEBUG:boto:Method: GET 2023-09-29T23:08:20.098 DEBUG:boto:Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.098 DEBUG:boto:Data: 2023-09-29T23:08:20.098 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.098 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.099 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.099 DEBUG:boto:Params: {} 2023-09-29T23:08:20.099 DEBUG:boto:Token: None 2023-09-29T23:08:20.099 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11?versionId=null 2023-09-29T23:08:20.099 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:08:20.099 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:08:20.102 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98c48ba6bc84784-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.103 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.103 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.103 DEBUG:boto:path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.103 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.103 DEBUG:boto:Method: GET 2023-09-29T23:08:20.103 DEBUG:boto:Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.103 DEBUG:boto:Data: 2023-09-29T23:08:20.104 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.104 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.104 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.104 DEBUG:boto:Params: {} 2023-09-29T23:08:20.104 DEBUG:boto:Token: None 2023-09-29T23:08:20.104 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.109 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018804391b7a798fc-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.110 DEBUG:boto:b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.110 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.110 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.110 DEBUG:boto:path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.111 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.111 DEBUG:boto:Method: GET 2023-09-29T23:08:20.111 DEBUG:boto:Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.111 DEBUG:boto:Data: 2023-09-29T23:08:20.111 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.111 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.111 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.111 DEBUG:boto:Params: {} 2023-09-29T23:08:20.111 DEBUG:boto:Token: None 2023-09-29T23:08:20.112 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.112 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.112 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.117 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f57d318606aed28-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.162 DEBUG:boto:b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.163 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.163 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.163 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.163 DEBUG:boto:Path: /hcnphz-64/obj11 2023-09-29T23:08:20.163 DEBUG:boto:Data: 2023-09-29T23:08:20.164 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.164 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.164 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.164 DEBUG:boto:Params: {} 2023-09-29T23:08:20.164 DEBUG:boto:Token: None 2023-09-29T23:08:20.164 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11 2023-09-29T23:08:20.164 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8= 2023-09-29T23:08:20.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8=', 'Content-Length': '0'} 2023-09-29T23:08:20.167 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059267b1b76200a7d-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.168 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.168 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.168 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.168 DEBUG:boto:Path: /hcnphz-64/obj11 2023-09-29T23:08:20.168 DEBUG:boto:Data: 2023-09-29T23:08:20.168 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.168 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.168 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.169 DEBUG:boto:Params: {} 2023-09-29T23:08:20.169 DEBUG:boto:Token: None 2023-09-29T23:08:20.169 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11 2023-09-29T23:08:20.169 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8= 2023-09-29T23:08:20.169 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8=', 'Content-Length': '0'} 2023-09-29T23:08:20.172 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c0ccc79487aa3d-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj11 2023-09-29T23:08:20.172 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.172 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.173 DEBUG:boto:Method: GET 2023-09-29T23:08:20.173 DEBUG:boto:Path: /hcnphz-64/obj11 2023-09-29T23:08:20.173 DEBUG:boto:Data: 2023-09-29T23:08:20.173 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.173 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.173 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.173 DEBUG:boto:Params: {} 2023-09-29T23:08:20.173 DEBUG:boto:Token: None 2023-09-29T23:08:20.174 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11 2023-09-29T23:08:20.174 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA= 2023-09-29T23:08:20.174 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA=', 'Content-Length': '0'} 2023-09-29T23:08:20.177 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daa19247e0b4ee7-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.218 DEBUG:boto:path=/hcnphz-64/obj11 2023-09-29T23:08:20.219 DEBUG:boto:auth_path=/hcnphz-64/obj11 2023-09-29T23:08:20.219 DEBUG:boto:Method: GET 2023-09-29T23:08:20.219 DEBUG:boto:Path: /hcnphz-64/obj11 2023-09-29T23:08:20.219 DEBUG:boto:Data: 2023-09-29T23:08:20.220 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.220 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.220 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.220 DEBUG:boto:Params: {} 2023-09-29T23:08:20.221 DEBUG:boto:Token: None 2023-09-29T23:08:20.221 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj11 2023-09-29T23:08:20.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA= 2023-09-29T23:08:20.221 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA=', 'Content-Length': '0'} 2023-09-29T23:08:20.224 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8aa16084aade9f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.224 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.225 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.225 DEBUG:boto:path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.225 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.225 DEBUG:boto:Method: GET 2023-09-29T23:08:20.225 DEBUG:boto:Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.226 DEBUG:boto:Data: 2023-09-29T23:08:20.226 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.226 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.226 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.226 DEBUG:boto:Params: {} 2023-09-29T23:08:20.227 DEBUG:boto:Token: None 2023-09-29T23:08:20.227 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.227 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.228 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.231 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007cec1790ec521a2-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.231 DEBUG:boto:b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.232 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.232 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.232 DEBUG:boto:path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.233 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.233 DEBUG:boto:Method: GET 2023-09-29T23:08:20.233 DEBUG:boto:Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:08:20.233 DEBUG:boto:Data: 2023-09-29T23:08:20.233 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.234 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.234 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.234 DEBUG:boto:Params: {} 2023-09-29T23:08:20.234 DEBUG:boto:Token: None 2023-09-29T23:08:20.235 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.235 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.235 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.239 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8340eb2629567d-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.240 DEBUG:boto:b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T23:08:20.241 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.241 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.241 DEBUG:boto:path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.241 DEBUG:boto:auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.241 DEBUG:boto:Method: GET 2023-09-29T23:08:20.242 DEBUG:boto:Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.242 DEBUG:boto:Data: 2023-09-29T23:08:20.242 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.242 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.242 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.243 DEBUG:boto:Params: {} 2023-09-29T23:08:20.243 DEBUG:boto:Token: None 2023-09-29T23:08:20.243 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.244 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:08:20.244 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:08:20.246 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c295662032753d54-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.247 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.247 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.247 DEBUG:boto:path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.248 DEBUG:boto:auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.248 DEBUG:boto:Method: GET 2023-09-29T23:08:20.248 DEBUG:boto:Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.248 DEBUG:boto:Data: 2023-09-29T23:08:20.249 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.249 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.249 DEBUG:boto:Params: {} 2023-09-29T23:08:20.249 DEBUG:boto:Token: None 2023-09-29T23:08:20.250 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.250 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:08:20.250 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:08:20.253 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14a40ad1e91eba3-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.294 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.295 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.295 DEBUG:boto:path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.295 DEBUG:boto:auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.295 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.296 DEBUG:boto:Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.296 DEBUG:boto:Data: 2023-09-29T23:08:20.296 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.296 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.297 DEBUG:boto:Params: {} 2023-09-29T23:08:20.297 DEBUG:boto:Token: None 2023-09-29T23:08:20.297 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY= 2023-09-29T23:08:20.298 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY=', 'Content-Length': '0'} 2023-09-29T23:08:20.300 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011773f0a91baec19-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.301 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.301 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.301 DEBUG:boto:path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.301 DEBUG:boto:auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.301 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.302 DEBUG:boto:Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.302 DEBUG:boto:Data: 2023-09-29T23:08:20.302 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.302 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.302 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.303 DEBUG:boto:Params: {} 2023-09-29T23:08:20.303 DEBUG:boto:Token: None 2023-09-29T23:08:20.303 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.303 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY= 2023-09-29T23:08:20.303 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY=', 'Content-Length': '0'} 2023-09-29T23:08:20.306 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000650467a1744f490b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T23:08:20.307 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.307 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.307 DEBUG:boto:path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.308 DEBUG:boto:auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.308 DEBUG:boto:Method: GET 2023-09-29T23:08:20.308 DEBUG:boto:Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.308 DEBUG:boto:Data: 2023-09-29T23:08:20.309 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.309 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.309 DEBUG:boto:Params: {} 2023-09-29T23:08:20.310 DEBUG:boto:Token: None 2023-09-29T23:08:20.310 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.310 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:08:20.310 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:08:20.313 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448f21448f66f838-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.313 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.314 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.314 DEBUG:boto:path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.314 DEBUG:boto:auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.314 DEBUG:boto:Method: GET 2023-09-29T23:08:20.315 DEBUG:boto:Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.315 DEBUG:boto:Data: 2023-09-29T23:08:20.315 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.315 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.315 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.315 DEBUG:boto:Params: {} 2023-09-29T23:08:20.315 DEBUG:boto:Token: None 2023-09-29T23:08:20.315 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22?versionId=null 2023-09-29T23:08:20.316 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:08:20.316 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:08:20.318 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af5f1de7ddcf1e23-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.319 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.319 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.319 DEBUG:boto:path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.319 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.319 DEBUG:boto:Method: GET 2023-09-29T23:08:20.319 DEBUG:boto:Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.319 DEBUG:boto:Data: 2023-09-29T23:08:20.320 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.320 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.320 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.320 DEBUG:boto:Params: {} 2023-09-29T23:08:20.320 DEBUG:boto:Token: None 2023-09-29T23:08:20.320 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.320 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.320 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.325 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fef5b6ad6f856e4f-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.325 DEBUG:boto:b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.325 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.325 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.326 DEBUG:boto:path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.326 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.326 DEBUG:boto:Method: GET 2023-09-29T23:08:20.326 DEBUG:boto:Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.326 DEBUG:boto:Data: 2023-09-29T23:08:20.326 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.326 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.326 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.326 DEBUG:boto:Params: {} 2023-09-29T23:08:20.326 DEBUG:boto:Token: None 2023-09-29T23:08:20.327 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.327 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.327 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.331 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62eb2f1cf14fc23-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.374 DEBUG:boto:b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.375 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.375 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.375 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.375 DEBUG:boto:Path: /hcnphz-64/obj22 2023-09-29T23:08:20.375 DEBUG:boto:Data: 2023-09-29T23:08:20.375 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.376 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.376 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.376 DEBUG:boto:Params: {} 2023-09-29T23:08:20.376 DEBUG:boto:Token: None 2023-09-29T23:08:20.376 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22 2023-09-29T23:08:20.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew= 2023-09-29T23:08:20.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew=', 'Content-Length': '0'} 2023-09-29T23:08:20.379 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca8e46435b2d831-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.379 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.380 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.380 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.380 DEBUG:boto:Path: /hcnphz-64/obj22 2023-09-29T23:08:20.380 DEBUG:boto:Data: 2023-09-29T23:08:20.380 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.380 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.380 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.380 DEBUG:boto:Params: {} 2023-09-29T23:08:20.380 DEBUG:boto:Token: None 2023-09-29T23:08:20.381 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22 2023-09-29T23:08:20.381 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew= 2023-09-29T23:08:20.381 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew=', 'Content-Length': '0'} 2023-09-29T23:08:20.383 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f8c86d327cc106-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj22 2023-09-29T23:08:20.384 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.384 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.384 DEBUG:boto:Method: GET 2023-09-29T23:08:20.384 DEBUG:boto:Path: /hcnphz-64/obj22 2023-09-29T23:08:20.384 DEBUG:boto:Data: 2023-09-29T23:08:20.384 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.384 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.385 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.385 DEBUG:boto:Params: {} 2023-09-29T23:08:20.385 DEBUG:boto:Token: None 2023-09-29T23:08:20.385 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22 2023-09-29T23:08:20.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ= 2023-09-29T23:08:20.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ=', 'Content-Length': '0'} 2023-09-29T23:08:20.388 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50885bb50e8b3a5-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.430 DEBUG:boto:path=/hcnphz-64/obj22 2023-09-29T23:08:20.431 DEBUG:boto:auth_path=/hcnphz-64/obj22 2023-09-29T23:08:20.431 DEBUG:boto:Method: GET 2023-09-29T23:08:20.431 DEBUG:boto:Path: /hcnphz-64/obj22 2023-09-29T23:08:20.431 DEBUG:boto:Data: 2023-09-29T23:08:20.431 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.431 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.431 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.431 DEBUG:boto:Params: {} 2023-09-29T23:08:20.431 DEBUG:boto:Token: None 2023-09-29T23:08:20.432 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj22 2023-09-29T23:08:20.432 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ= 2023-09-29T23:08:20.432 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ=', 'Content-Length': '0'} 2023-09-29T23:08:20.435 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf80f887baa23ed-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.435 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.435 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.435 DEBUG:boto:path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.435 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.435 DEBUG:boto:Method: GET 2023-09-29T23:08:20.436 DEBUG:boto:Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.436 DEBUG:boto:Data: 2023-09-29T23:08:20.436 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.436 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.436 DEBUG:boto:Params: {} 2023-09-29T23:08:20.436 DEBUG:boto:Token: None 2023-09-29T23:08:20.436 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.436 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.441 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464b265d465bfc94-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.441 DEBUG:boto:b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.442 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.442 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.442 DEBUG:boto:path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.442 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.442 DEBUG:boto:Method: GET 2023-09-29T23:08:20.443 DEBUG:boto:Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:08:20.443 DEBUG:boto:Data: 2023-09-29T23:08:20.443 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.443 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.443 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.443 DEBUG:boto:Params: {} 2023-09-29T23:08:20.443 DEBUG:boto:Token: None 2023-09-29T23:08:20.444 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.444 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.444 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.448 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1ff4eb47946cbe-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.449 DEBUG:boto:b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T23:08:20.450 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.450 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.450 DEBUG:boto:path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.450 DEBUG:boto:auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.450 DEBUG:boto:Method: GET 2023-09-29T23:08:20.451 DEBUG:boto:Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.451 DEBUG:boto:Data: 2023-09-29T23:08:20.451 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.451 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.452 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.452 DEBUG:boto:Params: {} 2023-09-29T23:08:20.452 DEBUG:boto:Token: None 2023-09-29T23:08:20.452 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.453 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:08:20.453 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:08:20.455 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33d996ef5557fd7-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.455 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.456 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.456 DEBUG:boto:path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.456 DEBUG:boto:auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.457 DEBUG:boto:Method: GET 2023-09-29T23:08:20.457 DEBUG:boto:Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.457 DEBUG:boto:Data: 2023-09-29T23:08:20.457 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.458 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.458 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.458 DEBUG:boto:Params: {} 2023-09-29T23:08:20.459 DEBUG:boto:Token: None 2023-09-29T23:08:20.459 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:08:20.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:08:20.463 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1444c1c10b40d4b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.506 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.507 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.507 DEBUG:boto:path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.507 DEBUG:boto:auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.507 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.507 DEBUG:boto:Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.508 DEBUG:boto:Data: 2023-09-29T23:08:20.508 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.508 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.508 DEBUG:boto:Params: {} 2023-09-29T23:08:20.508 DEBUG:boto:Token: None 2023-09-29T23:08:20.509 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI= 2023-09-29T23:08:20.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI=', 'Content-Length': '0'} 2023-09-29T23:08:20.512 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea4984875816f20-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.512 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.512 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.512 DEBUG:boto:path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.513 DEBUG:boto:auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.513 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.513 DEBUG:boto:Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.513 DEBUG:boto:Data: 2023-09-29T23:08:20.513 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.513 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.513 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.514 DEBUG:boto:Params: {} 2023-09-29T23:08:20.514 DEBUG:boto:Token: None 2023-09-29T23:08:20.514 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.514 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI= 2023-09-29T23:08:20.514 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI=', 'Content-Length': '0'} 2023-09-29T23:08:20.517 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ff99f125ab8d33-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T23:08:20.517 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.518 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.518 DEBUG:boto:path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.518 DEBUG:boto:auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.518 DEBUG:boto:Method: GET 2023-09-29T23:08:20.518 DEBUG:boto:Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.518 DEBUG:boto:Data: 2023-09-29T23:08:20.519 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.519 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.519 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.519 DEBUG:boto:Params: {} 2023-09-29T23:08:20.519 DEBUG:boto:Token: None 2023-09-29T23:08:20.520 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.520 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:08:20.520 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:08:20.522 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29d011778f52249-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.566 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.566 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.566 DEBUG:boto:path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.567 DEBUG:boto:auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.567 DEBUG:boto:Method: GET 2023-09-29T23:08:20.567 DEBUG:boto:Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.567 DEBUG:boto:Data: 2023-09-29T23:08:20.567 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.567 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.567 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.567 DEBUG:boto:Params: {} 2023-09-29T23:08:20.567 DEBUG:boto:Token: None 2023-09-29T23:08:20.568 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33?versionId=null 2023-09-29T23:08:20.568 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:08:20.568 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:08:20.571 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03ac08b21861ed8-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.571 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.571 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.571 DEBUG:boto:path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.571 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.571 DEBUG:boto:Method: GET 2023-09-29T23:08:20.572 DEBUG:boto:Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.572 DEBUG:boto:Data: 2023-09-29T23:08:20.572 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.572 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.572 DEBUG:boto:Params: {} 2023-09-29T23:08:20.572 DEBUG:boto:Token: None 2023-09-29T23:08:20.572 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.572 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.573 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.577 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccdc70020543a9f-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.577 DEBUG:boto:b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.578 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.578 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.578 DEBUG:boto:path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.578 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.579 DEBUG:boto:Method: GET 2023-09-29T23:08:20.579 DEBUG:boto:Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.579 DEBUG:boto:Data: 2023-09-29T23:08:20.579 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.579 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.579 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.579 DEBUG:boto:Params: {} 2023-09-29T23:08:20.580 DEBUG:boto:Token: None 2023-09-29T23:08:20.580 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.580 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.580 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.584 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196cbcd37e0f2b5d-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.584 DEBUG:boto:b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.585 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.585 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.585 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.585 DEBUG:boto:Path: /hcnphz-64/obj33 2023-09-29T23:08:20.586 DEBUG:boto:Data: 2023-09-29T23:08:20.586 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.586 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.586 DEBUG:boto:Params: {} 2023-09-29T23:08:20.587 DEBUG:boto:Token: None 2023-09-29T23:08:20.587 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33 2023-09-29T23:08:20.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8= 2023-09-29T23:08:20.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8=', 'Content-Length': '0'} 2023-09-29T23:08:20.590 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fbd2df1fb51879-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.590 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.590 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.590 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.590 DEBUG:boto:Path: /hcnphz-64/obj33 2023-09-29T23:08:20.590 DEBUG:boto:Data: 2023-09-29T23:08:20.591 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.591 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.591 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.591 DEBUG:boto:Params: {} 2023-09-29T23:08:20.591 DEBUG:boto:Token: None 2023-09-29T23:08:20.591 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33 2023-09-29T23:08:20.591 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8= 2023-09-29T23:08:20.591 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8=', 'Content-Length': '0'} 2023-09-29T23:08:20.593 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c0b3a8d3a47138-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj33 2023-09-29T23:08:20.594 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.594 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.594 DEBUG:boto:Method: GET 2023-09-29T23:08:20.594 DEBUG:boto:Path: /hcnphz-64/obj33 2023-09-29T23:08:20.594 DEBUG:boto:Data: 2023-09-29T23:08:20.594 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.594 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.595 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.595 DEBUG:boto:Params: {} 2023-09-29T23:08:20.595 DEBUG:boto:Token: None 2023-09-29T23:08:20.595 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33 2023-09-29T23:08:20.595 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c= 2023-09-29T23:08:20.595 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c=', 'Content-Length': '0'} 2023-09-29T23:08:20.597 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002489d0f695c53387-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.598 DEBUG:boto:path=/hcnphz-64/obj33 2023-09-29T23:08:20.598 DEBUG:boto:auth_path=/hcnphz-64/obj33 2023-09-29T23:08:20.598 DEBUG:boto:Method: GET 2023-09-29T23:08:20.598 DEBUG:boto:Path: /hcnphz-64/obj33 2023-09-29T23:08:20.598 DEBUG:boto:Data: 2023-09-29T23:08:20.598 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.599 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.599 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.599 DEBUG:boto:Params: {} 2023-09-29T23:08:20.599 DEBUG:boto:Token: None 2023-09-29T23:08:20.599 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj33 2023-09-29T23:08:20.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c= 2023-09-29T23:08:20.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c=', 'Content-Length': '0'} 2023-09-29T23:08:20.602 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f80dd6cac90417a-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.642 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.642 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.642 DEBUG:boto:path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.643 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.643 DEBUG:boto:Method: GET 2023-09-29T23:08:20.643 DEBUG:boto:Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.643 DEBUG:boto:Data: 2023-09-29T23:08:20.643 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.643 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.643 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.643 DEBUG:boto:Params: {} 2023-09-29T23:08:20.643 DEBUG:boto:Token: None 2023-09-29T23:08:20.644 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.648 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060cf2bc87f4bed00-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.649 DEBUG:boto:b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.649 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.649 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.649 DEBUG:boto:path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.649 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.650 DEBUG:boto:Method: GET 2023-09-29T23:08:20.650 DEBUG:boto:Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:08:20.650 DEBUG:boto:Data: 2023-09-29T23:08:20.650 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.650 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.650 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.650 DEBUG:boto:Params: {} 2023-09-29T23:08:20.650 DEBUG:boto:Token: None 2023-09-29T23:08:20.651 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.651 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.651 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.656 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b347724b622bd40-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.656 DEBUG:boto:b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T23:08:20.657 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.657 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.657 DEBUG:boto:path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.657 DEBUG:boto:auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.657 DEBUG:boto:Method: GET 2023-09-29T23:08:20.657 DEBUG:boto:Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.658 DEBUG:boto:Data: 2023-09-29T23:08:20.658 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.658 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.658 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.658 DEBUG:boto:Params: {} 2023-09-29T23:08:20.658 DEBUG:boto:Token: None 2023-09-29T23:08:20.658 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.658 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:08:20.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:08:20.661 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9e5c49430058be-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.702 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.703 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.703 DEBUG:boto:path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.703 DEBUG:boto:auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.703 DEBUG:boto:Method: GET 2023-09-29T23:08:20.703 DEBUG:boto:Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.703 DEBUG:boto:Data: 2023-09-29T23:08:20.703 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.703 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.703 DEBUG:boto:Params: {} 2023-09-29T23:08:20.704 DEBUG:boto:Token: None 2023-09-29T23:08:20.704 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:08:20.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:08:20.707 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1130b7aee650bf8-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.707 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.707 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.708 DEBUG:boto:path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.708 DEBUG:boto:auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.708 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.708 DEBUG:boto:Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.708 DEBUG:boto:Data: 2023-09-29T23:08:20.708 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.708 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.708 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.708 DEBUG:boto:Params: {} 2023-09-29T23:08:20.708 DEBUG:boto:Token: None 2023-09-29T23:08:20.709 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.709 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo= 2023-09-29T23:08:20.709 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo=', 'Content-Length': '0'} 2023-09-29T23:08:20.711 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e40215e0fcf62f-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.712 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.712 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.712 DEBUG:boto:path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.712 DEBUG:boto:auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.712 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.712 DEBUG:boto:Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.712 DEBUG:boto:Data: 2023-09-29T23:08:20.712 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.713 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.713 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.713 DEBUG:boto:Params: {} 2023-09-29T23:08:20.713 DEBUG:boto:Token: None 2023-09-29T23:08:20.713 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.713 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo= 2023-09-29T23:08:20.713 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo=', 'Content-Length': '0'} 2023-09-29T23:08:20.715 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039615b7e67f6ec38-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T23:08:20.716 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.716 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.716 DEBUG:boto:path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.716 DEBUG:boto:auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.716 DEBUG:boto:Method: GET 2023-09-29T23:08:20.716 DEBUG:boto:Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.717 DEBUG:boto:Data: 2023-09-29T23:08:20.717 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.717 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.717 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.717 DEBUG:boto:Params: {} 2023-09-29T23:08:20.717 DEBUG:boto:Token: None 2023-09-29T23:08:20.717 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.717 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:08:20.718 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:08:20.720 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094861147099ab769-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.720 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.721 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.721 DEBUG:boto:path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.721 DEBUG:boto:auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.721 DEBUG:boto:Method: GET 2023-09-29T23:08:20.721 DEBUG:boto:Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.721 DEBUG:boto:Data: 2023-09-29T23:08:20.721 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.721 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.721 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.721 DEBUG:boto:Params: {} 2023-09-29T23:08:20.722 DEBUG:boto:Token: None 2023-09-29T23:08:20.722 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44?versionId=null 2023-09-29T23:08:20.722 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:08:20.722 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:08:20.725 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e66d8746a9106b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.766 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.767 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.767 DEBUG:boto:path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.767 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.767 DEBUG:boto:Method: GET 2023-09-29T23:08:20.767 DEBUG:boto:Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.767 DEBUG:boto:Data: 2023-09-29T23:08:20.767 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.767 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.767 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.767 DEBUG:boto:Params: {} 2023-09-29T23:08:20.768 DEBUG:boto:Token: None 2023-09-29T23:08:20.768 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.768 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.772 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011cc2f53213d7ae5-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.772 DEBUG:boto:b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.773 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.773 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.773 DEBUG:boto:path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.773 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.773 DEBUG:boto:Method: GET 2023-09-29T23:08:20.773 DEBUG:boto:Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.773 DEBUG:boto:Data: 2023-09-29T23:08:20.773 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.774 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.774 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.774 DEBUG:boto:Params: {} 2023-09-29T23:08:20.774 DEBUG:boto:Token: None 2023-09-29T23:08:20.774 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.774 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.774 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.778 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bad3ca7861e9323-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.779 DEBUG:boto:b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.779 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.779 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.779 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.779 DEBUG:boto:Path: /hcnphz-64/obj44 2023-09-29T23:08:20.779 DEBUG:boto:Data: 2023-09-29T23:08:20.779 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.780 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.780 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.780 DEBUG:boto:Params: {} 2023-09-29T23:08:20.780 DEBUG:boto:Token: None 2023-09-29T23:08:20.780 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44 2023-09-29T23:08:20.780 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc= 2023-09-29T23:08:20.780 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc=', 'Content-Length': '0'} 2023-09-29T23:08:20.782 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952badfbf8aaa365-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.783 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.783 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.783 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.783 DEBUG:boto:Path: /hcnphz-64/obj44 2023-09-29T23:08:20.783 DEBUG:boto:Data: 2023-09-29T23:08:20.783 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.784 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.784 DEBUG:boto:Params: {} 2023-09-29T23:08:20.784 DEBUG:boto:Token: None 2023-09-29T23:08:20.784 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44 2023-09-29T23:08:20.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc= 2023-09-29T23:08:20.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc=', 'Content-Length': '0'} 2023-09-29T23:08:20.786 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000339fd83137d68b0f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj44 2023-09-29T23:08:20.787 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.787 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.787 DEBUG:boto:Method: GET 2023-09-29T23:08:20.787 DEBUG:boto:Path: /hcnphz-64/obj44 2023-09-29T23:08:20.787 DEBUG:boto:Data: 2023-09-29T23:08:20.787 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.787 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.788 DEBUG:boto:Params: {} 2023-09-29T23:08:20.788 DEBUG:boto:Token: None 2023-09-29T23:08:20.788 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44 2023-09-29T23:08:20.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs= 2023-09-29T23:08:20.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs=', 'Content-Length': '0'} 2023-09-29T23:08:20.790 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4a6f54647e233f-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.834 DEBUG:boto:path=/hcnphz-64/obj44 2023-09-29T23:08:20.835 DEBUG:boto:auth_path=/hcnphz-64/obj44 2023-09-29T23:08:20.835 DEBUG:boto:Method: GET 2023-09-29T23:08:20.835 DEBUG:boto:Path: /hcnphz-64/obj44 2023-09-29T23:08:20.835 DEBUG:boto:Data: 2023-09-29T23:08:20.835 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.835 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.835 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.835 DEBUG:boto:Params: {} 2023-09-29T23:08:20.836 DEBUG:boto:Token: None 2023-09-29T23:08:20.836 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj44 2023-09-29T23:08:20.836 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs= 2023-09-29T23:08:20.836 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs=', 'Content-Length': '0'} 2023-09-29T23:08:20.839 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c81ef947b2a0661-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.840 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.840 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.840 DEBUG:boto:path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.840 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.840 DEBUG:boto:Method: GET 2023-09-29T23:08:20.840 DEBUG:boto:Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.840 DEBUG:boto:Data: 2023-09-29T23:08:20.840 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.840 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.841 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.841 DEBUG:boto:Params: {} 2023-09-29T23:08:20.841 DEBUG:boto:Token: None 2023-09-29T23:08:20.841 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.841 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.841 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.846 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a5aaca2a4b1e5a-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.847 DEBUG:boto:b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.847 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.848 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.848 DEBUG:boto:path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.848 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.848 DEBUG:boto:Method: GET 2023-09-29T23:08:20.848 DEBUG:boto:Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:08:20.849 DEBUG:boto:Data: 2023-09-29T23:08:20.849 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.849 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.849 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.849 DEBUG:boto:Params: {} 2023-09-29T23:08:20.850 DEBUG:boto:Token: None 2023-09-29T23:08:20.850 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.850 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.850 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.855 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8551c08d628464-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.898 DEBUG:boto:b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T23:08:20.899 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.899 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.899 DEBUG:boto:path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.899 DEBUG:boto:auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.899 DEBUG:boto:Method: GET 2023-09-29T23:08:20.900 DEBUG:boto:Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.900 DEBUG:boto:Data: 2023-09-29T23:08:20.900 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.900 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.900 DEBUG:boto:Params: {} 2023-09-29T23:08:20.900 DEBUG:boto:Token: None 2023-09-29T23:08:20.900 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:08:20.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:08:20.904 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf29e7cef3f8d57-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.904 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.904 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.904 DEBUG:boto:path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.904 DEBUG:boto:auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.904 DEBUG:boto:Method: GET 2023-09-29T23:08:20.904 DEBUG:boto:Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.904 DEBUG:boto:Data: 2023-09-29T23:08:20.905 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.905 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.905 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.905 DEBUG:boto:Params: {} 2023-09-29T23:08:20.905 DEBUG:boto:Token: None 2023-09-29T23:08:20.905 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.905 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:08:20.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:08:20.908 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d79c93d172f66a2-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.909 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.909 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.909 DEBUG:boto:path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.909 DEBUG:boto:auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.909 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.909 DEBUG:boto:Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.909 DEBUG:boto:Data: 2023-09-29T23:08:20.909 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.910 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.910 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.910 DEBUG:boto:Params: {} 2023-09-29T23:08:20.910 DEBUG:boto:Token: None 2023-09-29T23:08:20.910 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.910 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM= 2023-09-29T23:08:20.910 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM=', 'Content-Length': '0'} 2023-09-29T23:08:20.913 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9cf5582ceb40ff-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.913 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.913 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.913 DEBUG:boto:path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.914 DEBUG:boto:auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.914 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.914 DEBUG:boto:Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.914 DEBUG:boto:Data: 2023-09-29T23:08:20.914 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.914 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.914 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.914 DEBUG:boto:Params: {} 2023-09-29T23:08:20.915 DEBUG:boto:Token: None 2023-09-29T23:08:20.915 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.915 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM= 2023-09-29T23:08:20.915 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM=', 'Content-Length': '0'} 2023-09-29T23:08:20.918 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f03ade4f92a20f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T23:08:20.918 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.918 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.919 DEBUG:boto:path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.919 DEBUG:boto:auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.919 DEBUG:boto:Method: GET 2023-09-29T23:08:20.919 DEBUG:boto:Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.919 DEBUG:boto:Data: 2023-09-29T23:08:20.919 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.919 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.919 DEBUG:boto:Params: {} 2023-09-29T23:08:20.920 DEBUG:boto:Token: None 2023-09-29T23:08:20.920 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:08:20.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:08:20.923 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6cefb24a297835-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.923 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.923 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.923 DEBUG:boto:path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.924 DEBUG:boto:auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.924 DEBUG:boto:Method: GET 2023-09-29T23:08:20.924 DEBUG:boto:Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.924 DEBUG:boto:Data: 2023-09-29T23:08:20.924 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.924 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.924 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.924 DEBUG:boto:Params: {} 2023-09-29T23:08:20.924 DEBUG:boto:Token: None 2023-09-29T23:08:20.925 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55?versionId=null 2023-09-29T23:08:20.925 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:08:20.925 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:08:20.928 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded2b2933a4c8e2f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.928 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.928 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.928 DEBUG:boto:path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:20.929 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:20.929 DEBUG:boto:Method: GET 2023-09-29T23:08:20.929 DEBUG:boto:Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:08:20.929 DEBUG:boto:Data: 2023-09-29T23:08:20.929 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.929 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.929 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.929 DEBUG:boto:Params: {} 2023-09-29T23:08:20.929 DEBUG:boto:Token: None 2023-09-29T23:08:20.930 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.930 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.930 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.935 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eca33e05f014ed0-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.982 DEBUG:boto:b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.983 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:20.983 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:20.983 DEBUG:boto:path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:20.983 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:20.983 DEBUG:boto:Method: GET 2023-09-29T23:08:20.983 DEBUG:boto:Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:08:20.983 DEBUG:boto:Data: 2023-09-29T23:08:20.984 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.984 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.984 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.984 DEBUG:boto:Params: {} 2023-09-29T23:08:20.984 DEBUG:boto:Token: None 2023-09-29T23:08:20.984 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/ 2023-09-29T23:08:20.984 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:08:20.985 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:08:20.990 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df00100b7e3dfb0d-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.990 DEBUG:boto:b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:20.990 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.991 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.991 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.991 DEBUG:boto:Path: /hcnphz-64/obj55 2023-09-29T23:08:20.991 DEBUG:boto:Data: 2023-09-29T23:08:20.991 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.992 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:20.992 DEBUG:boto:Port: 8001 2023-09-29T23:08:20.992 DEBUG:boto:Params: {} 2023-09-29T23:08:20.992 DEBUG:boto:Token: None 2023-09-29T23:08:20.992 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55 2023-09-29T23:08:20.993 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI= 2023-09-29T23:08:20.993 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI=', 'Content-Length': '0'} 2023-09-29T23:08:20.995 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142c6d5a10cd005b-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:20.996 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:20.996 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:20.996 DEBUG:boto:Method: HEAD 2023-09-29T23:08:20.996 DEBUG:boto:Path: /hcnphz-64/obj55 2023-09-29T23:08:20.997 DEBUG:boto:Data: 2023-09-29T23:08:20.997 DEBUG:boto:Headers: {} 2023-09-29T23:08:20.997 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:20.997 DEBUG:boto:Port: 8000 2023-09-29T23:08:20.997 DEBUG:boto:Params: {} 2023-09-29T23:08:20.997 DEBUG:boto:Token: None 2023-09-29T23:08:20.998 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:20 GMT /hcnphz-64/obj55 2023-09-29T23:08:20.998 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI= 2023-09-29T23:08:20.998 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI=', 'Content-Length': '0'} 2023-09-29T23:08:21.001 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007f747b6572d266d-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:21.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj55 2023-09-29T23:08:21.002 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:21.002 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:21.002 DEBUG:boto:Method: GET 2023-09-29T23:08:21.002 DEBUG:boto:Path: /hcnphz-64/obj55 2023-09-29T23:08:21.002 DEBUG:boto:Data: 2023-09-29T23:08:21.003 DEBUG:boto:Headers: {} 2023-09-29T23:08:21.003 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:21.003 DEBUG:boto:Port: 8001 2023-09-29T23:08:21.003 DEBUG:boto:Params: {} 2023-09-29T23:08:21.003 DEBUG:boto:Token: None 2023-09-29T23:08:21.004 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:21 GMT /hcnphz-64/obj55 2023-09-29T23:08:21.004 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE= 2023-09-29T23:08:21.004 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE=', 'Content-Length': '0'} 2023-09-29T23:08:21.007 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde0e29296f5d983-00651758e5-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:21.007 DEBUG:boto:path=/hcnphz-64/obj55 2023-09-29T23:08:21.007 DEBUG:boto:auth_path=/hcnphz-64/obj55 2023-09-29T23:08:21.007 DEBUG:boto:Method: GET 2023-09-29T23:08:21.008 DEBUG:boto:Path: /hcnphz-64/obj55 2023-09-29T23:08:21.008 DEBUG:boto:Data: 2023-09-29T23:08:21.008 DEBUG:boto:Headers: {} 2023-09-29T23:08:21.008 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:21.008 DEBUG:boto:Port: 8000 2023-09-29T23:08:21.009 DEBUG:boto:Params: {} 2023-09-29T23:08:21.009 DEBUG:boto:Token: None 2023-09-29T23:08:21.009 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:21 GMT /hcnphz-64/obj55 2023-09-29T23:08:21.009 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE= 2023-09-29T23:08:21.009 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE=', 'Content-Length': '0'} 2023-09-29T23:08:21.012 DEBUG:boto:Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e7d9a98367631c-00651758e5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:21.054 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:21.055 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:21.055 DEBUG:boto:path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:21.055 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:21.055 DEBUG:boto:Method: GET 2023-09-29T23:08:21.056 DEBUG:boto:Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:08:21.056 DEBUG:boto:Data: 2023-09-29T23:08:21.056 DEBUG:boto:Headers: {} 2023-09-29T23:08:21.056 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:21.056 DEBUG:boto:Port: 8001 2023-09-29T23:08:21.056 DEBUG:boto:Params: {} 2023-09-29T23:08:21.057 DEBUG:boto:Token: None 2023-09-29T23:08:21.057 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:21 GMT /hcnphz-64/ 2023-09-29T23:08:21.057 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY= 2023-09-29T23:08:21.057 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY=', 'Content-Length': '0'} 2023-09-29T23:08:21.062 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008fcdcecfe256e60-00651758e5-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:21.062 DEBUG:boto:b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:21.063 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:21.063 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:21.063 DEBUG:boto:path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:21.063 DEBUG:boto:auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:08:21.064 DEBUG:boto:Method: GET 2023-09-29T23:08:21.064 DEBUG:boto:Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:08:21.064 DEBUG:boto:Data: 2023-09-29T23:08:21.064 DEBUG:boto:Headers: {} 2023-09-29T23:08:21.064 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:21.065 DEBUG:boto:Port: 8000 2023-09-29T23:08:21.065 DEBUG:boto:Params: {} 2023-09-29T23:08:21.065 DEBUG:boto:Token: None 2023-09-29T23:08:21.065 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:08:21 GMT /hcnphz-64/ 2023-09-29T23:08:21.065 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY= 2023-09-29T23:08:21.066 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY=', 'Content-Length': '0'} 2023-09-29T23:08:21.070 DEBUG:boto:Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009431464e0d3aa018-00651758e5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:21.070 DEBUG:boto:b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:08:21.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hcnphz-64 zones={a2, a1} 2023-09-29T23:08:21.071 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T23:08:21.071 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:08:21.072 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:08:21.071+0000 7f4a52ee1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 79272) UID: 0 2023-09-29T23:08:21.072 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:08:21.071+0000 7f4a56740f80 -1 shutting down 2023-09-29T23:08:23.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T23:08:27.171 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T23:08:27.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:27.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:27.171 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:27.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:27.297 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:08:27.297 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-64 2023-09-29T23:08:27.305 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 5 2023-09-29T23:08:27.384 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.384 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.384 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.384 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.384 DEBUG:boto:Data: 2023-09-29T23:08:27.384 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.385 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.385 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.385 DEBUG:boto:Params: {} 2023-09-29T23:08:27.385 DEBUG:boto:Token: None 2023-09-29T23:08:27.385 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.385 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.385 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.389 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cff3d762e3b75169-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.390 DEBUG:boto:path=/hcnphz-64/obj17 2023-09-29T23:08:27.390 DEBUG:boto:auth_path=/hcnphz-64/obj17 2023-09-29T23:08:27.390 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.390 DEBUG:boto:Path: /hcnphz-64/obj17 2023-09-29T23:08:27.390 DEBUG:boto:Data: 2023-09-29T23:08:27.390 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.390 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.391 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.391 DEBUG:boto:Params: {} 2023-09-29T23:08:27.391 DEBUG:boto:Token: None 2023-09-29T23:08:27.391 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj17 2023-09-29T23:08:27.391 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:MrET2tVQQMpyzFXqkPaqZOMu76k= 2023-09-29T23:08:27.391 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MrET2tVQQMpyzFXqkPaqZOMu76k='} 2023-09-29T23:08:27.401 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006ce1e67f1244ab1-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.401 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.402 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.402 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.402 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.402 DEBUG:boto:Data: 2023-09-29T23:08:27.402 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.402 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.402 DEBUG:boto:Params: {} 2023-09-29T23:08:27.402 DEBUG:boto:Token: None 2023-09-29T23:08:27.403 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.403 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.407 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004930a8520e33ef5f-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.407 DEBUG:boto:path=/hcnphz-64/obj34 2023-09-29T23:08:27.407 DEBUG:boto:auth_path=/hcnphz-64/obj34 2023-09-29T23:08:27.408 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.408 DEBUG:boto:Path: /hcnphz-64/obj34 2023-09-29T23:08:27.408 DEBUG:boto:Data: 2023-09-29T23:08:27.408 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.408 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.408 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.408 DEBUG:boto:Params: {} 2023-09-29T23:08:27.408 DEBUG:boto:Token: None 2023-09-29T23:08:27.409 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj34 2023-09-29T23:08:27.409 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uNsLZa7sWvzgjO5PTjwO1kRzKg0= 2023-09-29T23:08:27.409 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uNsLZa7sWvzgjO5PTjwO1kRzKg0='} 2023-09-29T23:08:27.454 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035e847b471d6d244-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.455 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.455 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.455 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.455 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.455 DEBUG:boto:Data: 2023-09-29T23:08:27.455 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.456 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.456 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.456 DEBUG:boto:Params: {} 2023-09-29T23:08:27.456 DEBUG:boto:Token: None 2023-09-29T23:08:27.456 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.462 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002000272854156dae-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.462 DEBUG:boto:path=/hcnphz-64/obj51 2023-09-29T23:08:27.462 DEBUG:boto:auth_path=/hcnphz-64/obj51 2023-09-29T23:08:27.462 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.462 DEBUG:boto:Path: /hcnphz-64/obj51 2023-09-29T23:08:27.463 DEBUG:boto:Data: 2023-09-29T23:08:27.463 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.463 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.463 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.463 DEBUG:boto:Params: {} 2023-09-29T23:08:27.463 DEBUG:boto:Token: None 2023-09-29T23:08:27.463 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj51 2023-09-29T23:08:27.463 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FHN3efPYJCcZ6n4Jxp0Ng/BrVIQ= 2023-09-29T23:08:27.464 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FHN3efPYJCcZ6n4Jxp0Ng/BrVIQ='} 2023-09-29T23:08:27.475 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cc0cec6198ccf07-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.475 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.475 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.475 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.476 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.476 DEBUG:boto:Data: 2023-09-29T23:08:27.476 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.476 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.476 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.476 DEBUG:boto:Params: {} 2023-09-29T23:08:27.476 DEBUG:boto:Token: None 2023-09-29T23:08:27.476 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.477 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.477 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.480 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012d689b03a79737e-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.481 DEBUG:boto:path=/hcnphz-64/obj68 2023-09-29T23:08:27.481 DEBUG:boto:auth_path=/hcnphz-64/obj68 2023-09-29T23:08:27.481 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.481 DEBUG:boto:Path: /hcnphz-64/obj68 2023-09-29T23:08:27.481 DEBUG:boto:Data: 2023-09-29T23:08:27.481 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.482 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.482 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.482 DEBUG:boto:Params: {} 2023-09-29T23:08:27.482 DEBUG:boto:Token: None 2023-09-29T23:08:27.482 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj68 2023-09-29T23:08:27.482 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FiqbqHGdJJWkQVwAopGsVKgzzU4= 2023-09-29T23:08:27.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FiqbqHGdJJWkQVwAopGsVKgzzU4='} 2023-09-29T23:08:27.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b979194614f2ebab-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.527 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.527 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.527 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.527 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.527 DEBUG:boto:Data: 2023-09-29T23:08:27.527 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.528 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.528 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.528 DEBUG:boto:Params: {} 2023-09-29T23:08:27.528 DEBUG:boto:Token: None 2023-09-29T23:08:27.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008347f07c42dd4a0b-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.533 DEBUG:boto:path=/hcnphz-64/obj85 2023-09-29T23:08:27.534 DEBUG:boto:auth_path=/hcnphz-64/obj85 2023-09-29T23:08:27.534 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.534 DEBUG:boto:Path: /hcnphz-64/obj85 2023-09-29T23:08:27.534 DEBUG:boto:Data: 2023-09-29T23:08:27.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.534 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.534 DEBUG:boto:Params: {} 2023-09-29T23:08:27.535 DEBUG:boto:Token: None 2023-09-29T23:08:27.535 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj85 2023-09-29T23:08:27.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SShh1zayCDhUb1AA5Ia+eyJu9bc= 2023-09-29T23:08:27.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SShh1zayCDhUb1AA5Ia+eyJu9bc='} 2023-09-29T23:08:27.546 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059e9f906b811f3d9-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:27.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:27.547 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:27.593 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:27.674 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:08:27.674 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-64 2023-09-29T23:08:27.686 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 10 2023-09-29T23:08:27.781 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.781 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.782 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.782 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.782 DEBUG:boto:Data: 2023-09-29T23:08:27.782 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.782 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.782 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.782 DEBUG:boto:Params: {} 2023-09-29T23:08:27.782 DEBUG:boto:Token: None 2023-09-29T23:08:27.783 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.783 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.783 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.787 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005580344351cf044f-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.787 DEBUG:boto:path=/hcnphz-64/obj19 2023-09-29T23:08:27.787 DEBUG:boto:auth_path=/hcnphz-64/obj19 2023-09-29T23:08:27.787 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.788 DEBUG:boto:Path: /hcnphz-64/obj19 2023-09-29T23:08:27.788 DEBUG:boto:Data: 2023-09-29T23:08:27.788 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.788 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.788 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.788 DEBUG:boto:Params: {} 2023-09-29T23:08:27.788 DEBUG:boto:Token: None 2023-09-29T23:08:27.788 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj19 2023-09-29T23:08:27.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:7LWU167bq48cKm6Ly/X7/9xRZ5w= 2023-09-29T23:08:27.789 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7LWU167bq48cKm6Ly/X7/9xRZ5w='} 2023-09-29T23:08:27.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcdf75ec71f3f161-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.831 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.831 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.831 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.831 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.831 DEBUG:boto:Data: 2023-09-29T23:08:27.832 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.832 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.832 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.832 DEBUG:boto:Params: {} 2023-09-29T23:08:27.832 DEBUG:boto:Token: None 2023-09-29T23:08:27.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.833 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.838 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000520dbac2be8b2021-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.838 DEBUG:boto:path=/hcnphz-64/obj38 2023-09-29T23:08:27.839 DEBUG:boto:auth_path=/hcnphz-64/obj38 2023-09-29T23:08:27.839 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.839 DEBUG:boto:Path: /hcnphz-64/obj38 2023-09-29T23:08:27.839 DEBUG:boto:Data: 2023-09-29T23:08:27.839 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.839 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.839 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.839 DEBUG:boto:Params: {} 2023-09-29T23:08:27.839 DEBUG:boto:Token: None 2023-09-29T23:08:27.840 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj38 2023-09-29T23:08:27.840 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:q9sx11XIQu4AdH/if0/h1GAPobQ= 2023-09-29T23:08:27.840 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9sx11XIQu4AdH/if0/h1GAPobQ='} 2023-09-29T23:08:27.847 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000096d8c0e4855f15d-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.848 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.848 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.848 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.848 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.848 DEBUG:boto:Data: 2023-09-29T23:08:27.848 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.848 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.848 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.848 DEBUG:boto:Params: {} 2023-09-29T23:08:27.849 DEBUG:boto:Token: None 2023-09-29T23:08:27.849 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.849 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.849 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.854 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba616302b5d3f43c-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.854 DEBUG:boto:path=/hcnphz-64/obj57 2023-09-29T23:08:27.854 DEBUG:boto:auth_path=/hcnphz-64/obj57 2023-09-29T23:08:27.855 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.855 DEBUG:boto:Path: /hcnphz-64/obj57 2023-09-29T23:08:27.855 DEBUG:boto:Data: 2023-09-29T23:08:27.855 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.855 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.855 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.855 DEBUG:boto:Params: {} 2023-09-29T23:08:27.855 DEBUG:boto:Token: None 2023-09-29T23:08:27.855 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj57 2023-09-29T23:08:27.856 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lDP7LbJYU4c2KCwU3PhvwhtEdHM= 2023-09-29T23:08:27.856 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lDP7LbJYU4c2KCwU3PhvwhtEdHM='} 2023-09-29T23:08:27.898 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004efec346b392dd80-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.899 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.899 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.899 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.899 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.899 DEBUG:boto:Data: 2023-09-29T23:08:27.899 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.899 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.900 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.900 DEBUG:boto:Params: {} 2023-09-29T23:08:27.900 DEBUG:boto:Token: None 2023-09-29T23:08:27.900 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.900 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.900 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.905 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f128894b7407e10-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.906 DEBUG:boto:path=/hcnphz-64/obj76 2023-09-29T23:08:27.906 DEBUG:boto:auth_path=/hcnphz-64/obj76 2023-09-29T23:08:27.906 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.906 DEBUG:boto:Path: /hcnphz-64/obj76 2023-09-29T23:08:27.906 DEBUG:boto:Data: 2023-09-29T23:08:27.906 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.906 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.906 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.906 DEBUG:boto:Params: {} 2023-09-29T23:08:27.907 DEBUG:boto:Token: None 2023-09-29T23:08:27.907 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj76 2023-09-29T23:08:27.907 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:IzH5d+0rvAV+Ts95LMabeDOneJM= 2023-09-29T23:08:27.907 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IzH5d+0rvAV+Ts95LMabeDOneJM='} 2023-09-29T23:08:27.918 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f99c6838308edc01-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.918 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:27.918 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:27.918 DEBUG:boto:Method: HEAD 2023-09-29T23:08:27.918 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:27.919 DEBUG:boto:Data: 2023-09-29T23:08:27.919 DEBUG:boto:Headers: {} 2023-09-29T23:08:27.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.919 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.919 DEBUG:boto:Params: {} 2023-09-29T23:08:27.919 DEBUG:boto:Token: None 2023-09-29T23:08:27.919 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/ 2023-09-29T23:08:27.919 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:08:27.919 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:08:27.924 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000628e1208b04d3d4a-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.924 DEBUG:boto:path=/hcnphz-64/obj95 2023-09-29T23:08:27.925 DEBUG:boto:auth_path=/hcnphz-64/obj95 2023-09-29T23:08:27.925 DEBUG:boto:Method: PUT 2023-09-29T23:08:27.925 DEBUG:boto:Path: /hcnphz-64/obj95 2023-09-29T23:08:27.925 DEBUG:boto:Data: 2023-09-29T23:08:27.925 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:27.925 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:27.925 DEBUG:boto:Port: 8000 2023-09-29T23:08:27.925 DEBUG:boto:Params: {} 2023-09-29T23:08:27.925 DEBUG:boto:Token: None 2023-09-29T23:08:27.926 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:27 GMT /hcnphz-64/obj95 2023-09-29T23:08:27.926 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5+Ve9lauabMQalYkYEmsWVl7XH0= 2023-09-29T23:08:27.926 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5+Ve9lauabMQalYkYEmsWVl7XH0='} 2023-09-29T23:08:27.970 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003650fe82a5a7143b-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:27.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:27.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:27.971 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:28.020 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:28.021 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:28.104 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:08:28.104 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-64 2023-09-29T23:08:28.117 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 15 2023-09-29T23:08:28.222 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.222 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.222 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.222 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.222 DEBUG:boto:Data: 2023-09-29T23:08:28.223 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.223 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.223 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.223 DEBUG:boto:Params: {} 2023-09-29T23:08:28.223 DEBUG:boto:Token: None 2023-09-29T23:08:28.223 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.223 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.223 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.227 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008293c57f5cc96530-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.228 DEBUG:boto:path=/hcnphz-64/obj23 2023-09-29T23:08:28.228 DEBUG:boto:auth_path=/hcnphz-64/obj23 2023-09-29T23:08:28.228 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.228 DEBUG:boto:Path: /hcnphz-64/obj23 2023-09-29T23:08:28.228 DEBUG:boto:Data: 2023-09-29T23:08:28.228 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.228 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.229 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.229 DEBUG:boto:Params: {} 2023-09-29T23:08:28.229 DEBUG:boto:Token: None 2023-09-29T23:08:28.229 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj23 2023-09-29T23:08:28.229 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dgDHy46sV2hP+wCnOm0XgOFFKio= 2023-09-29T23:08:28.229 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dgDHy46sV2hP+wCnOm0XgOFFKio='} 2023-09-29T23:08:28.237 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010d626fcf3db0f0d-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.237 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.238 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.238 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.238 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.238 DEBUG:boto:Data: 2023-09-29T23:08:28.238 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.238 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.238 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.238 DEBUG:boto:Params: {} 2023-09-29T23:08:28.239 DEBUG:boto:Token: None 2023-09-29T23:08:28.239 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.239 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.239 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.243 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dde7ae51b32fcdcf-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.243 DEBUG:boto:path=/hcnphz-64/obj46 2023-09-29T23:08:28.243 DEBUG:boto:auth_path=/hcnphz-64/obj46 2023-09-29T23:08:28.243 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.245 DEBUG:boto:Path: /hcnphz-64/obj46 2023-09-29T23:08:28.245 DEBUG:boto:Data: 2023-09-29T23:08:28.245 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.245 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.245 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.245 DEBUG:boto:Params: {} 2023-09-29T23:08:28.245 DEBUG:boto:Token: None 2023-09-29T23:08:28.246 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj46 2023-09-29T23:08:28.246 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XdRDq9QCgPjpA4s+Qklp+LRzcyg= 2023-09-29T23:08:28.246 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XdRDq9QCgPjpA4s+Qklp+LRzcyg='} 2023-09-29T23:08:28.294 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059a13ef26bd3ac21-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.295 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.295 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.295 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.295 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.295 DEBUG:boto:Data: 2023-09-29T23:08:28.295 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.296 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.296 DEBUG:boto:Params: {} 2023-09-29T23:08:28.296 DEBUG:boto:Token: None 2023-09-29T23:08:28.296 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.296 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.296 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.303 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bd16b4eb472d005-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.303 DEBUG:boto:path=/hcnphz-64/obj69 2023-09-29T23:08:28.303 DEBUG:boto:auth_path=/hcnphz-64/obj69 2023-09-29T23:08:28.304 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.304 DEBUG:boto:Path: /hcnphz-64/obj69 2023-09-29T23:08:28.304 DEBUG:boto:Data: 2023-09-29T23:08:28.304 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.305 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.305 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.305 DEBUG:boto:Params: {} 2023-09-29T23:08:28.305 DEBUG:boto:Token: None 2023-09-29T23:08:28.306 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj69 2023-09-29T23:08:28.306 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uvFUKcJ1PWYvrpsFave8AbrGTCo= 2023-09-29T23:08:28.306 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uvFUKcJ1PWYvrpsFave8AbrGTCo='} 2023-09-29T23:08:28.317 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be7658a89d96ec68-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.317 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.317 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.317 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.317 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.317 DEBUG:boto:Data: 2023-09-29T23:08:28.317 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.318 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.318 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.318 DEBUG:boto:Params: {} 2023-09-29T23:08:28.318 DEBUG:boto:Token: None 2023-09-29T23:08:28.318 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.318 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.318 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.323 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041700fd4c27f9aa4-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.323 DEBUG:boto:path=/hcnphz-64/obj92 2023-09-29T23:08:28.324 DEBUG:boto:auth_path=/hcnphz-64/obj92 2023-09-29T23:08:28.324 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.324 DEBUG:boto:Path: /hcnphz-64/obj92 2023-09-29T23:08:28.324 DEBUG:boto:Data: 2023-09-29T23:08:28.324 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.324 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.324 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.324 DEBUG:boto:Params: {} 2023-09-29T23:08:28.324 DEBUG:boto:Token: None 2023-09-29T23:08:28.325 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj92 2023-09-29T23:08:28.325 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ox6yM9WyH9QenmachEelKuAbWZ0= 2023-09-29T23:08:28.325 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ox6yM9WyH9QenmachEelKuAbWZ0='} 2023-09-29T23:08:28.366 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d84b5ca71659926-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.367 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.367 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.367 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.367 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.367 DEBUG:boto:Data: 2023-09-29T23:08:28.367 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.367 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.367 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.367 DEBUG:boto:Params: {} 2023-09-29T23:08:28.368 DEBUG:boto:Token: None 2023-09-29T23:08:28.368 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.368 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.368 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.374 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043351201958f4477-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.374 DEBUG:boto:path=/hcnphz-64/obj115 2023-09-29T23:08:28.374 DEBUG:boto:auth_path=/hcnphz-64/obj115 2023-09-29T23:08:28.375 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.375 DEBUG:boto:Path: /hcnphz-64/obj115 2023-09-29T23:08:28.375 DEBUG:boto:Data: 2023-09-29T23:08:28.375 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.375 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.375 DEBUG:boto:Params: {} 2023-09-29T23:08:28.376 DEBUG:boto:Token: None 2023-09-29T23:08:28.376 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj115 2023-09-29T23:08:28.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yXJoh+VBQQmStytwBpqC7xB0xCU= 2023-09-29T23:08:28.376 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXJoh+VBQQmStytwBpqC7xB0xCU='} 2023-09-29T23:08:28.386 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004641adc5eae9dbe5-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.387 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:28.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:28.387 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:28.433 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:28.434 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:28.515 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:08:28.515 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-64 2023-09-29T23:08:28.532 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 20 2023-09-29T23:08:28.638 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.638 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.638 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.639 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.639 DEBUG:boto:Data: 2023-09-29T23:08:28.639 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.639 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.639 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.639 DEBUG:boto:Params: {} 2023-09-29T23:08:28.639 DEBUG:boto:Token: None 2023-09-29T23:08:28.639 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.640 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.640 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.645 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b52fa273bfe91892-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.645 DEBUG:boto:path=/hcnphz-64/obj29 2023-09-29T23:08:28.646 DEBUG:boto:auth_path=/hcnphz-64/obj29 2023-09-29T23:08:28.646 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.646 DEBUG:boto:Path: /hcnphz-64/obj29 2023-09-29T23:08:28.646 DEBUG:boto:Data: 2023-09-29T23:08:28.646 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.646 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.646 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.646 DEBUG:boto:Params: {} 2023-09-29T23:08:28.647 DEBUG:boto:Token: None 2023-09-29T23:08:28.647 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj29 2023-09-29T23:08:28.647 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Uwnj4mjedUputwQ+K+3F8M3kOII= 2023-09-29T23:08:28.647 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uwnj4mjedUputwQ+K+3F8M3kOII='} 2023-09-29T23:08:28.694 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037fca4e37768d0fc-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.695 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.695 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.695 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.695 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.695 DEBUG:boto:Data: 2023-09-29T23:08:28.695 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.696 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.696 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.696 DEBUG:boto:Params: {} 2023-09-29T23:08:28.696 DEBUG:boto:Token: None 2023-09-29T23:08:28.696 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.696 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.697 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.702 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adf31fe4319ba385-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.703 DEBUG:boto:path=/hcnphz-64/obj58 2023-09-29T23:08:28.703 DEBUG:boto:auth_path=/hcnphz-64/obj58 2023-09-29T23:08:28.703 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.703 DEBUG:boto:Path: /hcnphz-64/obj58 2023-09-29T23:08:28.703 DEBUG:boto:Data: 2023-09-29T23:08:28.703 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.703 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.704 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.704 DEBUG:boto:Params: {} 2023-09-29T23:08:28.704 DEBUG:boto:Token: None 2023-09-29T23:08:28.704 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj58 2023-09-29T23:08:28.704 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OmDsrddYYCrJRsUnU+oL+Mbfgx4= 2023-09-29T23:08:28.704 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OmDsrddYYCrJRsUnU+oL+Mbfgx4='} 2023-09-29T23:08:28.715 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7f0563ac79c39fb-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.715 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.715 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.715 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.715 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.715 DEBUG:boto:Data: 2023-09-29T23:08:28.716 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.716 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.716 DEBUG:boto:Params: {} 2023-09-29T23:08:28.716 DEBUG:boto:Token: None 2023-09-29T23:08:28.716 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.716 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.722 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000950368f051b1cb69-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.722 DEBUG:boto:path=/hcnphz-64/obj87 2023-09-29T23:08:28.722 DEBUG:boto:auth_path=/hcnphz-64/obj87 2023-09-29T23:08:28.723 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.723 DEBUG:boto:Path: /hcnphz-64/obj87 2023-09-29T23:08:28.723 DEBUG:boto:Data: 2023-09-29T23:08:28.723 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.723 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.723 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.723 DEBUG:boto:Params: {} 2023-09-29T23:08:28.724 DEBUG:boto:Token: None 2023-09-29T23:08:28.724 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj87 2023-09-29T23:08:28.724 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ruMT9q9TUznBT+Vh/q+TiGMP9TA= 2023-09-29T23:08:28.724 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ruMT9q9TUznBT+Vh/q+TiGMP9TA='} 2023-09-29T23:08:28.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095a27bb2e646bfc5-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.767 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.767 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.767 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.767 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.767 DEBUG:boto:Data: 2023-09-29T23:08:28.768 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.768 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.768 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.768 DEBUG:boto:Params: {} 2023-09-29T23:08:28.768 DEBUG:boto:Token: None 2023-09-29T23:08:28.768 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.768 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.769 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.774 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd4dc534c472d5b-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.775 DEBUG:boto:path=/hcnphz-64/obj116 2023-09-29T23:08:28.775 DEBUG:boto:auth_path=/hcnphz-64/obj116 2023-09-29T23:08:28.775 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.775 DEBUG:boto:Path: /hcnphz-64/obj116 2023-09-29T23:08:28.775 DEBUG:boto:Data: 2023-09-29T23:08:28.775 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.775 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.775 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.775 DEBUG:boto:Params: {} 2023-09-29T23:08:28.776 DEBUG:boto:Token: None 2023-09-29T23:08:28.776 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj116 2023-09-29T23:08:28.776 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:0cVGTpDjY3A33WOHnY65E6sv0bU= 2023-09-29T23:08:28.776 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0cVGTpDjY3A33WOHnY65E6sv0bU='} 2023-09-29T23:08:28.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd3f785daef3b47b-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.786 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:28.786 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:28.786 DEBUG:boto:Method: HEAD 2023-09-29T23:08:28.787 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:28.787 DEBUG:boto:Data: 2023-09-29T23:08:28.787 DEBUG:boto:Headers: {} 2023-09-29T23:08:28.787 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.787 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.787 DEBUG:boto:Params: {} 2023-09-29T23:08:28.788 DEBUG:boto:Token: None 2023-09-29T23:08:28.788 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/ 2023-09-29T23:08:28.788 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:08:28.788 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:08:28.793 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008e6aad89c1a76dc-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.794 DEBUG:boto:path=/hcnphz-64/obj145 2023-09-29T23:08:28.794 DEBUG:boto:auth_path=/hcnphz-64/obj145 2023-09-29T23:08:28.794 DEBUG:boto:Method: PUT 2023-09-29T23:08:28.794 DEBUG:boto:Path: /hcnphz-64/obj145 2023-09-29T23:08:28.794 DEBUG:boto:Data: 2023-09-29T23:08:28.794 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:28.795 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:28.795 DEBUG:boto:Port: 8000 2023-09-29T23:08:28.795 DEBUG:boto:Params: {} 2023-09-29T23:08:28.795 DEBUG:boto:Token: None 2023-09-29T23:08:28.796 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:28 GMT /hcnphz-64/obj145 2023-09-29T23:08:28.796 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:J+Tu5687J7vKbwYyZe5uAca0mNU= 2023-09-29T23:08:28.796 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J+Tu5687J7vKbwYyZe5uAca0mNU='} 2023-09-29T23:08:28.838 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e64f88adc5f20906-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:28.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:28.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:28.839 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:28.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:28.976 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:08:28.977 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-64 2023-09-29T23:08:28.998 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 25 2023-09-29T23:08:29.123 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.123 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.123 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.123 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.123 DEBUG:boto:Data: 2023-09-29T23:08:29.123 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.123 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.123 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.124 DEBUG:boto:Params: {} 2023-09-29T23:08:29.124 DEBUG:boto:Token: None 2023-09-29T23:08:29.124 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.124 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.124 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.129 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f1f0d68c2d6aadd-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.130 DEBUG:boto:path=/hcnphz-64/obj31 2023-09-29T23:08:29.130 DEBUG:boto:auth_path=/hcnphz-64/obj31 2023-09-29T23:08:29.130 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.130 DEBUG:boto:Path: /hcnphz-64/obj31 2023-09-29T23:08:29.130 DEBUG:boto:Data: 2023-09-29T23:08:29.130 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.130 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.131 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.131 DEBUG:boto:Params: {} 2023-09-29T23:08:29.131 DEBUG:boto:Token: None 2023-09-29T23:08:29.131 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj31 2023-09-29T23:08:29.131 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xFKAiNuF2JhqXFCMnnB+oJmcCoY= 2023-09-29T23:08:29.131 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xFKAiNuF2JhqXFCMnnB+oJmcCoY='} 2023-09-29T23:08:29.141 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6bf34768b218c93-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.141 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.142 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.142 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.142 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.142 DEBUG:boto:Data: 2023-09-29T23:08:29.142 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.142 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.142 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.142 DEBUG:boto:Params: {} 2023-09-29T23:08:29.142 DEBUG:boto:Token: None 2023-09-29T23:08:29.143 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.143 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.143 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.148 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1117808ff7d0699-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.148 DEBUG:boto:path=/hcnphz-64/obj62 2023-09-29T23:08:29.148 DEBUG:boto:auth_path=/hcnphz-64/obj62 2023-09-29T23:08:29.148 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.148 DEBUG:boto:Path: /hcnphz-64/obj62 2023-09-29T23:08:29.148 DEBUG:boto:Data: 2023-09-29T23:08:29.149 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.149 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.149 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.149 DEBUG:boto:Params: {} 2023-09-29T23:08:29.149 DEBUG:boto:Token: None 2023-09-29T23:08:29.149 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj62 2023-09-29T23:08:29.150 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:E5kPQwCvfIqMpaSSvQV6Jawzf9M= 2023-09-29T23:08:29.150 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E5kPQwCvfIqMpaSSvQV6Jawzf9M='} 2023-09-29T23:08:29.194 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000601211c5046f97-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.195 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.195 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.195 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.195 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.195 DEBUG:boto:Data: 2023-09-29T23:08:29.195 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.195 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.196 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.196 DEBUG:boto:Params: {} 2023-09-29T23:08:29.196 DEBUG:boto:Token: None 2023-09-29T23:08:29.196 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.196 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.203 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003608d865a54b388f-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.203 DEBUG:boto:path=/hcnphz-64/obj93 2023-09-29T23:08:29.204 DEBUG:boto:auth_path=/hcnphz-64/obj93 2023-09-29T23:08:29.204 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.204 DEBUG:boto:Path: /hcnphz-64/obj93 2023-09-29T23:08:29.204 DEBUG:boto:Data: 2023-09-29T23:08:29.204 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.204 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.204 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.204 DEBUG:boto:Params: {} 2023-09-29T23:08:29.204 DEBUG:boto:Token: None 2023-09-29T23:08:29.205 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj93 2023-09-29T23:08:29.205 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:z7W92xRkFg+mE4e0AHfYI49MJOE= 2023-09-29T23:08:29.205 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7W92xRkFg+mE4e0AHfYI49MJOE='} 2023-09-29T23:08:29.216 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000fdd5682d942aba-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.216 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.216 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.216 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.216 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.216 DEBUG:boto:Data: 2023-09-29T23:08:29.217 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.217 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.217 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.217 DEBUG:boto:Params: {} 2023-09-29T23:08:29.217 DEBUG:boto:Token: None 2023-09-29T23:08:29.217 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.218 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.223 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b92b82a4ca70fdd2-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.223 DEBUG:boto:path=/hcnphz-64/obj124 2023-09-29T23:08:29.224 DEBUG:boto:auth_path=/hcnphz-64/obj124 2023-09-29T23:08:29.224 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.224 DEBUG:boto:Path: /hcnphz-64/obj124 2023-09-29T23:08:29.224 DEBUG:boto:Data: 2023-09-29T23:08:29.224 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.224 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.224 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.224 DEBUG:boto:Params: {} 2023-09-29T23:08:29.224 DEBUG:boto:Token: None 2023-09-29T23:08:29.225 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj124 2023-09-29T23:08:29.225 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Q1F66Vm/gXtFwmgPXreIslGXyew= 2023-09-29T23:08:29.225 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q1F66Vm/gXtFwmgPXreIslGXyew='} 2023-09-29T23:08:29.266 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039afa990908cb3bc-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.267 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.267 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.267 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.268 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.268 DEBUG:boto:Data: 2023-09-29T23:08:29.268 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.268 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.268 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.269 DEBUG:boto:Params: {} 2023-09-29T23:08:29.269 DEBUG:boto:Token: None 2023-09-29T23:08:29.269 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.270 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.270 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.277 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6431d1a874ec7e5-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.278 DEBUG:boto:path=/hcnphz-64/obj155 2023-09-29T23:08:29.278 DEBUG:boto:auth_path=/hcnphz-64/obj155 2023-09-29T23:08:29.278 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.279 DEBUG:boto:Path: /hcnphz-64/obj155 2023-09-29T23:08:29.279 DEBUG:boto:Data: 2023-09-29T23:08:29.279 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.279 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.279 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.279 DEBUG:boto:Params: {} 2023-09-29T23:08:29.280 DEBUG:boto:Token: None 2023-09-29T23:08:29.280 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj155 2023-09-29T23:08:29.280 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:iumiTN4f7/qkLdyN0VuF2ybHYv4= 2023-09-29T23:08:29.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iumiTN4f7/qkLdyN0VuF2ybHYv4='} 2023-09-29T23:08:29.290 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026cd8deb670b33ac-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:29.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:29.290 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:29.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T23:08:29.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:29.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:29.428 INFO:teuthology.orchestra.run.smithi112.stdout:tenant: 2023-09-29T23:08:29.428 INFO:teuthology.orchestra.run.smithi112.stdout:bucket name: hcnphz-64 2023-09-29T23:08:29.452 INFO:teuthology.orchestra.run.smithi112.stdout:total entries: 30 2023-09-29T23:08:29.577 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.577 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.578 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.578 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.578 DEBUG:boto:Data: 2023-09-29T23:08:29.578 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.578 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.578 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.578 DEBUG:boto:Params: {} 2023-09-29T23:08:29.578 DEBUG:boto:Token: None 2023-09-29T23:08:29.579 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.579 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.579 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.585 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d4bfad0b95087a6-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.585 DEBUG:boto:path=/hcnphz-64/obj37 2023-09-29T23:08:29.585 DEBUG:boto:auth_path=/hcnphz-64/obj37 2023-09-29T23:08:29.585 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.586 DEBUG:boto:Path: /hcnphz-64/obj37 2023-09-29T23:08:29.586 DEBUG:boto:Data: 2023-09-29T23:08:29.586 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.586 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.586 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.586 DEBUG:boto:Params: {} 2023-09-29T23:08:29.586 DEBUG:boto:Token: None 2023-09-29T23:08:29.586 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj37 2023-09-29T23:08:29.587 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pq9fua7yfSlwKc354hDYdvZaEgA= 2023-09-29T23:08:29.587 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pq9fua7yfSlwKc354hDYdvZaEgA='} 2023-09-29T23:08:29.630 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070f35307a9d492c4-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.631 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.631 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.631 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.631 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.631 DEBUG:boto:Data: 2023-09-29T23:08:29.631 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.631 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.631 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.632 DEBUG:boto:Params: {} 2023-09-29T23:08:29.632 DEBUG:boto:Token: None 2023-09-29T23:08:29.632 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.632 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.632 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.639 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3164dfc2e5d43ba-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.639 DEBUG:boto:path=/hcnphz-64/obj74 2023-09-29T23:08:29.639 DEBUG:boto:auth_path=/hcnphz-64/obj74 2023-09-29T23:08:29.640 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.640 DEBUG:boto:Path: /hcnphz-64/obj74 2023-09-29T23:08:29.640 DEBUG:boto:Data: 2023-09-29T23:08:29.640 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.640 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.641 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.641 DEBUG:boto:Params: {} 2023-09-29T23:08:29.641 DEBUG:boto:Token: None 2023-09-29T23:08:29.641 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj74 2023-09-29T23:08:29.641 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yCafieqdpd0V3sM8DLoC0rjlpD0= 2023-09-29T23:08:29.641 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCafieqdpd0V3sM8DLoC0rjlpD0='} 2023-09-29T23:08:29.650 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca92505fed6fbc06-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.650 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.650 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.650 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.650 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.651 DEBUG:boto:Data: 2023-09-29T23:08:29.651 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.651 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.651 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.651 DEBUG:boto:Params: {} 2023-09-29T23:08:29.651 DEBUG:boto:Token: None 2023-09-29T23:08:29.651 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.652 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.652 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.658 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057d404f0e1d22159-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.658 DEBUG:boto:path=/hcnphz-64/obj111 2023-09-29T23:08:29.658 DEBUG:boto:auth_path=/hcnphz-64/obj111 2023-09-29T23:08:29.658 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.658 DEBUG:boto:Path: /hcnphz-64/obj111 2023-09-29T23:08:29.658 DEBUG:boto:Data: 2023-09-29T23:08:29.658 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.659 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.659 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.659 DEBUG:boto:Params: {} 2023-09-29T23:08:29.659 DEBUG:boto:Token: None 2023-09-29T23:08:29.659 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj111 2023-09-29T23:08:29.659 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LntyddlroZBjqNST93aVFituHFU= 2023-09-29T23:08:29.659 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LntyddlroZBjqNST93aVFituHFU='} 2023-09-29T23:08:29.706 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5cf61a9acf0076a-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.707 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.707 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.707 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.707 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.707 DEBUG:boto:Data: 2023-09-29T23:08:29.707 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.707 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.708 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.708 DEBUG:boto:Params: {} 2023-09-29T23:08:29.708 DEBUG:boto:Token: None 2023-09-29T23:08:29.708 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.708 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.708 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.715 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c98bbc2ec3b39029-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.716 DEBUG:boto:path=/hcnphz-64/obj148 2023-09-29T23:08:29.716 DEBUG:boto:auth_path=/hcnphz-64/obj148 2023-09-29T23:08:29.716 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.716 DEBUG:boto:Path: /hcnphz-64/obj148 2023-09-29T23:08:29.716 DEBUG:boto:Data: 2023-09-29T23:08:29.716 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.716 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.716 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.717 DEBUG:boto:Params: {} 2023-09-29T23:08:29.717 DEBUG:boto:Token: None 2023-09-29T23:08:29.717 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj148 2023-09-29T23:08:29.717 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yI75IkBDSiFKr6/nP0G8rHEu0xw= 2023-09-29T23:08:29.717 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yI75IkBDSiFKr6/nP0G8rHEu0xw='} 2023-09-29T23:08:29.728 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0c34c4de88268fd-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.728 DEBUG:boto:path=/hcnphz-64/ 2023-09-29T23:08:29.728 DEBUG:boto:auth_path=/hcnphz-64/ 2023-09-29T23:08:29.729 DEBUG:boto:Method: HEAD 2023-09-29T23:08:29.729 DEBUG:boto:Path: /hcnphz-64/ 2023-09-29T23:08:29.729 DEBUG:boto:Data: 2023-09-29T23:08:29.729 DEBUG:boto:Headers: {} 2023-09-29T23:08:29.729 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.729 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.729 DEBUG:boto:Params: {} 2023-09-29T23:08:29.730 DEBUG:boto:Token: None 2023-09-29T23:08:29.730 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/ 2023-09-29T23:08:29.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:08:29.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:08:29.736 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e10db53e11b8d6e7-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.737 DEBUG:boto:path=/hcnphz-64/obj185 2023-09-29T23:08:29.737 DEBUG:boto:auth_path=/hcnphz-64/obj185 2023-09-29T23:08:29.737 DEBUG:boto:Method: PUT 2023-09-29T23:08:29.737 DEBUG:boto:Path: /hcnphz-64/obj185 2023-09-29T23:08:29.737 DEBUG:boto:Data: 2023-09-29T23:08:29.738 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:08:29.738 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:08:29.738 DEBUG:boto:Port: 8000 2023-09-29T23:08:29.738 DEBUG:boto:Params: {} 2023-09-29T23:08:29.738 DEBUG:boto:Token: None 2023-09-29T23:08:29.739 DEBUG:boto:StringToSign: PUT rL0Y20zC+Fzt72VPzMSk2A== application/octet-stream Fri, 29 Sep 2023 23:08:29 GMT /hcnphz-64/obj185 2023-09-29T23:08:29.739 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jbyEuWCKZk2eHZ2HWc1P4DItY3o= 2023-09-29T23:08:29.739 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbyEuWCKZk2eHZ2HWc1P4DItY3o='} 2023-09-29T23:08:29.786 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb3c73ebaece6587-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:08:29.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:29.787 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:08:29.787 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:08:29.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:29.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:29.891 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:29.891+0000 7ff864fd5f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-29T23:08:29.893 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:29.891+0000 7ff864fd5f40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-29T23:08:29.932 INFO:tasks.rgw.c1.client.1:Restarting daemon 2023-09-29T23:08:29.932 DEBUG:teuthology.orchestra.run.smithi112:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T23:08:29.976 INFO:tasks.rgw.c1.client.1:Started 2023-09-29T23:08:29.994 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:30.005 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:08:30.005 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:08:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-09-29T23:08:30.008 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T23:08:30.008 INFO:teuthology.orchestra.run.smithi112.stderr:curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T23:08:31.009 DEBUG:teuthology.orchestra.run.smithi112:> curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T23:08:31.025 INFO:teuthology.orchestra.run.smithi112.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:08:31.025 INFO:teuthology.orchestra.run.smithi112.stderr: Dload Upload Total Spent Left Speed 2023-09-29T23:08:31.028 INFO:teuthology.orchestra.run.smithi112.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T23:08:31.029 INFO:teuthology.orchestra.run.smithi112.stdout:anonymous 2023-09-29T23:08:31.029 DEBUG:rgw_multi.tests:bucket checkpoint: source=a1 target=a2 bucket=hcnphz-64 2023-09-29T23:08:31.029 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:08:31.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:08:31.030 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:08:31.123 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:08:31.123 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:08:31.187 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:31.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:08:36.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:36.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:08:41.189 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:41.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:08:46.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:46.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:08:51.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:51.191+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:08:56.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:08:56.190+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:01.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:01.190+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:06.193 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:06.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:11.194 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:11.190+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:16.195 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:16.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:21.196 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:21.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:26.197 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:26.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:31.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:31.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:36.198 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:36.197+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:41.200 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:41.197+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:46.202 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:46.201+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:51.203 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:51.205+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:09:56.204 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:09:56.205+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:10:01.205 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:01.205+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:06.207 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:06.209+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:11.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:11.208+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:16.209 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:16.208+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:21.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:21.208+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:26.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:26.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:31.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:31.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:36.213 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:36.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:41.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:41.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:46.215 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:46.216+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:51.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:51.215+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:10:56.216 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:10:56.215+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:01.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:01.219+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:06.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:06.215+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:11.219 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:11.219+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:16.220 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:16.219+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:21.221 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:21.223+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:26.223 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:26.223+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:31.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:31.222+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:36.224 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:36.222+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:41.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:41.222+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:46.226 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:46.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:51.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:51.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:11:56.228 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:11:56.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:01.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:01.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:06.230 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:06.225+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:11.231 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:11.229+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:16.232 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:16.229+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:21.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:21.233+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:26.234 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:26.233+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:30.390 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:12:30.389+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:12:31.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:31.237+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:36.235 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:36.237+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:41.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:41.237+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:46.237 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:46.236+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:51.238 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:51.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:12:56.239 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:12:56.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:13:01.241 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:01.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:13:06.242 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:06.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:13:11.243 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:11.244+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:13:16.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:16.244+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:13:21.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:21.244+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:13:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:26.243+0000 7ff58b31cf40 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-09-29T23:13:26.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:26.243+0000 7ff58b31cf40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:13:26.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:26.243+0000 7ff58b31cf40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:13:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 110 2023-09-29T23:13:26.274 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init ... FAIL 2023-09-29T23:13:26.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:26.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:26.276 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:26.318 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:26.318 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:26.377 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:26.407 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:29.470637Z" 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:26.408 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:13:26.410 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:26.410 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:13:26.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:26.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:26.410 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:26.455 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:26.455 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:26.514 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:26.511+0000 7f0ef3816f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:26.516 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:26.515+0000 7f0ef3816f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:26.544 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:26.544 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:26.545 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:26.546 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:26.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:26.548 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:26.549 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:26.550 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:26Z" 2023-09-29T23:13:26.551 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:26.552 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:26Z" } 2023-09-29T23:13:26.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:26.553 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:26.553 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:26.553 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:13:26.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.553 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:26.595 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:26.685 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:13:26.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.686 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:26.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:26.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:26.824 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:13:26.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.825 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:26.867 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:26.867 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:26.951 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:26.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:26.952 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:26.994 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.089 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-29T23:13:27.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.090 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:27.169 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.257 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:27.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.258 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:27.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:27.307 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.400 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:27.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.401 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:27.448 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:27.449 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.555 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:27.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.556 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:27.601 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:27.601 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.692 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:13:27.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.693 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:27.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:27.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.824 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"} 2023-09-29T23:13:27.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:27.825 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:27.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 13, 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:13:27.983 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.984 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.985 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:13:27.986 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:13:27.987 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:13:27.988 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:13:27.989 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:13:27.990 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: "directional": [ 2023-09-29T23:13:27.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: "source_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: "dest_zone": "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:13:27.992 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:13:27.993 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.994 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:13:27.996 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:13:27.997 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:13:27.998 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:13:27.999 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:13:28.000 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:13:28.001 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.002 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:13:28.007 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:13:28.007 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:28.008 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:28.009 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:13:28.010 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:13:28.011 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:13:28.012 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:13:28.013 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:28.014 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:28.015 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-65 2023-09-29T23:13:28.015 DEBUG:boto:path=/hcnphz-65/ 2023-09-29T23:13:28.015 DEBUG:boto:auth_path=/hcnphz-65/ 2023-09-29T23:13:28.016 DEBUG:boto:Method: PUT 2023-09-29T23:13:28.016 DEBUG:boto:Path: /hcnphz-65/ 2023-09-29T23:13:28.016 DEBUG:boto:Data: 2023-09-29T23:13:28.016 DEBUG:boto:Headers: {} 2023-09-29T23:13:28.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:13:28.016 DEBUG:boto:Port: 8000 2023-09-29T23:13:28.016 DEBUG:boto:Params: {} 2023-09-29T23:13:28.017 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:13:28.017 DEBUG:boto:Token: None 2023-09-29T23:13:28.017 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:13:28 GMT /hcnphz-65/ 2023-09-29T23:13:28.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ghQmX8RZ6LvZZnySmiJoXy6zMvY= 2023-09-29T23:13:28.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:13:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ghQmX8RZ6LvZZnySmiJoXy6zMvY=', 'Content-Length': '0'} 2023-09-29T23:13:28.281 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000002687408fd922c2cb-0065175a18-206a-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:13:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:13:28.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.282 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:28.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:28.490 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:13:28.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.491 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:28.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:28.546 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:28.635 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:28.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.636 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:28.682 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:28.682 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:28.763 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"} 2023-09-29T23:13:28.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.764 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:28.807 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:28.807 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:28.878 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:28.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:28.879 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:28.925 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:28.926 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:29.020 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:29.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:29.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:29.020 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:29.071 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:29.072 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:29.173 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:13:29.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:29.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:29.174 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:29.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:29.222 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:29.320 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:13:29.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:29.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'get', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:13:29.321 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group get --group-id sync-bucket --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:13:29.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:29.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:29.454 INFO:teuthology.orchestra.run.smithi112.stdout:{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"} 2023-09-29T23:13:29.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:29.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:29.455 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:29.499 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:29.499 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:29.561 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029209.294756_704.1", 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:13:29.294756Z" 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:29.593 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:13:29.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:29.593 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:13:29.594 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:29.594 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:13:29.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:29.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:29.595 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:29.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:29.640 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:29.699 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:29.699+0000 7ff75a994f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:29.701 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:29.699+0000 7ff75a994f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:29.725 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:29.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:29.727 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:29.728 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:29.729 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:29.730 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:29.731 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:29Z" 2023-09-29T23:13:29.732 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:29.733 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:29Z" } 2023-09-29T23:13:29.733 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:29.734 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:29.734 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:29.734 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:13:34.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:34.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:34.736 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:34.781 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:34.781 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:34.835 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:34.835+0000 7fa020b54f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:34.836 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:34.835+0000 7fa020b54f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:34.865 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:34.866 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:34.867 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:34.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:34.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:34.870 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:34.871 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:34Z" 2023-09-29T23:13:34.872 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:34.873 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:34Z" } 2023-09-29T23:13:34.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:34.873 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:34.873 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:34.873 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:13:39.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:39.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:39.875 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:39.919 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:39.974 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:39.975+0000 7fb3ee8a7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:39.976 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:39.975+0000 7fb3ee8a7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:40.004 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:40.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:40.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:40.007 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:40.008 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:40.009 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:40.010 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:40.011 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:39Z" 2023-09-29T23:13:40.012 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:40.013 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:39Z" } 2023-09-29T23:13:40.013 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:40.013 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:40.013 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:40.013 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:13:45.014 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:45.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:45.015 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:45.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:45.061 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:45.119 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:45.119+0000 7fbd9eabff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:45.120 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:45.119+0000 7fbd9eabff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:45.147 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:45.147 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:45.147 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:45.147 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:45.147 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:45.147 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:45.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:45.149 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:45.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:45.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:45.152 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:45.153 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:45.154 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:45Z" 2023-09-29T23:13:45.154 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:45.154 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:45Z" } 2023-09-29T23:13:45.155 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:45.155 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:45.155 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:45.155 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:13:50.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:50.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:50.157 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:50.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:50.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:50.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:50.259+0000 7fe260cfef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:50.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:50.259+0000 7fe260cfef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:50.287 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:50.287 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:50.288 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:50.289 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:50.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:50.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:50.292 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:50.293 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:50.294 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:50.295 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:50Z" 2023-09-29T23:13:50.295 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:50.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:50Z" } 2023-09-29T23:13:50.296 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:50.296 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:50.297 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:50.297 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:13:55.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:55.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:13:55.299 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:13:55.345 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:13:55.346 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:13:55.404 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:55.403+0000 7f13dfc85f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:55.406 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:13:55.403+0000 7f13dfc85f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:13:55.433 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:13:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:13:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:13:55.433 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:55.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:13:55.435 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:55.436 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:55.437 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:13:55.438 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:13:55.439 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:13:55Z" 2023-09-29T23:13:55.440 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:13:55.440 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:13:55Z" } 2023-09-29T23:13:55.441 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:13:55.441 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:13:55.441 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:13:55.441 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:00.442 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:00.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:00.442 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:00.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:00.488 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:00.548 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:00.547+0000 7f1927d05f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:00.550 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:00.551+0000 7f1927d05f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:00.576 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:00.576 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:00.576 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:00.576 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:00.576 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:00.577 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:00.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:00.579 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:00.580 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:00.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:00.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:00Z" 2023-09-29T23:14:00.583 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:00.584 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:00Z" } 2023-09-29T23:14:00.585 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:00.585 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:00.585 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:00.585 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:05.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:05.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:05.587 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:05.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:05.634 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:05.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:05.686+0000 7f2318bddf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:05.687 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:05.686+0000 7f2318bddf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:05.713 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:05.713 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:05.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:05.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:05.716 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:05.717 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:05.718 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:05.719 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:05Z" 2023-09-29T23:14:05.720 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:05.721 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:05Z" } 2023-09-29T23:14:05.722 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:05.722 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:05.722 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:05.722 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:10.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:10.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:10.724 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:10.769 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:10.769 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:10.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:10.822+0000 7ff86fd36f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:10.827 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:10.826+0000 7ff86fd36f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:10.853 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:10.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:10.855 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:10.856 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:10.857 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:10.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:10.859 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:10.860 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:10Z" 2023-09-29T23:14:10.861 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:10.861 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:10Z" } 2023-09-29T23:14:10.862 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:10.862 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:10.862 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:10.862 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:15.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:15.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:15.864 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:15.909 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:15.909 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:15.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:15.958+0000 7f0c23babf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:15.963 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:15.962+0000 7f0c23babf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:15.991 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:15.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:15.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:15.994 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:15.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:15.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:15Z" 2023-09-29T23:14:15.997 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:15.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:15Z" } 2023-09-29T23:14:15.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:15.999 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:15.999 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:15.999 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:21.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:21.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:21.001 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:21.047 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:21.047 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:21.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:21.102+0000 7f0f9411af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:21.105 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:21.102+0000 7f0f9411af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:21.133 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:21.133 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:21.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:21.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:21.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:21.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:21.138 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:21.139 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:21Z" 2023-09-29T23:14:21.140 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:21.141 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:21Z" } 2023-09-29T23:14:21.142 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:21.142 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:21.142 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:21.142 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:26.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:26.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:26.143 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:26.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:26.189 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:26.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:26.246+0000 7fbf42555f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:26.248 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:26.246+0000 7fbf42555f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:26.273 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:26.274 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:26.275 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:26.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:26.277 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:26.278 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:26.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:26.280 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:26Z" 2023-09-29T23:14:26.280 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:26.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:26Z" } 2023-09-29T23:14:26.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:26.281 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:26.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:26.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:31.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:31.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:31.282 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:31.328 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:31.329 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:31.387 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:31.386+0000 7f9efb3fbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:31.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:31.386+0000 7f9efb3fbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:31.415 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:31.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:31.417 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:31.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:31Z" 2023-09-29T23:14:31.422 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:31.423 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:31Z" } 2023-09-29T23:14:31.423 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:31.423 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:31.423 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:31.423 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:36.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:36.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:36.426 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:36.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:36.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:36.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:36.530+0000 7f78dc39cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:36.533 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:36.530+0000 7f78dc39cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:36.559 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:36.559 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:36.559 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:36.560 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:36.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696028909.470637_669.1", 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:36.562 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.470637Z", 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:36.563 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:36.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:36.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:36.566 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:36.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:36.566 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:36.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:36.566 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:36Z" 2023-09-29T23:14:36.566 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:36.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696028909.470637_669.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:29.470637Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:36Z" } 2023-09-29T23:14:36.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028909.470637_669.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.470637Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:36.567 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:36.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028909.470637_669.1', 2: '', 3: ''} 2023-09-29T23:14:36.568 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696028909.470637_669.1 2023-09-29T23:14:41.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:41.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:41.569 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:41.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:41.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:41.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:41.669+0000 7f94d7641f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:41.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:41.669+0000 7f94d7641f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:41.697 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:41.697 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:41.698 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:41.699 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029208.260485_683.1", 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:41.700 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:13:28.260485Z", 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:41.701 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:41.702 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:41.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:41Z" 2023-09-29T23:14:41.704 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:41.705 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029208.260485_683.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:13:28.260485Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:41Z" } 2023-09-29T23:14:41.705 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029208.260485_683.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:13:28.260485Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:41.706 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:41.706 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029208.260485_683.1', 2: '', 3: ''} 2023-09-29T23:14:41.706 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696029208.260485_683.1 2023-09-29T23:14:46.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:46.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:46.708 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:46.755 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:46.755 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:46.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:46.809+0000 7fa72cfabf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:46.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:46.809+0000 7fa72cfabf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:46.842 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:46.842 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:46.842 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:46.842 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:46.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:46.844 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:46.845 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029208.991747_700.1", 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:13:28.991747Z", 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:46.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:46.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:46.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:46.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:46Z" 2023-09-29T23:14:46.850 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:46.852 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029208.991747_700.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:13:28.991747Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:46Z" } 2023-09-29T23:14:46.852 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029208.991747_700.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:13:28.991747Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:46.852 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:46.853 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029208.991747_700.1', 2: '', 3: ''} 2023-09-29T23:14:46.853 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029209.294756_704.1 target=1_1696029208.991747_700.1 2023-09-29T23:14:51.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:51.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:51.855 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:51.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:51.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:51.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:51.957+0000 7f0cf63f2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:51.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:51.961+0000 7f0cf63f2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:51.987 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:51.987 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:51.987 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:51.987 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:51.987 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:51.987 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:51.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:51.989 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029209.294756_704.1", 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:13:29.294756Z", 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:51.990 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:51.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:51.992 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:51.993 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:51Z" 2023-09-29T23:14:51.994 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:51.994 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029209.294756_704.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:13:29.294756Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:51Z" } 2023-09-29T23:14:51.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029209.294756_704.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:13:29.294756Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:51.995 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:51.995 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:51.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:14:51.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:51.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--bucket=hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:51.996 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-bucket --pipe-id sync-pipe --bucket=hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:14:52.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.149 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:14:52.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.149 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:14:52.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.298 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:14:52.298 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-bucket', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.299 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-bucket --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:14:52.349 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.350 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.456 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:14:52.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.457 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:14:52.499 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.500 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.594 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:14:52.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hcnphz-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:14:52.595 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hcnphz-65 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:14:52.638 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.638 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.731 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:14:52.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:52.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:52.732 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:52.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.834 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:14:52.861 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:14:52.861 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.861 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:52.861 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:14:52.861 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:14:52.567201Z" 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:14:52.862 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:52.863 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.863 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:52.863 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:14:52.863 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:52.863 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:14:52.864 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:14:52.864 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:14:52.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:52.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:52.864 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:52.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:52.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:52.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:52.961+0000 7f4f67396f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:52.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:52.965+0000 7f4f67396f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:52.993 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:52.994 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:52.994 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:52.994 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:52.994 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:52.994 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:52.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:52.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:52.995 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:52.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.995 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:52.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:52.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:52.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:52.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:52.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:52.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:52.997 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:52.997 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:52.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:52.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:52.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029209.294756_704.1", 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:52.998 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:13:29.294756Z", 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:52.999 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:53.000 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:53.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:53.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:53.002 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:53.002 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:53.002 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:53.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:53.002 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:52Z" 2023-09-29T23:14:53.003 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:53.004 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029209.294756_704.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:13:29.294756Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:52Z" } 2023-09-29T23:14:53.004 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029209.294756_704.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:13:29.294756Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:53.004 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:14:53.004 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:53.004 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029292.567201_712.1 target=1_1696029209.294756_704.1 2023-09-29T23:14:58.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:58.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:14:58.006 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:14:58.051 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:14:58.051 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:14:58.106 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:58.109+0000 7fc06e12ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:58.108 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:14:58.109+0000 7fc06e12ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:14:58.131 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:14:58.131 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:14:58.131 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:58.132 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:58.133 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029209.294756_704.1", 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:13:29.294756Z", 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:58.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:58.135 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:14:58.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:14:58.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:14:58.138 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:14:58Z" 2023-09-29T23:14:58.138 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:14:58.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029209.294756_704.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:13:29.294756Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:14:58Z" } 2023-09-29T23:14:58.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029209.294756_704.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:13:29.294756Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:14:58.139 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:14:58.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:14:58.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029292.567201_712.1 target=1_1696029209.294756_704.1 2023-09-29T23:14:59.674 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:14:59.670+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:15:03.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:03.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:03.140 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:03.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:03.185 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:03.244 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:03.245+0000 7f5be6bfef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:03.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:03.249+0000 7f5be6bfef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:03.273 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:03.273 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:03.273 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:03.273 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:03.273 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:03.274 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:03.275 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029209.294756_704.1", 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:13:29.294756Z", 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:03.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:03.277 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:03.278 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:03.279 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:03.280 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:03.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:03.280 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:03Z" 2023-09-29T23:15:03.280 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:03.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029209.294756_704.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:13:29.294756Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:03Z" } 2023-09-29T23:15:03.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029209.294756_704.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:13:29.294756Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:03.281 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:03.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029209.294756_704.1', 2: '', 3: ''} 2023-09-29T23:15:03.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029292.567201_712.1 target=1_1696029209.294756_704.1 2023-09-29T23:15:08.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:08.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:08.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:08.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:08.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:08.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:08.389+0000 7fe031570f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:08.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:08.389+0000 7fe031570f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:08.416 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:08.417 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.115339_705.1", 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:08.418 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.115339Z", 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:08.419 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:08.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:08.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:08Z" 2023-09-29T23:15:08.422 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:08.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.115339_705.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.115339Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:08Z" } 2023-09-29T23:15:08.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.115339_705.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.115339Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:08.424 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:08.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.115339_705.1', 2: '', 3: ''} 2023-09-29T23:15:08.424 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029292.567201_712.1 target=1_1696029292.115339_705.1 2023-09-29T23:15:13.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:13.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:13.426 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:13.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:13.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:13.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:13.533+0000 7f739686ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:13.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:13.533+0000 7f739686ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:13.560 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:13.560 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:13.561 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:13.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:13.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:13.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:13.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:13Z" 2023-09-29T23:15:13.566 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:13.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:13Z" } 2023-09-29T23:15:13.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:13.568 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:13.568 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:13.568 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:15:13.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:13.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:13.568 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:13.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:13.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:13.709 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}],"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:15:13.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:13.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow2', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:13.710 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow2 --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:13.804 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:13.804 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:13.894 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:15:13.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:13.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:13.895 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:13.941 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:13.941 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:14.036 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:15:14.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.037 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:14.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:14.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:14.165 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:15:14.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.166 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:14.252 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:14.252 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:14.337 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:15:14.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.338 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:14.384 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:14.499 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:14.499 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 14, 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:15:14.500 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:14.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:14.502 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.503 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:14.504 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:15:14.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:15:14.506 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:15:14.507 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.508 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:14.509 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:14.510 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:14.522 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:14.523 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:15:14.524 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:15:14.525 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:15:14.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:15:14.527 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:14.528 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:15:14.529 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:15:14.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:15:14.531 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:14.532 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:14.534 INFO:tasks.rgw_multisite_tests:test_sync_policy_config_zonegroup: ... ok 2023-09-29T23:15:14.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:14.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:14.535 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:14.577 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:14.577 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:14.640 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:15:14.667 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:15:14.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.667 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:14.667 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:14:52.567201Z" 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.668 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.669 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:14.669 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:15:14.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.669 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:15:14.670 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:14.670 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:15:14.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:14.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:14.671 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:14.764 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:14.764 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:14.852 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:14.853+0000 7f365aac0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:14.866 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:14.869+0000 7f365aac0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:14.904 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:14.904 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:14.905 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:14.907 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:14.908 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:14.909 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:14.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:14.911 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:14.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:14.911 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:14Z" 2023-09-29T23:15:14.911 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:14.912 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:14Z" } 2023-09-29T23:15:14.912 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:14.912 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:14.912 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:14.912 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:15:14.913 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:14.913 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:14.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:14.965 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.080 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:15:15.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.080 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:15.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:15.129 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.205 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:15:15.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.206 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:15.257 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:15.258 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.334 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:15:15.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.334 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:15.377 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:15.377 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.455 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:15:15.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.456 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:15.498 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:15.498 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.587 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:15:15.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.588 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:15.631 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:15.631 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 15, 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:15:15.763 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.764 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.765 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:15.766 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.767 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:15.768 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:15:15.769 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:15:15.770 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:15:15.773 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.774 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.775 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:15:15.776 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:15:15.777 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:15.778 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:15.779 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:15.779 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:15.779 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:15.779 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:15.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.780 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:15:15.780 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:15:15.780 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:15:15.780 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:15:15.781 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.782 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:15:15.783 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:15:15.784 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:15:15.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.785 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.785 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:15:15.785 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:15:15.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:15:15.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:15:15.789 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:15:15.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:15:15.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:15:15.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:15.793 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:15.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:15:15.795 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:15:15.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:15.837 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:15.917 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:15:15.918 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-66 2023-09-29T23:15:15.918 DEBUG:boto:path=/hcnphz-66/ 2023-09-29T23:15:15.918 DEBUG:boto:auth_path=/hcnphz-66/ 2023-09-29T23:15:15.918 DEBUG:boto:Method: PUT 2023-09-29T23:15:15.918 DEBUG:boto:Path: /hcnphz-66/ 2023-09-29T23:15:15.918 DEBUG:boto:Data: 2023-09-29T23:15:15.919 DEBUG:boto:Headers: {} 2023-09-29T23:15:15.919 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:15:15.919 DEBUG:boto:Port: 8000 2023-09-29T23:15:15.919 DEBUG:boto:Params: {} 2023-09-29T23:15:15.919 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:15:15.919 DEBUG:boto:Token: None 2023-09-29T23:15:15.920 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:15:15 GMT /hcnphz-66/ 2023-09-29T23:15:15.920 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yqT2WNtcCDqP+xtooI/q36ZoUIg= 2023-09-29T23:15:15.920 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yqT2WNtcCDqP+xtooI/q36ZoUIg=', 'Content-Length': '0'} 2023-09-29T23:15:16.025 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000cde69bf3867a1293-0065175a83-2217-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:15:16.025 DEBUG:boto:path=/hcnphz-66/ 2023-09-29T23:15:16.025 DEBUG:boto:auth_path=/hcnphz-66/ 2023-09-29T23:15:16.025 DEBUG:boto:Method: HEAD 2023-09-29T23:15:16.025 DEBUG:boto:Path: /hcnphz-66/ 2023-09-29T23:15:16.025 DEBUG:boto:Data: 2023-09-29T23:15:16.026 DEBUG:boto:Headers: {} 2023-09-29T23:15:16.026 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:15:16.026 DEBUG:boto:Port: 8000 2023-09-29T23:15:16.026 DEBUG:boto:Params: {} 2023-09-29T23:15:16.026 DEBUG:boto:Token: None 2023-09-29T23:15:16.026 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:15:16 GMT /hcnphz-66/ 2023-09-29T23:15:16.026 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bV/MxvkPOKI2058Y3erXXVQaMng= 2023-09-29T23:15:16.027 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bV/MxvkPOKI2058Y3erXXVQaMng=', 'Content-Length': '0'} 2023-09-29T23:15:16.030 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041c475760ae6a02a-0065175a84-2217-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:15:16.030 DEBUG:boto:path=/hcnphz-66/obj1 2023-09-29T23:15:16.031 DEBUG:boto:auth_path=/hcnphz-66/obj1 2023-09-29T23:15:16.031 DEBUG:boto:Method: PUT 2023-09-29T23:15:16.031 DEBUG:boto:Path: /hcnphz-66/obj1 2023-09-29T23:15:16.031 DEBUG:boto:Data: 2023-09-29T23:15:16.031 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:15:16.031 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:15:16.031 DEBUG:boto:Port: 8000 2023-09-29T23:15:16.031 DEBUG:boto:Params: {} 2023-09-29T23:15:16.031 DEBUG:boto:Token: None 2023-09-29T23:15:16.032 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:15:16 GMT /hcnphz-66/obj1 2023-09-29T23:15:16.032 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:toS9Pm3kwkPNtC7dVI3QyO+m6H8= 2023-09-29T23:15:16.032 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:toS9Pm3kwkPNtC7dVI3QyO+m6H8='} 2023-09-29T23:15:16.046 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003210490ace1674d4-0065175a84-2217-a1'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:15:16.046 INFO:rgw_multi.tests:create bucket zone=a2 name=hcnphz-67 2023-09-29T23:15:16.046 DEBUG:boto:path=/hcnphz-67/ 2023-09-29T23:15:16.046 DEBUG:boto:auth_path=/hcnphz-67/ 2023-09-29T23:15:16.046 DEBUG:boto:Method: PUT 2023-09-29T23:15:16.047 DEBUG:boto:Path: /hcnphz-67/ 2023-09-29T23:15:16.047 DEBUG:boto:Data: 2023-09-29T23:15:16.047 DEBUG:boto:Headers: {} 2023-09-29T23:15:16.047 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:15:16.047 DEBUG:boto:Port: 8001 2023-09-29T23:15:16.047 DEBUG:boto:Params: {} 2023-09-29T23:15:16.047 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:15:16.048 DEBUG:boto:Token: None 2023-09-29T23:15:16.048 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:15:16 GMT /hcnphz-67/ 2023-09-29T23:15:16.048 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QRvCHnh35mrMHpb5lywU9ccrFag= 2023-09-29T23:15:16.048 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QRvCHnh35mrMHpb5lywU9ccrFag=', 'Content-Length': '0'} 2023-09-29T23:15:16.179 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000b5963b7d5de1323a-0065175a84-221d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:15:16.180 DEBUG:boto:path=/hcnphz-67/ 2023-09-29T23:15:16.180 DEBUG:boto:auth_path=/hcnphz-67/ 2023-09-29T23:15:16.180 DEBUG:boto:Method: HEAD 2023-09-29T23:15:16.180 DEBUG:boto:Path: /hcnphz-67/ 2023-09-29T23:15:16.180 DEBUG:boto:Data: 2023-09-29T23:15:16.180 DEBUG:boto:Headers: {} 2023-09-29T23:15:16.180 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:15:16.181 DEBUG:boto:Port: 8001 2023-09-29T23:15:16.181 DEBUG:boto:Params: {} 2023-09-29T23:15:16.181 DEBUG:boto:Token: None 2023-09-29T23:15:16.181 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:15:16 GMT /hcnphz-67/ 2023-09-29T23:15:16.181 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/XoqtsCzRqKYmAkEfXtpKWJ4kBg= 2023-09-29T23:15:16.181 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/XoqtsCzRqKYmAkEfXtpKWJ4kBg=', 'Content-Length': '0'} 2023-09-29T23:15:16.185 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ea2e93878711df8-0065175a84-221d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:15:16.186 DEBUG:boto:path=/hcnphz-67/obj2 2023-09-29T23:15:16.186 DEBUG:boto:auth_path=/hcnphz-67/obj2 2023-09-29T23:15:16.186 DEBUG:boto:Method: PUT 2023-09-29T23:15:16.186 DEBUG:boto:Path: /hcnphz-67/obj2 2023-09-29T23:15:16.186 DEBUG:boto:Data: 2023-09-29T23:15:16.186 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:15:16.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:15:16.187 DEBUG:boto:Port: 8001 2023-09-29T23:15:16.187 DEBUG:boto:Params: {} 2023-09-29T23:15:16.187 DEBUG:boto:Token: None 2023-09-29T23:15:16.187 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:15:16 GMT /hcnphz-67/obj2 2023-09-29T23:15:16.187 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SXSD8vSZVrMHHyyBO4o+V1CW1dw= 2023-09-29T23:15:16.187 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SXSD8vSZVrMHHyyBO4o+V1CW1dw='} 2023-09-29T23:15:16.201 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015de03fa6e36adfd-0065175a84-221d-a2'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:15:16.201 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:16.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:16.201 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:16.254 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:16.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:16.316 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:15:16.340 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:15:16.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.340 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:15:16.160461Z" 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:16.341 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:15:16.342 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.342 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.342 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:16.342 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:15:16.342 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:16.342 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:15:16.343 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:16.343 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:15:16.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:16.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:16.344 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:16.389 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:16.389 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:16.437 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:16.437+0000 7f9ea52fff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:16.438 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:16.441+0000 7f9ea52fff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:16.460 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:16.460 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:16.461 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:16.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:16.463 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:16.464 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:16.465 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:16.466 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:16.467 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:16Z" 2023-09-29T23:15:16.467 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:16.468 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:16Z" } 2023-09-29T23:15:16.468 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:16.468 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:16.468 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:16.468 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:17.140 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-09-29T23:15:17.141+0000 7f4525c2c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:15:21.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:21.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:21.470 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:21.516 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:21.517 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:21.576 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:21.576+0000 7fa085659f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:21.578 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:21.580+0000 7fa085659f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:21.605 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:21.605 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:21.605 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:21.606 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:21.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:21.608 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:21.609 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:21.610 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:21.611 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:21.612 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:21Z" 2023-09-29T23:15:21.612 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:21.613 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:21Z" } 2023-09-29T23:15:21.613 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:21.613 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:21.613 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:21.613 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:26.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:26.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:26.615 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:26.660 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:26.660 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:26.717 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:26.716+0000 7f378444ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:26.719 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:26.720+0000 7f378444ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:26.746 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:26.746 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:26.746 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:26.746 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:26.747 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:26.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:26.749 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:26.750 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:26.751 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:26.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:26Z" 2023-09-29T23:15:26.753 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:26.754 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:26Z" } 2023-09-29T23:15:26.755 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:26.755 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:26.755 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:26.755 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:31.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:31.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:31.757 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:31.801 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:31.802 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:31.858 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:31.856+0000 7f06e6c9bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:31.859 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:31.860+0000 7f06e6c9bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:31.890 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:31.890 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:31.891 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:31.892 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:31.893 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:31.894 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:31.895 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:31.896 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:31.897 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:31Z" 2023-09-29T23:15:31.897 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:31.898 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:31Z" } 2023-09-29T23:15:31.898 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:31.898 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:31.898 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:31.899 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:36.900 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:36.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:36.900 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:36.946 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:36.947 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:37.003 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:37.004+0000 7fb3e6e11f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:37.004 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:37.004+0000 7fb3e6e11f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:37.030 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:37Z" 2023-09-29T23:15:37.037 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:37.038 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:37Z" } 2023-09-29T23:15:37.038 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:37.038 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:37.039 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:37.039 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:42.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:42.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:42.040 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:42.085 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:42.085 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:42.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:42.140+0000 7f5cd6dccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:42.142 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:42.144+0000 7f5cd6dccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:42.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:42.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:42.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:42.172 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:42.173 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:42.174 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:42Z" 2023-09-29T23:15:42.175 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:42.176 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:42Z" } 2023-09-29T23:15:42.176 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:42.177 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:42.177 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:42.177 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:47.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:47.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:47.178 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:47.224 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:47.225 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:47.280 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:47.280+0000 7f9acaaa1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:47.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:47.284+0000 7f9acaaa1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:47.309 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:47.309 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:47.310 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:47.311 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:47.312 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:47.313 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:47.314 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:47.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:47Z" 2023-09-29T23:15:47.316 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:47.317 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:47Z" } 2023-09-29T23:15:47.317 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:47.318 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:47.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:47.318 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:52.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:52.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:52.320 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:52.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:52.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:52.423 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:52.424+0000 7f3cf48f0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:52.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:52.424+0000 7f3cf48f0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:52.450 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:52.450 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:52.451 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:52.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:52.453 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:52.454 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:52.455 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:52.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:52.457 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:52.457 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:52.457 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:52.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:52.457 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:52Z" 2023-09-29T23:15:52.457 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:52.458 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:52Z" } 2023-09-29T23:15:52.458 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:52.458 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:52.459 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:52.459 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:15:57.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:57.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:15:57.460 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:15:57.505 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:15:57.505 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:15:57.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:57.564+0000 7faec6197f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:57.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:15:57.568+0000 7faec6197f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:15:57.594 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:57.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:57.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:15:57.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:57.598 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:15:57.599 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:15:57.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:15:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:15:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:15:57.601 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:15:57Z" 2023-09-29T23:15:57.601 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:15:57.602 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:15:57Z" } 2023-09-29T23:15:57.602 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:15:57.602 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:15:57.602 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:15:57.602 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:02.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:02.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:02.604 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:02.649 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:02.649 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:02.705 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:02.703+0000 7f4912670f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:02.707 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:02.707+0000 7f4912670f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:02.734 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:02.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:02.734 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:02.734 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:02.735 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:02.736 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:02.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:02.738 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:02.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:02.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:02.741 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:02Z" 2023-09-29T23:16:02.741 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:02.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:02Z" } 2023-09-29T23:16:02.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:02.742 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:02.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:02.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:07.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:07.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:07.744 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:07.791 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:07.844 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:07.843+0000 7f46dbbaef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:07.845 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:07.843+0000 7f46dbbaef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:07.873 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:07.873 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:07.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:07.875 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:07.876 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:07.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:07.878 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:07Z" 2023-09-29T23:16:07.879 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:07.880 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:07Z" } 2023-09-29T23:16:07.881 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:07.881 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:07.881 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:07.881 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:12.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:12.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:12.882 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:12.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:12.927 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:12.988 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:12.987+0000 7f27f0cbef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:12.989 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:12.987+0000 7f27f0cbef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:13.018 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:13.018 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:13.019 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:13.020 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:13.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:13.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:13.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:13.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:13.025 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:12Z" 2023-09-29T23:16:13.025 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:13.026 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:12Z" } 2023-09-29T23:16:13.026 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:13.026 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:13.026 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:13.026 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:18.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:18.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:18.028 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:18.074 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:18.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:18.134 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:18.135+0000 7f7ef2665f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:18.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:18.135+0000 7f7ef2665f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:18.161 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:18.161 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:18.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:18.163 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:18.164 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:18.165 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:18.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:18.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:18.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:18.169 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:18Z" 2023-09-29T23:16:18.169 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:18.170 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:18Z" } 2023-09-29T23:16:18.170 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:18.170 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:18.170 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:18.170 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:23.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:23.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:23.172 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:23.219 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:23.276 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:23.275+0000 7ff3a0e37f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:23.277 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:23.275+0000 7ff3a0e37f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:23.304 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:23.304 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:23.304 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:23.304 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:23.304 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:23.304 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:23.305 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:23.306 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:23.307 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:23.309 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:23.310 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:23.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:23.312 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:23.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:23.312 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:23Z" 2023-09-29T23:16:23.312 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:23.313 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:23Z" } 2023-09-29T23:16:23.313 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:23.313 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:23.313 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:23.313 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:28.315 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:28.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:28.315 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:28.362 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:28.363 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:28.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:28.419+0000 7fc9f73b3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:28.421 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:28.419+0000 7fc9f73b3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:28.448 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:28.448 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:28.448 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:28.448 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:28.448 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:28.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:28.450 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:28.451 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:28.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:28.453 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:28.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:28Z" 2023-09-29T23:16:28.455 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:28.456 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:28Z" } 2023-09-29T23:16:28.457 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:28.457 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:28.457 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:28.457 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:33.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:33.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:33.459 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:33.506 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:33.506 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:33.564 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:33.563+0000 7fdf1186ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:33.565 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:33.563+0000 7fdf1186ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:33.592 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:33.592 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:33.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:33.594 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:33.595 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:33.596 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:33.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:33.598 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:33Z" 2023-09-29T23:16:33.599 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:33.600 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:33Z" } 2023-09-29T23:16:33.600 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:33.601 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:33.601 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:33.601 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:38.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:38.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:38.602 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:38.648 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:38.648 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:38.706 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:38.706+0000 7f540134ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:38.708 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:38.706+0000 7f540134ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:38.734 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:38.735 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:38.736 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:38.737 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:38.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:38.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:38.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:38.741 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:38Z" 2023-09-29T23:16:38.741 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:38.742 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:38Z" } 2023-09-29T23:16:38.742 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:38.742 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:38.742 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:38.742 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:43.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:43.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:43.744 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:43.789 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:43.789 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:43.848 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:43.846+0000 7f8576554f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:43.850 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:43.850+0000 7f8576554f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:43.879 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:43.880 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:43.881 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:43.882 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:43.883 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:43.884 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:43Z" 2023-09-29T23:16:43.885 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:43.886 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:43Z" } 2023-09-29T23:16:43.886 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:43.886 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:43.887 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:43.887 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:48.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:48.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:48.888 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:48.933 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:48.933 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:48.997 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:48.994+0000 7f29b8d2ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:48.998 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:48.998+0000 7f29b8d2ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:49.025 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:49.025 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:49.025 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:49.026 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:49.027 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:49.028 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:49.029 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:49.030 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:49.031 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:49.032 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:49.033 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:49.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:49.033 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:49Z" 2023-09-29T23:16:49.033 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:49.034 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:49Z" } 2023-09-29T23:16:49.034 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:49.034 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:49.035 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:49.035 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:54.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:54.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:54.036 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:54.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:54.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:54.140 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:54.138+0000 7f3080855f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:54.141 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:54.138+0000 7f3080855f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:54.176 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:54.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:54.178 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:54.179 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:54.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:54.181 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:54.182 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:54.183 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:54Z" 2023-09-29T23:16:54.183 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:54.184 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:54Z" } 2023-09-29T23:16:54.184 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:54.184 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:54.184 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:54.184 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:16:59.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:59.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:16:59.185 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:16:59.232 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:16:59.232 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:16:59.288 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:59.286+0000 7f6004011f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:59.289 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:16:59.286+0000 7f6004011f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:16:59.315 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:16:59.315 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:16:59.315 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:16:59.315 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:59.316 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:59.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:16:59.318 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:59.319 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:16:59.320 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:59.321 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:16:59.322 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:16:59.322 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:16:59.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:16:59.322 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:16:59Z" 2023-09-29T23:16:59.322 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:16:59.323 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:16:59Z" } 2023-09-29T23:16:59.323 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:16:59.323 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:16:59.323 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:16:59.324 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:17:04.324 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:04.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:04.325 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:04.372 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:04.372 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:04.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:04.422+0000 7f38dabf3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:04.426 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:04.426+0000 7f38dabf3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:04.452 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:04.452 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:04.452 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:04.452 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:04.452 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:04.452 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:04.453 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:04.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:04.455 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:04.456 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:04.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:04.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:04.459 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:04.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:04.460 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:04Z" 2023-09-29T23:17:04.460 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:04.461 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:04Z" } 2023-09-29T23:17:04.461 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:04.461 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:04.461 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:17:04.461 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:17:09.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:09.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:09.463 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:09.510 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:09.510 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:09.568 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:09.566+0000 7f7490621f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:09.570 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:09.570+0000 7f7490621f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:09.597 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:09.598 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:09.599 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:09.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:09.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:09.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:09.603 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:09.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:09.605 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:09Z" 2023-09-29T23:17:09.605 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:09.606 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:09Z" } 2023-09-29T23:17:09.606 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:09.606 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:09.606 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:17:09.606 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:17:14.607 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:14.608 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:14.608 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:14.655 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:14.656 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:14.712 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:14.710+0000 7f6a53d5bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:14.713 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:14.710+0000 7f6a53d5bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:14.739 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:14.739 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:14.740 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:14.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:14.742 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:14.743 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:14.744 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:14.745 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:14Z" 2023-09-29T23:17:14.746 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:14.747 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:14Z" } 2023-09-29T23:17:14.747 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:14.748 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:14.748 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:17:14.748 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:17:17.172 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:17:17.169+0000 7f87f9cdf700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T23:17:17.383 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:17:17.379+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:17:19.749 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:19.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:19.749 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:19.794 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:19.794 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:19.855 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:19.853+0000 7f4bb9237f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:19.857 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:19.853+0000 7f4bb9237f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:19.882 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:19.882 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:19.882 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:19.882 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:19.882 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:19.882 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:19.883 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:17:19.884 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:19.885 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:19.886 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:19.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:19Z" 2023-09-29T23:17:19.888 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:19.889 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:19Z" } 2023-09-29T23:17:19.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:19.889 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:19.889 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:17:19.890 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:17:24.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:24.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:24.890 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:24.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:24.936 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:24.994 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:24.993+0000 7f4cbf1c0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:24.995 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:24.993+0000 7f4cbf1c0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:25.020 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:25.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:25.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:25.022 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:25.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:25.024 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:25.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:25.026 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:25Z" 2023-09-29T23:17:25.027 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:25.028 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029292.567201_712.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:14:52.567201Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:25Z" } 2023-09-29T23:17:25.029 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:25.029 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:25.029 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:17:25.029 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:17:30.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.031 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:30.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:30.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:30.137+0000 7fc8a34c9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:30.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:30.137+0000 7fc8a34c9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:30.163 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:30.163 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:30.164 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.160461Z", 2023-09-29T23:17:30.166 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.169 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.170 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:17:30.170 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:30.171 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029316.160461_736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:16.160461Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:30Z" } 2023-09-29T23:17:30.171 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029316.160461_736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.160461Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:30.171 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:30.171 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:30.172 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:17:30.172 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.172 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:30.217 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.217 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:30.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.313 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:17:30.313 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:30.314 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:17:30.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.314 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:30.404 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.405 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:30.491 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:30.491 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:30.492 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.493 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.494 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:30.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:17:30.497 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:30.498 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:30Z" } 2023-09-29T23:17:30.499 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:30.499 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:30.499 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:30.499 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:17:30.499 DEBUG:boto:path=/hcnphz-66/ 2023-09-29T23:17:30.499 DEBUG:boto:auth_path=/hcnphz-66/ 2023-09-29T23:17:30.499 DEBUG:boto:Method: HEAD 2023-09-29T23:17:30.500 DEBUG:boto:Path: /hcnphz-66/ 2023-09-29T23:17:30.500 DEBUG:boto:Data: 2023-09-29T23:17:30.500 DEBUG:boto:Headers: {} 2023-09-29T23:17:30.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:17:30.500 DEBUG:boto:Port: 8001 2023-09-29T23:17:30.500 DEBUG:boto:Params: {} 2023-09-29T23:17:30.500 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:17:30.501 DEBUG:boto:Token: None 2023-09-29T23:17:30.501 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:17:30 GMT /hcnphz-66/ 2023-09-29T23:17:30.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:K3LO7PDMrdsjmd1E0blhb1sx6hI= 2023-09-29T23:17:30.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K3LO7PDMrdsjmd1E0blhb1sx6hI=', 'Content-Length': '0'} 2023-09-29T23:17:30.506 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061589254f67d3604-0065175b0a-221d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:30.506 DEBUG:boto:path=/hcnphz-66/obj1 2023-09-29T23:17:30.507 DEBUG:boto:auth_path=/hcnphz-66/obj1 2023-09-29T23:17:30.507 DEBUG:boto:Method: HEAD 2023-09-29T23:17:30.507 DEBUG:boto:Path: /hcnphz-66/obj1 2023-09-29T23:17:30.507 DEBUG:boto:Data: 2023-09-29T23:17:30.507 DEBUG:boto:Headers: {} 2023-09-29T23:17:30.507 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:17:30.507 DEBUG:boto:Port: 8001 2023-09-29T23:17:30.507 DEBUG:boto:Params: {} 2023-09-29T23:17:30.507 DEBUG:boto:Token: None 2023-09-29T23:17:30.508 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:17:30 GMT /hcnphz-66/obj1 2023-09-29T23:17:30.508 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OrOb49b1R+pw/3q4hMndFfGdP9Y= 2023-09-29T23:17:30.508 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OrOb49b1R+pw/3q4hMndFfGdP9Y=', 'Content-Length': '0'} 2023-09-29T23:17:30.510 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000074e5ab68511bc5db-0065175b0a-221d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:17:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:30.512 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_all: ... FAIL 2023-09-29T23:17:30.513 INFO:tasks.rgw_multisite_tests:test_sync_flow_symmetrical_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-09-29T23:17:30.514 INFO:tasks.rgw_multisite_tests:test_sync_flow_directional_zonegroup_select: ... SKIP: test_sync_flow_symmetrical_zonegroup_select skipped. Requires 3 or more zones in master zonegroup. 2023-09-29T23:17:30.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.514 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.559 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:30.621 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:17:30.647 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:17:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.647 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:15:16.160461Z" 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:30.648 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:17:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:17:30.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.649 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:17:30.650 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:30.650 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:17:30.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:30.651 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:30.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.695 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:30.753+0000 7f2603656f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:30.757+0000 7f2603656f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:30.781 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.782 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:30.783 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:30.784 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.160461Z", 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.786 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:30.787 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:30.788 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:17:30.789 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:30.790 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029316.160461_736.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:16.160461Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:30Z" } 2023-09-29T23:17:30.791 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029316.160461_736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.160461Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:30.791 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:30.791 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:17:30.791 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:17:30.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:30.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:30.792 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:30.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.834 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:30.922 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:17:30.922 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:30.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:30.923 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:30.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:30.962 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.053 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:17:31.053 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-68 2023-09-29T23:17:31.054 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:17:31.054 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:17:31.054 DEBUG:boto:Method: PUT 2023-09-29T23:17:31.054 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:17:31.054 DEBUG:boto:Data: 2023-09-29T23:17:31.054 DEBUG:boto:Headers: {} 2023-09-29T23:17:31.055 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:31.055 DEBUG:boto:Port: 8000 2023-09-29T23:17:31.055 DEBUG:boto:Params: {} 2023-09-29T23:17:31.055 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:17:31.055 DEBUG:boto:Token: None 2023-09-29T23:17:31.056 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:17:31 GMT /hcnphz-68/ 2023-09-29T23:17:31.056 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ADcuToPTv3Aoa34neg/TWI/69Fo= 2023-09-29T23:17:31.056 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ADcuToPTv3Aoa34neg/TWI/69Fo=', 'Content-Length': '0'} 2023-09-29T23:17:31.089 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx00000dca03dd0c5f74435-0065175b0b-2217-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:31.089 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-69 2023-09-29T23:17:31.089 DEBUG:boto:path=/hcnphz-69/ 2023-09-29T23:17:31.090 DEBUG:boto:auth_path=/hcnphz-69/ 2023-09-29T23:17:31.090 DEBUG:boto:Method: PUT 2023-09-29T23:17:31.090 DEBUG:boto:Path: /hcnphz-69/ 2023-09-29T23:17:31.090 DEBUG:boto:Data: 2023-09-29T23:17:31.090 DEBUG:boto:Headers: {} 2023-09-29T23:17:31.090 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:31.090 DEBUG:boto:Port: 8000 2023-09-29T23:17:31.090 DEBUG:boto:Params: {} 2023-09-29T23:17:31.091 DEBUG:boto:Token: None 2023-09-29T23:17:31.091 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:17:31 GMT /hcnphz-69/ 2023-09-29T23:17:31.091 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:avg0gpXExj9dSh0pIviRJr6+lrA= 2023-09-29T23:17:31.091 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:31 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:avg0gpXExj9dSh0pIviRJr6+lrA=', 'Content-Length': '0'} 2023-09-29T23:17:31.119 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000021ebe41403e6cfdf-0065175b0b-2217-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:31 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:31.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:31.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:31.120 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:31.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:31.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.220 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:17:31.247 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:17:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:31.247 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:31.119832Z" 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:31.248 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:17:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:17:31.249 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:31.249 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:17:31.250 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:17:31.250 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:17:31.250 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:31.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:31.251 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:31.332 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:31.332 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:31.389+0000 7f71f765cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:31.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:31.393+0000 7f71f765cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:31.418 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:31.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:31.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:31.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:31.422 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:31.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:31.424 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:31.425 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:31Z" 2023-09-29T23:17:31.426 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:31.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:31Z" } 2023-09-29T23:17:31.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:31.427 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:17:31.427 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:17:31.427 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:17:31.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.428 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:31.471 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:31.472 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.564 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:17:31.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.565 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:31.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:31.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.714 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:17:31.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=hcnphz-68', '--dest-bucket=hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--source-bucket=hcnphz-68', '--dest-bucket=hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.714 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --source-bucket=hcnphz-68 --dest-bucket=hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:31.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:31.765 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.852 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-68","zones":["a2","a1"]},"dest":{"bucket":"hcnphz-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:17:31.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.853 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:31.898 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:31.898 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:31.988 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-68","zones":["a2","a1"]},"dest":{"bucket":"hcnphz-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:17:31.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:31.989 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:32.027 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:32.117 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-68","zones":["a2","a1"]},"dest":{"bucket":"hcnphz-68","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:17:32.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:32.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:32.118 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:32.203 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:32.203 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 16, 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:17:32.311 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:17:32.312 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:17:32.313 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:17:32.314 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:17:32.315 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:17:32.316 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.317 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:17:32.318 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:17:32.319 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-68", 2023-09-29T23:17:32.320 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-68", 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:17:32.321 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:17:32.322 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:17:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:17:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:17:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.323 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.324 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:17:32.325 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:17:32.326 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.327 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.328 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:17:32.329 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:17:32.330 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.331 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.332 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:17:32.333 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:17:32.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:17:32.335 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.336 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:17:32.337 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:17:32.338 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:32.339 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:32.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:32.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'info', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:17:32.340 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync info --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:17:32.378 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:32.378 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:32.457 INFO:teuthology.orchestra.run.smithi112.stdout:{"sources":[{"id":"sync-pipe","source":{"zone":"a2","bucket":"hcnphz-68"},"dest":{"zone":"a1","bucket":"hcnphz-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"dests":[{"id":"sync-pipe","source":{"zone":"a1","bucket":"hcnphz-68"},"dest":{"zone":"a2","bucket":"hcnphz-68"},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"hints":{"sources":[],"dests":[]},"resolved-hints-1":{"sources":[],"dests":[]},"resolved-hints":{"sources":[],"dests":[]}} 2023-09-29T23:17:32.458 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:17:32.458 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:17:32.458 DEBUG:boto:Method: HEAD 2023-09-29T23:17:32.458 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:17:32.458 DEBUG:boto:Data: 2023-09-29T23:17:32.459 DEBUG:boto:Headers: {} 2023-09-29T23:17:32.459 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.459 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.459 DEBUG:boto:Params: {} 2023-09-29T23:17:32.459 DEBUG:boto:Token: None 2023-09-29T23:17:32.459 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-68/ 2023-09-29T23:17:32.459 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WLIfPsbpdXRT9J2B+LdTDgRsALM= 2023-09-29T23:17:32.460 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WLIfPsbpdXRT9J2B+LdTDgRsALM=', 'Content-Length': '0'} 2023-09-29T23:17:32.613 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009a250c76041865c3-0065175b0c-237b-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.614 DEBUG:boto:path=/hcnphz-68/obj1 2023-09-29T23:17:32.614 DEBUG:boto:auth_path=/hcnphz-68/obj1 2023-09-29T23:17:32.614 DEBUG:boto:Method: PUT 2023-09-29T23:17:32.614 DEBUG:boto:Path: /hcnphz-68/obj1 2023-09-29T23:17:32.614 DEBUG:boto:Data: 2023-09-29T23:17:32.614 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:17:32.615 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.615 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.615 DEBUG:boto:Params: {} 2023-09-29T23:17:32.615 DEBUG:boto:Token: None 2023-09-29T23:17:32.615 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-68/obj1 2023-09-29T23:17:32.615 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pbpbX2NVH6Y8BHtdkSo9xTz9tic= 2023-09-29T23:17:32.615 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pbpbX2NVH6Y8BHtdkSo9xTz9tic='} 2023-09-29T23:17:32.637 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005a37d28e24e32bd4-0065175b0c-237b-a1'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.637 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:17:32.637 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:17:32.638 DEBUG:boto:Method: HEAD 2023-09-29T23:17:32.638 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:17:32.638 DEBUG:boto:Data: 2023-09-29T23:17:32.638 DEBUG:boto:Headers: {} 2023-09-29T23:17:32.638 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.638 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.639 DEBUG:boto:Params: {} 2023-09-29T23:17:32.639 DEBUG:boto:Token: None 2023-09-29T23:17:32.639 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-68/ 2023-09-29T23:17:32.639 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WLIfPsbpdXRT9J2B+LdTDgRsALM= 2023-09-29T23:17:32.639 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WLIfPsbpdXRT9J2B+LdTDgRsALM=', 'Content-Length': '0'} 2023-09-29T23:17:32.643 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000513d4edb5b1477e0-0065175b0c-237b-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.643 DEBUG:boto:path=/hcnphz-68/obj2 2023-09-29T23:17:32.643 DEBUG:boto:auth_path=/hcnphz-68/obj2 2023-09-29T23:17:32.643 DEBUG:boto:Method: PUT 2023-09-29T23:17:32.643 DEBUG:boto:Path: /hcnphz-68/obj2 2023-09-29T23:17:32.643 DEBUG:boto:Data: 2023-09-29T23:17:32.644 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:17:32.644 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.644 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.644 DEBUG:boto:Params: {} 2023-09-29T23:17:32.644 DEBUG:boto:Token: None 2023-09-29T23:17:32.644 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-68/obj2 2023-09-29T23:17:32.644 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SYdbji5GckXAn/PdHKxK0ozqVB4= 2023-09-29T23:17:32.644 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SYdbji5GckXAn/PdHKxK0ozqVB4='} 2023-09-29T23:17:32.690 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007dfb082b5fc0688a-0065175b0c-237b-a1'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.691 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:17:32.691 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:17:32.691 DEBUG:boto:Method: HEAD 2023-09-29T23:17:32.691 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:17:32.691 DEBUG:boto:Data: 2023-09-29T23:17:32.691 DEBUG:boto:Headers: {} 2023-09-29T23:17:32.692 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.692 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.692 DEBUG:boto:Params: {} 2023-09-29T23:17:32.692 DEBUG:boto:Token: None 2023-09-29T23:17:32.692 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-68/ 2023-09-29T23:17:32.692 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WLIfPsbpdXRT9J2B+LdTDgRsALM= 2023-09-29T23:17:32.692 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WLIfPsbpdXRT9J2B+LdTDgRsALM=', 'Content-Length': '0'} 2023-09-29T23:17:32.697 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b034269f9efdb2f-0065175b0c-237b-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.697 DEBUG:boto:path=/hcnphz-68/obj3 2023-09-29T23:17:32.697 DEBUG:boto:auth_path=/hcnphz-68/obj3 2023-09-29T23:17:32.697 DEBUG:boto:Method: PUT 2023-09-29T23:17:32.698 DEBUG:boto:Path: /hcnphz-68/obj3 2023-09-29T23:17:32.698 DEBUG:boto:Data: 2023-09-29T23:17:32.698 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:17:32.698 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.698 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.698 DEBUG:boto:Params: {} 2023-09-29T23:17:32.698 DEBUG:boto:Token: None 2023-09-29T23:17:32.698 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-68/obj3 2023-09-29T23:17:32.699 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pwoK7+wwhH4pfpqNvtei8OQD4Mo= 2023-09-29T23:17:32.699 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pwoK7+wwhH4pfpqNvtei8OQD4Mo='} 2023-09-29T23:17:32.710 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000967a284101eb4bae-0065175b0c-237b-a1'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.710 DEBUG:boto:path=/hcnphz-69/ 2023-09-29T23:17:32.711 DEBUG:boto:auth_path=/hcnphz-69/ 2023-09-29T23:17:32.711 DEBUG:boto:Method: HEAD 2023-09-29T23:17:32.711 DEBUG:boto:Path: /hcnphz-69/ 2023-09-29T23:17:32.711 DEBUG:boto:Data: 2023-09-29T23:17:32.711 DEBUG:boto:Headers: {} 2023-09-29T23:17:32.711 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.711 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.711 DEBUG:boto:Params: {} 2023-09-29T23:17:32.712 DEBUG:boto:Token: None 2023-09-29T23:17:32.712 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-69/ 2023-09-29T23:17:32.712 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X5QDknpeFDRig6DlPVDD20Kq7PI= 2023-09-29T23:17:32.712 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X5QDknpeFDRig6DlPVDD20Kq7PI=', 'Content-Length': '0'} 2023-09-29T23:17:32.718 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f4c7d57a0404f07b-0065175b0c-237b-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.719 DEBUG:boto:path=/hcnphz-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-29T23:17:32.719 DEBUG:boto:auth_path=/hcnphz-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-29T23:17:32.719 DEBUG:boto:Method: PUT 2023-09-29T23:17:32.719 DEBUG:boto:Path: /hcnphz-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-29T23:17:32.719 DEBUG:boto:Data: 2023-09-29T23:17:32.719 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:17:32.719 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:17:32.719 DEBUG:boto:Port: 8000 2023-09-29T23:17:32.720 DEBUG:boto:Params: {} 2023-09-29T23:17:32.720 DEBUG:boto:Token: None 2023-09-29T23:17:32.720 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:17:32 GMT /hcnphz-69/%5B%27obj1%27%2C%20%27obj2%27%2C%20%27obj3%27%5D 2023-09-29T23:17:32.720 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xNbQpMAgGULVy6M97+9SjVF3Rh0= 2023-09-29T23:17:32.720 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:17:32 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xNbQpMAgGULVy6M97+9SjVF3Rh0='} 2023-09-29T23:17:32.766 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019dc3efaade501bb-0065175b0c-237b-a1'), ('Date', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:17:32.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:32.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:32.767 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:32.821 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:32.821 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:32.906 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:31.119832Z" 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:32.941 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:17:32.942 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:32.942 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:17:32.943 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:17:32.943 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:17:32.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:32.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:32.943 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:32.995 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:32.995 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:33.045 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:33.045+0000 7fe74c0a3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:33.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:17:33.049+0000 7fe74c0a3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:17:33.068 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:33.068 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:33.068 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:33.069 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.070 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:33.071 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.074 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.075 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:33Z" 2023-09-29T23:17:33.076 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:33.077 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:17:33Z" } 2023-09-29T23:17:33.077 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:17:33.077 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:17:33.077 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:17:33.077 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:17:33.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:33.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:33.078 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:33.123 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:33.124 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051249", 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.711439Z" 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.202 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:33.203 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.204 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:17:33.204 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:33.205 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:17:33.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:33.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:33.205 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:33.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:33.293 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:33.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:33.382 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.383 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:33.384 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:33.385 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:33.386 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:33Z" 2023-09-29T23:17:33.387 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:33.388 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:33Z" } 2023-09-29T23:17:33.388 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:33.389 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:33.389 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:33.389 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:17:38.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:38.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:38.390 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:38.438 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:38.528 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:38.529 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:38.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:38.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:38.532 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:38.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:38.534 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:38Z" 2023-09-29T23:17:38.535 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:38.535 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:38Z" } 2023-09-29T23:17:38.536 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:38.536 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:38.536 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:38.536 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:17:43.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:43.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:43.537 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:43.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:43.584 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:43.673 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:43.673 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:43.674 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:43.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:43.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:43.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:43.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:43.679 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:43Z" 2023-09-29T23:17:43.680 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:43.680 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:43Z" } 2023-09-29T23:17:43.681 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:43.681 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:43.681 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:43.681 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:17:48.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:48.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:48.683 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:48.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:48.732 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:48.818 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:48.818 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:48.819 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:48.820 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:48.821 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:48.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:48.823 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:48.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:48.825 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:48Z" 2023-09-29T23:17:48.826 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:48.827 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:48Z" } 2023-09-29T23:17:48.827 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:48.828 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:48.828 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:48.828 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:17:53.828 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:53.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:53.829 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:53.875 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:53.876 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:53.963 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:53.963 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:53.964 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:53.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:53.966 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:53.967 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:53.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:53.969 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:53.970 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:53.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:53.971 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:53Z" 2023-09-29T23:17:53.971 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:53.972 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:53Z" } 2023-09-29T23:17:53.972 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:53.972 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:53.972 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:53.972 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:17:58.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:58.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:17:58.974 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:17:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:17:59.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:17:59.108 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:17:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:17:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:17:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:17:59.108 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:59.109 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:59.110 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:59.111 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:59.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:17:59.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:17:59Z" 2023-09-29T23:17:59.114 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:17:59.115 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:17:59Z" } 2023-09-29T23:17:59.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:17:59.116 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:59.116 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:17:59.116 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:04.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:04.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:04.117 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:04.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:04.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:04.254 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:04.254 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:04.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:04.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:04.257 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:04.258 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:04.259 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:04.260 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:04Z" 2023-09-29T23:18:04.260 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:04.261 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:04Z" } 2023-09-29T23:18:04.261 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:04.261 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:04.262 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:04.262 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:09.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:09.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:09.263 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:09.313 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:09.313 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:09.400 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:09.400 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:09.400 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:09.400 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:09.400 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:09.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:09.402 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:09.403 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:09.404 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:09.405 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:09Z" 2023-09-29T23:18:09.406 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:09.407 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:09Z" } 2023-09-29T23:18:09.408 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:09.408 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:09.408 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:09.408 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:14.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:14.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:14.410 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:14.458 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:14.459 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:14.544 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:14.545 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:14.546 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:14.547 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:14.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:14.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:14.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:14.551 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:14Z" 2023-09-29T23:18:14.551 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:14.551 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:14Z" } 2023-09-29T23:18:14.552 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:14.552 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:14.552 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:14.552 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:19.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:19.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:19.554 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:19.602 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:19.602 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:19.681 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:19.681 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:19.681 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:19.681 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:19.681 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:19.682 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:19.683 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:19.684 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:19.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:19.686 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:19Z" 2023-09-29T23:18:19.687 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:19.688 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:19Z" } 2023-09-29T23:18:19.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:19.689 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:19.689 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:19.689 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:24.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:24.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:24.691 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:24.737 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:24.823 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:24.824 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:24.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:24.826 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:24.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:24.828 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:24.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:24.830 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:24Z" 2023-09-29T23:18:24.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:24.831 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:24Z" } 2023-09-29T23:18:24.831 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:24.831 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:24.831 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:24.831 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:29.832 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:29.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:29.833 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:29.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:29.885 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:29.972 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:29.973 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:29.974 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:29.975 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:29.976 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:29.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:29.978 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:29.979 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:29.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:29.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:29.979 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:29.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:29.979 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:29.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:29.980 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:29.980 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:29.980 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:29Z" 2023-09-29T23:18:29.980 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:29.981 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:29Z" } 2023-09-29T23:18:29.981 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:29.982 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:29.982 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:29.982 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:34.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:34.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:34.984 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:35.030 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:35.031 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:35.118 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:35.119 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:35.120 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:35.121 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:35.122 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:35.123 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:35.124 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:35.125 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:35Z" 2023-09-29T23:18:35.125 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:35.125 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:35Z" } 2023-09-29T23:18:35.126 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:35.126 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:35.126 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:35.126 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:40.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:40.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:40.128 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:40.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:40.175 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:40.259 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:40.259 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:40.259 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:40.260 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:40.261 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:40.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:40.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:40.264 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:40Z" 2023-09-29T23:18:40.265 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:40.267 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:40Z" } 2023-09-29T23:18:40.267 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:40.267 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:40.267 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:40.267 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:45.269 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:45.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:45.269 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:45.317 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:45.318 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:45.410 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:45.411 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:45.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:45.413 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:45.414 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:45.415 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:45.416 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:45Z" 2023-09-29T23:18:45.417 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:45.418 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:45Z" } 2023-09-29T23:18:45.418 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:45.418 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:45.419 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:45.419 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:50.419 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:50.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:50.420 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:50.466 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:50.467 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:50.553 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:50.554 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:50.555 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:50.556 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:50.557 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:50.558 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:50.559 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:50.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:50.560 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:50Z" 2023-09-29T23:18:50.560 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:50.561 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:50Z" } 2023-09-29T23:18:50.561 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:50.561 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:50.561 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:50.561 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:18:55.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:55.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:18:55.563 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:18:55.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:18:55.612 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:18:55.702 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:18:55.702 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:55.703 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:55.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:55.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:55.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:18:55.707 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:18:55.708 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:18:55.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:18:55.709 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:18:55Z" 2023-09-29T23:18:55.709 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:18:55.710 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:18:55Z" } 2023-09-29T23:18:55.710 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:18:55.710 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:55.710 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:18:55.711 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:00.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:00.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:00.712 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:00.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:00.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:00.849 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:00.850 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:00.851 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:00.852 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:00.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:00.854 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:00Z" 2023-09-29T23:19:00.854 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:00.855 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:00Z" } 2023-09-29T23:19:00.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:00.855 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:00.855 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:00.856 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:05.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:05.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:05.857 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:05.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:05.997 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:05.997 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:05.997 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:05.997 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:05.997 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:05.998 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:05.999 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:06.000 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:06.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:06.002 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:05Z" 2023-09-29T23:19:06.003 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:06.004 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:05Z" } 2023-09-29T23:19:06.005 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:06.005 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:06.005 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:06.005 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:11.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:11.006 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:11.006 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:11.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:11.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:11.146 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:11.146 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:11.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:11.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:11.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:11.150 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:11.151 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:11Z" 2023-09-29T23:19:11.152 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:11.153 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:11Z" } 2023-09-29T23:19:11.154 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:11.154 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:11.154 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:11.154 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:16.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:16.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:16.155 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:16.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:16.202 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:16.293 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:16.293 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:16.294 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:16.295 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:16.296 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:16.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:16.298 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:16Z" 2023-09-29T23:19:16.299 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:16.300 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:16Z" } 2023-09-29T23:19:16.301 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:16.301 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:16.301 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:16.301 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:21.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:21.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:21.302 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:21.346 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:21.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:21.446 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:21.447 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:21.448 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:21.449 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:21.450 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:21.451 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:21Z" 2023-09-29T23:19:21.451 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:21.452 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:21Z" } 2023-09-29T23:19:21.452 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:21.452 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:21.452 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:21.452 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:26.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:26.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:26.454 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:26.503 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:26.503 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:26.590 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:26.590 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:26.591 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:26.592 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:26.593 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:26.594 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:26.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:26.596 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:26Z" 2023-09-29T23:19:26.596 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:26.597 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:26Z" } 2023-09-29T23:19:26.597 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:26.598 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:26.598 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:26.598 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:31.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:31.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:31.599 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:31.645 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:31.645 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:31.731 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:31.731 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:31.732 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:31.733 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:31.734 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:31.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:31.736 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:31.737 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:31Z" 2023-09-29T23:19:31.737 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:31.738 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000050969", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:15:16.048128Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045024", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:08:29.597662Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045517", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:15:19.183068Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:31Z" } 2023-09-29T23:19:31.738 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:31.738 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:31.738 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:31.738 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051249 target=00000000000000000000:00000000000000050969, shard=1 master=00000000000000000000:00000000000000045166 target=00000000000000000000:00000000000000045024, shard=2 master=00000000000000000000:00000000000000045657 target=00000000000000000000:00000000000000045517 2023-09-29T23:19:33.857 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:19:33.852+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:19:36.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:36.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:36.740 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:36.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:36.790 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:36.870 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:36.870 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:36.870 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:36.871 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051249", 2023-09-29T23:19:36.872 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:36.872 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:19:36.872 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:36.872 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.711439Z" 2023-09-29T23:19:36.872 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:19:36.873 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:36.874 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:36.875 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:36.876 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:36.877 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:36.877 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:19:36.877 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:19:36.877 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:36.877 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:36Z" 2023-09-29T23:19:36.877 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:36.878 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051249", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:17:32.711439Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:19:36Z" } 2023-09-29T23:19:36.878 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051249', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.711439Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:19:36.878 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:36.879 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051249', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:19:36.879 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:19:36.879 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:19:36.879 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:19:36.879 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.879 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:19:36.879 DEBUG:boto:Data: 2023-09-29T23:19:36.879 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.880 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.880 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.880 DEBUG:boto:Params: {} 2023-09-29T23:19:36.880 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:19:36.880 DEBUG:boto:Token: None 2023-09-29T23:19:36.880 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/ 2023-09-29T23:19:36.881 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hQTsT5fpVLEUOOd7mBVgEUShqjI= 2023-09-29T23:19:36.881 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hQTsT5fpVLEUOOd7mBVgEUShqjI=', 'Content-Length': '0'} 2023-09-29T23:19:36.890 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f2f2709c49d212e-0065175b88-2375-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.890 DEBUG:boto:path=/hcnphz-68/obj1 2023-09-29T23:19:36.890 DEBUG:boto:auth_path=/hcnphz-68/obj1 2023-09-29T23:19:36.890 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.890 DEBUG:boto:Path: /hcnphz-68/obj1 2023-09-29T23:19:36.891 DEBUG:boto:Data: 2023-09-29T23:19:36.891 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.891 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.891 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.891 DEBUG:boto:Params: {} 2023-09-29T23:19:36.891 DEBUG:boto:Token: None 2023-09-29T23:19:36.891 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/obj1 2023-09-29T23:19:36.891 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mBznQJ1Lj2stLG/bWZA/T9eAO3k= 2023-09-29T23:19:36.892 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mBznQJ1Lj2stLG/bWZA/T9eAO3k=', 'Content-Length': '0'} 2023-09-29T23:19:36.895 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a27765068f383b37-0065175b88-2375-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.895 DEBUG:boto:path=/hcnphz-68/obj1 2023-09-29T23:19:36.895 DEBUG:boto:auth_path=/hcnphz-68/obj1 2023-09-29T23:19:36.895 DEBUG:boto:Method: GET 2023-09-29T23:19:36.895 DEBUG:boto:Path: /hcnphz-68/obj1 2023-09-29T23:19:36.896 DEBUG:boto:Data: 2023-09-29T23:19:36.896 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.896 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.896 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.896 DEBUG:boto:Params: {} 2023-09-29T23:19:36.896 DEBUG:boto:Token: None 2023-09-29T23:19:36.896 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/obj1 2023-09-29T23:19:36.896 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:RGf6eT2RJNvSG1Mt+WY1W/K/ods= 2023-09-29T23:19:36.896 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RGf6eT2RJNvSG1Mt+WY1W/K/ods=', 'Content-Length': '0'} 2023-09-29T23:19:36.899 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005d2359eb833ce6b4-0065175b88-2375-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.899 DEBUG:boto:path=/hcnphz-68/obj2 2023-09-29T23:19:36.900 DEBUG:boto:auth_path=/hcnphz-68/obj2 2023-09-29T23:19:36.900 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.900 DEBUG:boto:Path: /hcnphz-68/obj2 2023-09-29T23:19:36.900 DEBUG:boto:Data: 2023-09-29T23:19:36.900 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.900 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.900 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.900 DEBUG:boto:Params: {} 2023-09-29T23:19:36.900 DEBUG:boto:Token: None 2023-09-29T23:19:36.901 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/obj2 2023-09-29T23:19:36.901 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:asSYBFJlMGPFsTI82xzDW0W81OE= 2023-09-29T23:19:36.901 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:asSYBFJlMGPFsTI82xzDW0W81OE=', 'Content-Length': '0'} 2023-09-29T23:19:36.903 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000005a742cd4988757f3-0065175b88-2375-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.903 DEBUG:boto:path=/hcnphz-68/obj2 2023-09-29T23:19:36.903 DEBUG:boto:auth_path=/hcnphz-68/obj2 2023-09-29T23:19:36.903 DEBUG:boto:Method: GET 2023-09-29T23:19:36.904 DEBUG:boto:Path: /hcnphz-68/obj2 2023-09-29T23:19:36.904 DEBUG:boto:Data: 2023-09-29T23:19:36.904 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.904 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.904 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.904 DEBUG:boto:Params: {} 2023-09-29T23:19:36.904 DEBUG:boto:Token: None 2023-09-29T23:19:36.904 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/obj2 2023-09-29T23:19:36.904 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vBqQ/D0FCxKE0C1WX1HPgi2sHc0= 2023-09-29T23:19:36.905 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vBqQ/D0FCxKE0C1WX1HPgi2sHc0=', 'Content-Length': '0'} 2023-09-29T23:19:36.907 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b190f7b702ad0339-0065175b88-2375-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.950 DEBUG:boto:path=/hcnphz-68/obj3 2023-09-29T23:19:36.951 DEBUG:boto:auth_path=/hcnphz-68/obj3 2023-09-29T23:19:36.951 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.951 DEBUG:boto:Path: /hcnphz-68/obj3 2023-09-29T23:19:36.951 DEBUG:boto:Data: 2023-09-29T23:19:36.951 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.951 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.951 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.951 DEBUG:boto:Params: {} 2023-09-29T23:19:36.951 DEBUG:boto:Token: None 2023-09-29T23:19:36.952 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/obj3 2023-09-29T23:19:36.952 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:LhjdAnb1IiQYYTy4sF45DQszz4I= 2023-09-29T23:19:36.952 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LhjdAnb1IiQYYTy4sF45DQszz4I=', 'Content-Length': '0'} 2023-09-29T23:19:36.955 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eea39b54b3caead7-0065175b88-2375-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.955 DEBUG:boto:path=/hcnphz-68/obj3 2023-09-29T23:19:36.955 DEBUG:boto:auth_path=/hcnphz-68/obj3 2023-09-29T23:19:36.955 DEBUG:boto:Method: GET 2023-09-29T23:19:36.955 DEBUG:boto:Path: /hcnphz-68/obj3 2023-09-29T23:19:36.955 DEBUG:boto:Data: 2023-09-29T23:19:36.955 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.956 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.956 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.956 DEBUG:boto:Params: {} 2023-09-29T23:19:36.956 DEBUG:boto:Token: None 2023-09-29T23:19:36.956 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-68/obj3 2023-09-29T23:19:36.956 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X8u5i/hsBzwsC08eht160VhZiDc= 2023-09-29T23:19:36.956 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8u5i/hsBzwsC08eht160VhZiDc=', 'Content-Length': '0'} 2023-09-29T23:19:36.959 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007540fc5aa5ed022c-0065175b88-2375-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.960 DEBUG:boto:path=/hcnphz-69/ 2023-09-29T23:19:36.960 DEBUG:boto:auth_path=/hcnphz-69/ 2023-09-29T23:19:36.960 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.960 DEBUG:boto:Path: /hcnphz-69/ 2023-09-29T23:19:36.960 DEBUG:boto:Data: 2023-09-29T23:19:36.960 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.960 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.960 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.961 DEBUG:boto:Params: {} 2023-09-29T23:19:36.961 DEBUG:boto:Token: None 2023-09-29T23:19:36.961 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-69/ 2023-09-29T23:19:36.961 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y1qCrWJu0/3WPTvQUcr/7iDnrL4= 2023-09-29T23:19:36.961 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y1qCrWJu0/3WPTvQUcr/7iDnrL4=', 'Content-Length': '0'} 2023-09-29T23:19:36.966 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000053164a506c7a0897-0065175b88-2375-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.967 DEBUG:boto:path=/hcnphz-69/obj1 2023-09-29T23:19:36.967 DEBUG:boto:auth_path=/hcnphz-69/obj1 2023-09-29T23:19:36.967 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.967 DEBUG:boto:Path: /hcnphz-69/obj1 2023-09-29T23:19:36.967 DEBUG:boto:Data: 2023-09-29T23:19:36.967 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.967 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.967 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.968 DEBUG:boto:Params: {} 2023-09-29T23:19:36.968 DEBUG:boto:Token: None 2023-09-29T23:19:36.968 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-69/obj1 2023-09-29T23:19:36.968 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wJHFXhZlxZF28bdqoDPTrJSTlyE= 2023-09-29T23:19:36.968 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wJHFXhZlxZF28bdqoDPTrJSTlyE=', 'Content-Length': '0'} 2023-09-29T23:19:36.970 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000042b09af326586e3e-0065175b88-2375-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.970 DEBUG:boto:path=/hcnphz-69/obj2 2023-09-29T23:19:36.970 DEBUG:boto:auth_path=/hcnphz-69/obj2 2023-09-29T23:19:36.971 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.971 DEBUG:boto:Path: /hcnphz-69/obj2 2023-09-29T23:19:36.971 DEBUG:boto:Data: 2023-09-29T23:19:36.971 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.971 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.971 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.971 DEBUG:boto:Params: {} 2023-09-29T23:19:36.971 DEBUG:boto:Token: None 2023-09-29T23:19:36.972 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-69/obj2 2023-09-29T23:19:36.972 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3roQx+beCNWwdCN3MDcuTUEdSno= 2023-09-29T23:19:36.972 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3roQx+beCNWwdCN3MDcuTUEdSno=', 'Content-Length': '0'} 2023-09-29T23:19:36.974 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008a53a35248c28521-0065175b88-2375-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.974 DEBUG:boto:path=/hcnphz-69/obj3 2023-09-29T23:19:36.974 DEBUG:boto:auth_path=/hcnphz-69/obj3 2023-09-29T23:19:36.974 DEBUG:boto:Method: HEAD 2023-09-29T23:19:36.975 DEBUG:boto:Path: /hcnphz-69/obj3 2023-09-29T23:19:36.975 DEBUG:boto:Data: 2023-09-29T23:19:36.975 DEBUG:boto:Headers: {} 2023-09-29T23:19:36.975 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:19:36.975 DEBUG:boto:Port: 8001 2023-09-29T23:19:36.975 DEBUG:boto:Params: {} 2023-09-29T23:19:36.975 DEBUG:boto:Token: None 2023-09-29T23:19:36.975 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:36 GMT /hcnphz-69/obj3 2023-09-29T23:19:36.976 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xifASyr8tD4Fo/LoHz8M3q8HIFo= 2023-09-29T23:19:36.976 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xifASyr8tD4Fo/LoHz8M3q8HIFo=', 'Content-Length': '0'} 2023-09-29T23:19:36.978 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000008271d927770a4d43-0065175b88-2375-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:19:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:36.978 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:36.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:36.978 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:37.024 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:37.025 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:37.122 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:19:37.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.122 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:37.167 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:37.168 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:37.264 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:19:37.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.265 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:37.311 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:37.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:37.405 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:19:37.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.406 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:37.450 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:37.450 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:37.571 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:37.571 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:37.571 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 17, 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:19:37.572 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:19:37.573 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.574 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:19:37.575 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:19:37.576 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:19:37.577 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.578 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:19:37.579 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:19:37.580 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.581 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.582 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:19:37.583 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:19:37.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:19:37.585 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:19:37.586 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.587 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:19:37.588 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:19:37.590 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:19:37.591 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:19:37.592 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:19:37.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:19:37.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:19:37.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:19:37.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:19:37.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:37.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:19:37.598 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:19:37.599 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:19:37.600 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:19:37.601 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:19:37.602 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:37.603 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:37.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.604 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:37.645 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:37.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:37.725 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:19:37.726 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.726 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:37.832 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:37.832 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:37.947 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:19:37.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:37.948 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:38.005 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:38.006 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:38.088 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:19:38.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:38.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:38.089 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:38.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:38.140 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:38.223 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:19:38.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:38.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:38.224 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:38.272 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:38.273 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:38.357 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:19:38.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:38.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:19:38.358 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:19:38.400 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:38.400 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:38.493 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:19:38.493 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:19:38.493 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:19:38.493 DEBUG:boto:Method: HEAD 2023-09-29T23:19:38.494 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:19:38.494 DEBUG:boto:Data: 2023-09-29T23:19:38.494 DEBUG:boto:Headers: {} 2023-09-29T23:19:38.494 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:19:38.494 DEBUG:boto:Port: 8000 2023-09-29T23:19:38.494 DEBUG:boto:Params: {} 2023-09-29T23:19:38.495 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:19:38.495 DEBUG:boto:Token: None 2023-09-29T23:19:38.495 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:38 GMT /hcnphz-68/ 2023-09-29T23:19:38.495 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Yu8+NmlIxtHqt0R8oZzZRqtt6Ck= 2023-09-29T23:19:38.495 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Yu8+NmlIxtHqt0R8oZzZRqtt6Ck=', 'Content-Length': '0'} 2023-09-29T23:19:38.504 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002451352c0daf8b44-0065175b8a-2488-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:19:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:38.504 DEBUG:boto:path=/hcnphz-68/obj3 2023-09-29T23:19:38.504 DEBUG:boto:auth_path=/hcnphz-68/obj3 2023-09-29T23:19:38.504 DEBUG:boto:Method: PUT 2023-09-29T23:19:38.504 DEBUG:boto:Path: /hcnphz-68/obj3 2023-09-29T23:19:38.504 DEBUG:boto:Data: 2023-09-29T23:19:38.505 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:19:38.505 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:19:38.505 DEBUG:boto:Port: 8000 2023-09-29T23:19:38.505 DEBUG:boto:Params: {} 2023-09-29T23:19:38.505 DEBUG:boto:Token: None 2023-09-29T23:19:38.505 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:19:38 GMT /hcnphz-68/obj3 2023-09-29T23:19:38.505 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CrUGpet8+HPheycP4QLWfqZsd/A= 2023-09-29T23:19:38.506 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:19:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CrUGpet8+HPheycP4QLWfqZsd/A='} 2023-09-29T23:19:38.526 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000007f9f2786d53d4d2c-0065175b8a-2488-a1'), ('Date', 'Fri, 29 Sep 2023 23:19:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:38.526 DEBUG:boto:path=/hcnphz-69/ 2023-09-29T23:19:38.527 DEBUG:boto:auth_path=/hcnphz-69/ 2023-09-29T23:19:38.527 DEBUG:boto:Method: HEAD 2023-09-29T23:19:38.527 DEBUG:boto:Path: /hcnphz-69/ 2023-09-29T23:19:38.527 DEBUG:boto:Data: 2023-09-29T23:19:38.527 DEBUG:boto:Headers: {} 2023-09-29T23:19:38.527 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:19:38.527 DEBUG:boto:Port: 8000 2023-09-29T23:19:38.527 DEBUG:boto:Params: {} 2023-09-29T23:19:38.528 DEBUG:boto:Token: None 2023-09-29T23:19:38.528 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:19:38 GMT /hcnphz-69/ 2023-09-29T23:19:38.528 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:dzpxtBhGDRtrVyFzdBv0bDAVja4= 2023-09-29T23:19:38.528 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:19:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dzpxtBhGDRtrVyFzdBv0bDAVja4=', 'Content-Length': '0'} 2023-09-29T23:19:38.533 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008c0ae8c99048c1ef-0065175b8a-2488-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:19:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:38.533 DEBUG:boto:path=/hcnphz-69/obj3 2023-09-29T23:19:38.534 DEBUG:boto:auth_path=/hcnphz-69/obj3 2023-09-29T23:19:38.534 DEBUG:boto:Method: PUT 2023-09-29T23:19:38.534 DEBUG:boto:Path: /hcnphz-69/obj3 2023-09-29T23:19:38.534 DEBUG:boto:Data: 2023-09-29T23:19:38.534 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:19:38.534 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:19:38.534 DEBUG:boto:Port: 8000 2023-09-29T23:19:38.534 DEBUG:boto:Params: {} 2023-09-29T23:19:38.534 DEBUG:boto:Token: None 2023-09-29T23:19:38.535 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:19:38 GMT /hcnphz-69/obj3 2023-09-29T23:19:38.535 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1FVI8aS4j3xpICOD3Xt06mZlJMw= 2023-09-29T23:19:38.535 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:19:38 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1FVI8aS4j3xpICOD3Xt06mZlJMw='} 2023-09-29T23:19:38.578 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e2d819da27b5c1d5-0065175b8a-2488-a1'), ('Date', 'Fri, 29 Sep 2023 23:19:38 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:19:38.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:38.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:38.579 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:38.623 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:38.623 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:38.682 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:19:38.711 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:19:38.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.711 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:19:38.711 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:19:38.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029578.333519_780.1", 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:19:38.333519Z" 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:38.713 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:19:38.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:38.713 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:19:38.714 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:19:38.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:19:38.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:38.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:38.717 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:38.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:38.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:38.818+0000 7fb191ad8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:38.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:38.818+0000 7fb191ad8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:38.840 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:38.840 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:38.840 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:38.841 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:38.842 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:38.843 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:38.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:38.845 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:38.846 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:38Z" 2023-09-29T23:19:38.847 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:38.848 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:19:38Z" } 2023-09-29T23:19:38.848 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:19:38.849 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:19:38.849 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:19:38.849 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:19:43.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:43.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:43.850 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:43.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:43.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:43.947 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:43.946+0000 7f9689360f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:43.949 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:43.946+0000 7f9689360f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:43.973 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:43.974 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:43.974 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:43.974 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:43.974 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:43.974 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:43.974 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:43.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:43.975 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:43.975 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:43.975 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:43.975 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:43.975 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:43.976 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:19:43.976 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:43.976 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:19:43.976 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:43.976 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:19:43.976 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:43.977 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:43.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:19:43.978 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:43.978 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:43.978 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:43.978 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:19:43.978 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:43.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:43.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:43.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:43.979 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:43.979 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:43.979 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:43.980 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:43.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:43.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:43.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:43.981 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:43.981 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:43.981 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:43.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:43.982 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:43.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:43.982 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:43.982 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:43.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:43.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:43.983 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:43.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:43.983 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:43.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:19:43.984 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:19:43.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:43.984 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:43Z" 2023-09-29T23:19:43.984 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:43.985 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:19:43Z" } 2023-09-29T23:19:43.985 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:19:43.985 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:19:43.986 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:19:43.986 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:19:48.987 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:48.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:48.987 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:49.032 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:49.032 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:49.085 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:49.082+0000 7fb05c06df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:49.087 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:49.086+0000 7fb05c06df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:49.113 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:49.113 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:49.113 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:49.113 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:49.114 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:49.114 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:49.114 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:49.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:49.114 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:49.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:49.115 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:49.115 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:49.115 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:49.115 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:19:49.115 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:49.116 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:49.117 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:49.118 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:49.118 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:49.118 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:49.118 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:49.118 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:49.119 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:49.119 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:49.119 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:49.119 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:49.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:49.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:49.120 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:49.120 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:49.120 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:49.120 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:49.120 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:49.120 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:49.121 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:49.122 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:49.122 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:49.122 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:49.122 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:19:49.123 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:19:49.123 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:49.123 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:49Z" 2023-09-29T23:19:49.123 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:49.124 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:19:49Z" } 2023-09-29T23:19:49.124 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:19:49.125 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:19:49.125 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:19:49.125 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:19:54.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:54.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:54.126 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:54.171 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:54.172 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:54.229 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:54.225+0000 7fcc8319cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:54.233 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:54.229+0000 7fcc8319cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:54.251 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:54.251 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:54.251 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:54.251 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:54.252 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:54.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:54.254 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:54.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:54.256 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:54.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:54Z" 2023-09-29T23:19:54.258 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:54.259 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:19:54Z" } 2023-09-29T23:19:54.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:19:54.260 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:19:54.260 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:19:54.260 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:19:59.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:59.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:19:59.262 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:19:59.308 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:19:59.308 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:19:59.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:59.361+0000 7f5ce8105f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:59.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:19:59.361+0000 7f5ce8105f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:19:59.391 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:19:59.391 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:19:59.391 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:19:59.391 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:19:59.391 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:19:59.391 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:19:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:19:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:19:59.392 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:59.393 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:59.393 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:19:59.393 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:59.393 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:19:59.393 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:59.393 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:19:59.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:59.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:59.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:59.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:59.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:19:59.394 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:19:59.395 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:59.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:59.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:59.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:19:59.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:19:59.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:19:59.398 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:19:59.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:19:59.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:19:59.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:19:59.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:19:59.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:19:59.399 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:19:59.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:59.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:19:59.400 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:19:59Z" 2023-09-29T23:19:59.401 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:19:59.401 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:19:59Z" } 2023-09-29T23:19:59.402 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:19:59.402 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:19:59.402 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:19:59.402 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:04.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:04.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:04.403 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:04.448 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:04.449 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:04.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:04.505+0000 7f4afd90bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:04.507 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:04.505+0000 7f4afd90bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:04.535 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:04.535 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:04.536 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:04.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:04.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:04.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:04.540 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:04Z" 2023-09-29T23:20:04.541 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:04.542 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:04Z" } 2023-09-29T23:20:04.542 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:04.543 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:04.543 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:04.543 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:09.544 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:09.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:09.544 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:09.589 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:09.589 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:09.647 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:09.649+0000 7f311dbacf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:09.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:09.649+0000 7f311dbacf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:09.675 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:09.675 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:09.676 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:09.677 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:09.678 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:09.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:09.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:09Z" 2023-09-29T23:20:09.681 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:09.682 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:09Z" } 2023-09-29T23:20:09.686 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:09.686 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:09.686 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:09.686 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:14.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:14.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:14.689 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:14.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:14.791 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:14.793+0000 7f68e4a66f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:14.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:14.793+0000 7f68e4a66f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:14.819 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:14.819 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:14.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:14.821 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:14.822 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:14.823 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:14.824 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:14.825 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:14.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:14.827 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:14Z" 2023-09-29T23:20:14.827 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:14.828 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:14Z" } 2023-09-29T23:20:14.828 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:14.828 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:14.829 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:14.829 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:19.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:19.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:19.830 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:19.876 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:19.877 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:19.932 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:19.933+0000 7fec8d183f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:19.934 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:19.937+0000 7fec8d183f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:19.961 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:19.961 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:19.962 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:19.962 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:19.962 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:19.962 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:19.963 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:19.963 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:19.963 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:19.963 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:19.964 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:19.964 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:19.964 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:19.964 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:19.964 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:19.965 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:19.965 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:19.965 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:19.965 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:19.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:19.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:19.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:19.966 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:19.967 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:19.967 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:19.967 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:19.967 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:19.967 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:19.968 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:19.968 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:19.968 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:19.968 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:19.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:19.969 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:19.969 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:19.969 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:19.969 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:19.970 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:19.970 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:19.970 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:19.970 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:19.970 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:19.971 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:19.971 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:19.971 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:19.971 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:19.971 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:19.972 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:19.972 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:19.972 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:19.972 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:19.973 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:19.973 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:19.973 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:19.973 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:19.973 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:19.974 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:19.974 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:19.974 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:19.974 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:19.974 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:19.975 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:19.975 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:19.975 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:19Z" 2023-09-29T23:20:19.975 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:19.977 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:19Z" } 2023-09-29T23:20:19.977 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:19.977 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:19.978 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:19.978 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:24.979 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:24.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:24.980 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:25.026 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:25.082 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:25.085+0000 7fd7eeb2df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:25.084 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:25.085+0000 7fd7eeb2df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:25.111 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:25.111 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:25.111 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:25.111 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:25.111 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:25.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:25.113 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:25.114 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:25.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:25.116 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:25Z" 2023-09-29T23:20:25.117 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:25.118 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:25Z" } 2023-09-29T23:20:25.119 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:25.119 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:25.119 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:25.119 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:30.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:30.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:30.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:30.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:30.165 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:30.217 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:30.217+0000 7fe52bb16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:30.218 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:30.221+0000 7fe52bb16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:30.254 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:30.255 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:30.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:30.257 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:30.258 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:30.259 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:30.260 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:30Z" 2023-09-29T23:20:30.261 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:30.262 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:30Z" } 2023-09-29T23:20:30.263 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:30.263 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:30.263 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:30.263 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:35.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:35.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:35.264 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:35.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:35.312 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:35.364 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:35.364+0000 7f5dbe903f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:35.365 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:35.364+0000 7f5dbe903f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:35.392 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:35.392 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:35.393 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:35.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:35.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:35.396 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:35.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:35.398 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:35.399 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:35Z" 2023-09-29T23:20:35.399 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:35.400 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:35Z" } 2023-09-29T23:20:35.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:35.400 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:35.400 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:35.400 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:40.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:40.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:40.402 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:40.449 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:40.449 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:40.504 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:40.504+0000 7fc2475eaf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:40.506 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:40.508+0000 7fc2475eaf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:40.533 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:40.534 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:40.535 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:40.536 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:40.537 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:40.538 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:40.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:40.540 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:40.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:40.540 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:40.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:40.540 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:40Z" 2023-09-29T23:20:40.540 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:40.541 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:40Z" } 2023-09-29T23:20:40.541 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:40.541 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:40.541 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:40.542 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:45.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:45.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:45.543 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:45.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:45.590 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:45.648 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:45.648+0000 7f1b32edef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:45.649 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:45.648+0000 7f1b32edef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:45.677 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:45.677 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:45.678 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:45.679 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:45.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:45.681 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:45.682 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:45.683 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:45.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:45Z" 2023-09-29T23:20:45.685 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:45.686 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:45Z" } 2023-09-29T23:20:45.687 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:45.687 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:45.687 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:45.687 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:50.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:50.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:50.688 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:50.733 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:50.733 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:50.793 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:50.792+0000 7fcc66699f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:50.795 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:50.796+0000 7fcc66699f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:50.823 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:50.824 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:50.825 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:50.826 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:50.827 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:50.828 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:50.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:50Z" 2023-09-29T23:20:50.830 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:50.831 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:50Z" } 2023-09-29T23:20:50.832 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:50.832 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:50.832 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:50.832 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:20:55.833 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:55.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:20:55.833 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:20:55.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:20:55.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:20:55.936 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:55.936+0000 7fe5b6ca9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:55.937 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:20:55.940+0000 7fe5b6ca9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:20:55.963 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:20:55.964 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:55.965 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:55.966 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:55.967 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:20:55.968 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:20:55.969 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:20:55Z" 2023-09-29T23:20:55.970 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:20:55.970 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:20:55Z" } 2023-09-29T23:20:55.971 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:20:55.971 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:20:55.971 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:20:55.971 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:00.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:00.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:00.973 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:01.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:01.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:01.074 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:01.076+0000 7f26bfb12f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:01.076 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:01.076+0000 7f26bfb12f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:01.101 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:01.102 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:01.103 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:01.104 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:01.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:01.106 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:01.107 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:01.108 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:01.109 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:01Z" 2023-09-29T23:21:01.109 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:01.110 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:01Z" } 2023-09-29T23:21:01.110 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:01.110 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:01.110 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:01.110 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:06.111 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:06.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:06.112 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:06.156 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:06.157 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:06.212 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:06.212+0000 7f0921459f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:06.214 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:06.216+0000 7f0921459f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:06.242 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:06.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:06.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:06.245 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:06.246 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:06.247 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:06.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:06.249 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:06.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:06.249 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:06Z" 2023-09-29T23:21:06.249 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:06.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:06Z" } 2023-09-29T23:21:06.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:06.250 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:06.250 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:06.250 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:11.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:11.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:11.252 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:11.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:11.297 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:11.351 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:11.351+0000 7f722ada6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:11.352 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:11.351+0000 7f722ada6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:11.376 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:11.376 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:11.377 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:11.378 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:11.379 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:11.380 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:11.381 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:11.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:11.383 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:11Z" 2023-09-29T23:21:11.383 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:11.383 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:11Z" } 2023-09-29T23:21:11.384 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:11.384 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:11.384 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:11.384 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:16.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:16.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:16.386 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:16.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:16.432 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:16.489 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:16.487+0000 7efd7923ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:16.490 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:16.491+0000 7efd7923ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:16.519 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:16.520 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:16.521 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:16.522 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:16.523 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:16.524 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:16.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:16.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:16.527 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:16.527 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:16.527 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:16.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:16.527 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:16Z" 2023-09-29T23:21:16.527 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:16.528 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:16Z" } 2023-09-29T23:21:16.528 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:16.529 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:16.529 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:16.529 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:21.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:21.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:21.530 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:21.575 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:21.575 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:21.633 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:21.631+0000 7f1925830f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:21.635 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:21.635+0000 7f1925830f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:21.661 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:21.661 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:21.661 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:21.662 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:21.662 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:21.662 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:21.662 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:21.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:21.663 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:21.663 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:21.663 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:21.663 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:21.663 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:21.664 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:21.665 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:21.666 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:21.667 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:21.668 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:21Z" 2023-09-29T23:21:21.669 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:21.673 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:21Z" } 2023-09-29T23:21:21.674 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:21.674 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:21.675 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:21.675 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:26.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:26.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:26.677 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:26.721 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:26.721 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:26.778 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:26.779+0000 7f4b20dd1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:26.779 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:26.779+0000 7f4b20dd1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:26.806 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:26.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:26.808 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:26.809 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:26.810 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:26.811 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:26.812 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:26.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:26.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:26.812 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:26.813 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:26.814 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:26Z" 2023-09-29T23:21:26.815 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:26.817 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:26Z" } 2023-09-29T23:21:26.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:26.817 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:26.818 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:26.818 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:31.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:31.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:31.819 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:31.866 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:31.866 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:31.927 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:31.927+0000 7fa47d517f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:31.928 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:31.927+0000 7fa47d517f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:31.955 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:31.955 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:31.956 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:31.957 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:31.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:31.959 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:31.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:31.961 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:31.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:31.962 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:31Z" 2023-09-29T23:21:31.962 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:31.963 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:31Z" } 2023-09-29T23:21:31.963 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:31.963 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:31.963 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:31.964 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:36.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:36.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:36.966 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:37.012 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:37.012 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:37.071 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:37.071+0000 7f97966ecf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:37.072 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:37.071+0000 7f97966ecf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:37.097 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:37.097 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:37.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:37.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029451.119832_750.1", 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:37.100 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:31.119832Z", 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:37.101 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:37.102 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:37.103 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:37Z" 2023-09-29T23:21:37.104 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:37.105 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029451.119832_750.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:31.119832Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:37Z" } 2023-09-29T23:21:37.106 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029451.119832_750.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:31.119832Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:37.106 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:37.106 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029451.119832_750.1', 2: '', 3: ''} 2023-09-29T23:21:37.106 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029578.333519_780.1 target=1_1696029451.119832_750.1 2023-09-29T23:21:42.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.107 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:42.153 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:42.153 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:42.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:42.211+0000 7f3737a7ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:42.211 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:42.211+0000 7f3737a7ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:42.242 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:42.242 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:42.242 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:42.243 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.244 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029578.333519_780.1", 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.333519Z", 2023-09-29T23:21:42.245 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.246 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:42.247 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.248 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:42.249 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:42.249 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:42.249 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.249 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:42Z" 2023-09-29T23:21:42.249 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:42.250 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029578.333519_780.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:19:38.333519Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:42Z" } 2023-09-29T23:21:42.250 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029578.333519_780.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.333519Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:42.251 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:42.251 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:42.251 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:21:42.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.251 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:42.290 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:42.380 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.381 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:21:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:21:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:21:42.382 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:21:42.383 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.383 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:21:42.384 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:21:42.384 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:21:42.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.384 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:42.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:42.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:42.564 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:42.564 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:21:42.566 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:21:42.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:42.568 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:42.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:21:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:21:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:42.570 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:42Z" 2023-09-29T23:21:42.570 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:42.571 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:21:42Z" } 2023-09-29T23:21:42.571 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:21:42.571 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:21:42.571 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:21:42.572 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:21:42.572 DEBUG:boto:path=/hcnphz-68/ 2023-09-29T23:21:42.572 DEBUG:boto:auth_path=/hcnphz-68/ 2023-09-29T23:21:42.572 DEBUG:boto:Method: HEAD 2023-09-29T23:21:42.572 DEBUG:boto:Path: /hcnphz-68/ 2023-09-29T23:21:42.572 DEBUG:boto:Data: 2023-09-29T23:21:42.572 DEBUG:boto:Headers: {} 2023-09-29T23:21:42.572 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:21:42.572 DEBUG:boto:Port: 8001 2023-09-29T23:21:42.573 DEBUG:boto:Params: {} 2023-09-29T23:21:42.573 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:21:42.573 DEBUG:boto:Token: None 2023-09-29T23:21:42.573 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:21:42 GMT /hcnphz-68/ 2023-09-29T23:21:42.573 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5O4Bsidtk9RTchFyn7UzrD0//6A= 2023-09-29T23:21:42.574 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:21:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5O4Bsidtk9RTchFyn7UzrD0//6A=', 'Content-Length': '0'} 2023-09-29T23:21:42.582 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e124cd8f69d75168-0065175c06-247e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:21:42.582 DEBUG:boto:path=/hcnphz-68/obj3 2023-09-29T23:21:42.582 DEBUG:boto:auth_path=/hcnphz-68/obj3 2023-09-29T23:21:42.582 DEBUG:boto:Method: HEAD 2023-09-29T23:21:42.582 DEBUG:boto:Path: /hcnphz-68/obj3 2023-09-29T23:21:42.582 DEBUG:boto:Data: 2023-09-29T23:21:42.583 DEBUG:boto:Headers: {} 2023-09-29T23:21:42.583 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:21:42.583 DEBUG:boto:Port: 8001 2023-09-29T23:21:42.583 DEBUG:boto:Params: {} 2023-09-29T23:21:42.583 DEBUG:boto:Token: None 2023-09-29T23:21:42.584 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:21:42 GMT /hcnphz-68/obj3 2023-09-29T23:21:42.584 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:bs5Xj6l8Fo/0e32J7QvEvqzNzco= 2023-09-29T23:21:42.584 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:21:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bs5Xj6l8Fo/0e32J7QvEvqzNzco=', 'Content-Length': '0'} 2023-09-29T23:21:42.587 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000010ea2bd1381cf069-0065175c06-247e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:21:42.587 DEBUG:boto:path=/hcnphz-68/obj3 2023-09-29T23:21:42.587 DEBUG:boto:auth_path=/hcnphz-68/obj3 2023-09-29T23:21:42.588 DEBUG:boto:Method: GET 2023-09-29T23:21:42.588 DEBUG:boto:Path: /hcnphz-68/obj3 2023-09-29T23:21:42.588 DEBUG:boto:Data: 2023-09-29T23:21:42.588 DEBUG:boto:Headers: {} 2023-09-29T23:21:42.588 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:21:42.588 DEBUG:boto:Port: 8001 2023-09-29T23:21:42.588 DEBUG:boto:Params: {} 2023-09-29T23:21:42.588 DEBUG:boto:Token: None 2023-09-29T23:21:42.588 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:21:42 GMT /hcnphz-68/obj3 2023-09-29T23:21:42.589 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rQ7hHijVyN3w87F37sTY0kZbn+0= 2023-09-29T23:21:42.589 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:21:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rQ7hHijVyN3w87F37sTY0kZbn+0=', 'Content-Length': '0'} 2023-09-29T23:21:42.591 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:17:32 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-68:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.8731.3'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007173133dae1dc4e3-0065175c06-247e-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:21:42.591 DEBUG:boto:path=/hcnphz-69/ 2023-09-29T23:21:42.591 DEBUG:boto:auth_path=/hcnphz-69/ 2023-09-29T23:21:42.592 DEBUG:boto:Method: HEAD 2023-09-29T23:21:42.592 DEBUG:boto:Path: /hcnphz-69/ 2023-09-29T23:21:42.592 DEBUG:boto:Data: 2023-09-29T23:21:42.592 DEBUG:boto:Headers: {} 2023-09-29T23:21:42.592 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:21:42.592 DEBUG:boto:Port: 8001 2023-09-29T23:21:42.592 DEBUG:boto:Params: {} 2023-09-29T23:21:42.592 DEBUG:boto:Token: None 2023-09-29T23:21:42.592 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:21:42 GMT /hcnphz-69/ 2023-09-29T23:21:42.593 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:WIWVWVwxm2gas13cZm+utTGwJCI= 2023-09-29T23:21:42.593 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:21:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:WIWVWVwxm2gas13cZm+utTGwJCI=', 'Content-Length': '0'} 2023-09-29T23:21:42.597 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006a92ed1e7dd0dbe0-0065175c06-247e-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:21:42.598 DEBUG:boto:path=/hcnphz-69/obj3 2023-09-29T23:21:42.598 DEBUG:boto:auth_path=/hcnphz-69/obj3 2023-09-29T23:21:42.598 DEBUG:boto:Method: HEAD 2023-09-29T23:21:42.598 DEBUG:boto:Path: /hcnphz-69/obj3 2023-09-29T23:21:42.598 DEBUG:boto:Data: 2023-09-29T23:21:42.598 DEBUG:boto:Headers: {} 2023-09-29T23:21:42.598 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:21:42.598 DEBUG:boto:Port: 8001 2023-09-29T23:21:42.599 DEBUG:boto:Params: {} 2023-09-29T23:21:42.599 DEBUG:boto:Token: None 2023-09-29T23:21:42.599 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:21:42 GMT /hcnphz-69/obj3 2023-09-29T23:21:42.599 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:jUWSRFQiEZ1PcrFRqaAGCLvGry8= 2023-09-29T23:21:42.599 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:21:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jUWSRFQiEZ1PcrFRqaAGCLvGry8=', 'Content-Length': '0'} 2023-09-29T23:21:42.601 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000028efd691902127d8-0065175c06-247e-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:21:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:21:42.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:21:42.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:21:42.602 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hcnphz-68 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:21:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:42.647 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:42.745 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:21:42.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:21:42.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:21:42.746 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:21:42.788 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:42.788 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:42.882 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:21:42.883 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket: ... ok 2023-09-29T23:21:42.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:42.884 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:42.971 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:42.972 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:43.033 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:21:43.060 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:21:43.060 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.060 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:43.060 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:21:43.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029702.717573_782.1", 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:21:42.717573Z" 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.061 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:43.062 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:21:43.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:43.062 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:21:43.063 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:21:43.063 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:21:43.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:43.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:43.063 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:43.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:43.109 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:43.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:43.167+0000 7f68c0f17f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:43.167 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:43.167+0000 7f68c0f17f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:43.194 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:43.194 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:43.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:43.196 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029578.333519_780.1", 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:43.197 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.333519Z", 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:43.198 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:43.199 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:43.200 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:43Z" 2023-09-29T23:21:43.201 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:43.202 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029578.333519_780.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:19:38.333519Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:43Z" } 2023-09-29T23:21:43.202 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029578.333519_780.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.333519Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:43.203 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:21:43.203 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:43.203 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029702.717573_782.1 target=1_1696029578.333519_780.1 2023-09-29T23:21:48.204 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:48.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:48.205 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:48.255 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:48.303 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:48.303+0000 7f8b5d7f0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:48.304 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:48.303+0000 7f8b5d7f0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:48.331 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:48.332 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:48.333 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029578.333519_780.1", 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:48.334 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.333519Z", 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:48.335 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:48.336 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:48.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:48.338 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:48Z" 2023-09-29T23:21:48.338 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:48.339 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029578.333519_780.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:19:38.333519Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:48Z" } 2023-09-29T23:21:48.340 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029578.333519_780.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.333519Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:48.340 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:21:48.340 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:48.340 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029702.717573_782.1 target=1_1696029578.333519_780.1 2023-09-29T23:21:53.341 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:53.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:53.342 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:53.388 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:53.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:53.446+0000 7f750b87df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:53.448 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:53.446+0000 7f750b87df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:53.472 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:53.472 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:53.472 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:53.473 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:53.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029578.333519_780.1", 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.333519Z", 2023-09-29T23:21:53.475 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:53.476 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:53.477 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:53.478 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:53.479 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:53.479 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:53.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:53.479 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:53Z" 2023-09-29T23:21:53.479 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:53.480 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029578.333519_780.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:19:38.333519Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:53Z" } 2023-09-29T23:21:53.480 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029578.333519_780.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.333519Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:53.480 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:21:53.480 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:53.480 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029702.717573_782.1 target=1_1696029578.333519_780.1 2023-09-29T23:21:58.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:58.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:21:58.482 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:21:58.528 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:21:58.529 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:21:58.587 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:58.586+0000 7f44b427af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:58.588 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:21:58.586+0000 7f44b427af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:21:58.614 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:58.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:21:58.616 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029578.333519_780.1", 2023-09-29T23:21:58.617 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.333519Z", 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:21:58.618 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:58.619 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:21:58.620 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:21:58.621 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:21:58.622 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:21:58Z" 2023-09-29T23:21:58.622 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:21:58.623 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029578.333519_780.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:19:38.333519Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:21:58Z" } 2023-09-29T23:21:58.623 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029578.333519_780.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.333519Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:21:58.623 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:21:58.623 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029578.333519_780.1', 2: '', 3: ''} 2023-09-29T23:21:58.623 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029702.717573_782.1 target=1_1696029578.333519_780.1 2023-09-29T23:22:03.624 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:03.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:03.625 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:03.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:03.671 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:03.729 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:22:03.726+0000 7fa253cccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:22:03.730 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:22:03.730+0000 7fa253cccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:22:03.756 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:03.756 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:03.757 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:03.758 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:03.759 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029702.717573_782.1", 2023-09-29T23:22:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:03.761 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:21:42.717573Z", 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:03.762 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:22:03.763 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:03.764 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:22:03.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:03.766 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:03Z" 2023-09-29T23:22:03.766 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:03.767 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029702.717573_782.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:21:42.717573Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:22:03Z" } 2023-09-29T23:22:03.767 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029702.717573_782.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:21:42.717573Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:22:03.767 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:22:03.767 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029702.717573_782.1', 2: '', 3: ''} 2023-09-29T23:22:03.767 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:22:03.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:03.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:03.768 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:03.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:03.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:03.901 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:22:03.901 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-70 2023-09-29T23:22:03.901 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:22:03.901 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:22:03.901 DEBUG:boto:Method: PUT 2023-09-29T23:22:03.902 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:22:03.902 DEBUG:boto:Data: 2023-09-29T23:22:03.902 DEBUG:boto:Headers: {} 2023-09-29T23:22:03.902 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:22:03.902 DEBUG:boto:Port: 8000 2023-09-29T23:22:03.902 DEBUG:boto:Params: {} 2023-09-29T23:22:03.902 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:22:03.903 DEBUG:boto:Token: None 2023-09-29T23:22:03.903 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:22:03 GMT /hcnphz-70/ 2023-09-29T23:22:03.903 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OJtZhteg4FguNgPZiUYESU0sB1s= 2023-09-29T23:22:03.903 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:22:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OJtZhteg4FguNgPZiUYESU0sB1s=', 'Content-Length': '0'} 2023-09-29T23:22:03.938 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000005446e552a3d8de63-0065175c1b-2488-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:22:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:22:03.938 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-71 2023-09-29T23:22:03.939 DEBUG:boto:path=/hcnphz-71/ 2023-09-29T23:22:03.939 DEBUG:boto:auth_path=/hcnphz-71/ 2023-09-29T23:22:03.939 DEBUG:boto:Method: PUT 2023-09-29T23:22:03.939 DEBUG:boto:Path: /hcnphz-71/ 2023-09-29T23:22:03.940 DEBUG:boto:Data: 2023-09-29T23:22:03.940 DEBUG:boto:Headers: {} 2023-09-29T23:22:03.940 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:22:03.940 DEBUG:boto:Port: 8000 2023-09-29T23:22:03.940 DEBUG:boto:Params: {} 2023-09-29T23:22:03.941 DEBUG:boto:Token: None 2023-09-29T23:22:03.941 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:22:03 GMT /hcnphz-71/ 2023-09-29T23:22:03.941 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uE5CSnbddYGivuGTc+AK4D/t78k= 2023-09-29T23:22:03.942 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:22:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uE5CSnbddYGivuGTc+AK4D/t78k=', 'Content-Length': '0'} 2023-09-29T23:22:03.971 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000052efab609dfb470f-0065175c1b-2488-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:22:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:22:03.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:03.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:03.973 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:04.016 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.017 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:04.077 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:22:04.109 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:22:04.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.109 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:22:04.110 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:22:04.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.110 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:22:04.110 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:22:03.971870Z" 2023-09-29T23:22:04.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.111 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:04.111 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:22:04.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:04.112 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:22:04.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:04.113 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:22:04.114 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:22:04.114 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:22:04.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:04.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:04.115 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:04.157 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.158 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:04.208 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:22:04.206+0000 7f6bee890f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:22:04.210 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:22:04.210+0000 7f6bee890f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:22:04.235 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.236 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:22:04.237 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:04.238 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:04.239 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:04.240 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.241 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:22:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:22:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:04.242 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:04Z" 2023-09-29T23:22:04.242 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:04.243 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:22:04Z" } 2023-09-29T23:22:04.243 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:22:04.243 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:22:04.243 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:22:04.243 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:22:04.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.244 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:04.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.331 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:04.422 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:22:04.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.423 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:04.473 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.474 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:04.569 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:22:04.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-70', '--dest-bucket=hcnphz-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-70', '--dest-bucket=hcnphz-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.570 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hcnphz-70 --dest-bucket=hcnphz-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:04.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:04.708 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-70","zones":["a1"]},"dest":{"bucket":"hcnphz-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:22:04.708 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.709 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:04.753 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.754 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:04.845 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-70","zones":["a1"]},"dest":{"bucket":"hcnphz-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:22:04.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:04.846 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:04.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:04.934 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 18, 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:22:05.055 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.056 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.057 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:22:05.058 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.059 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:22:05.060 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.061 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:22:05.062 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:22:05.063 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-70", 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:22:05.064 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-71", 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:22:05.065 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.067 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:22:05.068 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:22:05.069 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:22:05.071 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.072 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:22:05.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:22:05.074 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:22:05.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:22:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:22:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:22:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:22:05.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:22:05.080 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:22:05.081 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.082 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:22:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:22:05.083 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:05.083 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:05.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:05.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:22:05.084 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:22:05.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:05.129 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:05.204 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-70","zones":["a1"]},"dest":{"bucket":"hcnphz-71","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:22:05.205 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:22:05.205 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:22:05.205 DEBUG:boto:Method: HEAD 2023-09-29T23:22:05.205 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:22:05.205 DEBUG:boto:Data: 2023-09-29T23:22:05.205 DEBUG:boto:Headers: {} 2023-09-29T23:22:05.205 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:22:05.205 DEBUG:boto:Port: 8000 2023-09-29T23:22:05.206 DEBUG:boto:Params: {} 2023-09-29T23:22:05.206 DEBUG:boto:Token: None 2023-09-29T23:22:05.206 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:22:05 GMT /hcnphz-70/ 2023-09-29T23:22:05.206 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2GeBAUJUXi7B4D6yJROcDlfJuuU= 2023-09-29T23:22:05.206 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:22:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2GeBAUJUXi7B4D6yJROcDlfJuuU=', 'Content-Length': '0'} 2023-09-29T23:22:05.230 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004ce77ef6b0e42c3b-0065175c1d-2615-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:22:05.231 DEBUG:boto:path=/hcnphz-70/obj1 2023-09-29T23:22:05.231 DEBUG:boto:auth_path=/hcnphz-70/obj1 2023-09-29T23:22:05.231 DEBUG:boto:Method: PUT 2023-09-29T23:22:05.231 DEBUG:boto:Path: /hcnphz-70/obj1 2023-09-29T23:22:05.231 DEBUG:boto:Data: 2023-09-29T23:22:05.231 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:22:05.231 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:22:05.231 DEBUG:boto:Port: 8000 2023-09-29T23:22:05.232 DEBUG:boto:Params: {} 2023-09-29T23:22:05.232 DEBUG:boto:Token: None 2023-09-29T23:22:05.232 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:22:05 GMT /hcnphz-70/obj1 2023-09-29T23:22:05.232 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PMT2XODlC3LSAna6vemfVfvvzYY= 2023-09-29T23:22:05.232 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:22:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PMT2XODlC3LSAna6vemfVfvvzYY='} 2023-09-29T23:22:05.248 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e353bf5e91266821-0065175c1d-2615-a1'), ('Date', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:22:05.248 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:22:05.248 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:22:05.248 DEBUG:boto:Method: HEAD 2023-09-29T23:22:05.248 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:22:05.248 DEBUG:boto:Data: 2023-09-29T23:22:05.249 DEBUG:boto:Headers: {} 2023-09-29T23:22:05.249 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:22:05.249 DEBUG:boto:Port: 8000 2023-09-29T23:22:05.249 DEBUG:boto:Params: {} 2023-09-29T23:22:05.249 DEBUG:boto:Token: None 2023-09-29T23:22:05.249 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:22:05 GMT /hcnphz-70/ 2023-09-29T23:22:05.249 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:2GeBAUJUXi7B4D6yJROcDlfJuuU= 2023-09-29T23:22:05.249 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:22:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2GeBAUJUXi7B4D6yJROcDlfJuuU=', 'Content-Length': '0'} 2023-09-29T23:22:05.254 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004b1ba747dc998673-0065175c1d-2615-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:22:05.254 DEBUG:boto:path=/hcnphz-70/obj2 2023-09-29T23:22:05.255 DEBUG:boto:auth_path=/hcnphz-70/obj2 2023-09-29T23:22:05.255 DEBUG:boto:Method: PUT 2023-09-29T23:22:05.255 DEBUG:boto:Path: /hcnphz-70/obj2 2023-09-29T23:22:05.255 DEBUG:boto:Data: 2023-09-29T23:22:05.255 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:22:05.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:22:05.255 DEBUG:boto:Port: 8000 2023-09-29T23:22:05.255 DEBUG:boto:Params: {} 2023-09-29T23:22:05.255 DEBUG:boto:Token: None 2023-09-29T23:22:05.256 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:22:05 GMT /hcnphz-70/obj2 2023-09-29T23:22:05.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:QbzS6cB5VQ3sQZynPTa+kDFgvqA= 2023-09-29T23:22:05.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:22:05 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QbzS6cB5VQ3sQZynPTa+kDFgvqA='} 2023-09-29T23:22:05.298 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e9f7834f49c3f61-0065175c1d-2615-a1'), ('Date', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:22:05.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.299 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:05.352 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:05.352 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:05.445 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:22:05.478 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:22:03.971870Z" 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.479 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.480 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:22:05.481 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:22:05.481 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:22:05.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.482 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:05.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:05.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:22:05.614+0000 7f50d6d6ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:22:05.617 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:22:05.614+0000 7f50d6d6ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:22:05.647 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:05.647 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:05.647 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.648 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.649 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:22:05.650 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:05.651 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:22:05.653 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:05Z" 2023-09-29T23:22:05.654 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:05.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:22:05Z" } 2023-09-29T23:22:05.656 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:22:05.656 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:22:05.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:22:05.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:22:05.656 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.657 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:05.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:05.706 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:05.807 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:22:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051669", 2023-09-29T23:22:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:22:05.249921Z" 2023-09-29T23:22:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.808 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.809 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:22:05.270925Z" 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:05.810 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.811 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:22:05.812 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:05.812 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:22:05.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:05.813 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:05.858 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:05.859 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:05.926 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:05.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.928 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.929 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:05.930 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:05.931 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:05.932 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:05.933 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:05Z" 2023-09-29T23:22:05.933 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:05.934 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:05Z" } 2023-09-29T23:22:05.934 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:05.934 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:05.934 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:05.934 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:10.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:10.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:10.935 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:10.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:10.984 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:11.071 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:11.072 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:11.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:11.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:11.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:11.076 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:11.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:11.078 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:11Z" 2023-09-29T23:22:11.078 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:11.079 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:11Z" } 2023-09-29T23:22:11.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:11.080 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:11.080 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:11.080 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:16.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:16.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:16.082 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:16.130 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:16.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:16.215 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:16.215 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:16.215 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:16.215 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:16.216 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:16.216 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:16.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:16.216 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:16.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:16.217 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:16.217 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:16.217 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:16.217 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:16.217 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:16.218 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:16.218 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:16.218 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:16.218 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:16.218 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:16.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:16.219 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:16.219 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:16.219 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:16.219 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:16.220 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:16.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:16.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:16.220 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:16.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:16.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:16.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:16.221 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:16.221 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:16.221 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:16.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:16.222 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:16.222 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:16.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:16.222 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:16.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:16.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:16.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:16.223 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:16.223 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:16.223 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:16.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:16.224 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:16.224 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:16.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:16.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:16.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:16.225 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:16.225 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:16.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:16.225 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:16.225 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:16.226 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:16Z" 2023-09-29T23:22:16.226 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:16.228 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:16Z" } 2023-09-29T23:22:16.228 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:16.228 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:16.229 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:16.229 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:21.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:21.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:21.231 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:21.278 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:21.278 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:21.365 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:21.366 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:21.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:21.368 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:21.369 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:21.370 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:21.371 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:21.372 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:21Z" 2023-09-29T23:22:21.372 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:21.373 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:21Z" } 2023-09-29T23:22:21.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:21.373 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:21.374 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:21.374 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:26.374 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:26.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:26.375 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:26.424 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:26.510 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:26.510 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:26.510 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:26.511 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:26.512 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:26.513 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:26.514 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:26.515 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:26.516 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:26.517 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:26.518 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:26Z" 2023-09-29T23:22:26.518 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:26.519 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:26Z" } 2023-09-29T23:22:26.519 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:26.519 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:26.519 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:26.519 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:31.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:31.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:31.521 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:31.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:31.567 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:31.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:31.657 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:31.658 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:31.659 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:31.660 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:31.661 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:31.662 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:31Z" 2023-09-29T23:22:31.662 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:31.663 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:31Z" } 2023-09-29T23:22:31.663 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:31.663 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:31.663 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:31.664 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:36.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:36.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:36.665 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:36.714 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:36.803 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:36.803 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:36.804 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:36.805 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:36.806 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:36.807 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:36.808 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:36.809 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:36.810 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:36.811 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:36.812 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:36.813 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:36.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:36.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:36.816 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:36Z" 2023-09-29T23:22:36.816 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:36.817 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:36Z" } 2023-09-29T23:22:36.817 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:36.818 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:36.818 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:36.818 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:41.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:41.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:41.820 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:41.868 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:41.869 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:41.952 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:41.953 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:41.954 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:41.955 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:41.956 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:41.957 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:41.958 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:41.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:41.958 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:41Z" 2023-09-29T23:22:41.958 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:41.959 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:41Z" } 2023-09-29T23:22:41.959 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:41.959 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:41.959 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:41.959 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:46.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:46.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:46.961 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:47.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:47.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:47.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:47.098 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:47.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:47.100 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:47.101 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:47.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:47.103 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:47.103 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:47.103 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:47.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:47.103 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:47Z" 2023-09-29T23:22:47.103 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:47.104 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:47Z" } 2023-09-29T23:22:47.104 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:47.105 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:47.105 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:47.105 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:52.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:52.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:52.106 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:52.154 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:52.154 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:52.250 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:52.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:52.252 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:52.253 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:52.254 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:52.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:52.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:52.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:52.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:52.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:52.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:52.256 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:52.257 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:52Z" 2023-09-29T23:22:52.258 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:52.259 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:52Z" } 2023-09-29T23:22:52.259 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:52.260 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:52.260 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:52.260 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:22:57.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:57.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:22:57.262 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:22:57.310 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:22:57.311 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:22:57.393 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:22:57.393 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:22:57.393 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:22:57.393 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:22:57.393 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:22:57.394 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:57.395 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:22:57.396 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:22:57.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:57.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:22:57Z" 2023-09-29T23:22:57.399 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:22:57.400 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:22:57Z" } 2023-09-29T23:22:57.400 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:22:57.401 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:57.401 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:22:57.401 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:02.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:02.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:02.402 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:02.451 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:02.451 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:02.529 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:02.530 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:02.531 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:02.532 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:02.533 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:02.534 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:02Z" 2023-09-29T23:23:02.535 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:02.536 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:02Z" } 2023-09-29T23:23:02.537 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:02.537 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:02.537 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:02.537 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:07.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:07.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:07.538 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:07.586 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:07.587 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:07.674 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:07.674 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:07.674 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:07.674 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:07.675 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:07.676 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:07.677 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:07.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:07.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:07.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:07.679 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:07.679 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:07.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:07.679 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:07.680 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:07.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:07.681 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:07Z" 2023-09-29T23:23:07.681 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:07.682 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:07Z" } 2023-09-29T23:23:07.682 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:07.682 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:07.682 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:07.683 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:12.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:12.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:12.684 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:12.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:12.730 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:12.812 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:12.813 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:12.814 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:12.815 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:12.816 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:12.817 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:12.818 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:12Z" 2023-09-29T23:23:12.818 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:12.819 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:12Z" } 2023-09-29T23:23:12.819 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:12.819 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:12.819 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:12.819 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:17.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:17.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:17.821 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:17.869 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:17.869 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:17.956 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:17.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:17.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:17.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:17.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:17.961 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:17.962 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:17Z" 2023-09-29T23:23:17.963 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:17.963 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:17Z" } 2023-09-29T23:23:17.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:17.964 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:17.964 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:17.965 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:22.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:22.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:22.967 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:23.016 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:23.016 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:23.105 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:23.106 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:23.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:23.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:23.109 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:23.110 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:23.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:23.112 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:23Z" 2023-09-29T23:23:23.112 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:23.113 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:23Z" } 2023-09-29T23:23:23.113 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:23.113 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:23.113 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:23.113 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:28.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:28.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:28.115 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:28.162 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:28.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:28.247 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:28.249 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:28.249 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:28.249 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:28.249 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:28.250 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:28.251 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:28.252 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:28.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:28.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:28Z" 2023-09-29T23:23:28.255 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:28.256 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:28Z" } 2023-09-29T23:23:28.256 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:28.257 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:28.257 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:28.257 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:33.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:33.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:33.258 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:33.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:33.306 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:33.393 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:33.394 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:33.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:33.396 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:33.397 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:33.398 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:33.399 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:33.400 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:33Z" 2023-09-29T23:23:33.400 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:33.401 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:33Z" } 2023-09-29T23:23:33.401 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:33.401 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:33.401 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:33.402 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:38.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:38.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:38.403 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:38.451 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:38.538 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:38.539 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:38.540 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:38.541 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:38.542 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:38.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:38.544 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:38.545 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:38.546 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:38Z" 2023-09-29T23:23:38.546 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:38.547 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:38Z" } 2023-09-29T23:23:38.547 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:38.547 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:38.547 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:38.547 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:43.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:43.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:43.549 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:43.596 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:43.596 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:43.681 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:43.681 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:43.682 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:43.683 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:43.684 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:43.685 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:43.686 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:43Z" 2023-09-29T23:23:43.687 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:43.688 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:43Z" } 2023-09-29T23:23:43.689 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:43.689 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:43.689 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:43.689 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:48.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:48.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:48.691 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:48.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:48.740 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:48.826 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:48.826 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:48.827 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:48.828 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:48.829 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:48.830 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:48.831 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:48.832 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:48.833 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:48.833 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:48.833 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:48.833 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:48.833 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:48.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:48.834 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:48.834 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:48.834 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:48.834 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:48.835 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:48.835 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:48.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:48.835 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:48.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:48.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:48.836 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:48.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:48.836 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:48.837 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:48.837 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:48.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:48.837 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:48Z" 2023-09-29T23:23:48.837 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:48.839 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:48Z" } 2023-09-29T23:23:48.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:48.840 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:48.840 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:48.840 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:53.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:53.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:53.842 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:53.890 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:53.891 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:53.978 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:53.978 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:53.979 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:53.979 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:53.979 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:53.979 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:53.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:53.979 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:53.980 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:53.981 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:53.982 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:53.983 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:53.984 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:53.985 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:53.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:53.986 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:53Z" 2023-09-29T23:23:53.986 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:53.987 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:53Z" } 2023-09-29T23:23:53.987 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:53.988 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:53.988 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:53.988 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:23:58.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:58.988 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:23:58.989 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:23:59.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:23:59.037 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:23:59.124 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:59.125 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:23:59.126 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:59.127 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:59.128 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:23:59.129 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:23:59.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:23:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:23:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:23:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:23:59.131 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:23:59Z" 2023-09-29T23:23:59.131 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:23:59.132 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:23:59Z" } 2023-09-29T23:23:59.132 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:23:59.132 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:59.132 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:23:59.133 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:24:04.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:04.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:04.134 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:04.182 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:04.183 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:04.269 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:04.269 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:04.270 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051529", 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:19:38.549429Z" 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:04.271 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:04.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045657", 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.660161Z" 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:04.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:24:04.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:04Z" 2023-09-29T23:24:04.275 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:04.276 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051529", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:19:38.549429Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045657", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:17:32.660161Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:24:04Z" } 2023-09-29T23:24:04.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051529', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:19:38.549429Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045657', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.660161Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:24:04.277 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:24:04.277 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051529', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045657', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:24:04.277 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051669 target=00000000000000000000:00000000000000051529, shard=2 master=00000000000000000000:00000000000000045797 target=00000000000000000000:00000000000000045657 2023-09-29T23:24:06.496 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T23:24:06.450+0000 7efd3a414700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:24:09.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:09.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:09.278 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:09.326 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:09.326 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:09.416 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:09.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051669", 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.249921Z" 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:09.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:09.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:09.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.270925Z" 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:09.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:24:09.422 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:24:09.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:09.423 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:09Z" 2023-09-29T23:24:09.423 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:09.424 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051669", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:22:05.249921Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:05.270925Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:24:09Z" } 2023-09-29T23:24:09.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051669', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.249921Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.270925Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:24:09.424 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:24:09.424 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:24:09.424 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:24:09.425 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:24:09.425 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:24:09.425 DEBUG:boto:Method: HEAD 2023-09-29T23:24:09.425 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:24:09.425 DEBUG:boto:Data: 2023-09-29T23:24:09.425 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.425 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.425 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.425 DEBUG:boto:Params: {} 2023-09-29T23:24:09.426 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:24:09.426 DEBUG:boto:Token: None 2023-09-29T23:24:09.426 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-70/ 2023-09-29T23:24:09.426 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nqfH0/5Mt4buTPcGYkeR+h5gfdA= 2023-09-29T23:24:09.427 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nqfH0/5Mt4buTPcGYkeR+h5gfdA=', 'Content-Length': '0'} 2023-09-29T23:24:09.435 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009ee202489482729e-0065175c99-2621-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.435 DEBUG:boto:path=/hcnphz-70/obj1 2023-09-29T23:24:09.436 DEBUG:boto:auth_path=/hcnphz-70/obj1 2023-09-29T23:24:09.436 DEBUG:boto:Method: HEAD 2023-09-29T23:24:09.436 DEBUG:boto:Path: /hcnphz-70/obj1 2023-09-29T23:24:09.436 DEBUG:boto:Data: 2023-09-29T23:24:09.436 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.436 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.436 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.436 DEBUG:boto:Params: {} 2023-09-29T23:24:09.437 DEBUG:boto:Token: None 2023-09-29T23:24:09.437 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-70/obj1 2023-09-29T23:24:09.437 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vujkUI4UNSzDwkmdkk4lKaSvPyM= 2023-09-29T23:24:09.437 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vujkUI4UNSzDwkmdkk4lKaSvPyM=', 'Content-Length': '0'} 2023-09-29T23:24:09.439 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000015406ad41b49b4ba-0065175c99-2621-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.440 DEBUG:boto:path=/hcnphz-70/obj2 2023-09-29T23:24:09.440 DEBUG:boto:auth_path=/hcnphz-70/obj2 2023-09-29T23:24:09.440 DEBUG:boto:Method: HEAD 2023-09-29T23:24:09.440 DEBUG:boto:Path: /hcnphz-70/obj2 2023-09-29T23:24:09.440 DEBUG:boto:Data: 2023-09-29T23:24:09.440 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.440 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.440 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.441 DEBUG:boto:Params: {} 2023-09-29T23:24:09.441 DEBUG:boto:Token: None 2023-09-29T23:24:09.441 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-70/obj2 2023-09-29T23:24:09.441 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mL/GNnSG/VtMlChw9iHyIJsn64Y= 2023-09-29T23:24:09.441 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mL/GNnSG/VtMlChw9iHyIJsn64Y=', 'Content-Length': '0'} 2023-09-29T23:24:09.443 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000002bc3699fa589d6ca-0065175c99-2621-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.444 DEBUG:boto:path=/hcnphz-71/ 2023-09-29T23:24:09.444 DEBUG:boto:auth_path=/hcnphz-71/ 2023-09-29T23:24:09.444 DEBUG:boto:Method: HEAD 2023-09-29T23:24:09.444 DEBUG:boto:Path: /hcnphz-71/ 2023-09-29T23:24:09.444 DEBUG:boto:Data: 2023-09-29T23:24:09.444 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.444 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.444 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.444 DEBUG:boto:Params: {} 2023-09-29T23:24:09.444 DEBUG:boto:Token: None 2023-09-29T23:24:09.445 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-71/ 2023-09-29T23:24:09.445 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1Q8CQ6SCHOWrzYUmzlnzw1+a7PU= 2023-09-29T23:24:09.445 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1Q8CQ6SCHOWrzYUmzlnzw1+a7PU=', 'Content-Length': '0'} 2023-09-29T23:24:09.449 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000908f47b09b1c6a57-0065175c99-2621-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.450 DEBUG:boto:path=/hcnphz-71/obj1 2023-09-29T23:24:09.450 DEBUG:boto:auth_path=/hcnphz-71/obj1 2023-09-29T23:24:09.450 DEBUG:boto:Method: HEAD 2023-09-29T23:24:09.450 DEBUG:boto:Path: /hcnphz-71/obj1 2023-09-29T23:24:09.450 DEBUG:boto:Data: 2023-09-29T23:24:09.450 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.450 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.450 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.450 DEBUG:boto:Params: {} 2023-09-29T23:24:09.451 DEBUG:boto:Token: None 2023-09-29T23:24:09.451 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-71/obj1 2023-09-29T23:24:09.451 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:27MtGKA2WnXycPLlku9v/w+RTxA= 2023-09-29T23:24:09.451 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:27MtGKA2WnXycPLlku9v/w+RTxA=', 'Content-Length': '0'} 2023-09-29T23:24:09.454 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b30e8db0b4b94d05-0065175c99-2621-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.455 DEBUG:boto:path=/hcnphz-71/obj1 2023-09-29T23:24:09.455 DEBUG:boto:auth_path=/hcnphz-71/obj1 2023-09-29T23:24:09.455 DEBUG:boto:Method: GET 2023-09-29T23:24:09.455 DEBUG:boto:Path: /hcnphz-71/obj1 2023-09-29T23:24:09.455 DEBUG:boto:Data: 2023-09-29T23:24:09.455 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.455 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.455 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.455 DEBUG:boto:Params: {} 2023-09-29T23:24:09.456 DEBUG:boto:Token: None 2023-09-29T23:24:09.456 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-71/obj1 2023-09-29T23:24:09.456 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PQud1mVzAnosH5yZAs2RPkvPss0= 2023-09-29T23:24:09.456 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PQud1mVzAnosH5yZAs2RPkvPss0=', 'Content-Length': '0'} 2023-09-29T23:24:09.459 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c66d3f6ec30d3d76-0065175c99-2621-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.502 DEBUG:boto:path=/hcnphz-71/obj2 2023-09-29T23:24:09.503 DEBUG:boto:auth_path=/hcnphz-71/obj2 2023-09-29T23:24:09.503 DEBUG:boto:Method: HEAD 2023-09-29T23:24:09.503 DEBUG:boto:Path: /hcnphz-71/obj2 2023-09-29T23:24:09.503 DEBUG:boto:Data: 2023-09-29T23:24:09.503 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.503 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.503 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.503 DEBUG:boto:Params: {} 2023-09-29T23:24:09.504 DEBUG:boto:Token: None 2023-09-29T23:24:09.504 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-71/obj2 2023-09-29T23:24:09.504 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/S9gDIRGAZGIEdr+ysjNKII53NE= 2023-09-29T23:24:09.504 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/S9gDIRGAZGIEdr+ysjNKII53NE=', 'Content-Length': '0'} 2023-09-29T23:24:09.507 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000363980d5998d3921-0065175c99-2621-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.507 DEBUG:boto:path=/hcnphz-71/obj2 2023-09-29T23:24:09.507 DEBUG:boto:auth_path=/hcnphz-71/obj2 2023-09-29T23:24:09.507 DEBUG:boto:Method: GET 2023-09-29T23:24:09.508 DEBUG:boto:Path: /hcnphz-71/obj2 2023-09-29T23:24:09.508 DEBUG:boto:Data: 2023-09-29T23:24:09.508 DEBUG:boto:Headers: {} 2023-09-29T23:24:09.508 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:24:09.508 DEBUG:boto:Port: 8001 2023-09-29T23:24:09.508 DEBUG:boto:Params: {} 2023-09-29T23:24:09.508 DEBUG:boto:Token: None 2023-09-29T23:24:09.508 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:24:09 GMT /hcnphz-71/obj2 2023-09-29T23:24:09.509 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:gMLqgJNz8Nd9pN1ilV2o+Q/1low= 2023-09-29T23:24:09.509 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:24:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gMLqgJNz8Nd9pN1ilV2o+Q/1low=', 'Content-Length': '0'} 2023-09-29T23:24:09.512 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:22:05 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f302242fd726d146-0065175c99-2621-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:24:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:24:09.512 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.512 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:09.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:09.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:09.654 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:24:09.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.655 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:09.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:09.703 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:09.800 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:24:09.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.801 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:09.844 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:09.845 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:09.935 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:24:09.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:09.936 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:09.979 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:09.979 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:10.090 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:10.090 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:10.090 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 19, 2023-09-29T23:24:10.090 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:24:10.090 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:24:10.091 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:24:10.092 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:24:10.093 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:24:10.094 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:24:10.095 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.096 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.097 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:24:10.098 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:24:10.099 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:24:10.100 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.101 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.102 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:24:10.103 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:24:10.104 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:24:10.105 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:24:10.106 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:24:10.107 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:24:10.108 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:24:10.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:24:10.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:24:10.111 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:24:10.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:24:10.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:24:10.115 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.116 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:24:10.117 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:24:10.118 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:24:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:24:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:24:10.119 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:10.119 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:10.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.121 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:10.161 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:10.253 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:24:10.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.254 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:10.349 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:10.350 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:10.470 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:24:10.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.471 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:10.539 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:10.539 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:10.642 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:24:10.642 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-70', '--source-bucket=*', '--dest-bucket=hcnphz-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipeA', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-70', '--source-bucket=*', '--dest-bucket=hcnphz-71', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.643 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipeA --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hcnphz-70 '--source-bucket=*' --dest-bucket=hcnphz-71 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:10.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:10.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:10.785 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"hcnphz-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:24:10.786 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.786 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:10.833 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:10.833 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:10.915 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"hcnphz-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:24:10.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:24:10.916 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:24:10.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:10.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:11.037 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipeA","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"hcnphz-71","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:24:11.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:11.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:11.038 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:11.075 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:11.076 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:11.132 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:24:11.161 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:24:10.896029Z" 2023-09-29T23:24:11.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:11.163 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:24:11.164 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:11.165 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:24:11.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:11.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:11.165 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:11.249 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:11.249 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:11.308 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:11.307+0000 7f303bf19f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:11.309 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:11.311+0000 7f303bf19f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:11.336 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:11.336 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:11.336 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.337 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:11.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:11.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:11.340 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.341 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:11.342 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:11.343 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:11Z" 2023-09-29T23:24:11.344 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:11.345 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:11Z" } 2023-09-29T23:24:11.345 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:11.345 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:11.345 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:11.345 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:16.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:16.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:16.347 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:16.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:16.393 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:16.451 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:16.451+0000 7f7849a96f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:16.453 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:16.451+0000 7f7849a96f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:16.494 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:16.494 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:16.495 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:16.496 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:16.497 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:16.498 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:16.499 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:16.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:16Z" 2023-09-29T23:24:16.501 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:16.502 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:16Z" } 2023-09-29T23:24:16.502 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:16.503 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:16.503 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:16.503 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:21.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:21.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:21.504 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:21.548 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:21.549 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:21.607 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:21.607+0000 7f33595e5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:21.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:21.607+0000 7f33595e5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:21.636 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:21.636 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:21.636 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:21.636 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:21.637 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:21.638 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:21.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:21.640 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:21.641 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:21.642 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:21Z" 2023-09-29T23:24:21.643 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:21.643 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:21Z" } 2023-09-29T23:24:21.644 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:21.644 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:21.644 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:21.644 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:26.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:26.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:26.646 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:26.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:26.694 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:26.744 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:26.742+0000 7f90fedb4f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:26.746 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:26.746+0000 7f90fedb4f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:26.771 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:26.772 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:26.773 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:26.774 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:26.775 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:26.776 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:26.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:26.778 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:26.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:26.780 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:26Z" 2023-09-29T23:24:26.780 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:26.781 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:26Z" } 2023-09-29T23:24:26.781 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:26.782 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:26.782 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:26.782 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:31.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:31.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:31.783 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:31.828 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:31.829 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:31.886 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:31.886+0000 7f7c5590ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:31.887 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:31.886+0000 7f7c5590ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:31.914 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:31.915 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:31.916 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:31.917 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:31.918 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:31.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:31.920 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:31Z" 2023-09-29T23:24:31.921 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:31.922 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:31Z" } 2023-09-29T23:24:31.923 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:31.923 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:31.923 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:31.923 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:36.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:36.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:36.925 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:36.973 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:36.974 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:37.031 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:37.030+0000 7f3a1cb67f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:37.032 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:37.030+0000 7f3a1cb67f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:37.060 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:37.060 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:37.060 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:37.060 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:37.061 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:37.062 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:37.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:37.064 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:37.065 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:37.066 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:37.067 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:37Z" 2023-09-29T23:24:37.067 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:37.068 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:37Z" } 2023-09-29T23:24:37.068 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:37.069 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:37.069 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:37.069 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:42.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:42.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:42.070 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:42.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:42.117 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:42.178 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:42.178+0000 7f7773d1af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:42.180 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:42.178+0000 7f7773d1af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:42.207 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:42.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:42.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:42.210 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:42.211 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:42.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:42.213 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:42Z" 2023-09-29T23:24:42.214 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:42.215 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:42Z" } 2023-09-29T23:24:42.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:42.216 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:42.216 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:42.216 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:47.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:47.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:47.218 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:47.265 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:47.266 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:47.325 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:47.322+0000 7f3a2d77bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:47.326 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:47.326+0000 7f3a2d77bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:47.350 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:47.350 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:47.350 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:47.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:47.352 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:47.353 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:47.354 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:47.355 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:47.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:47.357 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:47.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:47.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:47.360 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:47Z" 2023-09-29T23:24:47.360 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:47.361 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:47Z" } 2023-09-29T23:24:47.361 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:47.362 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:47.362 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:47.362 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:52.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:52.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:52.364 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:52.409 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:52.409 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:52.465 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:52.462+0000 7fe9a2a78f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:52.466 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:52.466+0000 7fe9a2a78f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:52.497 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:52.498 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:52.499 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:52.500 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:52.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:52.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:52Z" 2023-09-29T23:24:52.503 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:52.504 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:52Z" } 2023-09-29T23:24:52.505 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:52.505 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:52.505 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:52.505 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:24:57.506 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:57.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:24:57.507 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:24:57.553 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:24:57.553 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:24:57.608 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:57.606+0000 7f7689117f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:57.609 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:24:57.610+0000 7f7689117f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:24:57.636 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:24:57.637 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:24:57.637 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:24:57.637 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:24:57.637 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:24:57.637 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:24:57.637 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:24:57.638 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:57.639 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:57.640 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:57.641 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:24:57.642 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:24:57.643 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:24:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:24:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:24:57.644 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:24:57Z" 2023-09-29T23:24:57.644 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:24:57.645 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:24:57Z" } 2023-09-29T23:24:57.645 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:24:57.645 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:24:57.646 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:24:57.646 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:02.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:02.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:02.647 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:02.693 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:02.750 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:02.750+0000 7efdcba1df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:02.752 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:02.750+0000 7efdcba1df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:02.778 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:02.779 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:02.779 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:02.779 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:02.779 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:02.779 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:02.779 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:02.780 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:02.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:02.782 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:02.783 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:02.784 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:02.785 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:02.786 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:02.786 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:02.786 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:02.786 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:02.786 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:02Z" 2023-09-29T23:25:02.786 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:02.787 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:02Z" } 2023-09-29T23:25:02.787 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:02.788 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:02.788 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:02.788 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:07.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:07.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:07.790 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:07.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:07.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:07.896 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:07.893+0000 7f34bda83f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:07.897 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:07.897+0000 7f34bda83f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:07.925 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:07.926 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:07.927 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:07.928 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:07.929 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:07.930 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:07.931 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:07.932 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:07.932 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:07.932 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:07Z" 2023-09-29T23:25:07.932 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:07.933 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:07Z" } 2023-09-29T23:25:07.933 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:07.933 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:07.933 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:07.933 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:12.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:12.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:12.935 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:12.982 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:12.983 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:13.042 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:13.041+0000 7ffac8ccff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:13.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:13.041+0000 7ffac8ccff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:13.071 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:13.071 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:13.072 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:13.073 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:13.074 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:13.075 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:13.076 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:13.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:13.078 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:13Z" 2023-09-29T23:25:13.078 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:13.079 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:13Z" } 2023-09-29T23:25:13.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:13.079 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:13.080 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:13.080 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:18.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:18.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:18.081 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:18.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:18.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:18.188 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:18.185+0000 7f1749002f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:18.190 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:18.189+0000 7f1749002f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:18.215 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:18.215 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:18.216 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:18.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:18.218 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:18.219 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:18.220 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:18.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:18.222 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:18Z" 2023-09-29T23:25:18.222 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:18.223 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:18Z" } 2023-09-29T23:25:18.223 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:18.223 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:18.223 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:18.224 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:23.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:23.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:23.225 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:23.271 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:23.272 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:23.327 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:23.329+0000 7efd5dcddf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:23.328 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:23.329+0000 7efd5dcddf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:23.355 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:23.356 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:23.357 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:23.358 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:23.359 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:23.360 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:23.361 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:23Z" 2023-09-29T23:25:23.362 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:23.363 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:23Z" } 2023-09-29T23:25:23.364 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:23.364 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:23.364 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:23.364 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:28.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:28.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:28.366 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:28.412 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:28.412 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:28.472 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:28.473+0000 7fb3f927ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:28.473 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:28.477+0000 7fb3f927ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:28.499 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:28.499 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:28.499 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:28.499 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:28.499 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:28.500 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:28.501 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:28.502 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:28.503 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:28.504 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:28.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:28.506 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:28Z" 2023-09-29T23:25:28.506 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:28.507 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:28Z" } 2023-09-29T23:25:28.508 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:28.508 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:28.508 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:28.508 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:33.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:33.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:33.509 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:33.555 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:33.556 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:33.613 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:33.613+0000 7f16cba53f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:33.615 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:33.617+0000 7f16cba53f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:33.641 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:33.641 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:33.642 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:33.642 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:33.642 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:33.642 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:33.643 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:33.643 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:33.643 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:33.643 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:33.644 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:33.644 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:33.644 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:33.644 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:33.644 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:33.645 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:33.645 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:33.645 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:33.645 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:33.645 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:33.646 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:33.646 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:33.646 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:33.646 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:33.647 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:33.647 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:33.647 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:33.647 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:33.647 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:33.648 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:33.648 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:33.648 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:33.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:33.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:33.649 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:33.649 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:33.649 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:33.649 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:33.649 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:33.650 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:33.650 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:33.650 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:33.650 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:33.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:33.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:33.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:33.651 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:33.653 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:33.653 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:33.653 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:33.653 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:33.653 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:33.654 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:33.654 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:33.654 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:33.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:33.654 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:33.655 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:33.655 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:33.655 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:33.655 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:33.655 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:33.656 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:33.656 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:33Z" 2023-09-29T23:25:33.656 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:33.657 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:33Z" } 2023-09-29T23:25:33.658 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:33.658 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:33.658 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:33.658 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:38.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:38.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:38.659 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:38.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:38.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:38.759 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:38.761+0000 7fd2b3d49f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:38.761+0000 7fd2b3d49f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:38.788 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:38.788 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:38.789 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:38.790 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:38.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:38.792 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:38.793 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:38.794 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:38.795 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:38Z" 2023-09-29T23:25:38.795 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:38.796 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:38Z" } 2023-09-29T23:25:38.796 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:38.796 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:38.796 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:38.796 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:43.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:43.798 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:43.798 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:43.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:43.846 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:43.903 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:43.905+0000 7f6a487c8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:43.905 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:43.905+0000 7f6a487c8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:43.933 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:43.933 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:43.934 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:43.935 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:43.936 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:43.937 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:43.938 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:43.939 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:43.940 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:43Z" 2023-09-29T23:25:43.940 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:43.941 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:43Z" } 2023-09-29T23:25:43.941 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:43.941 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:43.941 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:43.942 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:48.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:48.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:48.943 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:48.988 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:48.988 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:49.044 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:49.044+0000 7f1e480b5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:49.046 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:49.048+0000 7f1e480b5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:49.071 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:49.072 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:49.073 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:49.074 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:49.075 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:49.076 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:49.077 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:49.078 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:49Z" 2023-09-29T23:25:49.078 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:49.079 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:49Z" } 2023-09-29T23:25:49.079 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:49.079 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:49.079 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:49.080 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:54.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:54.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:54.081 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:54.127 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:54.128 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:54.191 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:54.192+0000 7f7c82006f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:54.192 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:54.192+0000 7f7c82006f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:54.219 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:54.219 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:54.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:54.221 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:54.222 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:54.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:54.224 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:54.225 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:54.226 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:54.227 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:54Z" 2023-09-29T23:25:54.228 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:54.229 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:54Z" } 2023-09-29T23:25:54.230 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:54.230 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:54.230 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:54.230 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:25:59.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:59.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:25:59.231 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:25:59.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:25:59.276 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:25:59.332 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:59.332+0000 7f8fc08f2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:59.334 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:25:59.336+0000 7f8fc08f2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:25:59.361 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:25:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:25:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:25:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:25:59.361 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:59.362 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:59.363 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:59.364 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:59.365 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:25:59.366 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:25:59.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:25:59.368 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:25:59Z" 2023-09-29T23:25:59.368 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:25:59.369 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:25:59Z" } 2023-09-29T23:25:59.369 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:25:59.369 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:25:59.369 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:25:59.369 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:26:04.370 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:04.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:04.370 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:04.417 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:04.417 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:04.475 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:04.476+0000 7f656dcd2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:04.476 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:04.476+0000 7f656dcd2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:04.504 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:04.504 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:04.505 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:04.506 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:04.507 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:04.508 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:04.509 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:04.510 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:04.511 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:04Z" 2023-09-29T23:26:04.511 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:04.512 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:04Z" } 2023-09-29T23:26:04.512 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:04.512 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:04.512 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:26:04.512 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:26:09.513 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:09.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:09.514 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:09.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:09.561 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:09.618 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:09.620+0000 7ff385b61f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:09.620 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:09.620+0000 7ff385b61f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:09.647 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:09.647 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:09.648 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:09.649 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029723.971870_790.1", 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:09.650 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:03.971870Z", 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:09.651 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:09.652 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:09.653 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:09Z" 2023-09-29T23:26:09.654 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:09.655 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029723.971870_790.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:03.971870Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:09Z" } 2023-09-29T23:26:09.655 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029723.971870_790.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:03.971870Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:09.656 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:09.656 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029723.971870_790.1', 2: '', 3: ''} 2023-09-29T23:26:09.656 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029850.896029_818.1 target=1_1696029723.971870_790.1 2023-09-29T23:26:11.900 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:26:11.896+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:26:14.657 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:14.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:14.657 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:14.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:14.702 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:14.761 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:14.764+0000 7fdab35eff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:14.763 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:14.764+0000 7fdab35eff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:14.789 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:14.789 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:14.789 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:14.789 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:14.789 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:14.790 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:14.791 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:14.792 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:24:10.896029Z", 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:14.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:14.794 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:14.795 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:14.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:14.797 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:14.798 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:14Z" 2023-09-29T23:26:14.798 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:14.799 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029850.896029_818.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:24:10.896029Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:14Z" } 2023-09-29T23:26:14.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029850.896029_818.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:24:10.896029Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:14.799 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:14.799 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:14.799 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:26:14.800 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:26:14.800 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:26:14.800 DEBUG:boto:Method: HEAD 2023-09-29T23:26:14.800 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:26:14.800 DEBUG:boto:Data: 2023-09-29T23:26:14.800 DEBUG:boto:Headers: {} 2023-09-29T23:26:14.800 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:26:14.801 DEBUG:boto:Port: 8000 2023-09-29T23:26:14.801 DEBUG:boto:Params: {} 2023-09-29T23:26:14.801 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:26:14.801 DEBUG:boto:Token: None 2023-09-29T23:26:14.801 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:14 GMT /hcnphz-70/ 2023-09-29T23:26:14.802 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j5/mBiTruDXzM4EsXj6A+H6A2Ik= 2023-09-29T23:26:14.802 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j5/mBiTruDXzM4EsXj6A+H6A2Ik=', 'Content-Length': '0'} 2023-09-29T23:26:14.811 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dff300639574bf1c-0065175d16-272d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:14.812 DEBUG:boto:path=/hcnphz-70/obj5 2023-09-29T23:26:14.812 DEBUG:boto:auth_path=/hcnphz-70/obj5 2023-09-29T23:26:14.812 DEBUG:boto:Method: PUT 2023-09-29T23:26:14.812 DEBUG:boto:Path: /hcnphz-70/obj5 2023-09-29T23:26:14.812 DEBUG:boto:Data: 2023-09-29T23:26:14.812 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:26:14.812 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:26:14.813 DEBUG:boto:Port: 8000 2023-09-29T23:26:14.813 DEBUG:boto:Params: {} 2023-09-29T23:26:14.813 DEBUG:boto:Token: None 2023-09-29T23:26:14.813 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:26:14 GMT /hcnphz-70/obj5 2023-09-29T23:26:14.813 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nQ4X/tjDCc1GxYklis6M/OxmEbQ= 2023-09-29T23:26:14.813 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:26:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nQ4X/tjDCc1GxYklis6M/OxmEbQ='} 2023-09-29T23:26:14.830 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000dc67194b77c264e3-0065175d16-272d-a1'), ('Date', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:14.830 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:26:14.831 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:26:14.831 DEBUG:boto:Method: HEAD 2023-09-29T23:26:14.831 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:26:14.831 DEBUG:boto:Data: 2023-09-29T23:26:14.831 DEBUG:boto:Headers: {} 2023-09-29T23:26:14.831 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:26:14.831 DEBUG:boto:Port: 8000 2023-09-29T23:26:14.831 DEBUG:boto:Params: {} 2023-09-29T23:26:14.832 DEBUG:boto:Token: None 2023-09-29T23:26:14.832 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:14 GMT /hcnphz-70/ 2023-09-29T23:26:14.832 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:j5/mBiTruDXzM4EsXj6A+H6A2Ik= 2023-09-29T23:26:14.832 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:j5/mBiTruDXzM4EsXj6A+H6A2Ik=', 'Content-Length': '0'} 2023-09-29T23:26:14.835 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097ec446dd152792c-0065175d16-272d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:14.835 DEBUG:boto:path=/hcnphz-70/obj6 2023-09-29T23:26:14.835 DEBUG:boto:auth_path=/hcnphz-70/obj6 2023-09-29T23:26:14.835 DEBUG:boto:Method: PUT 2023-09-29T23:26:14.836 DEBUG:boto:Path: /hcnphz-70/obj6 2023-09-29T23:26:14.836 DEBUG:boto:Data: 2023-09-29T23:26:14.836 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:26:14.836 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:26:14.836 DEBUG:boto:Port: 8000 2023-09-29T23:26:14.836 DEBUG:boto:Params: {} 2023-09-29T23:26:14.836 DEBUG:boto:Token: None 2023-09-29T23:26:14.837 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:26:14 GMT /hcnphz-70/obj6 2023-09-29T23:26:14.837 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HBIG3gLcLDbRSjoVJ+Zt7lqJARU= 2023-09-29T23:26:14.837 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:26:14 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HBIG3gLcLDbRSjoVJ+Zt7lqJARU='} 2023-09-29T23:26:14.883 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000019f7a8b8bcedccc3-0065175d16-272d-a1'), ('Date', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:14.883 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:14.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:14.884 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:14.928 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:14.929 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:14.990 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:26:15.017 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:26:15.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:24:10.896029Z" 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.019 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:26:15.020 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:15.020 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:26:15.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:15.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:15.021 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:15.104 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:15.105 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:15.165 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:15.164+0000 7f95d3fbef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:15.166 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:15.168+0000 7f95d3fbef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:15.191 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.192 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:15.193 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.194 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:24:10.896029Z", 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:15.195 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:15.196 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.197 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:15Z" 2023-09-29T23:26:15.198 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:15.199 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029850.896029_818.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:24:10.896029Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:15Z" } 2023-09-29T23:26:15.199 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029850.896029_818.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:24:10.896029Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:15.199 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:15.200 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:15.200 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:26:15.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:15.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:15.200 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:15.243 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:15.243 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:15.343 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:26:15.343 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.343 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:26:15.343 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:26:15.343 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.344 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.345 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:26:15.345 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:26:15.345 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.345 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:26:15.346 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:15.346 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:26:15.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:15.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:15.347 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:15.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:15.435 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:26:15.524 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051669", 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.525 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.249921Z" 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:26:15.526 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.527 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.270925Z" 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.528 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:26:15.529 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:15Z" 2023-09-29T23:26:15.530 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:15.531 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051669", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:22:05.249921Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:05.270925Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:26:15Z" } 2023-09-29T23:26:15.532 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051669', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.249921Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.270925Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:26:15.532 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:15.532 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:15.532 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051809 target=00000000000000000000:00000000000000051669, shard=2 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797 2023-09-29T23:26:20.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:20.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:20.533 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:20.581 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:20.582 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:20.669 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:20.670 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:20.670 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:20.670 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:20.670 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:20.670 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:26:20.670 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051669", 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:26:20.671 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.249921Z" 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:20.672 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:20.673 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.270925Z" 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:20.674 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:20.675 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:20.676 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:20Z" 2023-09-29T23:26:20.677 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:20.678 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051669", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:22:05.249921Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:05.270925Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:26:20Z" } 2023-09-29T23:26:20.678 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051669', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.249921Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.270925Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:26:20.679 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:20.679 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:20.679 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051809 target=00000000000000000000:00000000000000051669, shard=2 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797 2023-09-29T23:26:25.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:25.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:25.680 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:25.729 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:25.816 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:25.816 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051669", 2023-09-29T23:26:25.817 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.249921Z" 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:25.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:25.819 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.270925Z" 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:25.820 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:25.821 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:25Z" 2023-09-29T23:26:25.822 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:25.823 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051669", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:22:05.249921Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:05.270925Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:26:25Z" } 2023-09-29T23:26:25.823 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051669', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.249921Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.270925Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:26:25.823 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:25.824 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:25.824 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051809 target=00000000000000000000:00000000000000051669, shard=2 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797 2023-09-29T23:26:30.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:30.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:30.825 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:30.871 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:30.871 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:30.956 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:30.956 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:30.957 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051669", 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.249921Z" 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:30.958 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:30.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045797", 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:30.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:22:05.270925Z" 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:30.961 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:30.962 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:30Z" 2023-09-29T23:26:30.963 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:30.963 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051669", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:22:05.249921Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045797", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:22:05.270925Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:26:30Z" } 2023-09-29T23:26:30.964 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051669', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.249921Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045797', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:22:05.270925Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:26:30.964 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:30.964 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051669', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045797', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:30.964 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000051809 target=00000000000000000000:00000000000000051669, shard=2 master=00000000000000000000:00000000000000045937 target=00000000000000000000:00000000000000045797 2023-09-29T23:26:35.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:35.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:35.966 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:36.014 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:36.100 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:36.100 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:36.100 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:36.100 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:36.101 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.102 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045166", 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:17:32.739987Z" 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.103 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.104 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:36.105 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.106 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:36Z" 2023-09-29T23:26:36.107 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:36.107 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045166", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:17:32.739987Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031496", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:08:33.687356Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:26:36Z" } 2023-09-29T23:26:36.108 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045166', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:17:32.739987Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:26:36.108 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:36.108 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045166', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:26:36.108 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:26:36.108 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:26:36.108 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:26:36.109 DEBUG:boto:Method: HEAD 2023-09-29T23:26:36.109 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:26:36.109 DEBUG:boto:Data: 2023-09-29T23:26:36.109 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.109 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.109 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.109 DEBUG:boto:Params: {} 2023-09-29T23:26:36.109 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:26:36.110 DEBUG:boto:Token: None 2023-09-29T23:26:36.111 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-70/ 2023-09-29T23:26:36.111 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n0P388ZsFJlHZrzdOMWagSMvBC8= 2023-09-29T23:26:36.111 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n0P388ZsFJlHZrzdOMWagSMvBC8=', 'Content-Length': '0'} 2023-09-29T23:26:36.120 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e5ff7c0db8588274-0065175d2c-272b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.120 DEBUG:boto:path=/hcnphz-70/obj5 2023-09-29T23:26:36.121 DEBUG:boto:auth_path=/hcnphz-70/obj5 2023-09-29T23:26:36.121 DEBUG:boto:Method: HEAD 2023-09-29T23:26:36.121 DEBUG:boto:Path: /hcnphz-70/obj5 2023-09-29T23:26:36.121 DEBUG:boto:Data: 2023-09-29T23:26:36.121 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.121 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.121 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.121 DEBUG:boto:Params: {} 2023-09-29T23:26:36.122 DEBUG:boto:Token: None 2023-09-29T23:26:36.122 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-70/obj5 2023-09-29T23:26:36.122 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:h8XQUs7XGQKhlRqfTzhBrIpf6Eg= 2023-09-29T23:26:36.122 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:h8XQUs7XGQKhlRqfTzhBrIpf6Eg=', 'Content-Length': '0'} 2023-09-29T23:26:36.124 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000978186b91364036c-0065175d2c-272b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.124 DEBUG:boto:path=/hcnphz-70/obj6 2023-09-29T23:26:36.124 DEBUG:boto:auth_path=/hcnphz-70/obj6 2023-09-29T23:26:36.125 DEBUG:boto:Method: HEAD 2023-09-29T23:26:36.125 DEBUG:boto:Path: /hcnphz-70/obj6 2023-09-29T23:26:36.125 DEBUG:boto:Data: 2023-09-29T23:26:36.125 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.125 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.125 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.125 DEBUG:boto:Params: {} 2023-09-29T23:26:36.125 DEBUG:boto:Token: None 2023-09-29T23:26:36.126 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-70/obj6 2023-09-29T23:26:36.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:si/V/YFKaalg9YllQ4Tnc2Q9ZQY= 2023-09-29T23:26:36.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:si/V/YFKaalg9YllQ4Tnc2Q9ZQY=', 'Content-Length': '0'} 2023-09-29T23:26:36.128 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000046c10813baef9f4e-0065175d2c-272b-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.128 DEBUG:boto:path=/hcnphz-71/ 2023-09-29T23:26:36.128 DEBUG:boto:auth_path=/hcnphz-71/ 2023-09-29T23:26:36.128 DEBUG:boto:Method: HEAD 2023-09-29T23:26:36.128 DEBUG:boto:Path: /hcnphz-71/ 2023-09-29T23:26:36.129 DEBUG:boto:Data: 2023-09-29T23:26:36.129 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.129 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.129 DEBUG:boto:Params: {} 2023-09-29T23:26:36.129 DEBUG:boto:Token: None 2023-09-29T23:26:36.129 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-71/ 2023-09-29T23:26:36.129 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:AzAEKiopGZoSO/iCzEb2PYHiCSE= 2023-09-29T23:26:36.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AzAEKiopGZoSO/iCzEb2PYHiCSE=', 'Content-Length': '0'} 2023-09-29T23:26:36.133 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bb1e0776f482da85-0065175d2c-272b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.133 DEBUG:boto:path=/hcnphz-71/obj5 2023-09-29T23:26:36.134 DEBUG:boto:auth_path=/hcnphz-71/obj5 2023-09-29T23:26:36.134 DEBUG:boto:Method: HEAD 2023-09-29T23:26:36.134 DEBUG:boto:Path: /hcnphz-71/obj5 2023-09-29T23:26:36.134 DEBUG:boto:Data: 2023-09-29T23:26:36.134 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.134 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.134 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.134 DEBUG:boto:Params: {} 2023-09-29T23:26:36.135 DEBUG:boto:Token: None 2023-09-29T23:26:36.135 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-71/obj5 2023-09-29T23:26:36.135 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SI64MB8iOvxZTilj+HiTtLOBPsw= 2023-09-29T23:26:36.135 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SI64MB8iOvxZTilj+HiTtLOBPsw=', 'Content-Length': '0'} 2023-09-29T23:26:36.138 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000761515c6ec42375-0065175d2c-272b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.138 DEBUG:boto:path=/hcnphz-71/obj5 2023-09-29T23:26:36.138 DEBUG:boto:auth_path=/hcnphz-71/obj5 2023-09-29T23:26:36.139 DEBUG:boto:Method: GET 2023-09-29T23:26:36.139 DEBUG:boto:Path: /hcnphz-71/obj5 2023-09-29T23:26:36.139 DEBUG:boto:Data: 2023-09-29T23:26:36.139 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.139 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.140 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.140 DEBUG:boto:Params: {} 2023-09-29T23:26:36.140 DEBUG:boto:Token: None 2023-09-29T23:26:36.140 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-71/obj5 2023-09-29T23:26:36.140 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:R2oJxhygO+ZWrfsWyKuhlepKqC0= 2023-09-29T23:26:36.141 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:R2oJxhygO+ZWrfsWyKuhlepKqC0=', 'Content-Length': '0'} 2023-09-29T23:26:36.143 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000db82b48ed630cffc-0065175d2c-272b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.143 DEBUG:boto:path=/hcnphz-71/obj6 2023-09-29T23:26:36.143 DEBUG:boto:auth_path=/hcnphz-71/obj6 2023-09-29T23:26:36.144 DEBUG:boto:Method: HEAD 2023-09-29T23:26:36.144 DEBUG:boto:Path: /hcnphz-71/obj6 2023-09-29T23:26:36.144 DEBUG:boto:Data: 2023-09-29T23:26:36.144 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.144 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.144 DEBUG:boto:Params: {} 2023-09-29T23:26:36.144 DEBUG:boto:Token: None 2023-09-29T23:26:36.144 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-71/obj6 2023-09-29T23:26:36.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:X3aw2/QoZkYSmLxseBtTvG24EI4= 2023-09-29T23:26:36.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X3aw2/QoZkYSmLxseBtTvG24EI4=', 'Content-Length': '0'} 2023-09-29T23:26:36.147 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002cdfd88d8485b2ba-0065175d2c-272b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.147 DEBUG:boto:path=/hcnphz-71/obj6 2023-09-29T23:26:36.147 DEBUG:boto:auth_path=/hcnphz-71/obj6 2023-09-29T23:26:36.148 DEBUG:boto:Method: GET 2023-09-29T23:26:36.148 DEBUG:boto:Path: /hcnphz-71/obj6 2023-09-29T23:26:36.148 DEBUG:boto:Data: 2023-09-29T23:26:36.148 DEBUG:boto:Headers: {} 2023-09-29T23:26:36.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:36.148 DEBUG:boto:Port: 8001 2023-09-29T23:26:36.148 DEBUG:boto:Params: {} 2023-09-29T23:26:36.148 DEBUG:boto:Token: None 2023-09-29T23:26:36.148 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:26:36 GMT /hcnphz-71/obj6 2023-09-29T23:26:36.149 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ufkc7U2BMdUilhoZZJsAbkB0dL0= 2023-09-29T23:26:36.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:36 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ufkc7U2BMdUilhoZZJsAbkB0dL0=', 'Content-Length': '0'} 2023-09-29T23:26:36.151 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:14 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-70:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000001bb99fec007e5c4e-0065175d2c-272b-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:26:36 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:36.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:36.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:36.195 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:36.244 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:36.244 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:36.357 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:26:36.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:36.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:36.358 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:36.402 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:36.462 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:26:36.488 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:26:36.488 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.488 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:36.488 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:26:36.488 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029996.317879_822.1", 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:36.317879Z" 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.489 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:36.490 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:26:36.490 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.490 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:26:36.491 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:36.491 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:26:36.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:36.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:36.492 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:36.536 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:36.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:36.597 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:36.595+0000 7f9cac6b0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:36.598 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:36.599+0000 7f9cac6b0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:36.622 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:36.623 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:36.623 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:36.623 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:36.623 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:36.623 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:36.623 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:36.624 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.625 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.626 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:24:10.896029Z", 2023-09-29T23:26:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.627 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:36.628 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:36.629 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:36.630 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:36.631 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:36.632 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:36Z" 2023-09-29T23:26:36.632 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:36.633 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029850.896029_818.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:24:10.896029Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:36Z" } 2023-09-29T23:26:36.634 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029850.896029_818.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:24:10.896029Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:36.634 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:36.634 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:36.634 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029996.317879_822.1 target=1_1696029850.896029_818.1 2023-09-29T23:26:41.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:41.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:41.636 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:41.681 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:41.740 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:41.739+0000 7f799a420f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:41.742 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:41.743+0000 7f799a420f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:41.769 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:41.769 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:41.769 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:41.769 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:41.769 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:41.770 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:41.771 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:24:10.896029Z", 2023-09-29T23:26:41.772 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:41.773 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:41.774 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:41.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:41.776 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:41Z" 2023-09-29T23:26:41.777 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:41.778 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029850.896029_818.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:24:10.896029Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:41Z" } 2023-09-29T23:26:41.778 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029850.896029_818.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:24:10.896029Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:41.778 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:41.778 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:41.779 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029996.317879_822.1 target=1_1696029850.896029_818.1 2023-09-29T23:26:46.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:46.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:46.780 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:46.824 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:46.825 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:46.876 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:46.875+0000 7f3ec10cbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:46.878 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:46.879+0000 7f3ec10cbf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:46.905 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:46.905 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:46.905 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:46.905 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:46.906 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:46.907 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:24:10.896029Z", 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:46.908 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:46.909 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:46.910 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:46.911 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:46.912 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:46.912 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:46Z" 2023-09-29T23:26:46.912 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:46.913 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029850.896029_818.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:24:10.896029Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:46Z" } 2023-09-29T23:26:46.913 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029850.896029_818.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:24:10.896029Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:46.913 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:46.913 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:46.913 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029996.317879_822.1 target=1_1696029850.896029_818.1 2023-09-29T23:26:51.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:51.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:51.914 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:51.960 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:51.961 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:52.023 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:52.023+0000 7f609aa5ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:52.025 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:52.023+0000 7f609aa5ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:52.053 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:52.054 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:52.055 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:52.056 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029850.896029_818.1", 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:24:10.896029Z", 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:52.057 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:52.058 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:52.059 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:52.060 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:52.061 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:52.061 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:52Z" 2023-09-29T23:26:52.061 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:52.062 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029850.896029_818.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:24:10.896029Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:52Z" } 2023-09-29T23:26:52.062 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029850.896029_818.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:24:10.896029Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:52.062 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:52.062 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029850.896029_818.1', 2: '', 3: ''} 2023-09-29T23:26:52.062 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696029996.317879_822.1 target=1_1696029850.896029_818.1 2023-09-29T23:26:57.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:57.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:57.063 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:57.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:57.108 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:57.177 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:57.175+0000 7fe6a6707f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:57.179 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:57.179+0000 7fe6a6707f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:57.206 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:57.207 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:57.208 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:57.209 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029996.317879_822.1", 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:57.210 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:36.317879Z", 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:57.211 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:57.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:57.213 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:57.214 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:57.215 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:57.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:57.215 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:57Z" 2023-09-29T23:26:57.215 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:57.216 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029996.317879_822.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:36.317879Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:57Z" } 2023-09-29T23:26:57.216 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029996.317879_822.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:36.317879Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:57.216 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:57.217 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:57.217 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:26:57.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.217 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:57.259 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:57.260 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:57.361 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:26:57.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.362 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:57.408 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:57.408 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:57.509 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:26:57.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.510 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:57.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:57.560 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:57.663 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:26:57.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-70', '--source-bucket=hcnphz-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--bucket=hcnphz-70', '--source-bucket=hcnphz-71', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.664 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --bucket=hcnphz-70 --source-bucket=hcnphz-71 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:57.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:57.713 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:57.827 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:26:57.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.827 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:57.873 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:57.874 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:57.973 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:26:57.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:26:57.974 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:26:58.016 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:58.016 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:58.118 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"hcnphz-71","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:26:58.118 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:26:58.118 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:26:58.118 DEBUG:boto:Method: HEAD 2023-09-29T23:26:58.119 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:26:58.119 DEBUG:boto:Data: 2023-09-29T23:26:58.119 DEBUG:boto:Headers: {} 2023-09-29T23:26:58.119 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.119 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.119 DEBUG:boto:Params: {} 2023-09-29T23:26:58.119 DEBUG:boto:Token: None 2023-09-29T23:26:58.119 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-70/ 2023-09-29T23:26:58.120 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pA+9KI8wFP4JqNTcA2AQRkCUiaI= 2023-09-29T23:26:58.120 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pA+9KI8wFP4JqNTcA2AQRkCUiaI=', 'Content-Length': '0'} 2023-09-29T23:26:58.123 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000480cce7be15e7458-0065175d42-272b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.124 DEBUG:boto:path=/hcnphz-70/obj7 2023-09-29T23:26:58.124 DEBUG:boto:auth_path=/hcnphz-70/obj7 2023-09-29T23:26:58.124 DEBUG:boto:Method: PUT 2023-09-29T23:26:58.124 DEBUG:boto:Path: /hcnphz-70/obj7 2023-09-29T23:26:58.124 DEBUG:boto:Data: 2023-09-29T23:26:58.124 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:26:58.124 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.124 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.125 DEBUG:boto:Params: {} 2023-09-29T23:26:58.125 DEBUG:boto:Token: None 2023-09-29T23:26:58.125 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-70/obj7 2023-09-29T23:26:58.125 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fZ0bBgfj+uXjdIe3fXsbBER+kcU= 2023-09-29T23:26:58.125 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fZ0bBgfj+uXjdIe3fXsbBER+kcU='} 2023-09-29T23:26:58.137 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000002832823e77404e18-0065175d42-272b-a2'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.137 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:26:58.137 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:26:58.138 DEBUG:boto:Method: HEAD 2023-09-29T23:26:58.138 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:26:58.138 DEBUG:boto:Data: 2023-09-29T23:26:58.138 DEBUG:boto:Headers: {} 2023-09-29T23:26:58.138 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.138 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.138 DEBUG:boto:Params: {} 2023-09-29T23:26:58.138 DEBUG:boto:Token: None 2023-09-29T23:26:58.139 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-70/ 2023-09-29T23:26:58.139 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pA+9KI8wFP4JqNTcA2AQRkCUiaI= 2023-09-29T23:26:58.139 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pA+9KI8wFP4JqNTcA2AQRkCUiaI=', 'Content-Length': '0'} 2023-09-29T23:26:58.143 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c551ebf0f56de18d-0065175d42-272b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.143 DEBUG:boto:path=/hcnphz-70/obj8 2023-09-29T23:26:58.143 DEBUG:boto:auth_path=/hcnphz-70/obj8 2023-09-29T23:26:58.144 DEBUG:boto:Method: PUT 2023-09-29T23:26:58.144 DEBUG:boto:Path: /hcnphz-70/obj8 2023-09-29T23:26:58.144 DEBUG:boto:Data: 2023-09-29T23:26:58.144 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:26:58.144 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.144 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.144 DEBUG:boto:Params: {} 2023-09-29T23:26:58.144 DEBUG:boto:Token: None 2023-09-29T23:26:58.145 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-70/obj8 2023-09-29T23:26:58.145 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:nnWyKxn/PnEJ9R3FElB0Sb3LhXc= 2023-09-29T23:26:58.145 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nnWyKxn/PnEJ9R3FElB0Sb3LhXc='} 2023-09-29T23:26:58.190 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000916cd91ebc9400ac-0065175d42-272b-a2'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.191 DEBUG:boto:path=/hcnphz-71/ 2023-09-29T23:26:58.191 DEBUG:boto:auth_path=/hcnphz-71/ 2023-09-29T23:26:58.191 DEBUG:boto:Method: HEAD 2023-09-29T23:26:58.191 DEBUG:boto:Path: /hcnphz-71/ 2023-09-29T23:26:58.191 DEBUG:boto:Data: 2023-09-29T23:26:58.191 DEBUG:boto:Headers: {} 2023-09-29T23:26:58.191 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.192 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.192 DEBUG:boto:Params: {} 2023-09-29T23:26:58.192 DEBUG:boto:Token: None 2023-09-29T23:26:58.192 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-71/ 2023-09-29T23:26:58.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ys/350Vva/UGqOCWcrRaWaf/8AA= 2023-09-29T23:26:58.192 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ys/350Vva/UGqOCWcrRaWaf/8AA=', 'Content-Length': '0'} 2023-09-29T23:26:58.196 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e95d8797d16f79cc-0065175d42-272b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.196 DEBUG:boto:path=/hcnphz-71/obj9 2023-09-29T23:26:58.196 DEBUG:boto:auth_path=/hcnphz-71/obj9 2023-09-29T23:26:58.197 DEBUG:boto:Method: PUT 2023-09-29T23:26:58.197 DEBUG:boto:Path: /hcnphz-71/obj9 2023-09-29T23:26:58.197 DEBUG:boto:Data: 2023-09-29T23:26:58.197 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:26:58.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.197 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.197 DEBUG:boto:Params: {} 2023-09-29T23:26:58.197 DEBUG:boto:Token: None 2023-09-29T23:26:58.198 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-71/obj9 2023-09-29T23:26:58.198 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:TkWm2zedQiR7j8P9MnF6pZFFy3A= 2023-09-29T23:26:58.198 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TkWm2zedQiR7j8P9MnF6pZFFy3A='} 2023-09-29T23:26:58.210 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004e8f6daa9a9549f6-0065175d42-272b-a2'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.210 DEBUG:boto:path=/hcnphz-71/ 2023-09-29T23:26:58.211 DEBUG:boto:auth_path=/hcnphz-71/ 2023-09-29T23:26:58.211 DEBUG:boto:Method: HEAD 2023-09-29T23:26:58.211 DEBUG:boto:Path: /hcnphz-71/ 2023-09-29T23:26:58.211 DEBUG:boto:Data: 2023-09-29T23:26:58.211 DEBUG:boto:Headers: {} 2023-09-29T23:26:58.211 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.211 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.211 DEBUG:boto:Params: {} 2023-09-29T23:26:58.211 DEBUG:boto:Token: None 2023-09-29T23:26:58.212 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-71/ 2023-09-29T23:26:58.212 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ys/350Vva/UGqOCWcrRaWaf/8AA= 2023-09-29T23:26:58.212 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ys/350Vva/UGqOCWcrRaWaf/8AA=', 'Content-Length': '0'} 2023-09-29T23:26:58.215 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076b64bf41991928a-0065175d42-272b-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.216 DEBUG:boto:path=/hcnphz-71/obj10 2023-09-29T23:26:58.216 DEBUG:boto:auth_path=/hcnphz-71/obj10 2023-09-29T23:26:58.216 DEBUG:boto:Method: PUT 2023-09-29T23:26:58.216 DEBUG:boto:Path: /hcnphz-71/obj10 2023-09-29T23:26:58.216 DEBUG:boto:Data: 2023-09-29T23:26:58.216 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:26:58.216 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:26:58.217 DEBUG:boto:Port: 8001 2023-09-29T23:26:58.217 DEBUG:boto:Params: {} 2023-09-29T23:26:58.217 DEBUG:boto:Token: None 2023-09-29T23:26:58.217 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:26:58 GMT /hcnphz-71/obj10 2023-09-29T23:26:58.217 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pCUGI66XaXUBpd4Xc3zuIgJWLME= 2023-09-29T23:26:58.217 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:26:58 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pCUGI66XaXUBpd4Xc3zuIgJWLME='} 2023-09-29T23:26:58.262 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006992de982c5dd478-0065175d42-272b-a2'), ('Date', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:26:58.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:58.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:58.263 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:58.304 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:58.305 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:58.361 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:26:58.390 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030017.946534_830.1", 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:57.946534Z" 2023-09-29T23:26:58.391 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:26:58.392 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:58.393 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:26:58.394 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:26:58.394 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:26:58.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:58.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:26:58.394 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:26:58.482 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:26:58.482 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:26:58.536 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:58.535+0000 7f0f99be6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:58.538 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:26:58.539+0000 7f0f99be6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:26:58.564 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:26:58.564 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:26:58.564 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:58.565 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.566 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029996.317879_822.1", 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:36.317879Z", 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:58.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:58.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:26:58.569 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:26:58.570 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:26:58.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:26:58.571 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:26:58Z" 2023-09-29T23:26:58.571 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:26:58.572 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029996.317879_822.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:36.317879Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:26:58Z" } 2023-09-29T23:26:58.572 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029996.317879_822.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:36.317879Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:26:58.572 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:26:58.572 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:26:58.572 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030017.946534_830.1 target=1_1696029996.317879_822.1 2023-09-29T23:27:03.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:03.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:03.574 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:03.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:03.620 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:03.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:03.674+0000 7f6547f02f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:03.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:03.674+0000 7f6547f02f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:03.703 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:03.703 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:03.704 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:03.705 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:03.706 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029996.317879_822.1", 2023-09-29T23:27:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:03.707 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:36.317879Z", 2023-09-29T23:27:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:03.708 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:03.709 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:03.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:03.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:03.712 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:03.713 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:03Z" 2023-09-29T23:27:03.713 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:03.714 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029996.317879_822.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:36.317879Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:03Z" } 2023-09-29T23:27:03.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029996.317879_822.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:36.317879Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:03.716 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:03.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:27:03.716 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030017.946534_830.1 target=1_1696029996.317879_822.1 2023-09-29T23:27:08.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:08.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:08.718 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:08.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:08.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:08.822 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:08.822+0000 7f661d636f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:08.823 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:08.822+0000 7f661d636f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:08.846 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:08.846 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:08.846 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:08.846 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:08.846 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:08.847 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:08.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696029996.317879_822.1", 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:08.849 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:36.317879Z", 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:08.850 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:08.851 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:08.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:08Z" 2023-09-29T23:27:08.853 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:08.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696029996.317879_822.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:36.317879Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:08Z" } 2023-09-29T23:27:08.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029996.317879_822.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:36.317879Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:08.855 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:08.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029996.317879_822.1', 2: '', 3: ''} 2023-09-29T23:27:08.855 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030017.946534_830.1 target=1_1696029996.317879_822.1 2023-09-29T23:27:13.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:13.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:13.856 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:13.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:13.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:13.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:13.966+0000 7f65ce6f2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:13.967 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:13.966+0000 7f65ce6f2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:14.003 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:14.003 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:14.003 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:14.003 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:14.003 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:14.003 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:14.004 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030017.475513_823.1", 2023-09-29T23:27:14.005 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:57.475513Z", 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:14.006 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:14.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:14.008 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:13Z" 2023-09-29T23:27:14.009 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:14.010 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030017.475513_823.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:57.475513Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:13Z" } 2023-09-29T23:27:14.010 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030017.475513_823.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:57.475513Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:14.011 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:14.011 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.475513_823.1', 2: '', 3: ''} 2023-09-29T23:27:14.011 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030017.946534_830.1 target=1_1696030017.475513_823.1 2023-09-29T23:27:19.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.014 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:19.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:19.060 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:19.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:19.114+0000 7fba5f0d8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:19.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:19.114+0000 7fba5f0d8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:19.140 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:19.141 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:19.141 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:19.141 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:19.141 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:19.141 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:19.141 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:19.142 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:19.143 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030017.946534_830.1", 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:57.946534Z", 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:19.144 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.145 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.146 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.147 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:19Z" 2023-09-29T23:27:19.148 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:19.149 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030017.946534_830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:57.946534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:19Z" } 2023-09-29T23:27:19.149 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030017.946534_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:57.946534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:19.150 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:19.150 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:19.150 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:27:19.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.150 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:19.192 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:19.296 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000016994", 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:58.139152Z" 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000016155", 2023-09-29T23:27:19.297 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:58.212429Z" 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013514", 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:31.516304Z" 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006149", 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:26:58.229874Z" 2023-09-29T23:27:19.298 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.299 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:27:19.299 DEBUG:rgw_multi.tests:data markers for zone=a2 markers={0: '00000000000000000000:00000000000000016994', 1: '00000000000000000000:00000000000000016155', 2: '00000000000000000000:00000000000000013514', 3: '00000000000000000000:00000000000000006149'} 2023-09-29T23:27:19.300 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T23:27:19.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.300 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --source-zone a2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:19.386 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:19.387 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:19.472 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:19.472 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:19.472 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:19.472 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:19.472 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:19.472 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 14104852016160026348 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000016994", 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.473 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:58.139152Z" 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:19.474 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000016155", 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:58.212429Z" 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:19.475 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000013514", 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:31.516304Z" 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:19.476 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000006149", 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:58.229874Z" 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:19.477 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 0, 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 0 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:19Z" 2023-09-29T23:27:19.478 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:19.479 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 14104852016160026348 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016994", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:58.139152Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000016155", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:58.212429Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000013514", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:31.516304Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000006149", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:58.229874Z" } } ] }, "full_sync": { "total": 0, "complete": 0 }, "current_time": "2023-09-29T23:27:19Z" } 2023-09-29T23:27:19.479 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016994', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:58.139152Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000016155', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:58.212429Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000013514', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:31.516304Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000006149', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:58.229874Z'}}] 2023-09-29T23:27:19.480 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000016994', 1: '00000000000000000000:00000000000000016155', 2: '00000000000000000000:00000000000000013514', 3: '00000000000000000000:00000000000000006149'} 2023-09-29T23:27:19.480 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000016994', 1: '00000000000000000000:00000000000000016155', 2: '00000000000000000000:00000000000000013514', 3: '00000000000000000000:00000000000000006149'} 2023-09-29T23:27:19.480 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a1 source_zone=a2 2023-09-29T23:27:19.480 DEBUG:boto:path=/hcnphz-70/ 2023-09-29T23:27:19.480 DEBUG:boto:auth_path=/hcnphz-70/ 2023-09-29T23:27:19.480 DEBUG:boto:Method: HEAD 2023-09-29T23:27:19.481 DEBUG:boto:Path: /hcnphz-70/ 2023-09-29T23:27:19.481 DEBUG:boto:Data: 2023-09-29T23:27:19.481 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.481 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.481 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.481 DEBUG:boto:Params: {} 2023-09-29T23:27:19.481 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:27:19.482 DEBUG:boto:Token: None 2023-09-29T23:27:19.482 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/ 2023-09-29T23:27:19.482 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:EMNL6ZsnJwDchH96KdEyn4syNn4= 2023-09-29T23:27:19.482 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EMNL6ZsnJwDchH96KdEyn4syNn4=', 'Content-Length': '0'} 2023-09-29T23:27:19.487 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008716a2bc21d39473-0065175d57-272d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.488 DEBUG:boto:path=/hcnphz-70/obj7 2023-09-29T23:27:19.488 DEBUG:boto:auth_path=/hcnphz-70/obj7 2023-09-29T23:27:19.488 DEBUG:boto:Method: HEAD 2023-09-29T23:27:19.488 DEBUG:boto:Path: /hcnphz-70/obj7 2023-09-29T23:27:19.488 DEBUG:boto:Data: 2023-09-29T23:27:19.488 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.488 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.488 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.488 DEBUG:boto:Params: {} 2023-09-29T23:27:19.489 DEBUG:boto:Token: None 2023-09-29T23:27:19.489 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/obj7 2023-09-29T23:27:19.489 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:D8VpyNNZ/XHb2UOUbdb0hlLpF4g= 2023-09-29T23:27:19.489 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:D8VpyNNZ/XHb2UOUbdb0hlLpF4g=', 'Content-Length': '0'} 2023-09-29T23:27:19.491 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000d1135cb6580d9733-0065175d57-272d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.492 DEBUG:boto:path=/hcnphz-70/obj8 2023-09-29T23:27:19.492 DEBUG:boto:auth_path=/hcnphz-70/obj8 2023-09-29T23:27:19.492 DEBUG:boto:Method: HEAD 2023-09-29T23:27:19.492 DEBUG:boto:Path: /hcnphz-70/obj8 2023-09-29T23:27:19.492 DEBUG:boto:Data: 2023-09-29T23:27:19.492 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.492 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.492 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.492 DEBUG:boto:Params: {} 2023-09-29T23:27:19.493 DEBUG:boto:Token: None 2023-09-29T23:27:19.493 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/obj8 2023-09-29T23:27:19.493 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:PAJ3Ouiz5jpVtEXJPKg4Zb4+7Mc= 2023-09-29T23:27:19.493 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PAJ3Ouiz5jpVtEXJPKg4Zb4+7Mc=', 'Content-Length': '0'} 2023-09-29T23:27:19.495 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c850e3584bbb1d65-0065175d57-272d-a1'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.495 DEBUG:boto:path=/hcnphz-70/obj9 2023-09-29T23:27:19.496 DEBUG:boto:auth_path=/hcnphz-70/obj9 2023-09-29T23:27:19.496 DEBUG:boto:Method: HEAD 2023-09-29T23:27:19.496 DEBUG:boto:Path: /hcnphz-70/obj9 2023-09-29T23:27:19.496 DEBUG:boto:Data: 2023-09-29T23:27:19.496 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.496 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.496 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.496 DEBUG:boto:Params: {} 2023-09-29T23:27:19.496 DEBUG:boto:Token: None 2023-09-29T23:27:19.497 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/obj9 2023-09-29T23:27:19.497 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:myDQhwCffpdUIhB8X6j1TUTAgyY= 2023-09-29T23:27:19.497 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:myDQhwCffpdUIhB8X6j1TUTAgyY=', 'Content-Length': '0'} 2023-09-29T23:27:19.499 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-71:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000006933d9e2717d9720-0065175d57-272d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.500 DEBUG:boto:path=/hcnphz-70/obj9 2023-09-29T23:27:19.500 DEBUG:boto:auth_path=/hcnphz-70/obj9 2023-09-29T23:27:19.500 DEBUG:boto:Method: GET 2023-09-29T23:27:19.500 DEBUG:boto:Path: /hcnphz-70/obj9 2023-09-29T23:27:19.500 DEBUG:boto:Data: 2023-09-29T23:27:19.500 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.500 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.500 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.501 DEBUG:boto:Params: {} 2023-09-29T23:27:19.501 DEBUG:boto:Token: None 2023-09-29T23:27:19.501 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/obj9 2023-09-29T23:27:19.501 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5ck7dL4BvKJLJIFsTs3lLAUxmew= 2023-09-29T23:27:19.501 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5ck7dL4BvKJLJIFsTs3lLAUxmew=', 'Content-Length': '0'} 2023-09-29T23:27:19.504 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-71:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000016bb2ff15ba4c946-0065175d57-272d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.546 DEBUG:boto:path=/hcnphz-70/obj10 2023-09-29T23:27:19.547 DEBUG:boto:auth_path=/hcnphz-70/obj10 2023-09-29T23:27:19.547 DEBUG:boto:Method: HEAD 2023-09-29T23:27:19.547 DEBUG:boto:Path: /hcnphz-70/obj10 2023-09-29T23:27:19.547 DEBUG:boto:Data: 2023-09-29T23:27:19.547 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.547 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.547 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.547 DEBUG:boto:Params: {} 2023-09-29T23:27:19.548 DEBUG:boto:Token: None 2023-09-29T23:27:19.548 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/obj10 2023-09-29T23:27:19.548 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:rf13ts9J4MFQeLnEEGURvaasUYQ= 2023-09-29T23:27:19.548 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rf13ts9J4MFQeLnEEGURvaasUYQ=', 'Content-Length': '0'} 2023-09-29T23:27:19.551 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-71:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000014950b0437b8d390-0065175d57-272d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.551 DEBUG:boto:path=/hcnphz-70/obj10 2023-09-29T23:27:19.551 DEBUG:boto:auth_path=/hcnphz-70/obj10 2023-09-29T23:27:19.552 DEBUG:boto:Method: GET 2023-09-29T23:27:19.552 DEBUG:boto:Path: /hcnphz-70/obj10 2023-09-29T23:27:19.552 DEBUG:boto:Data: 2023-09-29T23:27:19.552 DEBUG:boto:Headers: {} 2023-09-29T23:27:19.552 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:19.552 DEBUG:boto:Port: 8000 2023-09-29T23:27:19.552 DEBUG:boto:Params: {} 2023-09-29T23:27:19.553 DEBUG:boto:Token: None 2023-09-29T23:27:19.553 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:27:19 GMT /hcnphz-70/obj10 2023-09-29T23:27:19.553 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3okwtgYjhtB3MTZQVpjLm5UQdmc= 2023-09-29T23:27:19.553 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3okwtgYjhtB3MTZQVpjLm5UQdmc=', 'Content-Length': '0'} 2023-09-29T23:27:19.557 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:26:58 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', '272b60f1-3a4d-4009-a171-ab410395ba9a:hcnphz-71:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.9356.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000003eaf7ab16a830f67-0065175d57-272d-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:27:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:19.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:19.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:19.558 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hcnphz-70 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:19.606 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:19.606 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:19.718 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:27:19.719 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:19.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:19.720 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:19.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:19.763 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:19.855 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:27:19.856 INFO:tasks.rgw_multisite_tests:test_sync_different_buckets: ... ok 2023-09-29T23:27:19.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:19.857 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:19.943 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:20.008 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:27:20.037 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:27:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:20.037 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030039.682587_833.1", 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:19.682587Z" 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:20.038 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:27:20.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:20.039 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:27:20.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:20.039 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:27:20.040 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030039.682587_833.1', 2: '', 3: ''} 2023-09-29T23:27:20.040 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:27:20.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:20.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:20.041 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:20.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:20.087 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:20.145 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:20.146+0000 7fe6f1c08f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:20.147 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:20.146+0000 7fe6f1c08f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:20.175 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:20.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:20.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.178 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030017.946534_830.1", 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:57.946534Z", 2023-09-29T23:27:20.179 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:20.180 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:20.181 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:20.182 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:20.183 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:20Z" 2023-09-29T23:27:20.184 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:20.185 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030017.946534_830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:57.946534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:20Z" } 2023-09-29T23:27:20.185 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030017.946534_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:57.946534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:20.185 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030039.682587_833.1', 2: '', 3: ''} 2023-09-29T23:27:20.185 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:20.186 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030039.682587_833.1 target=1_1696030017.946534_830.1 2023-09-29T23:27:25.186 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:25.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:25.187 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:25.235 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:25.235 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:25.282 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:25.282+0000 7f4320a86f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:25.283 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:25.282+0000 7f4320a86f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:25.309 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:25.309 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:25.310 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:25.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030017.946534_830.1", 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:57.946534Z", 2023-09-29T23:27:25.312 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:25.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:25.314 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:25.315 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:25.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:25.317 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:25Z" 2023-09-29T23:27:25.317 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:25.318 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030017.946534_830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:57.946534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:25Z" } 2023-09-29T23:27:25.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030017.946534_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:57.946534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:25.318 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030039.682587_833.1', 2: '', 3: ''} 2023-09-29T23:27:25.318 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:25.319 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030039.682587_833.1 target=1_1696030017.946534_830.1 2023-09-29T23:27:30.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:30.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:30.320 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:30.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:30.424 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:30.422+0000 7fc1da2d8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:30.425 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:30.422+0000 7fc1da2d8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:30.456 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:30.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030017.946534_830.1", 2023-09-29T23:27:30.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:57.946534Z", 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:30.459 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:30.460 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:30.461 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:30Z" 2023-09-29T23:27:30.462 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:30.463 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030017.946534_830.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:57.946534Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:30Z" } 2023-09-29T23:27:30.464 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030017.946534_830.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:57.946534Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:30.464 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030039.682587_833.1', 2: '', 3: ''} 2023-09-29T23:27:30.464 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030017.946534_830.1', 2: '', 3: ''} 2023-09-29T23:27:30.464 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030039.682587_833.1 target=1_1696030017.946534_830.1 2023-09-29T23:27:35.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:35.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:35.466 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:35.510 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:35.511 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:35.567 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:35.566+0000 7f3ccce17f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:35.569 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:35.566+0000 7f3ccce17f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:35.595 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:35.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:35.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:35.598 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030039.682587_833.1", 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:19.682587Z", 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.599 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:35.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:35.601 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:35.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.603 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:35Z" 2023-09-29T23:27:35.603 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:35.604 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030039.682587_833.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:19.682587Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:35Z" } 2023-09-29T23:27:35.604 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030039.682587_833.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:19.682587Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:35.604 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030039.682587_833.1', 2: '', 3: ''} 2023-09-29T23:27:35.604 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030039.682587_833.1', 2: '', 3: ''} 2023-09-29T23:27:35.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:27:35.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:35.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:35.605 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:35.687 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:35.687 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:35.782 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:27:35.782 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-72 2023-09-29T23:27:35.782 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:27:35.782 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:27:35.783 DEBUG:boto:Method: PUT 2023-09-29T23:27:35.783 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:27:35.783 DEBUG:boto:Data: 2023-09-29T23:27:35.783 DEBUG:boto:Headers: {} 2023-09-29T23:27:35.783 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:35.783 DEBUG:boto:Port: 8000 2023-09-29T23:27:35.783 DEBUG:boto:Params: {} 2023-09-29T23:27:35.783 DEBUG:boto:Token: None 2023-09-29T23:27:35.784 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:27:35 GMT /hcnphz-72/ 2023-09-29T23:27:35.784 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:3blMUX2+OgQK6HMyNSKfiEhkqZU= 2023-09-29T23:27:35.784 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3blMUX2+OgQK6HMyNSKfiEhkqZU=', 'Content-Length': '0'} 2023-09-29T23:27:35.814 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000000563a18b8a79693-0065175d67-272d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:35.814 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-73 2023-09-29T23:27:35.814 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:27:35.814 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:27:35.814 DEBUG:boto:Method: PUT 2023-09-29T23:27:35.814 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:27:35.814 DEBUG:boto:Data: 2023-09-29T23:27:35.815 DEBUG:boto:Headers: {} 2023-09-29T23:27:35.815 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:35.815 DEBUG:boto:Port: 8000 2023-09-29T23:27:35.815 DEBUG:boto:Params: {} 2023-09-29T23:27:35.815 DEBUG:boto:Token: None 2023-09-29T23:27:35.815 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:27:35 GMT /hcnphz-73/ 2023-09-29T23:27:35.815 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:pYnHkFeRPrQx0JMDxiEbTGnOT1s= 2023-09-29T23:27:35.815 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:35 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pYnHkFeRPrQx0JMDxiEbTGnOT1s=', 'Content-Length': '0'} 2023-09-29T23:27:35.844 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx000009e87c5c24d741fe0-0065175d67-272d-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:35 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:35.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:35.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:35.845 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:35.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:35.887 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:35.956 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:27:35.977 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:27:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.977 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:35.844914Z" 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.978 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:35.979 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:27:35.980 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:27:35.980 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:27:35.981 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:35.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:35.981 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:36.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.023 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:36.080 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:36.078+0000 7f8f40fc1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:36.081 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:36.082+0000 7f8f40fc1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:36.108 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:36.108 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:36.109 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:36.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:36.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:36.112 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:36.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:36.114 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:36.115 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:36.115 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:36Z" 2023-09-29T23:27:36.115 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:36.116 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:36Z" } 2023-09-29T23:27:36.116 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:36.116 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:27:36.116 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:27:36.116 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:27:36.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.117 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:36.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:36.290 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:27:36.290 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.291 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:36.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.340 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:36.435 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:27:36.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-72', '--dest-bucket=hcnphz-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-72', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-72', '--dest-bucket=hcnphz-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.435 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hcnphz-72 --source-zones=a1 --dest-zones=a2 --source-bucket=hcnphz-72 --dest-bucket=hcnphz-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:36.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:36.579 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:27:36.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-73', '--dest-bucket=hcnphz-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-73', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-73', '--dest-bucket=hcnphz-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.580 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hcnphz-73 --source-zones=a1 --dest-zones=a2 --source-bucket=hcnphz-73 --dest-bucket=hcnphz-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:36.629 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.630 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:36.727 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-73","source":{"bucket":"hcnphz-73","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:27:36.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.728 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:36.771 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.772 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:36.858 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-73","source":{"bucket":"hcnphz-73","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:27:36.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:36.859 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:36.901 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:36.901 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:37.015 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:37.016 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:37.016 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 20, 2023-09-29T23:27:37.016 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:27:37.016 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:27:37.016 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:27:37.016 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:27:37.017 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:27:37.018 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:27:37.019 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:27:37.020 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:27:37.021 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:27:37.022 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.023 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.024 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:27:37.025 INFO:teuthology.orchestra.run.smithi112.stdout: "directional": [ 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "source_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "dest_zone": "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.026 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-hcnphz-72", 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-72", 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-73", 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:27:37.027 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.028 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-hcnphz-73", 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:27:37.029 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-73", 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-73", 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:27:37.030 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:27:37.031 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.032 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:27:37.033 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:27:37.034 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:27:37.035 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.036 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:27:37.037 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:27:37.038 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:27:37.038 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:27:37.038 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:27:37.038 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:27:37.038 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:27:37.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.040 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.041 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.043 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:27:37.044 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:27:37.045 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:27:37.046 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:27:37.047 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:27:37.048 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:27:37.049 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.049 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.049 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:27:37.049 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:27:37.049 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:37.049 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:37.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:37.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:27:37.051 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:27:37.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:37.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:37.219 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-73","source":{"bucket":"hcnphz-73","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:27:37.220 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:27:37.220 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:27:37.220 DEBUG:boto:Method: HEAD 2023-09-29T23:27:37.220 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:27:37.220 DEBUG:boto:Data: 2023-09-29T23:27:37.220 DEBUG:boto:Headers: {} 2023-09-29T23:27:37.221 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.221 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.221 DEBUG:boto:Params: {} 2023-09-29T23:27:37.221 DEBUG:boto:Token: None 2023-09-29T23:27:37.221 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-72/ 2023-09-29T23:27:37.221 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n8YHQnkCEMq0RndzgtEvhb9tMmM= 2023-09-29T23:27:37.222 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n8YHQnkCEMq0RndzgtEvhb9tMmM=', 'Content-Length': '0'} 2023-09-29T23:27:37.276 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075e6139fd08e4d6d-0065175d69-2998-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.276 DEBUG:boto:path=/hcnphz-72/obj1 2023-09-29T23:27:37.276 DEBUG:boto:auth_path=/hcnphz-72/obj1 2023-09-29T23:27:37.277 DEBUG:boto:Method: PUT 2023-09-29T23:27:37.277 DEBUG:boto:Path: /hcnphz-72/obj1 2023-09-29T23:27:37.277 DEBUG:boto:Data: 2023-09-29T23:27:37.277 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:27:37.277 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.278 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.278 DEBUG:boto:Params: {} 2023-09-29T23:27:37.278 DEBUG:boto:Token: None 2023-09-29T23:27:37.278 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-72/obj1 2023-09-29T23:27:37.279 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:NGTxfZcO1PrRUvA800SLNn6I+Nk= 2023-09-29T23:27:37.279 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGTxfZcO1PrRUvA800SLNn6I+Nk='} 2023-09-29T23:27:37.300 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e3088b177fa8f64e-0065175d69-2998-a1'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.300 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:27:37.300 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:27:37.301 DEBUG:boto:Method: HEAD 2023-09-29T23:27:37.301 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:27:37.301 DEBUG:boto:Data: 2023-09-29T23:27:37.301 DEBUG:boto:Headers: {} 2023-09-29T23:27:37.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.301 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.301 DEBUG:boto:Params: {} 2023-09-29T23:27:37.301 DEBUG:boto:Token: None 2023-09-29T23:27:37.301 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-72/ 2023-09-29T23:27:37.302 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n8YHQnkCEMq0RndzgtEvhb9tMmM= 2023-09-29T23:27:37.302 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n8YHQnkCEMq0RndzgtEvhb9tMmM=', 'Content-Length': '0'} 2023-09-29T23:27:37.307 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008016554bf41c993f-0065175d69-2998-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.308 DEBUG:boto:path=/hcnphz-72/obj2 2023-09-29T23:27:37.308 DEBUG:boto:auth_path=/hcnphz-72/obj2 2023-09-29T23:27:37.308 DEBUG:boto:Method: PUT 2023-09-29T23:27:37.308 DEBUG:boto:Path: /hcnphz-72/obj2 2023-09-29T23:27:37.308 DEBUG:boto:Data: 2023-09-29T23:27:37.308 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:27:37.309 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.309 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.309 DEBUG:boto:Params: {} 2023-09-29T23:27:37.309 DEBUG:boto:Token: None 2023-09-29T23:27:37.309 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-72/obj2 2023-09-29T23:27:37.309 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:1B6X5qkm19iUW+lKX/IJHq2w+CQ= 2023-09-29T23:27:37.309 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:1B6X5qkm19iUW+lKX/IJHq2w+CQ='} 2023-09-29T23:27:37.358 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fbd1da2fad537650-0065175d69-2998-a1'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.359 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:27:37.359 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:27:37.359 DEBUG:boto:Method: HEAD 2023-09-29T23:27:37.359 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:27:37.359 DEBUG:boto:Data: 2023-09-29T23:27:37.359 DEBUG:boto:Headers: {} 2023-09-29T23:27:37.359 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.359 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.360 DEBUG:boto:Params: {} 2023-09-29T23:27:37.360 DEBUG:boto:Token: None 2023-09-29T23:27:37.360 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-73/ 2023-09-29T23:27:37.360 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bab2EFXcs5G+D1KOkjLyExdfqRY= 2023-09-29T23:27:37.360 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bab2EFXcs5G+D1KOkjLyExdfqRY=', 'Content-Length': '0'} 2023-09-29T23:27:37.368 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005bb40600a11da3f3-0065175d69-2998-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.368 DEBUG:boto:path=/hcnphz-73/obj3 2023-09-29T23:27:37.368 DEBUG:boto:auth_path=/hcnphz-73/obj3 2023-09-29T23:27:37.369 DEBUG:boto:Method: PUT 2023-09-29T23:27:37.369 DEBUG:boto:Path: /hcnphz-73/obj3 2023-09-29T23:27:37.369 DEBUG:boto:Data: 2023-09-29T23:27:37.369 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:27:37.369 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.369 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.369 DEBUG:boto:Params: {} 2023-09-29T23:27:37.369 DEBUG:boto:Token: None 2023-09-29T23:27:37.370 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-73/obj3 2023-09-29T23:27:37.370 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:+0N9gytJwHJ9n+Q8wjRi6c3ETWQ= 2023-09-29T23:27:37.370 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:+0N9gytJwHJ9n+Q8wjRi6c3ETWQ='} 2023-09-29T23:27:37.385 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006e4c096e4acffed2-0065175d69-2998-a1'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.385 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:27:37.385 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:27:37.385 DEBUG:boto:Method: HEAD 2023-09-29T23:27:37.385 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:27:37.386 DEBUG:boto:Data: 2023-09-29T23:27:37.386 DEBUG:boto:Headers: {} 2023-09-29T23:27:37.386 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.386 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.386 DEBUG:boto:Params: {} 2023-09-29T23:27:37.386 DEBUG:boto:Token: None 2023-09-29T23:27:37.386 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-73/ 2023-09-29T23:27:37.387 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Bab2EFXcs5G+D1KOkjLyExdfqRY= 2023-09-29T23:27:37.387 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bab2EFXcs5G+D1KOkjLyExdfqRY=', 'Content-Length': '0'} 2023-09-29T23:27:37.392 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000082de8639b3048ae6-0065175d69-2998-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.392 DEBUG:boto:path=/hcnphz-73/obj4 2023-09-29T23:27:37.392 DEBUG:boto:auth_path=/hcnphz-73/obj4 2023-09-29T23:27:37.392 DEBUG:boto:Method: PUT 2023-09-29T23:27:37.392 DEBUG:boto:Path: /hcnphz-73/obj4 2023-09-29T23:27:37.392 DEBUG:boto:Data: 2023-09-29T23:27:37.393 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:27:37.393 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:27:37.393 DEBUG:boto:Port: 8000 2023-09-29T23:27:37.393 DEBUG:boto:Params: {} 2023-09-29T23:27:37.393 DEBUG:boto:Token: None 2023-09-29T23:27:37.393 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:27:37 GMT /hcnphz-73/obj4 2023-09-29T23:27:37.393 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:lJkZ7WRN0sHjkcFL5hy/r27tVrE= 2023-09-29T23:27:37.393 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:27:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lJkZ7WRN0sHjkcFL5hy/r27tVrE='} 2023-09-29T23:27:37.438 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006893f51af1335e51-0065175d69-2998-a1'), ('Date', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:27:37.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.439 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:37.491 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:37.491 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:37.577 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:27:37.614 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:27:37.614 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.614 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:37.614 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:27:37.614 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:35.844914Z" 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.615 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.616 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:37.616 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:27:37.616 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.616 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:27:37.616 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:27:37.616 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:27:37.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.617 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:37.670 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:37.735 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:37.734+0000 7f4eefb91f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:37.736 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:27:37.734+0000 7f4eefb91f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:27:37.759 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:37.760 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:37.760 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:37.760 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:37.760 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:37.760 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:27:37.760 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:37.761 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.761 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:37.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.761 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:37.761 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:37.761 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:37.762 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:27:37.762 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:37.762 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:27:37.762 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:37.762 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:37.763 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:37.764 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.765 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:37.766 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:27:37.767 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.768 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:37Z" 2023-09-29T23:27:37.768 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:37.769 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:27:37Z" } 2023-09-29T23:27:37.769 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:27:37.769 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:27:37.769 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:27:37.769 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:27:37.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.770 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:37.814 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:37.815 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:37.898 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:27:37.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.898 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:37.386807Z" 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:37.332389Z" 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031777", 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:37.411698Z" 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:37.900 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:27:37.902 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:27:37.902 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:27:37.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:37.903 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:37.944 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:37.945 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:38.023 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:38.024 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:27:38.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:38.026 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:38.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:38.028 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:38.029 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:38Z" 2023-09-29T23:27:38.030 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:38.030 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:27:38Z" } 2023-09-29T23:27:38.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:27:38.031 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:27:38.031 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:27:38.031 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:27:43.033 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:43.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:43.033 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:43.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:43.081 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:43.168 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:43.168 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:43.168 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:43.168 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:43.169 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:43.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:43.171 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:43.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:43.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:43Z" 2023-09-29T23:27:43.174 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:43.175 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:27:43Z" } 2023-09-29T23:27:43.175 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:27:43.176 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:27:43.176 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:27:43.176 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:27:48.177 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:48.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:48.178 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:48.225 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:48.226 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:48.311 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:48.312 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:27:48.313 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:27:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:27:48.317 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:27:48.317 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:48.317 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:48Z" 2023-09-29T23:27:48.317 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:48.318 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:27:48Z" } 2023-09-29T23:27:48.318 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:27:48.321 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:27:48.321 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:27:48.321 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:27:53.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:53.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:53.322 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:53.365 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:53.366 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:53.452 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:53.452 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:53.452 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:53.453 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:53.454 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:53.455 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:53.456 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:53.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:53.458 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:53.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:27:53.458 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:27:53.458 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:53.458 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:53Z" 2023-09-29T23:27:53.458 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:53.459 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:27:53Z" } 2023-09-29T23:27:53.459 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:27:53.459 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:27:53.459 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:27:53.460 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:27:58.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:58.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:27:58.461 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:27:58.508 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:27:58.508 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:27:58.594 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:27:58.594 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:27:58.594 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:27:58.594 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:27:58.594 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:58.595 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:27:58.596 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:58.597 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:27:58.598 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:58.599 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:27:58.600 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:27:58.600 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:27:58.600 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:27:58.600 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:27:58Z" 2023-09-29T23:27:58.600 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:27:58.601 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:27:58Z" } 2023-09-29T23:27:58.601 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:27:58.601 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:27:58.601 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:27:58.601 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:03.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:03.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:03.603 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:03.650 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:03.650 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:03.736 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:03.736 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:03.737 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:03.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:03.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:03.740 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:03.741 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:03.742 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:03Z" 2023-09-29T23:28:03.743 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:03.743 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:03Z" } 2023-09-29T23:28:03.744 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:03.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:03.744 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:03.744 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:08.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:08.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:08.746 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:08.796 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:08.797 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:08.879 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:08.879 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:08.880 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:08.881 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:08.882 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:08.883 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:08.884 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:08.885 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:08.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:08.886 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:08Z" 2023-09-29T23:28:08.886 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:08.887 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:08Z" } 2023-09-29T23:28:08.887 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:08.887 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:08.887 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:08.887 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:13.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:13.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:13.889 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:13.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:13.937 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:14.022 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:14.023 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:14.024 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:14.025 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:14.026 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:14.027 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:14.028 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:14.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:14.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:14.029 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:14.029 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:14.029 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:14.029 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:14.030 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:14.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:14.030 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:13Z" 2023-09-29T23:28:14.030 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:14.031 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:13Z" } 2023-09-29T23:28:14.031 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:14.032 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:14.032 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:14.032 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:19.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:19.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:19.033 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:19.082 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:19.082 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:19.168 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:19.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:19.170 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:19.171 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:19.172 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:19.173 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:19.173 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:19.173 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:19.173 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:19.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:19.173 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:19.174 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:19.175 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:19.175 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:19.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:19.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:19.175 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:19.175 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:19.176 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:19.176 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:19.176 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:19.176 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:19.176 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:19.176 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:19.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:19.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:19.177 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:19.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:19.177 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:19.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:19.178 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:19.178 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:19.178 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:19Z" 2023-09-29T23:28:19.178 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:19.179 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:19Z" } 2023-09-29T23:28:19.179 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:19.179 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:19.180 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:19.180 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:24.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:24.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:24.181 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:24.228 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:24.229 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:24.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:24.312 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:24.313 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:24.314 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:24.317 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:24.317 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:24.317 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:24.317 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:24.317 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:24.317 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:24.318 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:24Z" 2023-09-29T23:28:24.319 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:24.320 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:24Z" } 2023-09-29T23:28:24.321 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:24.321 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:24.321 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:24.321 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:29.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:29.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:29.323 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:29.369 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:29.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:29.460 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:29.460 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:29.460 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:29.461 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:29.461 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:29.461 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:29.461 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:29.461 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:29.461 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:29.462 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:29.462 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:29.462 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:29.462 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:29.462 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:29.463 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:29.464 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:29.465 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:29.466 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:29.467 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:29Z" 2023-09-29T23:28:29.468 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:29.468 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:29Z" } 2023-09-29T23:28:29.469 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:29.469 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:29.469 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:29.469 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:34.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:34.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:34.471 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:34.519 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:34.520 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:34.606 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:34.607 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:34.608 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:34.609 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:34.610 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:34.611 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:34.612 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:34.613 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:34Z" 2023-09-29T23:28:34.613 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:34.614 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:34Z" } 2023-09-29T23:28:34.614 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:34.614 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:34.614 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:34.614 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:39.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:39.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:39.616 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:39.663 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:39.663 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:39.750 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:39.750 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:39.751 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:39.752 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:39.753 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:39.754 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:39.755 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:39.756 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:39Z" 2023-09-29T23:28:39.757 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:39.758 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:39Z" } 2023-09-29T23:28:39.758 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:39.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:39.758 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:39.758 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:44.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:44.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:44.760 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:44.808 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:44.809 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:44.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:44.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:44.898 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:44.899 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:44.900 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:44.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:44.902 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:44Z" 2023-09-29T23:28:44.902 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:44.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:44Z" } 2023-09-29T23:28:44.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:44.903 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:44.903 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:44.903 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:49.904 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:49.904 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:49.904 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:49.952 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:49.952 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:50.036 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:50.036 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:50.036 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:50.036 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:50.036 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:50.036 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:50.037 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:50.038 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:50.039 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:50.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:50.041 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:50Z" 2023-09-29T23:28:50.042 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:50.043 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:50Z" } 2023-09-29T23:28:50.044 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:50.044 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:50.044 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:50.044 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:28:55.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:55.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:28:55.046 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:28:55.094 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:28:55.094 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:28:55.182 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:28:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:28:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:28:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:28:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:28:55.182 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:28:55.183 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:55.184 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:55.185 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:55.186 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:55.187 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:28:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:28:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:28:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:28:55.188 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:28:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:28:55.189 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:28:55Z" 2023-09-29T23:28:55.189 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:28:55.190 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:28:55Z" } 2023-09-29T23:28:55.190 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:28:55.191 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:28:55.191 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:28:55.191 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:00.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:00.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:00.192 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:00.238 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:00.239 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:00.319 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:00.320 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:00.321 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:00.322 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:00.323 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:00.324 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:00.325 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:00Z" 2023-09-29T23:29:00.325 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:00.325 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:00Z" } 2023-09-29T23:29:00.326 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:00.326 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:00.326 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:00.326 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:05.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:05.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:05.327 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:05.370 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:05.371 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:05.453 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:05.454 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:05.455 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:05.456 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:05.457 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:05.458 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:05Z" 2023-09-29T23:29:05.459 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:05.460 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:05Z" } 2023-09-29T23:29:05.460 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:05.461 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:05.461 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:05.461 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:10.461 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:10.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:10.462 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:10.510 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:10.510 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:10.600 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:10.600 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:10.600 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:10.600 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:10.600 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:10.601 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:10.602 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:10.603 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:10.604 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:10.605 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:10.606 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:10.606 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:10.606 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:10.606 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:10.606 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:10Z" 2023-09-29T23:29:10.606 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:10.607 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:10Z" } 2023-09-29T23:29:10.608 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:10.608 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:10.608 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:10.608 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:15.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:15.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:15.610 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:15.659 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:15.659 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:15.747 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:15.747 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:15.747 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:15.747 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:15.747 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:15.748 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:15.749 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:15.749 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:15.749 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:15.749 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:15.749 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:15.749 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:15.750 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:15.750 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:15.750 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:15.750 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:15.750 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:15.750 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:15.751 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:15.751 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:15.751 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:15.751 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:15.751 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:15.751 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:15.752 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:15.752 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:15.752 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:15.752 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:15.752 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:15.752 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:15.753 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:15.754 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:15.754 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:15.754 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:15.754 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:15.754 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:15.754 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:15.755 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:15.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:15.755 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:15.755 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:15.755 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:15.755 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:15.756 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:15.756 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:15.756 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:15.756 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:15Z" 2023-09-29T23:29:15.756 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:15.757 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:15Z" } 2023-09-29T23:29:15.757 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:15.758 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:15.758 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:15.758 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:20.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:20.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:20.760 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:20.807 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:20.807 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:20.895 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:20.895 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:20.896 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:20.897 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:20.898 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:20.899 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:20.900 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:20.901 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:20.902 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:20.902 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:20Z" 2023-09-29T23:29:20.902 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:20.903 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:20Z" } 2023-09-29T23:29:20.903 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:20.903 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:20.903 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:20.903 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:25.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:25.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:25.905 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:25.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:25.953 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:26.035 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:26.035 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:26.035 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:26.035 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:26.036 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:26.036 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:26.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:26.036 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:26.036 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:26.036 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:26.037 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:26.037 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:26.037 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:26.037 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:26.037 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:26.038 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:26.038 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:26.038 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:26.038 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:26.038 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:26.038 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:26.039 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:26.039 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:26.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:26.039 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:26.039 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:26.040 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:26.040 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:26.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:26.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:26.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:26.040 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:26.041 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:26.041 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:26.041 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:26.041 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:26.041 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:26.041 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:26.042 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:26.043 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:26.044 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:26.044 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:26.044 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:26.044 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:26Z" 2023-09-29T23:29:26.044 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:26.045 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:26Z" } 2023-09-29T23:29:26.045 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:26.045 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:26.045 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:26.046 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:31.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:31.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:31.650 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:31.699 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:31.766 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:31.788 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:31.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:31.790 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:31.791 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:31.792 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:31.793 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:31.794 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:31Z" 2023-09-29T23:29:31.795 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:31.796 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:31Z" } 2023-09-29T23:29:31.806 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:31.807 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:31.807 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:31.807 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:36.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:36.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:36.808 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:36.860 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:36.860 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:36.954 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:36.955 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000051809", 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.832479Z" 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:36.956 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:36.957 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045937", 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:36.958 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:26:14.848608Z" 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031637", 2023-09-29T23:29:36.959 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490559Z" 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:36.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:36.961 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:36.961 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:36.961 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:36Z" 2023-09-29T23:29:36.961 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:36.962 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000051809", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:26:14.832479Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045937", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:26:14.848608Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031637", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:10.490559Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:36Z" } 2023-09-29T23:29:36.962 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000051809', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.832479Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045937', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:26:14.848608Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031637', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490559Z'}}] 2023-09-29T23:29:36.962 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:36.962 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000051809', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000045937', 3: '00000000000000000000:00000000000000031637'} 2023-09-29T23:29:36.962 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052090 target=00000000000000000000:00000000000000051809, shard=2 master=00000000000000000000:00000000000000046077 target=00000000000000000000:00000000000000045937, shard=3 master=00000000000000000000:00000000000000031777 target=00000000000000000000:00000000000000031637 2023-09-29T23:29:41.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:41.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:41.964 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:42.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:42.010 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:42.094 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:42.094 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:42.094 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:42.094 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:42.095 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.386807Z" 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:42.096 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:42.097 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.332389Z" 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:42.098 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031777", 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.411698Z" 2023-09-29T23:29:42.099 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:42Z" 2023-09-29T23:29:42.100 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:42.101 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:27:37.386807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.332389Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.411698Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:29:42Z" } 2023-09-29T23:29:42.101 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.386807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.332389Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.411698Z'}}] 2023-09-29T23:29:42.102 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:42.102 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:29:42.102 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:29:42.102 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:29:42.102 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:29:42.102 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.102 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:29:42.103 DEBUG:boto:Data: 2023-09-29T23:29:42.103 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.103 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.103 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.103 DEBUG:boto:Params: {} 2023-09-29T23:29:42.103 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:29:42.104 DEBUG:boto:Token: None 2023-09-29T23:29:42.104 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-72/ 2023-09-29T23:29:42.104 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cyRVasphPR3AqOsPpNRRXX6e1AQ= 2023-09-29T23:29:42.104 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cyRVasphPR3AqOsPpNRRXX6e1AQ=', 'Content-Length': '0'} 2023-09-29T23:29:42.113 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000457069fe98bd2ce3-0065175de6-2996-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.114 DEBUG:boto:path=/hcnphz-72/obj1 2023-09-29T23:29:42.114 DEBUG:boto:auth_path=/hcnphz-72/obj1 2023-09-29T23:29:42.114 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.114 DEBUG:boto:Path: /hcnphz-72/obj1 2023-09-29T23:29:42.114 DEBUG:boto:Data: 2023-09-29T23:29:42.114 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.114 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.114 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.114 DEBUG:boto:Params: {} 2023-09-29T23:29:42.115 DEBUG:boto:Token: None 2023-09-29T23:29:42.115 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-72/obj1 2023-09-29T23:29:42.115 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v5D/aOTBxCJMOjaaLQojBleR8c0= 2023-09-29T23:29:42.115 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v5D/aOTBxCJMOjaaLQojBleR8c0=', 'Content-Length': '0'} 2023-09-29T23:29:42.117 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000005978f29ac11cd42d-0065175de6-2996-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.117 DEBUG:boto:path=/hcnphz-72/obj2 2023-09-29T23:29:42.117 DEBUG:boto:auth_path=/hcnphz-72/obj2 2023-09-29T23:29:42.118 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.118 DEBUG:boto:Path: /hcnphz-72/obj2 2023-09-29T23:29:42.118 DEBUG:boto:Data: 2023-09-29T23:29:42.118 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.118 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.118 DEBUG:boto:Params: {} 2023-09-29T23:29:42.118 DEBUG:boto:Token: None 2023-09-29T23:29:42.118 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-72/obj2 2023-09-29T23:29:42.119 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Ob8DX+1ikBw/jKpFUiDB13njamk= 2023-09-29T23:29:42.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ob8DX+1ikBw/jKpFUiDB13njamk=', 'Content-Length': '0'} 2023-09-29T23:29:42.121 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000c260b59d2ce72e40-0065175de6-2996-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.121 DEBUG:boto:path=/hcnphz-72/obj3 2023-09-29T23:29:42.121 DEBUG:boto:auth_path=/hcnphz-72/obj3 2023-09-29T23:29:42.121 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.122 DEBUG:boto:Path: /hcnphz-72/obj3 2023-09-29T23:29:42.122 DEBUG:boto:Data: 2023-09-29T23:29:42.122 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.122 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.122 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.122 DEBUG:boto:Params: {} 2023-09-29T23:29:42.122 DEBUG:boto:Token: None 2023-09-29T23:29:42.122 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-72/obj3 2023-09-29T23:29:42.123 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:v0Ta2RxrjnhN7ZliuJNkpQnDPHY= 2023-09-29T23:29:42.123 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v0Ta2RxrjnhN7ZliuJNkpQnDPHY=', 'Content-Length': '0'} 2023-09-29T23:29:42.125 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000063cb139543ba96ac-0065175de6-2996-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.125 DEBUG:boto:path=/hcnphz-72/obj4 2023-09-29T23:29:42.125 DEBUG:boto:auth_path=/hcnphz-72/obj4 2023-09-29T23:29:42.125 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.125 DEBUG:boto:Path: /hcnphz-72/obj4 2023-09-29T23:29:42.125 DEBUG:boto:Data: 2023-09-29T23:29:42.125 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.126 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.126 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.126 DEBUG:boto:Params: {} 2023-09-29T23:29:42.126 DEBUG:boto:Token: None 2023-09-29T23:29:42.126 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-72/obj4 2023-09-29T23:29:42.126 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:wYvRiyHlQrQwdFqVFAzi6452ysE= 2023-09-29T23:29:42.126 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wYvRiyHlQrQwdFqVFAzi6452ysE=', 'Content-Length': '0'} 2023-09-29T23:29:42.128 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000076b5ef0669e92d63-0065175de6-2996-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.129 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:29:42.129 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:29:42.129 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.129 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:29:42.129 DEBUG:boto:Data: 2023-09-29T23:29:42.129 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.129 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.129 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.129 DEBUG:boto:Params: {} 2023-09-29T23:29:42.130 DEBUG:boto:Token: None 2023-09-29T23:29:42.130 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/ 2023-09-29T23:29:42.130 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:xbZXBR5jAzEZACsQjURnS2p5/rY= 2023-09-29T23:29:42.130 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xbZXBR5jAzEZACsQjURnS2p5/rY=', 'Content-Length': '0'} 2023-09-29T23:29:42.135 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000076b8623b2b9299b8-0065175de6-2996-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.135 DEBUG:boto:path=/hcnphz-73/obj1 2023-09-29T23:29:42.135 DEBUG:boto:auth_path=/hcnphz-73/obj1 2023-09-29T23:29:42.135 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.135 DEBUG:boto:Path: /hcnphz-73/obj1 2023-09-29T23:29:42.136 DEBUG:boto:Data: 2023-09-29T23:29:42.136 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.136 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.136 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.136 DEBUG:boto:Params: {} 2023-09-29T23:29:42.136 DEBUG:boto:Token: None 2023-09-29T23:29:42.136 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj1 2023-09-29T23:29:42.136 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:VODztK5sDHfhf5OdHNWmVRziSOg= 2023-09-29T23:29:42.137 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VODztK5sDHfhf5OdHNWmVRziSOg=', 'Content-Length': '0'} 2023-09-29T23:29:42.140 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-72:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000059a90466681d1bb1-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.140 DEBUG:boto:path=/hcnphz-73/obj1 2023-09-29T23:29:42.140 DEBUG:boto:auth_path=/hcnphz-73/obj1 2023-09-29T23:29:42.140 DEBUG:boto:Method: GET 2023-09-29T23:29:42.140 DEBUG:boto:Path: /hcnphz-73/obj1 2023-09-29T23:29:42.141 DEBUG:boto:Data: 2023-09-29T23:29:42.141 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.141 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.141 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.141 DEBUG:boto:Params: {} 2023-09-29T23:29:42.141 DEBUG:boto:Token: None 2023-09-29T23:29:42.141 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj1 2023-09-29T23:29:42.141 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:UPmC1vowuhTR4IUEjsfB8yq90Xk= 2023-09-29T23:29:42.142 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:UPmC1vowuhTR4IUEjsfB8yq90Xk=', 'Content-Length': '0'} 2023-09-29T23:29:42.144 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-72:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e99087fff96311b6-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.186 DEBUG:boto:path=/hcnphz-73/obj2 2023-09-29T23:29:42.187 DEBUG:boto:auth_path=/hcnphz-73/obj2 2023-09-29T23:29:42.187 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.187 DEBUG:boto:Path: /hcnphz-73/obj2 2023-09-29T23:29:42.187 DEBUG:boto:Data: 2023-09-29T23:29:42.187 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.187 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.187 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.187 DEBUG:boto:Params: {} 2023-09-29T23:29:42.187 DEBUG:boto:Token: None 2023-09-29T23:29:42.188 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj2 2023-09-29T23:29:42.188 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:eRsRqceQwq/7UKMQ+88OkAUuqnM= 2023-09-29T23:29:42.188 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eRsRqceQwq/7UKMQ+88OkAUuqnM=', 'Content-Length': '0'} 2023-09-29T23:29:42.191 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-72:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000a23a3d9ab25f56b-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.191 DEBUG:boto:path=/hcnphz-73/obj2 2023-09-29T23:29:42.191 DEBUG:boto:auth_path=/hcnphz-73/obj2 2023-09-29T23:29:42.191 DEBUG:boto:Method: GET 2023-09-29T23:29:42.191 DEBUG:boto:Path: /hcnphz-73/obj2 2023-09-29T23:29:42.192 DEBUG:boto:Data: 2023-09-29T23:29:42.192 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.192 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.192 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.192 DEBUG:boto:Params: {} 2023-09-29T23:29:42.192 DEBUG:boto:Token: None 2023-09-29T23:29:42.192 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj2 2023-09-29T23:29:42.192 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ITZXsR8Fx9OoFjIXhC3ze+crdhg= 2023-09-29T23:29:42.193 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITZXsR8Fx9OoFjIXhC3ze+crdhg=', 'Content-Length': '0'} 2023-09-29T23:29:42.196 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-72:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000410b4e55965bae87-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.196 DEBUG:boto:path=/hcnphz-73/obj3 2023-09-29T23:29:42.196 DEBUG:boto:auth_path=/hcnphz-73/obj3 2023-09-29T23:29:42.196 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.196 DEBUG:boto:Path: /hcnphz-73/obj3 2023-09-29T23:29:42.196 DEBUG:boto:Data: 2023-09-29T23:29:42.197 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.197 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.197 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.197 DEBUG:boto:Params: {} 2023-09-29T23:29:42.197 DEBUG:boto:Token: None 2023-09-29T23:29:42.197 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj3 2023-09-29T23:29:42.197 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:uaAVO/GxzYaZG6sy0aN9NwdsxVU= 2023-09-29T23:29:42.197 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uaAVO/GxzYaZG6sy0aN9NwdsxVU=', 'Content-Length': '0'} 2023-09-29T23:29:42.200 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000004e492fa01487fb25-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.200 DEBUG:boto:path=/hcnphz-73/obj3 2023-09-29T23:29:42.200 DEBUG:boto:auth_path=/hcnphz-73/obj3 2023-09-29T23:29:42.200 DEBUG:boto:Method: GET 2023-09-29T23:29:42.200 DEBUG:boto:Path: /hcnphz-73/obj3 2023-09-29T23:29:42.201 DEBUG:boto:Data: 2023-09-29T23:29:42.201 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.201 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.201 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.201 DEBUG:boto:Params: {} 2023-09-29T23:29:42.201 DEBUG:boto:Token: None 2023-09-29T23:29:42.201 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj3 2023-09-29T23:29:42.201 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:43u/CEmUArJAMc8CfZMDd02VUsI= 2023-09-29T23:29:42.202 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:43u/CEmUArJAMc8CfZMDd02VUsI=', 'Content-Length': '0'} 2023-09-29T23:29:42.204 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b5dd2f8241424af5-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.250 DEBUG:boto:path=/hcnphz-73/obj4 2023-09-29T23:29:42.250 DEBUG:boto:auth_path=/hcnphz-73/obj4 2023-09-29T23:29:42.251 DEBUG:boto:Method: HEAD 2023-09-29T23:29:42.251 DEBUG:boto:Path: /hcnphz-73/obj4 2023-09-29T23:29:42.251 DEBUG:boto:Data: 2023-09-29T23:29:42.251 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.251 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.251 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.251 DEBUG:boto:Params: {} 2023-09-29T23:29:42.251 DEBUG:boto:Token: None 2023-09-29T23:29:42.252 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj4 2023-09-29T23:29:42.252 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zDp+od2F2hE0L2J9N5IbEZZgZfE= 2023-09-29T23:29:42.252 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zDp+od2F2hE0L2J9N5IbEZZgZfE=', 'Content-Length': '0'} 2023-09-29T23:29:42.254 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000c2844115e78c28b9-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.255 DEBUG:boto:path=/hcnphz-73/obj4 2023-09-29T23:29:42.255 DEBUG:boto:auth_path=/hcnphz-73/obj4 2023-09-29T23:29:42.255 DEBUG:boto:Method: GET 2023-09-29T23:29:42.255 DEBUG:boto:Path: /hcnphz-73/obj4 2023-09-29T23:29:42.255 DEBUG:boto:Data: 2023-09-29T23:29:42.255 DEBUG:boto:Headers: {} 2023-09-29T23:29:42.255 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:29:42.256 DEBUG:boto:Port: 8001 2023-09-29T23:29:42.256 DEBUG:boto:Params: {} 2023-09-29T23:29:42.256 DEBUG:boto:Token: None 2023-09-29T23:29:42.256 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:29:42 GMT /hcnphz-73/obj4 2023-09-29T23:29:42.256 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:XKZjywlJ8TK34TX50wgFlSLimlU= 2023-09-29T23:29:42.256 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:29:42 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XKZjywlJ8TK34TX50wgFlSLimlU=', 'Content-Length': '0'} 2023-09-29T23:29:42.259 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:27:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000007c879d6664c0bb1b-0065175de6-2996-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:29:42 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:29:42.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.262 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-hcnphz-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:42.308 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:42.309 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:42.398 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[{"id":"sync-pipe-hcnphz-73","source":{"bucket":"hcnphz-73","zones":["a1"]},"dest":{"bucket":"hcnphz-73","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:29:42.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-73', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.398 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe-hcnphz-73 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:42.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:42.441 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:42.529 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"directional":[{"source_zone":"a1","dest_zone":"a2"}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:29:42.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'remove', '--group-id', 'sync-group', '--flow-id', 'sync-flow', '--flow-type', 'directional', '--source-zone=a1', '--dest-zone=a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.530 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow remove --group-id sync-group --flow-id sync-flow --flow-type directional --source-zone=a1 --dest-zone=a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:42.579 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:42.579 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:42.670 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"enabled"}]} 2023-09-29T23:29:42.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.671 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:42.719 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:42.720 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:42.813 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"enabled"}]} 2023-09-29T23:29:42.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.814 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:42.862 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:42.863 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:42.956 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:29:42.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:42.957 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.002 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.003 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:43.103 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:29:43.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.103 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.143 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.144 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:43.254 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 21, 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.255 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:29:43.256 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:29:43.257 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:29:43.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:29:43.259 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:29:43.260 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:29:43.261 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:29:43.262 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.263 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:29:43.264 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:29:43.265 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:29:43.266 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.267 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.268 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.269 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:29:43.270 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.271 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:29:43.272 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:29:43.273 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:29:43.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:29:43.275 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:29:43.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:29:43.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:29:43.278 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:29:43.279 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:29:43.280 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:29:43.281 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:43.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:43.282 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:29:43.282 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:29:43.282 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:43.282 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:43.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.284 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.321 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.321 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:43.410 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:29:43.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.411 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hcnphz-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:43.496 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-09-29T23:29:43.494+0000 7f4525c2c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:29:43.596 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:29:43.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.597 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.657 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:43.750 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:29:43.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-72', '--source-bucket=hcnphz-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.750 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-72', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-72', '--source-bucket=hcnphz-72', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.750 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hcnphz-72 --source-zones=a1 --dest-zones=a2 --bucket=hcnphz-72 --source-bucket=hcnphz-72 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.805 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:43.894 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:29:43.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-72', '--source-bucket=hcnphz-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-73', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-72', '--source-bucket=hcnphz-73', '--dest-bucket=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:43.894 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hcnphz-73 --source-zones=a1 --dest-zones=a2 --bucket=hcnphz-72 --source-bucket=hcnphz-73 '--dest-bucket=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:43.947 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:43.948 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:44.040 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hcnphz-73","source":{"bucket":"hcnphz-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:29:44.041 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:44.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:44.041 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hcnphz-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:44.085 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:44.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:44.187 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-72","source":{"bucket":"hcnphz-72","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hcnphz-73","source":{"bucket":"hcnphz-73","zones":["a1"]},"dest":{"bucket":"*","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:29:44.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:44.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:29:44.188 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:29:44.228 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:44.228 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:44.314 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:29:44.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:44.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:44.315 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:44.357 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:44.358 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:44.416 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:29:44.444 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:29:44.158375Z" 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:44.446 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:29:44.448 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:29:44.448 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:29:44.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:44.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:44.449 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:44.489 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:44.489 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:44.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:44.542+0000 7f83b6af2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:44.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:44.542+0000 7f83b6af2f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:29:44.570 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:29:44.571 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:44.572 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:44.573 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:44.574 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:44.575 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:44Z" 2023-09-29T23:29:44.576 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:44.577 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:29:44Z" } 2023-09-29T23:29:44.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:29:44.577 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:29:44.578 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:29:44.578 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:29:49.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:49.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:49.579 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:49.626 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:49.626 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:49.684 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:49.682+0000 7f14fcfcff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:49.686 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:49.686+0000 7f14fcfcff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:49.713 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:49.714 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:49.715 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:49.716 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:49.717 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:49.718 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:49Z" 2023-09-29T23:29:49.719 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:49.720 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:29:49Z" } 2023-09-29T23:29:49.720 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:29:49.720 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:29:49.720 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:29:49.721 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:29:54.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:54.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:54.722 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:54.770 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:54.770 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:54.825 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:54.822+0000 7f66949cef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:54.826 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:54.826+0000 7f66949cef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:54.853 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:54.853 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:54.853 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:54.853 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:54.853 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:54.853 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:29:54.854 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:54.855 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:54.856 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:54.857 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:54.858 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:54Z" 2023-09-29T23:29:54.859 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:54.860 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:29:54Z" } 2023-09-29T23:29:54.860 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:29:54.861 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:29:54.861 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:29:54.861 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:29:59.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:59.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:29:59.863 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:29:59.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:29:59.908 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:29:59.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:59.962+0000 7f7081a68f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:59.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:29:59.962+0000 7f7081a68f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:29:59.990 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:59.991 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:59.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:29:59.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:59.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:29:59.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:59.996 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:29:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:29:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:29:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:29:59.997 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:29:59Z" 2023-09-29T23:29:59.997 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:29:59.998 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:29:59Z" } 2023-09-29T23:29:59.998 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:29:59.998 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:29:59.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:29:59.999 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:05.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:05.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:05.000 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:05.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:05.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:05.102 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:05.102+0000 7f3759d90f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:05.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:05.102+0000 7f3759d90f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:05.131 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:05.131 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:05.131 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:05.132 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:05.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:05.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:05.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:05.136 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:05.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:05.138 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:05.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:05.138 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:05Z" 2023-09-29T23:30:05.138 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:05.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:05Z" } 2023-09-29T23:30:05.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:05.139 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:05.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:05.139 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:10.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:10.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:10.141 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:10.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:10.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:10.245 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:10.242+0000 7ff6c3b4cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:10.246 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:10.246+0000 7ff6c3b4cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:10.273 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:10.273 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:10.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:10.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:10.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:10.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:10.278 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:10.279 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:10.280 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:10.280 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:10.280 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:10.280 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:10Z" 2023-09-29T23:30:10.280 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:10.281 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:10Z" } 2023-09-29T23:30:10.281 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:10.281 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:10.281 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:10.281 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:15.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:15.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:15.283 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:15.330 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:15.388 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:15.386+0000 7fb60f7c0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:15.389 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:15.390+0000 7fb60f7c0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:15.416 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:15.416 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:15.416 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:15.416 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:15.416 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:15.416 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:15.417 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:15.418 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:15.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:15.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:15.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:15.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:15.423 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:15Z" 2023-09-29T23:30:15.423 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:15.424 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:15Z" } 2023-09-29T23:30:15.424 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:15.424 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:15.424 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:15.425 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:20.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:20.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:20.427 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:20.474 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:20.474 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:20.531 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:20.529+0000 7f1dd29e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:20.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:20.529+0000 7f1dd29e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:20.559 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:20.559 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:20.560 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:20.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:20.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:20.563 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:20.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:20.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:20.566 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:20.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:20.566 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:20.566 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:20.566 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:20Z" 2023-09-29T23:30:20.566 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:20.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:20Z" } 2023-09-29T23:30:20.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:20.567 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:20.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:20.568 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:25.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:25.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:25.569 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:25.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:25.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:25.670 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:25.669+0000 7ff081fadf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:25.672 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:25.669+0000 7ff081fadf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:25.698 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:25.698 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:25.699 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:25.700 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:25.701 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:25.702 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:25.703 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:25.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:25.705 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:25.705 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:25.705 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:25.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:25.705 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:25Z" 2023-09-29T23:30:25.705 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:25.706 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:25Z" } 2023-09-29T23:30:25.706 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:25.706 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:25.707 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:25.707 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:30.707 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:30.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:30.708 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:30.754 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:30.755 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:30.812 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:30.809+0000 7f1e233c8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:30.813 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:30.813+0000 7f1e233c8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:30.844 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:30.844 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:30.844 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:30.844 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:30.845 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:30.845 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:30.845 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:30.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:30.845 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:30.845 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:30.846 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:30.846 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:30.846 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:30.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:30.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:30.846 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:30.847 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:30.847 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:30.847 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:30.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:30.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:30.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:30.848 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:30.849 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:30.849 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:30.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:30.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:30.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:30.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:30.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:30.850 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:30.850 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:30.850 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:30.850 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:30.850 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:30.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:30.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:30.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:30.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:30.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:30.851 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:30.852 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:30.852 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:30.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:30.852 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:30.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:30.853 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:30.854 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:30Z" 2023-09-29T23:30:30.855 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:30.856 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:30Z" } 2023-09-29T23:30:30.856 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:30.856 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:30.856 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:30.857 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:35.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:35.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:35.858 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:35.904 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:35.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:35.965+0000 7f67504bdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:35.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:35.965+0000 7f67504bdf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:35.991 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:35.992 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:35.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:35.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:35.995 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:35.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:35.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:35.998 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:35.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:35.998 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:35.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:35.998 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:35Z" 2023-09-29T23:30:35.998 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:35.999 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:35Z" } 2023-09-29T23:30:35.999 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:35.999 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:36.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:36.000 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:41.000 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:41.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:41.001 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:41.048 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:41.048 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:41.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:41.105+0000 7fed0dc35f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:41.104 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:41.105+0000 7fed0dc35f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:41.132 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:41.132 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:41.132 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:41.132 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:41.132 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:41.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:41.134 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:41.135 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:41.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:41.137 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:41.138 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:41.139 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:41.139 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:41.139 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:41.139 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:41.139 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:41Z" 2023-09-29T23:30:41.139 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:41.140 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:41Z" } 2023-09-29T23:30:41.140 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:41.141 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:41.141 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:41.141 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:46.142 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:46.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:46.143 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:46.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:46.189 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:46.247 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:46.249+0000 7f6990ceef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:46.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:46.249+0000 7f6990ceef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:46.284 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:46.285 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:46.286 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:46.287 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:46.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:46.289 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:46.290 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:46Z" 2023-09-29T23:30:46.291 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:46.291 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:46Z" } 2023-09-29T23:30:46.292 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:46.292 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:46.292 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:46.292 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:51.293 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:51.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:51.293 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:51.337 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:51.338 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:51.391 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:51.393+0000 7f94cae02f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:51.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:51.393+0000 7f94cae02f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:51.419 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:51.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:51.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:51.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:51.423 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:51.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:51.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:51Z" 2023-09-29T23:30:51.426 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:51.427 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:51Z" } 2023-09-29T23:30:51.427 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:51.428 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:51.428 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:51.428 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:30:56.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:56.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:30:56.430 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:30:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:30:56.477 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:30:56.530 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:56.533+0000 7f4d8daf5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:56.532 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:30:56.533+0000 7f4d8daf5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:30:56.559 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:30:56.559 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:30:56.559 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:30:56.559 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:30:56.559 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:30:56.560 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:56.561 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:56.562 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:56.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:30:56.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:30:56.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:30:56.566 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:30:56Z" 2023-09-29T23:30:56.566 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:30:56.567 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:30:56Z" } 2023-09-29T23:30:56.567 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:30:56.567 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:30:56.567 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:30:56.567 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:01.568 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:01.569 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:01.569 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:01.615 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:01.616 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:01.674 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:01.676+0000 7fdf64456f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:01.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:01.676+0000 7fdf64456f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:01.702 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:01.703 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:01.704 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:01.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:01.706 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:01.707 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:01.708 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:01.709 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:01.710 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:01.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:01.711 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:01Z" 2023-09-29T23:31:01.711 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:01.712 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:01Z" } 2023-09-29T23:31:01.712 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:01.712 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:01.712 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:01.713 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:06.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:06.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:06.714 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:06.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:06.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:06.816 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:06.816+0000 7f0ca33d3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:06.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:06.820+0000 7f0ca33d3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:06.845 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:06.845 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:06.846 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:06.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:06.848 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:06.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:06.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:06.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:06.853 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:06.854 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:06Z" 2023-09-29T23:31:06.854 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:06.855 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:06Z" } 2023-09-29T23:31:06.855 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:06.855 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:06.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:06.855 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:11.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:11.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:11.857 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:11.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:11.903 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:11.958 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:11.960+0000 7f8f044b6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:11.960 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:11.960+0000 7f8f044b6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:11.988 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:11.988 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:11.988 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:11.988 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:11.988 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:11.989 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:11.990 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:11.991 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:11.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:11.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:11.994 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:11.995 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:11.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:11.996 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:11.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:11.996 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:11Z" 2023-09-29T23:31:11.996 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:11.997 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:11Z" } 2023-09-29T23:31:11.997 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:11.997 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:11.998 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:11.998 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:16.998 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:16.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:16.999 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:17.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:17.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:17.101 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:17.100+0000 7f3bb36f8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:17.103 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:17.104+0000 7f3bb36f8f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:17.131 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:17.131 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:17.131 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:17.131 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:17.131 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:17.132 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:17.133 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:17.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:17.135 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:17.136 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:17.137 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:17.138 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:17Z" 2023-09-29T23:31:17.138 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:17.139 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:17Z" } 2023-09-29T23:31:17.139 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:17.139 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:17.139 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:17.140 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:22.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:22.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:22.141 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:22.187 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:22.188 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:22.249 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:22.248+0000 7f04f4846f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:22.250 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:22.252+0000 7f04f4846f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:22.278 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:22.278 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:22.278 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:22.279 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:22.280 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:22.281 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:22.282 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:22.283 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:22.284 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:22.285 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:22.285 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:22.285 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:22.285 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:22Z" 2023-09-29T23:31:22.285 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:22.286 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:22Z" } 2023-09-29T23:31:22.286 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:22.287 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:22.287 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:22.287 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:27.288 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:27.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:27.288 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:27.334 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:27.334 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:27.390 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:27.392+0000 7f09fe041f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:27.392 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:27.392+0000 7f09fe041f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:27.418 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:27.419 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:27.420 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:27.421 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:27.422 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:27.423 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:27.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:27.425 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:27Z" 2023-09-29T23:31:27.425 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:27.425 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:27Z" } 2023-09-29T23:31:27.426 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:27.426 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:27.426 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:27.426 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:32.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:32.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:32.428 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:32.474 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:32.475 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:32.534 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:32.536+0000 7fd947b5af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:32.535 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:32.536+0000 7fd947b5af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:32.562 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:32.562 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:32.562 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:32.562 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:32.562 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:32.563 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:32.564 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:32.565 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:32.566 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:32.567 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:32.568 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:32.569 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:32Z" 2023-09-29T23:31:32.569 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:32.570 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:32Z" } 2023-09-29T23:31:32.570 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:32.570 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:32.570 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:32.570 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:37.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:37.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:37.572 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:37.618 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:37.619 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:37.675 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:37.676+0000 7ff2d31f6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:37.676 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:37.676+0000 7ff2d31f6f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:37.703 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:37.703 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:37.704 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:37.705 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:37.706 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:37.707 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:37.708 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:37.709 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:37.710 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:37.710 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:37.710 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:37.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:37.710 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:37Z" 2023-09-29T23:31:37.710 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:37.711 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:37Z" } 2023-09-29T23:31:37.711 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:37.711 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:37.711 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:37.712 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:42.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:42.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:42.713 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:42.759 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:42.760 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:42.818 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:42.819+0000 7f84c4f72f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:42.820 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:42.819+0000 7f84c4f72f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:42.848 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:42.848 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:42.849 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:42.849 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:42.849 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:42.849 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:42.849 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:42.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:42.850 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:42.851 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030055.844914_841.1", 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:35.844914Z", 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:42.852 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:42.853 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:42.854 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:42.855 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:42.856 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:42.856 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:42.856 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:42Z" 2023-09-29T23:31:42.856 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:42.857 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030055.844914_841.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:35.844914Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:42Z" } 2023-09-29T23:31:42.857 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030055.844914_841.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:35.844914Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:42.857 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:42.857 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030055.844914_841.1', 2: '', 3: ''} 2023-09-29T23:31:42.858 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030184.158375_875.1 target=1_1696030055.844914_841.1 2023-09-29T23:31:47.859 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:47.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:47.859 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:47.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:47.907 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:47.966 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:47.967+0000 7fd45fe40f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:47.968 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:47.967+0000 7fd45fe40f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:47.995 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:47.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:31:47.997 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:29:44.158375Z", 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:47.998 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:47.999 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.000 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:48.001 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:48.002 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:48.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.002 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:47Z" 2023-09-29T23:31:48.002 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:48.003 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030184.158375_875.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:29:44.158375Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:47Z" } 2023-09-29T23:31:48.003 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030184.158375_875.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:29:44.158375Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:48.003 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:48.003 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:48.003 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:31:48.003 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:31:48.004 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:31:48.004 DEBUG:boto:Method: HEAD 2023-09-29T23:31:48.004 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:31:48.004 DEBUG:boto:Data: 2023-09-29T23:31:48.004 DEBUG:boto:Headers: {} 2023-09-29T23:31:48.004 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.004 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.004 DEBUG:boto:Params: {} 2023-09-29T23:31:48.005 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:31:48.005 DEBUG:boto:Token: None 2023-09-29T23:31:48.005 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-72/ 2023-09-29T23:31:48.005 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n3S8n8XsVGLfgZ1wb0o2EPfLfms= 2023-09-29T23:31:48.005 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n3S8n8XsVGLfgZ1wb0o2EPfLfms=', 'Content-Length': '0'} 2023-09-29T23:31:48.015 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000aa56f8abbe665ffc-0065175e64-2ac0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.015 DEBUG:boto:path=/hcnphz-72/obj5 2023-09-29T23:31:48.015 DEBUG:boto:auth_path=/hcnphz-72/obj5 2023-09-29T23:31:48.016 DEBUG:boto:Method: PUT 2023-09-29T23:31:48.016 DEBUG:boto:Path: /hcnphz-72/obj5 2023-09-29T23:31:48.016 DEBUG:boto:Data: 2023-09-29T23:31:48.016 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:31:48.016 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.016 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.016 DEBUG:boto:Params: {} 2023-09-29T23:31:48.016 DEBUG:boto:Token: None 2023-09-29T23:31:48.017 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-72/obj5 2023-09-29T23:31:48.017 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/oxsSqt9yAqBh00Dursvkj1QLgI= 2023-09-29T23:31:48.017 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/oxsSqt9yAqBh00Dursvkj1QLgI='} 2023-09-29T23:31:48.035 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fe146837e33e9ef1-0065175e64-2ac0-a1'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.035 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:31:48.035 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:31:48.035 DEBUG:boto:Method: HEAD 2023-09-29T23:31:48.035 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:31:48.035 DEBUG:boto:Data: 2023-09-29T23:31:48.035 DEBUG:boto:Headers: {} 2023-09-29T23:31:48.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.035 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.036 DEBUG:boto:Params: {} 2023-09-29T23:31:48.036 DEBUG:boto:Token: None 2023-09-29T23:31:48.036 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-72/ 2023-09-29T23:31:48.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:n3S8n8XsVGLfgZ1wb0o2EPfLfms= 2023-09-29T23:31:48.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:n3S8n8XsVGLfgZ1wb0o2EPfLfms=', 'Content-Length': '0'} 2023-09-29T23:31:48.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001c460ce004da902-0065175e64-2ac0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.039 DEBUG:boto:path=/hcnphz-72/obj6 2023-09-29T23:31:48.040 DEBUG:boto:auth_path=/hcnphz-72/obj6 2023-09-29T23:31:48.040 DEBUG:boto:Method: PUT 2023-09-29T23:31:48.040 DEBUG:boto:Path: /hcnphz-72/obj6 2023-09-29T23:31:48.040 DEBUG:boto:Data: 2023-09-29T23:31:48.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:31:48.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.040 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.040 DEBUG:boto:Params: {} 2023-09-29T23:31:48.040 DEBUG:boto:Token: None 2023-09-29T23:31:48.041 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-72/obj6 2023-09-29T23:31:48.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:GIgMBaFwDtndQRuegW5aAfFjAJc= 2023-09-29T23:31:48.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GIgMBaFwDtndQRuegW5aAfFjAJc='} 2023-09-29T23:31:48.091 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4bade863de6a17c-0065175e64-2ac0-a1'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.092 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:31:48.092 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:31:48.092 DEBUG:boto:Method: HEAD 2023-09-29T23:31:48.092 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:31:48.092 DEBUG:boto:Data: 2023-09-29T23:31:48.092 DEBUG:boto:Headers: {} 2023-09-29T23:31:48.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.092 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.092 DEBUG:boto:Params: {} 2023-09-29T23:31:48.093 DEBUG:boto:Token: None 2023-09-29T23:31:48.093 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-73/ 2023-09-29T23:31:48.093 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lbd9u6EOP40EQOo9MpzzxaM9//E= 2023-09-29T23:31:48.093 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lbd9u6EOP40EQOo9MpzzxaM9//E=', 'Content-Length': '0'} 2023-09-29T23:31:48.098 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d3b87aa89a1b17dd-0065175e64-2ac0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.099 DEBUG:boto:path=/hcnphz-73/obj7 2023-09-29T23:31:48.099 DEBUG:boto:auth_path=/hcnphz-73/obj7 2023-09-29T23:31:48.099 DEBUG:boto:Method: PUT 2023-09-29T23:31:48.099 DEBUG:boto:Path: /hcnphz-73/obj7 2023-09-29T23:31:48.099 DEBUG:boto:Data: 2023-09-29T23:31:48.099 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:31:48.099 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.099 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.099 DEBUG:boto:Params: {} 2023-09-29T23:31:48.100 DEBUG:boto:Token: None 2023-09-29T23:31:48.100 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-73/obj7 2023-09-29T23:31:48.100 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vxPLmxp9Fv27LiDIWy946cuqXKc= 2023-09-29T23:31:48.100 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vxPLmxp9Fv27LiDIWy946cuqXKc='} 2023-09-29T23:31:48.112 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000ccd046b1c8656b5-0065175e64-2ac0-a1'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.112 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:31:48.112 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:31:48.113 DEBUG:boto:Method: HEAD 2023-09-29T23:31:48.113 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:31:48.113 DEBUG:boto:Data: 2023-09-29T23:31:48.113 DEBUG:boto:Headers: {} 2023-09-29T23:31:48.113 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.113 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.113 DEBUG:boto:Params: {} 2023-09-29T23:31:48.114 DEBUG:boto:Token: None 2023-09-29T23:31:48.114 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-73/ 2023-09-29T23:31:48.114 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Lbd9u6EOP40EQOo9MpzzxaM9//E= 2023-09-29T23:31:48.114 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Lbd9u6EOP40EQOo9MpzzxaM9//E=', 'Content-Length': '0'} 2023-09-29T23:31:48.117 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bfd9f576fd9fab60-0065175e64-2ac0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.117 DEBUG:boto:path=/hcnphz-73/obj8 2023-09-29T23:31:48.117 DEBUG:boto:auth_path=/hcnphz-73/obj8 2023-09-29T23:31:48.117 DEBUG:boto:Method: PUT 2023-09-29T23:31:48.117 DEBUG:boto:Path: /hcnphz-73/obj8 2023-09-29T23:31:48.118 DEBUG:boto:Data: 2023-09-29T23:31:48.118 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:31:48.118 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:31:48.118 DEBUG:boto:Port: 8000 2023-09-29T23:31:48.118 DEBUG:boto:Params: {} 2023-09-29T23:31:48.118 DEBUG:boto:Token: None 2023-09-29T23:31:48.118 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:31:48 GMT /hcnphz-73/obj8 2023-09-29T23:31:48.119 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:HU5qYRc+xtYbwsMIeiD/5Dv4N+0= 2023-09-29T23:31:48.119 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:31:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HU5qYRc+xtYbwsMIeiD/5Dv4N+0='} 2023-09-29T23:31:48.166 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ec7eed99505aec6-0065175e64-2ac0-a1'), ('Date', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:31:48.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.167 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:48.211 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:48.211 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:48.290 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:31:48.314 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:31:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:48.314 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:29:44.158375Z" 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.315 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:31:48.316 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.316 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:31:48.317 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:48.317 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:31:48.317 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.318 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:48.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:48.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:48.419 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:48.419+0000 7f717695ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:48.420 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:31:48.419+0000 7f717695ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:31:48.446 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:48.446 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:48.446 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:48.446 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:48.446 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:48.447 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:48.448 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:29:44.158375Z", 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.449 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.450 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.451 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.452 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:48Z" 2023-09-29T23:31:48.453 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:48.453 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030184.158375_875.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:29:44.158375Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:31:48Z" } 2023-09-29T23:31:48.454 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030184.158375_875.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:29:44.158375Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:31:48.454 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:48.454 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:31:48.454 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:31:48.454 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.455 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:48.494 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:48.494 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:48.592 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:31:48.593 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.594 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:31:48.595 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:31:48.595 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:31:48.596 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:48.596 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:48.687 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:48.688 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:48.792 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:48.792 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:48.792 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.793 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.386807Z" 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:48.794 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.795 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.332389Z" 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.796 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031777", 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.411698Z" 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.797 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:48Z" 2023-09-29T23:31:48.798 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:48.799 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:27:37.386807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.332389Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.411698Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:31:48Z" } 2023-09-29T23:31:48.799 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.386807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.332389Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.411698Z'}}] 2023-09-29T23:31:48.800 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:31:48.800 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:31:48.800 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052372 target=00000000000000000000:00000000000000052090, shard=2 master=00000000000000000000:00000000000000046218 target=00000000000000000000:00000000000000046077, shard=3 master=00000000000000000000:00000000000000031918 target=00000000000000000000:00000000000000031777 2023-09-29T23:31:53.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:53.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:53.801 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:53.851 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:53.936 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:53.936 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:53.936 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:53.936 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-09-29T23:31:53.937 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.386807Z" 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:53.938 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:53.939 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.332389Z" 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:53.940 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031777", 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.411698Z" 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:53.941 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:53.942 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:31:53.942 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:31:53.942 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:53.942 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:53Z" 2023-09-29T23:31:53.942 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:53.943 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:27:37.386807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.332389Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.411698Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:31:53Z" } 2023-09-29T23:31:53.943 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.386807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.332389Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.411698Z'}}] 2023-09-29T23:31:53.943 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:31:53.943 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:31:53.943 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052372 target=00000000000000000000:00000000000000052090, shard=2 master=00000000000000000000:00000000000000046218 target=00000000000000000000:00000000000000046077, shard=3 master=00000000000000000000:00000000000000031918 target=00000000000000000000:00000000000000031777 2023-09-29T23:31:58.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:58.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:31:58.944 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:31:58.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:31:58.993 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:31:59.079 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:59.080 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.386807Z" 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:31:59.081 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:59.082 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.332389Z" 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031777", 2023-09-29T23:31:59.083 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.411698Z" 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:31:59.084 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:31:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:31:59.085 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:31:59Z" 2023-09-29T23:31:59.085 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:31:59.086 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:27:37.386807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.332389Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.411698Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:31:59Z" } 2023-09-29T23:31:59.086 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.386807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.332389Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.411698Z'}}] 2023-09-29T23:31:59.086 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:31:59.086 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:31:59.086 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052372 target=00000000000000000000:00000000000000052090, shard=2 master=00000000000000000000:00000000000000046218 target=00000000000000000000:00000000000000046077, shard=3 master=00000000000000000000:00000000000000031918 target=00000000000000000000:00000000000000031777 2023-09-29T23:32:04.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:04.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:04.087 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:04.136 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:04.137 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:04.219 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:04.219 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:04.219 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:04.219 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:04.219 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:04.219 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052090", 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:04.220 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.386807Z" 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:04.221 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:04.222 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046077", 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.332389Z" 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:04.223 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031777", 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:37.411698Z" 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:04.224 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:04.225 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:04.225 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:04.225 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:04.225 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:04.225 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:04Z" 2023-09-29T23:32:04.225 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:04.226 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052090", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:27:37.386807Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046077", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.332389Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031777", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:27:37.411698Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:04Z" } 2023-09-29T23:32:04.226 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052090', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.386807Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046077', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.332389Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031777', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:27:37.411698Z'}}] 2023-09-29T23:32:04.226 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:04.227 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052090', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046077', 3: '00000000000000000000:00000000000000031777'} 2023-09-29T23:32:04.227 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052372 target=00000000000000000000:00000000000000052090, shard=2 master=00000000000000000000:00000000000000046218 target=00000000000000000000:00000000000000046077, shard=3 master=00000000000000000000:00000000000000031918 target=00000000000000000000:00000000000000031777 2023-09-29T23:32:09.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:09.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:09.229 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:09.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:09.279 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:09.366 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:09.366 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:09.367 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:09.368 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:09.369 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:09.370 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:09.371 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:09Z" 2023-09-29T23:32:09.372 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:09.373 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:09Z" } 2023-09-29T23:32:09.373 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:32:09.374 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:09.374 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:09.374 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:32:09.375 DEBUG:boto:path=/hcnphz-73/ 2023-09-29T23:32:09.375 DEBUG:boto:auth_path=/hcnphz-73/ 2023-09-29T23:32:09.375 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.375 DEBUG:boto:Path: /hcnphz-73/ 2023-09-29T23:32:09.375 DEBUG:boto:Data: 2023-09-29T23:32:09.375 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.375 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.375 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.376 DEBUG:boto:Params: {} 2023-09-29T23:32:09.376 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:32:09.376 DEBUG:boto:Token: None 2023-09-29T23:32:09.376 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-73/ 2023-09-29T23:32:09.376 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FcOgKMG/m52MevcUz/Yf7O/Qe9I= 2023-09-29T23:32:09.377 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FcOgKMG/m52MevcUz/Yf7O/Qe9I=', 'Content-Length': '0'} 2023-09-29T23:32:09.386 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d17b6906b46e16bd-0065175e79-2abe-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.386 DEBUG:boto:path=/hcnphz-73/obj5 2023-09-29T23:32:09.386 DEBUG:boto:auth_path=/hcnphz-73/obj5 2023-09-29T23:32:09.386 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.387 DEBUG:boto:Path: /hcnphz-73/obj5 2023-09-29T23:32:09.387 DEBUG:boto:Data: 2023-09-29T23:32:09.387 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.387 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.387 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.387 DEBUG:boto:Params: {} 2023-09-29T23:32:09.387 DEBUG:boto:Token: None 2023-09-29T23:32:09.387 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-73/obj5 2023-09-29T23:32:09.388 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9JkbMhRH1BrPRRfonFZTg6FAXYI= 2023-09-29T23:32:09.388 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9JkbMhRH1BrPRRfonFZTg6FAXYI=', 'Content-Length': '0'} 2023-09-29T23:32:09.390 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx00000e5da37d39ba91b8a-0065175e79-2abe-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.390 DEBUG:boto:path=/hcnphz-73/obj6 2023-09-29T23:32:09.390 DEBUG:boto:auth_path=/hcnphz-73/obj6 2023-09-29T23:32:09.390 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.391 DEBUG:boto:Path: /hcnphz-73/obj6 2023-09-29T23:32:09.391 DEBUG:boto:Data: 2023-09-29T23:32:09.391 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.391 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.391 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.391 DEBUG:boto:Params: {} 2023-09-29T23:32:09.391 DEBUG:boto:Token: None 2023-09-29T23:32:09.391 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-73/obj6 2023-09-29T23:32:09.392 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4Tje+bX4dWrLydJ4TIvLdyNa7Aw= 2023-09-29T23:32:09.392 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4Tje+bX4dWrLydJ4TIvLdyNa7Aw=', 'Content-Length': '0'} 2023-09-29T23:32:09.394 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx000006dd35cfb67af4c18-0065175e79-2abe-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.394 DEBUG:boto:path=/hcnphz-73/obj7 2023-09-29T23:32:09.394 DEBUG:boto:auth_path=/hcnphz-73/obj7 2023-09-29T23:32:09.394 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.394 DEBUG:boto:Path: /hcnphz-73/obj7 2023-09-29T23:32:09.394 DEBUG:boto:Data: 2023-09-29T23:32:09.395 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.395 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.395 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.395 DEBUG:boto:Params: {} 2023-09-29T23:32:09.395 DEBUG:boto:Token: None 2023-09-29T23:32:09.395 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-73/obj7 2023-09-29T23:32:09.395 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:/s6vKa7CDj2q9FVnwlTCPM4TsYc= 2023-09-29T23:32:09.395 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/s6vKa7CDj2q9FVnwlTCPM4TsYc=', 'Content-Length': '0'} 2023-09-29T23:32:09.397 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000078691a6996e762bf-0065175e79-2abe-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.397 DEBUG:boto:path=/hcnphz-73/obj8 2023-09-29T23:32:09.398 DEBUG:boto:auth_path=/hcnphz-73/obj8 2023-09-29T23:32:09.398 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.398 DEBUG:boto:Path: /hcnphz-73/obj8 2023-09-29T23:32:09.398 DEBUG:boto:Data: 2023-09-29T23:32:09.398 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.398 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.398 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.398 DEBUG:boto:Params: {} 2023-09-29T23:32:09.399 DEBUG:boto:Token: None 2023-09-29T23:32:09.399 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-73/obj8 2023-09-29T23:32:09.399 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:cS1e/5ivSxJxeogiSdPnaiS59GM= 2023-09-29T23:32:09.399 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cS1e/5ivSxJxeogiSdPnaiS59GM=', 'Content-Length': '0'} 2023-09-29T23:32:09.401 DEBUG:boto:Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000055bb591b26c1f156-0065175e79-2abe-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.401 DEBUG:boto:path=/hcnphz-72/ 2023-09-29T23:32:09.402 DEBUG:boto:auth_path=/hcnphz-72/ 2023-09-29T23:32:09.402 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.402 DEBUG:boto:Path: /hcnphz-72/ 2023-09-29T23:32:09.402 DEBUG:boto:Data: 2023-09-29T23:32:09.402 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.402 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.402 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.402 DEBUG:boto:Params: {} 2023-09-29T23:32:09.402 DEBUG:boto:Token: None 2023-09-29T23:32:09.403 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/ 2023-09-29T23:32:09.403 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:4xd9Ox1hAci9ZdGPzyW5Zyd12C8= 2023-09-29T23:32:09.403 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4xd9Ox1hAci9ZdGPzyW5Zyd12C8=', 'Content-Length': '0'} 2023-09-29T23:32:09.409 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000053a6a3ee740f893-0065175e79-2abe-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.409 DEBUG:boto:path=/hcnphz-72/obj7 2023-09-29T23:32:09.409 DEBUG:boto:auth_path=/hcnphz-72/obj7 2023-09-29T23:32:09.409 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.410 DEBUG:boto:Path: /hcnphz-72/obj7 2023-09-29T23:32:09.410 DEBUG:boto:Data: 2023-09-29T23:32:09.410 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.410 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.410 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.410 DEBUG:boto:Params: {} 2023-09-29T23:32:09.410 DEBUG:boto:Token: None 2023-09-29T23:32:09.410 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj7 2023-09-29T23:32:09.411 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:az9KQNEG6KLbLVUt9AqjX0jrMRE= 2023-09-29T23:32:09.411 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:az9KQNEG6KLbLVUt9AqjX0jrMRE=', 'Content-Length': '0'} 2023-09-29T23:32:09.414 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000dc70f25e04af9bc6-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.414 DEBUG:boto:path=/hcnphz-72/obj7 2023-09-29T23:32:09.414 DEBUG:boto:auth_path=/hcnphz-72/obj7 2023-09-29T23:32:09.415 DEBUG:boto:Method: GET 2023-09-29T23:32:09.415 DEBUG:boto:Path: /hcnphz-72/obj7 2023-09-29T23:32:09.415 DEBUG:boto:Data: 2023-09-29T23:32:09.415 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.415 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.415 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.415 DEBUG:boto:Params: {} 2023-09-29T23:32:09.415 DEBUG:boto:Token: None 2023-09-29T23:32:09.415 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj7 2023-09-29T23:32:09.416 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Csxy2ag50HDlbMThAt+yifjhtek= 2023-09-29T23:32:09.416 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Csxy2ag50HDlbMThAt+yifjhtek=', 'Content-Length': '0'} 2023-09-29T23:32:09.418 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000002068b220716a81c9-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.419 DEBUG:boto:path=/hcnphz-72/obj8 2023-09-29T23:32:09.419 DEBUG:boto:auth_path=/hcnphz-72/obj8 2023-09-29T23:32:09.419 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.419 DEBUG:boto:Path: /hcnphz-72/obj8 2023-09-29T23:32:09.419 DEBUG:boto:Data: 2023-09-29T23:32:09.419 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.419 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.419 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.419 DEBUG:boto:Params: {} 2023-09-29T23:32:09.420 DEBUG:boto:Token: None 2023-09-29T23:32:09.420 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj8 2023-09-29T23:32:09.420 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tPXlKc3mvndZw/16yIFIs/V7Gzs= 2023-09-29T23:32:09.420 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tPXlKc3mvndZw/16yIFIs/V7Gzs=', 'Content-Length': '0'} 2023-09-29T23:32:09.423 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f3e35390d37d2737-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.423 DEBUG:boto:path=/hcnphz-72/obj8 2023-09-29T23:32:09.423 DEBUG:boto:auth_path=/hcnphz-72/obj8 2023-09-29T23:32:09.423 DEBUG:boto:Method: GET 2023-09-29T23:32:09.423 DEBUG:boto:Path: /hcnphz-72/obj8 2023-09-29T23:32:09.423 DEBUG:boto:Data: 2023-09-29T23:32:09.423 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.424 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.424 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.424 DEBUG:boto:Params: {} 2023-09-29T23:32:09.424 DEBUG:boto:Token: None 2023-09-29T23:32:09.424 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj8 2023-09-29T23:32:09.424 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znf7xhzVCv3bhULQAZi2Xa5Ju4M= 2023-09-29T23:32:09.424 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znf7xhzVCv3bhULQAZi2Xa5Ju4M=', 'Content-Length': '0'} 2023-09-29T23:32:09.428 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000899091d8be6e4aba-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.470 DEBUG:boto:path=/hcnphz-72/obj7 2023-09-29T23:32:09.471 DEBUG:boto:auth_path=/hcnphz-72/obj7 2023-09-29T23:32:09.471 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.471 DEBUG:boto:Path: /hcnphz-72/obj7 2023-09-29T23:32:09.471 DEBUG:boto:Data: 2023-09-29T23:32:09.471 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.471 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.471 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.471 DEBUG:boto:Params: {} 2023-09-29T23:32:09.472 DEBUG:boto:Token: None 2023-09-29T23:32:09.472 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj7 2023-09-29T23:32:09.472 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:az9KQNEG6KLbLVUt9AqjX0jrMRE= 2023-09-29T23:32:09.472 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:az9KQNEG6KLbLVUt9AqjX0jrMRE=', 'Content-Length': '0'} 2023-09-29T23:32:09.474 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000037abd9e2082c5dba-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.475 DEBUG:boto:path=/hcnphz-72/obj7 2023-09-29T23:32:09.475 DEBUG:boto:auth_path=/hcnphz-72/obj7 2023-09-29T23:32:09.475 DEBUG:boto:Method: GET 2023-09-29T23:32:09.475 DEBUG:boto:Path: /hcnphz-72/obj7 2023-09-29T23:32:09.475 DEBUG:boto:Data: 2023-09-29T23:32:09.475 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.475 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.476 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.476 DEBUG:boto:Params: {} 2023-09-29T23:32:09.476 DEBUG:boto:Token: None 2023-09-29T23:32:09.476 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj7 2023-09-29T23:32:09.476 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Csxy2ag50HDlbMThAt+yifjhtek= 2023-09-29T23:32:09.476 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Csxy2ag50HDlbMThAt+yifjhtek=', 'Content-Length': '0'} 2023-09-29T23:32:09.479 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000fff72645760d09c-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.479 DEBUG:boto:path=/hcnphz-72/obj8 2023-09-29T23:32:09.479 DEBUG:boto:auth_path=/hcnphz-72/obj8 2023-09-29T23:32:09.479 DEBUG:boto:Method: HEAD 2023-09-29T23:32:09.479 DEBUG:boto:Path: /hcnphz-72/obj8 2023-09-29T23:32:09.479 DEBUG:boto:Data: 2023-09-29T23:32:09.480 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.480 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.480 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.480 DEBUG:boto:Params: {} 2023-09-29T23:32:09.480 DEBUG:boto:Token: None 2023-09-29T23:32:09.480 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj8 2023-09-29T23:32:09.480 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tPXlKc3mvndZw/16yIFIs/V7Gzs= 2023-09-29T23:32:09.480 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tPXlKc3mvndZw/16yIFIs/V7Gzs=', 'Content-Length': '0'} 2023-09-29T23:32:09.483 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a0cec44eca3b9c53-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.483 DEBUG:boto:path=/hcnphz-72/obj8 2023-09-29T23:32:09.483 DEBUG:boto:auth_path=/hcnphz-72/obj8 2023-09-29T23:32:09.483 DEBUG:boto:Method: GET 2023-09-29T23:32:09.483 DEBUG:boto:Path: /hcnphz-72/obj8 2023-09-29T23:32:09.484 DEBUG:boto:Data: 2023-09-29T23:32:09.484 DEBUG:boto:Headers: {} 2023-09-29T23:32:09.484 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:32:09.484 DEBUG:boto:Port: 8001 2023-09-29T23:32:09.484 DEBUG:boto:Params: {} 2023-09-29T23:32:09.484 DEBUG:boto:Token: None 2023-09-29T23:32:09.484 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:32:09 GMT /hcnphz-72/obj8 2023-09-29T23:32:09.484 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:znf7xhzVCv3bhULQAZi2Xa5Ju4M= 2023-09-29T23:32:09.485 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:09 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:znf7xhzVCv3bhULQAZi2Xa5Ju4M=', 'Content-Length': '0'} 2023-09-29T23:32:09.487 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:31:48 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-73:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10033.2'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000ac4ce0bd253be84-0065175e79-2abe-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:32:09 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:09.530 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:09.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:09.531 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hcnphz-72 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:09.576 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:09.576 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:09.692 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:32:09.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:09.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:09.692 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:09.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:09.830 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:32:09.830 INFO:tasks.rgw_multisite_tests:test_sync_multiple_buckets_to_single: ... ok 2023-09-29T23:32:09.831 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:09.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:09.832 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:09.915 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:09.916 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:09.976 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:32:10.005 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:32:10.005 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030329.651174_877.1", 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:32:09.651174Z" 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:10.006 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:32:10.007 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.007 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.007 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:10.007 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:32:10.007 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:10.007 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:32:10.008 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:10.009 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:32:10.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:10.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:10.009 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:10.053 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:10.113 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:10.111+0000 7f10235aaf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:10.114 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:10.115+0000 7f10235aaf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:10.142 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:10.143 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:10.144 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:10.145 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:29:44.158375Z", 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:10.146 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:10.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:10.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:10Z" 2023-09-29T23:32:10.149 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:10.150 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030184.158375_875.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:29:44.158375Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:10Z" } 2023-09-29T23:32:10.150 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030184.158375_875.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:29:44.158375Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:10.151 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:10.151 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:32:10.151 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030329.651174_877.1 target=1_1696030184.158375_875.1 2023-09-29T23:32:15.153 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:15.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:15.154 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:15.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:15.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:15.258 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:15.259+0000 7f37eaa2ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:15.259 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:15.259+0000 7f37eaa2ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:15.286 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:15.286 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:15.287 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:15.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:15.289 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:29:44.158375Z", 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:15.290 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:15.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:15.292 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:15.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:15.294 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:15Z" 2023-09-29T23:32:15.294 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:15.295 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030184.158375_875.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:29:44.158375Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:15Z" } 2023-09-29T23:32:15.295 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030184.158375_875.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:29:44.158375Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:15.295 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:15.295 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:32:15.295 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030329.651174_877.1 target=1_1696030184.158375_875.1 2023-09-29T23:32:20.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:20.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:20.297 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:20.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:20.341 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:20.398 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:20.398+0000 7f9ddb21ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:20.399 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:20.398+0000 7f9ddb21ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:20.426 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:20.427 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:20.428 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:20.429 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:29:44.158375Z", 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:20.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:20.431 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:20.432 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:20.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:20Z" 2023-09-29T23:32:20.434 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:20.435 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030184.158375_875.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:29:44.158375Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:20Z" } 2023-09-29T23:32:20.436 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030184.158375_875.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:29:44.158375Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:20.436 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:20.436 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:32:20.436 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030329.651174_877.1 target=1_1696030184.158375_875.1 2023-09-29T23:32:25.437 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:25.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:25.438 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:25.485 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:25.486 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:25.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:25.542+0000 7f5339267f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:25.543 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:25.542+0000 7f5339267f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:25.571 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:25.571 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:25.571 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:25.571 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:25.571 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:25.572 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:25.573 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030184.158375_875.1", 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:29:44.158375Z", 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:25.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:25.575 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:25.576 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:25.577 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:25.578 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:25.578 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:25Z" 2023-09-29T23:32:25.578 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:25.579 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030184.158375_875.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:29:44.158375Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:25Z" } 2023-09-29T23:32:25.579 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030184.158375_875.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:29:44.158375Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:25.579 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:25.579 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030184.158375_875.1', 2: '', 3: ''} 2023-09-29T23:32:25.579 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030329.651174_877.1 target=1_1696030184.158375_875.1 2023-09-29T23:32:30.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:30.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:30.581 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:30.627 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:30.627 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:30.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:30.678+0000 7f1f8b769f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:30.682 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:30.682+0000 7f1f8b769f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:30.709 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:30.709 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:30.710 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:30.711 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030329.651174_877.1", 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:09.651174Z", 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:30.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:30.713 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:30.714 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:30.715 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:30.716 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:30Z" 2023-09-29T23:32:30.716 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:30.717 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030329.651174_877.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:09.651174Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:30Z" } 2023-09-29T23:32:30.717 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030329.651174_877.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:09.651174Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:30.717 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:30.717 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030329.651174_877.1', 2: '', 3: ''} 2023-09-29T23:32:30.718 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:32:30.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:30.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:30.718 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:30.758 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:30.851 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[]} 2023-09-29T23:32:30.852 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-74 2023-09-29T23:32:30.852 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:32:30.852 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:32:30.852 DEBUG:boto:Method: PUT 2023-09-29T23:32:30.852 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:32:30.852 DEBUG:boto:Data: 2023-09-29T23:32:30.852 DEBUG:boto:Headers: {} 2023-09-29T23:32:30.852 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:32:30.853 DEBUG:boto:Port: 8000 2023-09-29T23:32:30.853 DEBUG:boto:Params: {} 2023-09-29T23:32:30.853 DEBUG:boto:Token: None 2023-09-29T23:32:30.853 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:32:30 GMT /hcnphz-74/ 2023-09-29T23:32:30.853 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:FaIY4lxkhL0f15nD6NyHqEwNuTE= 2023-09-29T23:32:30.853 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FaIY4lxkhL0f15nD6NyHqEwNuTE=', 'Content-Length': '0'} 2023-09-29T23:32:30.883 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000018e793553355b269-0065175e8e-2ac0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:32:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:30.883 INFO:rgw_multi.tests:create bucket zone=a1 name=hcnphz-75 2023-09-29T23:32:30.883 DEBUG:boto:path=/hcnphz-75/ 2023-09-29T23:32:30.883 DEBUG:boto:auth_path=/hcnphz-75/ 2023-09-29T23:32:30.883 DEBUG:boto:Method: PUT 2023-09-29T23:32:30.884 DEBUG:boto:Path: /hcnphz-75/ 2023-09-29T23:32:30.884 DEBUG:boto:Data: 2023-09-29T23:32:30.884 DEBUG:boto:Headers: {} 2023-09-29T23:32:30.884 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:32:30.884 DEBUG:boto:Port: 8000 2023-09-29T23:32:30.884 DEBUG:boto:Params: {} 2023-09-29T23:32:30.884 DEBUG:boto:Token: None 2023-09-29T23:32:30.884 DEBUG:boto:StringToSign: PUT Fri, 29 Sep 2023 23:32:30 GMT /hcnphz-75/ 2023-09-29T23:32:30.884 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:zbnqLcIVR7xKzDj5IZ0JcdYw8IM= 2023-09-29T23:32:30.885 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zbnqLcIVR7xKzDj5IZ0JcdYw8IM=', 'Content-Length': '0'} 2023-09-29T23:32:30.915 DEBUG:boto:Response headers: [('x-amz-request-id', 'tx0000086a8b4e2c15686a3-0065175e8e-2ac0-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:32:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:30.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:30.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:30.916 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:30.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:30.959 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:31.019 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.039 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:32:30.916514Z" 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:32:31.040 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:31.041 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:32:31.041 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:31.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:32:31.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:31.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:31.042 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:31.134 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:31.135 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:31.182 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:31.182+0000 7f9892b0cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:31.184 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:31.182+0000 7f9892b0cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:31.210 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:31.210 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.211 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:31.212 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.874022_879.1", 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.874022Z", 2023-09-29T23:32:31.213 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:31.214 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:31.215 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:31.216 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:31.217 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:31Z" 2023-09-29T23:32:31.217 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:31.218 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.874022_879.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.874022Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:31Z" } 2023-09-29T23:32:31.218 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.874022_879.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.874022Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:31.218 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:31.218 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.874022_879.1', 2: '', 3: ''} 2023-09-29T23:32:31.218 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030350.916514_886.1 target=1_1696030350.874022_879.1 2023-09-29T23:32:36.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:36.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:36.220 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:36.265 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:36.265 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:36.320 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:36.318+0000 7fab6eff3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:36.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:36.322+0000 7fab6eff3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:36.348 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:36.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:36.350 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:32:36.351 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:36.352 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:36.353 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:36.354 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:36.355 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:36.356 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:36Z" 2023-09-29T23:32:36.356 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:36.356 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:36Z" } 2023-09-29T23:32:36.357 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:36.357 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:36.357 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:36.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:32:36.357 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.358 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:36.445 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:36.445 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:36.540 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:32:36.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.541 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:36.591 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:36.591 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:36.684 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:32:36.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-74', '--dest-bucket=hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-74', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-74', '--dest-bucket=hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.685 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hcnphz-74 --source-zones=a1 --dest-zones=a2 --source-bucket=hcnphz-74 --dest-bucket=hcnphz-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:36.733 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:36.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:36.830 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:32:36.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-74', '--dest-bucket=hcnphz-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe-hcnphz-75', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-74', '--dest-bucket=hcnphz-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.830 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe-hcnphz-75 --source-zones=a1 --dest-zones=a2 --source-bucket=hcnphz-74 --dest-bucket=hcnphz-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:36.878 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:36.879 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:36.968 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:32:36.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-74', '--dest-bucket=hcnphz-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1', '--dest-zones=a2', '--source-bucket=hcnphz-74', '--dest-bucket=hcnphz-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:36.969 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1 --dest-zones=a2 --source-bucket=hcnphz-74 --dest-bucket=hcnphz-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:37.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:37.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:37.111 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:32:37.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:37.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:37.112 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:37.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:37.155 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:37.244 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:32:37.244 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:37.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:37.245 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:37.283 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:37.284 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:37.394 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:37.394 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:37.394 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 22, 2023-09-29T23:32:37.394 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:32:37.395 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:32:37.396 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:32:37.397 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:32:37.398 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:32:37.399 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.400 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:32:37.401 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:32:37.402 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-hcnphz-74", 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:32:37.403 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:32:37.404 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-hcnphz-75", 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:32:37.406 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-75", 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:32:37.408 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-75", 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.409 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:32:37.410 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "enabled" 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:32:37.411 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:32:37.412 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:32:37.413 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:32:37.413 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.417 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:32:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:32:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:32:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:32:37.418 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.419 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:32:37.420 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.421 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.422 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:32:37.423 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.424 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:32:37.425 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:32:37.426 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:32:37.427 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.428 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:32:37.429 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:37.430 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:37.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:37.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:32:37.431 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:32:37.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:37.513 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:37.604 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:32:37.605 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:32:37.605 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:32:37.605 DEBUG:boto:Method: HEAD 2023-09-29T23:32:37.605 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:32:37.605 DEBUG:boto:Data: 2023-09-29T23:32:37.606 DEBUG:boto:Headers: {} 2023-09-29T23:32:37.606 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:32:37.606 DEBUG:boto:Port: 8000 2023-09-29T23:32:37.606 DEBUG:boto:Params: {} 2023-09-29T23:32:37.606 DEBUG:boto:Token: None 2023-09-29T23:32:37.606 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:37 GMT /hcnphz-74/ 2023-09-29T23:32:37.606 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L8qywe3SGkv3Iei8MQFwORsl//A= 2023-09-29T23:32:37.607 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L8qywe3SGkv3Iei8MQFwORsl//A=', 'Content-Length': '0'} 2023-09-29T23:32:37.728 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003ada1a8a1ea14b33-0065175e95-2c94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:37.729 DEBUG:boto:path=/hcnphz-74/obj1 2023-09-29T23:32:37.729 DEBUG:boto:auth_path=/hcnphz-74/obj1 2023-09-29T23:32:37.729 DEBUG:boto:Method: PUT 2023-09-29T23:32:37.729 DEBUG:boto:Path: /hcnphz-74/obj1 2023-09-29T23:32:37.729 DEBUG:boto:Data: 2023-09-29T23:32:37.729 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:32:37.730 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:32:37.730 DEBUG:boto:Port: 8000 2023-09-29T23:32:37.730 DEBUG:boto:Params: {} 2023-09-29T23:32:37.730 DEBUG:boto:Token: None 2023-09-29T23:32:37.730 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:32:37 GMT /hcnphz-74/obj1 2023-09-29T23:32:37.730 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:aP7IK7tXQF6ieTdqdfMntHx0LQY= 2023-09-29T23:32:37.730 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:32:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aP7IK7tXQF6ieTdqdfMntHx0LQY='} 2023-09-29T23:32:37.751 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d474f708443c2c5f-0065175e95-2c94-a1'), ('Date', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:37.751 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:32:37.751 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:32:37.751 DEBUG:boto:Method: HEAD 2023-09-29T23:32:37.751 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:32:37.751 DEBUG:boto:Data: 2023-09-29T23:32:37.751 DEBUG:boto:Headers: {} 2023-09-29T23:32:37.751 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:32:37.752 DEBUG:boto:Port: 8000 2023-09-29T23:32:37.752 DEBUG:boto:Params: {} 2023-09-29T23:32:37.752 DEBUG:boto:Token: None 2023-09-29T23:32:37.752 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:32:37 GMT /hcnphz-74/ 2023-09-29T23:32:37.752 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:L8qywe3SGkv3Iei8MQFwORsl//A= 2023-09-29T23:32:37.752 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:32:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L8qywe3SGkv3Iei8MQFwORsl//A=', 'Content-Length': '0'} 2023-09-29T23:32:37.756 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008e03344e30b365c1-0065175e95-2c94-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:37.756 DEBUG:boto:path=/hcnphz-74/obj2 2023-09-29T23:32:37.757 DEBUG:boto:auth_path=/hcnphz-74/obj2 2023-09-29T23:32:37.757 DEBUG:boto:Method: PUT 2023-09-29T23:32:37.757 DEBUG:boto:Path: /hcnphz-74/obj2 2023-09-29T23:32:37.757 DEBUG:boto:Data: 2023-09-29T23:32:37.757 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:32:37.757 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:32:37.757 DEBUG:boto:Port: 8000 2023-09-29T23:32:37.757 DEBUG:boto:Params: {} 2023-09-29T23:32:37.757 DEBUG:boto:Token: None 2023-09-29T23:32:37.758 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:32:37 GMT /hcnphz-74/obj2 2023-09-29T23:32:37.758 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:YR0fDq/O7BC1HmkeWjpSPexnVbA= 2023-09-29T23:32:37.758 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:32:37 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YR0fDq/O7BC1HmkeWjpSPexnVbA='} 2023-09-29T23:32:37.802 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000177cdbab2e809bd1-0065175e95-2c94-a1'), ('Date', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:32:37.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:37.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:37.803 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:37.865 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:37.865 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:37.967 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:32:38.001 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:32:38.001 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:32:30.916514Z" 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.002 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.003 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:32:38.004 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:38.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:32:38.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:38.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:38.005 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:38.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:38.056 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:38.135 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:38.134+0000 7f1b11613f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:38.136 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:32:38.134+0000 7f1b11613f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:32:38.165 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:38.166 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.167 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.168 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:38.169 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.170 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.171 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:32:38.172 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:32:38.173 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.173 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:38Z" 2023-09-29T23:32:38.173 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:38.174 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:32:38Z" } 2023-09-29T23:32:38.174 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:32:38.174 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:38.174 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:32:38.175 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:32:38.175 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:38.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:38.175 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:38.220 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:38.310 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:32:38.310 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:32:38.311 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:32:38.312 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.312 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.312 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:38.312 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:38.312 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.312 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:32:38.313 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:38.313 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:32:38.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:38.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:38.314 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:38.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:38.361 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:38.431 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:38.431 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:38.431 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:38.432 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.433 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.434 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:32:38.435 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:32:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.436 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:38.437 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:38.438 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:38Z" 2023-09-29T23:32:38.439 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:38.439 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:38Z" } 2023-09-29T23:32:38.440 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:32:38.440 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:38.440 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:38.440 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:32:43.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:43.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:43.442 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:43.491 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:43.492 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:43.580 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:43.580 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:43.580 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:43.581 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:43.582 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:43.583 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:43.584 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:43.585 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:43Z" 2023-09-29T23:32:43.586 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:43.587 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:43Z" } 2023-09-29T23:32:43.588 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:32:43.588 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:43.588 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:43.588 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:32:48.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:48.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:48.589 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:48.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:48.637 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:48.722 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:48.722 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:48.722 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:48.722 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:48.722 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:48.722 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:48.723 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:48.724 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:48.725 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:48.726 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:48.727 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:48Z" 2023-09-29T23:32:48.728 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:48.729 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:48Z" } 2023-09-29T23:32:48.730 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:32:48.730 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:48.730 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:48.730 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:32:53.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:53.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:53.731 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:53.778 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:53.779 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:53.864 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:53.864 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:53.864 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:53.864 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:53.865 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:53.866 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:53.867 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:53.868 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:53.869 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:53.870 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:53.870 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:53.870 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:53.870 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:53.870 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:53.870 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:53Z" 2023-09-29T23:32:53.871 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:53.872 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:53Z" } 2023-09-29T23:32:53.873 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:32:53.873 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:53.873 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:53.873 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:32:58.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:58.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:32:58.875 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:32:58.926 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:32:58.926 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:32:59.012 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:32:59.013 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:59.014 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:59.015 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:59.016 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:32:59.017 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:32:59.018 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:32:59.019 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:32:58Z" 2023-09-29T23:32:59.019 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:32:59.019 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:32:58Z" } 2023-09-29T23:32:59.020 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:32:59.020 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:59.020 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:32:59.020 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:04.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:04.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:04.021 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:04.072 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:04.072 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:04.161 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:04.161 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:04.161 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:04.161 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:04.162 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:04.163 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:04.164 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:04.165 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:04.166 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:04.167 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:04.168 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:04.168 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:04.168 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:04.168 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:04Z" 2023-09-29T23:33:04.168 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:04.169 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:04Z" } 2023-09-29T23:33:04.169 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:04.169 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:04.169 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:04.169 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:09.170 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:09.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:09.171 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:09.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:09.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:09.299 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:09.299 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:09.300 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:09.301 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:09.302 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:09.303 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:09.304 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:09.305 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:09Z" 2023-09-29T23:33:09.305 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:09.306 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:09Z" } 2023-09-29T23:33:09.306 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:09.306 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:09.306 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:09.307 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:14.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:14.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:14.308 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:14.354 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:14.355 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:14.443 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:14.444 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:14.445 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:14.446 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:14.447 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:14.448 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:14.449 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:14.449 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:14Z" 2023-09-29T23:33:14.449 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:14.450 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:14Z" } 2023-09-29T23:33:14.450 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:14.450 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:14.450 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:14.450 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:19.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:19.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:19.452 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:19.502 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:19.503 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:19.586 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:19.587 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:19.588 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:19.589 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:19.590 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:19.591 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:19Z" 2023-09-29T23:33:19.592 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:19.593 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:19Z" } 2023-09-29T23:33:19.594 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:19.594 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:19.594 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:19.594 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:24.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:24.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:24.596 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:24.645 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:24.645 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:24.734 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:24.734 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:24.734 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:24.734 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:24.735 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:24.736 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:24.737 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:24.738 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:24.739 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:24.740 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:24.741 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:24.742 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:24Z" 2023-09-29T23:33:24.742 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:24.743 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:24Z" } 2023-09-29T23:33:24.743 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:24.744 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:24.744 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:24.744 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:29.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:29.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:29.745 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:29.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:29.793 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:29.882 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:29.883 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:29.884 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:29.885 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:29.886 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:29.887 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:29.888 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:29.888 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:29.888 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:29Z" 2023-09-29T23:33:29.888 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:29.889 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:29Z" } 2023-09-29T23:33:29.889 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:29.889 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:29.889 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:29.889 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:34.891 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:34.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:34.891 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:34.938 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:35.027 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:35.027 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:35.028 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:35.029 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:35.030 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:35.031 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:35.033 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:35.033 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:35.033 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:35.033 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:35.033 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:35.034 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:35.035 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:35.036 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:35.036 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:35Z" 2023-09-29T23:33:35.036 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:35.037 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:35Z" } 2023-09-29T23:33:35.037 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:35.037 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:35.037 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:35.038 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:40.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:40.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:40.039 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:40.086 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:40.174 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:40.175 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:40.176 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:40.177 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:40.178 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:40.179 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:40.180 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:40.181 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:40Z" 2023-09-29T23:33:40.181 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:40.183 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:40Z" } 2023-09-29T23:33:40.183 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:40.183 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:40.184 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:40.184 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:45.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:45.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:45.185 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:45.233 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:45.233 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:45.335 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:45.336 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:45.337 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:45.338 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:45.339 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:45.340 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:45Z" 2023-09-29T23:33:45.341 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:45.342 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:45Z" } 2023-09-29T23:33:45.343 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:45.343 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:45.343 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:45.343 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:50.344 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:50.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:50.345 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:50.391 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:50.392 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:50.478 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:50.479 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:50.479 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:50.479 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:50.479 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:50.479 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:50.479 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:50.480 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:50.481 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:50.481 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:50.481 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:50.481 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:50.481 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:50.481 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:50.482 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:50.483 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:50.484 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:50.485 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:50.486 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:50Z" 2023-09-29T23:33:50.486 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:50.487 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:50Z" } 2023-09-29T23:33:50.487 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:50.487 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:50.487 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:50.487 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:33:55.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:55.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:33:55.488 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:33:55.537 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:33:55.625 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:33:55.625 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:33:55.625 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:33:55.625 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:33:55.625 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:33:55.626 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:33:55.626 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:55.626 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:33:55.626 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:55.627 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:33:55.627 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:55.627 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:55.627 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:33:55.627 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:55.628 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:33:55.628 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:55.628 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:33:55.628 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:55.628 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:55.628 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:55.629 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:33:55.629 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:55.629 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:55.629 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:33:55.629 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:55.630 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:33:55.630 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:55.630 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:33:55.630 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:55.630 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:55.631 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:55.631 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:33:55.631 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:55.631 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:55.631 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:33:55.631 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:55.632 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:55.632 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:55.632 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:33:55.632 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:55.632 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:55.633 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:33:55.633 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:33:55.633 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:33:55.633 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:33:55.633 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:33:55.634 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:33:55.634 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:33:55.634 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:33:55.634 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:33:55.634 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:55.635 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:33:55.635 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:33:55.635 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:55.635 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:33:55.635 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:33:55.635 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:33:55.636 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:33:55.636 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:33:55Z" 2023-09-29T23:33:55.636 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:33:55.637 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:33:55Z" } 2023-09-29T23:33:55.637 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:33:55.638 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:55.638 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:33:55.638 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:34:00.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:00.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:00.640 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:00.687 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:00.688 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:00.774 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:00.774 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:34:00.775 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:00.776 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:34:00.777 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:34:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:00.778 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:00.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:00.780 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:00Z" 2023-09-29T23:34:00.781 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:00.781 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:34:00Z" } 2023-09-29T23:34:00.782 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:34:00.782 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:00.782 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:00.782 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:34:05.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:05.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:05.783 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:05.830 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:05.831 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:34:05.919 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052372", 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:34:05.920 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.115299Z" 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:05.921 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:05.922 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046218", 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.055073Z" 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:05.923 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:05.924 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:34:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:34:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:05.925 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:05Z" 2023-09-29T23:34:05.925 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:05.926 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052372", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:31:48.115299Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046218", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.055073Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:34:05Z" } 2023-09-29T23:34:05.926 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052372', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.115299Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046218', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.055073Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:34:05.926 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:05.927 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052372', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046218', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:05.927 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052513 target=00000000000000000000:00000000000000052372, shard=2 master=00000000000000000000:00000000000000046359 target=00000000000000000000:00000000000000046218 2023-09-29T23:34:09.327 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:34:09.322+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:34:10.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:10.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:10.928 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:10.974 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:10.975 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:11.061 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:11.061 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:11.061 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:11.061 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:11.061 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:11.062 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.063 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:34:11.064 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:11.065 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:34:11.066 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:11.066 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:11.066 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:11.069 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:34:11.069 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.069 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:11.070 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:11Z" 2023-09-29T23:34:11.071 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:11.072 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:34:11Z" } 2023-09-29T23:34:11.073 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:34:11.073 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:11.073 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:11.073 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:34:11.073 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:34:11.073 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:34:11.074 DEBUG:boto:Method: HEAD 2023-09-29T23:34:11.074 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:34:11.074 DEBUG:boto:Data: 2023-09-29T23:34:11.074 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.074 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.074 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.074 DEBUG:boto:Params: {} 2023-09-29T23:34:11.075 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:34:11.075 DEBUG:boto:Token: None 2023-09-29T23:34:11.075 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-74/ 2023-09-29T23:34:11.075 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:9prh6LsCsWTpsbhDah4hJBzhuvU= 2023-09-29T23:34:11.075 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9prh6LsCsWTpsbhDah4hJBzhuvU=', 'Content-Length': '0'} 2023-09-29T23:34:11.085 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021bf6ecdb709d5d0-0065175ef3-2c96-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.085 DEBUG:boto:path=/hcnphz-74/obj1 2023-09-29T23:34:11.085 DEBUG:boto:auth_path=/hcnphz-74/obj1 2023-09-29T23:34:11.085 DEBUG:boto:Method: HEAD 2023-09-29T23:34:11.086 DEBUG:boto:Path: /hcnphz-74/obj1 2023-09-29T23:34:11.086 DEBUG:boto:Data: 2023-09-29T23:34:11.086 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.086 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.086 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.086 DEBUG:boto:Params: {} 2023-09-29T23:34:11.086 DEBUG:boto:Token: None 2023-09-29T23:34:11.086 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-74/obj1 2023-09-29T23:34:11.087 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:Y+YnhidQjTSCSPJuc2WCLxIMAjs= 2023-09-29T23:34:11.087 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+YnhidQjTSCSPJuc2WCLxIMAjs=', 'Content-Length': '0'} 2023-09-29T23:34:11.091 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000412167b6ebbb660-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.091 DEBUG:boto:path=/hcnphz-74/obj1 2023-09-29T23:34:11.091 DEBUG:boto:auth_path=/hcnphz-74/obj1 2023-09-29T23:34:11.091 DEBUG:boto:Method: GET 2023-09-29T23:34:11.091 DEBUG:boto:Path: /hcnphz-74/obj1 2023-09-29T23:34:11.091 DEBUG:boto:Data: 2023-09-29T23:34:11.092 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.092 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.092 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.092 DEBUG:boto:Params: {} 2023-09-29T23:34:11.092 DEBUG:boto:Token: None 2023-09-29T23:34:11.092 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-74/obj1 2023-09-29T23:34:11.092 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:SkuQJwVYyEO/tlaf4AHrfSTC/4E= 2023-09-29T23:34:11.092 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SkuQJwVYyEO/tlaf4AHrfSTC/4E=', 'Content-Length': '0'} 2023-09-29T23:34:11.095 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000020f931610c50a1f-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.096 DEBUG:boto:path=/hcnphz-74/obj2 2023-09-29T23:34:11.096 DEBUG:boto:auth_path=/hcnphz-74/obj2 2023-09-29T23:34:11.096 DEBUG:boto:Method: HEAD 2023-09-29T23:34:11.096 DEBUG:boto:Path: /hcnphz-74/obj2 2023-09-29T23:34:11.096 DEBUG:boto:Data: 2023-09-29T23:34:11.096 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.096 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.096 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.097 DEBUG:boto:Params: {} 2023-09-29T23:34:11.097 DEBUG:boto:Token: None 2023-09-29T23:34:11.097 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-74/obj2 2023-09-29T23:34:11.097 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DyiL0M2B7Y15T9PK0N9HGZ90hSI= 2023-09-29T23:34:11.097 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DyiL0M2B7Y15T9PK0N9HGZ90hSI=', 'Content-Length': '0'} 2023-09-29T23:34:11.100 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000034bb7706ac1974c3-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.100 DEBUG:boto:path=/hcnphz-74/obj2 2023-09-29T23:34:11.100 DEBUG:boto:auth_path=/hcnphz-74/obj2 2023-09-29T23:34:11.100 DEBUG:boto:Method: GET 2023-09-29T23:34:11.100 DEBUG:boto:Path: /hcnphz-74/obj2 2023-09-29T23:34:11.100 DEBUG:boto:Data: 2023-09-29T23:34:11.100 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.101 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.101 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.101 DEBUG:boto:Params: {} 2023-09-29T23:34:11.101 DEBUG:boto:Token: None 2023-09-29T23:34:11.101 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-74/obj2 2023-09-29T23:34:11.102 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fchcQxErIyK+jpHqOEhyZIh5Kx4= 2023-09-29T23:34:11.102 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fchcQxErIyK+jpHqOEhyZIh5Kx4=', 'Content-Length': '0'} 2023-09-29T23:34:11.104 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000025dd81f04b860982-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.146 DEBUG:boto:path=/hcnphz-75/ 2023-09-29T23:34:11.147 DEBUG:boto:auth_path=/hcnphz-75/ 2023-09-29T23:34:11.147 DEBUG:boto:Method: HEAD 2023-09-29T23:34:11.147 DEBUG:boto:Path: /hcnphz-75/ 2023-09-29T23:34:11.147 DEBUG:boto:Data: 2023-09-29T23:34:11.148 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.148 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.148 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.148 DEBUG:boto:Params: {} 2023-09-29T23:34:11.148 DEBUG:boto:Token: None 2023-09-29T23:34:11.148 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-75/ 2023-09-29T23:34:11.148 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:yQuoV3jJ9dQXuKCy3gjvFzaBCiY= 2023-09-29T23:34:11.149 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yQuoV3jJ9dQXuKCy3gjvFzaBCiY=', 'Content-Length': '0'} 2023-09-29T23:34:11.153 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000086c54bae6ac481b9-0065175ef3-2c96-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.153 DEBUG:boto:path=/hcnphz-75/obj1 2023-09-29T23:34:11.153 DEBUG:boto:auth_path=/hcnphz-75/obj1 2023-09-29T23:34:11.154 DEBUG:boto:Method: HEAD 2023-09-29T23:34:11.154 DEBUG:boto:Path: /hcnphz-75/obj1 2023-09-29T23:34:11.154 DEBUG:boto:Data: 2023-09-29T23:34:11.154 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.154 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.154 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.154 DEBUG:boto:Params: {} 2023-09-29T23:34:11.154 DEBUG:boto:Token: None 2023-09-29T23:34:11.154 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-75/obj1 2023-09-29T23:34:11.155 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DQCS70gIC4o6OuW46cQDbHpSly8= 2023-09-29T23:34:11.155 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQCS70gIC4o6OuW46cQDbHpSly8=', 'Content-Length': '0'} 2023-09-29T23:34:11.157 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000a6e87741343b742e-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.158 DEBUG:boto:path=/hcnphz-75/obj1 2023-09-29T23:34:11.158 DEBUG:boto:auth_path=/hcnphz-75/obj1 2023-09-29T23:34:11.158 DEBUG:boto:Method: GET 2023-09-29T23:34:11.158 DEBUG:boto:Path: /hcnphz-75/obj1 2023-09-29T23:34:11.158 DEBUG:boto:Data: 2023-09-29T23:34:11.158 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.158 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.158 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.158 DEBUG:boto:Params: {} 2023-09-29T23:34:11.159 DEBUG:boto:Token: None 2023-09-29T23:34:11.159 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-75/obj1 2023-09-29T23:34:11.159 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:W/Pa7oYJfuV72Tl1E1aN8z+NNJ0= 2023-09-29T23:34:11.159 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W/Pa7oYJfuV72Tl1E1aN8z+NNJ0=', 'Content-Length': '0'} 2023-09-29T23:34:11.162 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000759b53a7fab6e770-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.162 DEBUG:boto:path=/hcnphz-75/obj2 2023-09-29T23:34:11.162 DEBUG:boto:auth_path=/hcnphz-75/obj2 2023-09-29T23:34:11.162 DEBUG:boto:Method: HEAD 2023-09-29T23:34:11.163 DEBUG:boto:Path: /hcnphz-75/obj2 2023-09-29T23:34:11.163 DEBUG:boto:Data: 2023-09-29T23:34:11.163 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.163 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.163 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.163 DEBUG:boto:Params: {} 2023-09-29T23:34:11.163 DEBUG:boto:Token: None 2023-09-29T23:34:11.163 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-75/obj2 2023-09-29T23:34:11.163 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:fIjldhWEBj3shu+RZ4jxZRNZggA= 2023-09-29T23:34:11.164 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fIjldhWEBj3shu+RZ4jxZRNZggA=', 'Content-Length': '0'} 2023-09-29T23:34:11.166 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000008c06d07876a84781-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.166 DEBUG:boto:path=/hcnphz-75/obj2 2023-09-29T23:34:11.166 DEBUG:boto:auth_path=/hcnphz-75/obj2 2023-09-29T23:34:11.167 DEBUG:boto:Method: GET 2023-09-29T23:34:11.167 DEBUG:boto:Path: /hcnphz-75/obj2 2023-09-29T23:34:11.167 DEBUG:boto:Data: 2023-09-29T23:34:11.167 DEBUG:boto:Headers: {} 2023-09-29T23:34:11.167 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:34:11.167 DEBUG:boto:Port: 8001 2023-09-29T23:34:11.167 DEBUG:boto:Params: {} 2023-09-29T23:34:11.167 DEBUG:boto:Token: None 2023-09-29T23:34:11.167 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:34:11 GMT /hcnphz-75/obj2 2023-09-29T23:34:11.168 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:6+C2UKf9XJVdneYA/wgh09YEnbs= 2023-09-29T23:34:11.168 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:11 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6+C2UKf9XJVdneYA/wgh09YEnbs=', 'Content-Length': '0'} 2023-09-29T23:34:11.170 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:32:37 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000f357f8f75bd869c9-0065175ef3-2c96-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:34:11 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:11.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'remove', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.171 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe remove --group-id sync-group --pipe-id sync-pipe --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:11.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:11.218 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:11.314 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:34:11.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=*', '--dest-zones=*', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.314 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe '--source-zones=*' '--dest-zones=*' --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:11.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:11.360 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:11.427 INFO:teuthology.orchestra.run.smithi112.stderr:WARNING: cannot find source zone id for name=* 2023-09-29T23:34:11.427 INFO:teuthology.orchestra.run.smithi112.stderr:WARNING: cannot find source zone id for name=* 2023-09-29T23:34:11.462 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:34:11.463 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.463 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:11.507 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:11.508 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:11.602 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:34:11.602 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.602 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:11.645 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:11.646 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:11.759 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:11.759 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:11.759 INFO:teuthology.orchestra.run.smithi112.stdout: "epoch": 23, 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": [], 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "period_map": { 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "zonegroups": [ 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a", 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a", 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:34:11.760 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": true, 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:34:11.761 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a2", 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:34:11.762 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.763 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "a1", 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:34:11.764 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:34:11.765 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:34:11.765 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:34:11.765 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:34:11.765 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:34:11.765 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.765 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.766 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [ 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.767 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-group", 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: "data_flow": { 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: "symmetrical": [ 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-flow1", 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.768 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.769 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: "pipes": [ 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-hcnphz-74", 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:34:11.770 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.771 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:34:11.772 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe-hcnphz-75", 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-74", 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.773 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "hcnphz-75", 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "272b60f1-3a4d-4009-a171-ab410395ba9a" 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:34:11.774 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.775 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "sync-pipe", 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "*" 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": { 2023-09-29T23:34:11.776 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket": "*", 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: "*" 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: "params": { 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: "source": { 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: "filter": { 2023-09-29T23:34:11.777 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [] 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: "dest": {}, 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: "priority": 0, 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: "mode": "system", 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: "user": "" 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.778 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "allowed" 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.779 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "b", 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b", 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "api_name": "", 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "is_master": false, 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames": [], 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "hostnames_s3website": [], 2023-09-29T23:34:11.780 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:34:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: "zones": [ 2023-09-29T23:34:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.781 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:34:11.782 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b2", 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:34:11.783 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.784 INFO:teuthology.orchestra.run.smithi112.stdout: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "b1", 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "endpoints": [ 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "log_meta": false, 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "log_data": true, 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_index_max_shards": 11, 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "read_only": false, 2023-09-29T23:34:11.785 INFO:teuthology.orchestra.run.smithi112.stdout: "tier_type": "", 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from_all": true, 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_from": [], 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "redirect_zone": "", 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "supported_features": [ 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "compress-encrypted", 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.786 INFO:teuthology.orchestra.run.smithi112.stdout: "placement_targets": [ 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: "name": "default-placement", 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: "tags": [], 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: "storage_classes": [ 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: "STANDARD" 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.787 INFO:teuthology.orchestra.run.smithi112.stdout: "default_placement": "default-placement", 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_policy": { 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: "groups": [] 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled_features": [ 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: "resharding" 2023-09-29T23:34:11.788 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: ], 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: "short_zone_ids": [ 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 1230604351 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:34:11.789 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 192990612 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3868157510 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: "val": 3032081421 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.790 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zonegroup": "a", 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "period_config": { 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_quota": { 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:34:11.791 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "user_quota": { 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false, 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "check_on_raw": false, 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size": -1, 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_size_kb": 0, 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_objects": -1 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "user_ratelimit": { 2023-09-29T23:34:11.792 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "bucket_ratelimit": { 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:34:11.793 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "anonymous_ratelimit": { 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_ops": 0, 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_ops": 0, 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "max_read_bytes": 0, 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "max_write_bytes": 0, 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: "enabled": false 2023-09-29T23:34:11.794 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:34:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_name": "test-realm", 2023-09-29T23:34:11.795 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:11.795 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:11.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.797 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:11.836 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:11.925 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:34:11.925 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-bucket', '--status', 'allowed', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:11.926 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-bucket --status allowed --bucket hcnphz-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:11.980 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:11.981 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.069 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:34:12.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-bucket', '--flow-id', 'sync-flowA', '--flow-type', 'symmetrical', '--zones=a1,a2', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.070 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-bucket --flow-id sync-flowA --flow-type symmetrical --zones=a1,a2 --bucket hcnphz-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:12.142 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.256 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:34:12.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-74', '--source-bucket=*', '--dest-bucket=hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-74', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-74', '--source-bucket=*', '--dest-bucket=hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.257 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hcnphz-74 --source-zones=a1 --dest-zones=a2 --bucket=hcnphz-74 '--source-bucket=*' --dest-bucket=hcnphz-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:12.316 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:12.316 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.408 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:34:12.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-74', '--source-bucket=*', '--dest-bucket=hcnphz-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-bucket', '--pipe-id', 'sync-pipe-hcnphz-75', '--source-zones=a1', '--dest-zones=a2', '--bucket=hcnphz-74', '--source-bucket=*', '--dest-bucket=hcnphz-75', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.409 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-bucket --pipe-id sync-pipe-hcnphz-75 --source-zones=a1 --dest-zones=a2 --bucket=hcnphz-74 '--source-bucket=*' --dest-bucket=hcnphz-75 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:12.463 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:12.464 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.562 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"allowed"}]} 2023-09-29T23:34:12.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-bucket', '--status', 'enabled', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.563 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-bucket --status enabled --bucket hcnphz-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:12.611 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.689 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-bucket","data_flow":{"symmetrical":[{"id":"sync-flowA","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"*","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":"rgw-multisite-test-user"}}],"status":"enabled"}]} 2023-09-29T23:34:12.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:34:12.690 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:34:12.727 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:12.728 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.819 INFO:teuthology.orchestra.run.smithi112.stdout:{"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe-hcnphz-74","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-74","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe-hcnphz-75","source":{"bucket":"hcnphz-74","zones":["a1"]},"dest":{"bucket":"hcnphz-75","zones":["a2"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}},{"id":"sync-pipe","source":{"bucket":"*","zones":["*"]},"dest":{"bucket":"*","zones":["*"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:34:12.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:12.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:12.820 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:12.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:12.906 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:12.965 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:34:12.995 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030452.667146_918.1", 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:34:12.667146Z" 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:12.996 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:12.997 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:34:12.999 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:12.999 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:34:12.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:12.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:12.999 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:13.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:13.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:13.115 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:13.116+0000 7f9fa2b4cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:13.117 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:13.116+0000 7f9fa2b4cf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:13.145 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:13.146 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:13.147 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:13.148 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:13.149 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:13.150 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:13.151 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:13.152 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:13Z" 2023-09-29T23:34:13.152 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:13.153 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:13Z" } 2023-09-29T23:34:13.153 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:13.154 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:13.154 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:34:13.154 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030452.667146_918.1 target=1_1696030350.916514_886.1 2023-09-29T23:34:18.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:18.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:18.155 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:18.200 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:18.201 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:18.260 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:18.259+0000 7f7e3dc16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:18.261 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:18.263+0000 7f7e3dc16f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:18.285 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:18.285 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:18.286 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:18.287 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:18.288 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:34:18.289 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:18.290 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:18.291 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:18.292 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:18.293 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:18.294 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:18Z" 2023-09-29T23:34:18.295 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:18.296 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:18Z" } 2023-09-29T23:34:18.297 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:18.297 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:18.298 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:34:18.298 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030452.667146_918.1 target=1_1696030350.916514_886.1 2023-09-29T23:34:23.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:23.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:23.300 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:23.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:23.344 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:23.401 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:23.399+0000 7f180a6d0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:23.402 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:23.403+0000 7f180a6d0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:23.429 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:23.429 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:23.430 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:23.431 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:23.432 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:23.433 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:23.434 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:23Z" 2023-09-29T23:34:23.435 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:23.437 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:23Z" } 2023-09-29T23:34:23.437 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:23.437 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:23.437 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:34:23.437 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030452.667146_918.1 target=1_1696030350.916514_886.1 2023-09-29T23:34:28.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:28.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:28.439 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:28.484 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:28.484 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:28.540 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:28.539+0000 7fbc01b4af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:28.542 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:28.543+0000 7fbc01b4af40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:28.568 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:28.569 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:28.570 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:28.571 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:28.572 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:28.573 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:28.574 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:28Z" 2023-09-29T23:34:28.575 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:28.576 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:28Z" } 2023-09-29T23:34:28.577 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:28.577 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:28.577 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:34:28.577 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030452.667146_918.1 target=1_1696030350.916514_886.1 2023-09-29T23:34:33.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:33.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:33.578 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:33.621 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:33.622 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:33.680 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:33.679+0000 7fb3d9660f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:33.681 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:33.683+0000 7fb3d9660f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:33.708 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:33.709 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:33.710 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:33.711 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:33.712 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:33.713 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:33Z" 2023-09-29T23:34:33.714 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:33.715 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:33Z" } 2023-09-29T23:34:33.715 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:33.715 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:33.716 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:34:33.716 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030452.667146_918.1 target=1_1696030350.916514_886.1 2023-09-29T23:34:38.716 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:38.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:38.717 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:38.761 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:38.817 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:38.819+0000 7f72780d5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:38.819 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:38.819+0000 7f72780d5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:38.846 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:38.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:38.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030350.916514_886.1", 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:38.849 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:30.916514Z", 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:38.850 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:38.851 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:38.852 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:38Z" 2023-09-29T23:34:38.853 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:38.854 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030350.916514_886.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:30.916514Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:38Z" } 2023-09-29T23:34:38.854 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030350.916514_886.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:30.916514Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:38.854 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:38.855 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030350.916514_886.1', 2: '', 3: ''} 2023-09-29T23:34:38.855 WARNING:rgw_multi.tests:zone a2 behind master: shard=1 master=1_1696030452.667146_918.1 target=1_1696030350.916514_886.1 2023-09-29T23:34:43.856 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:43.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:43.856 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:43.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:43.905 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:43.962 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:43.963+0000 7fa066e89f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:43.964 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:43.963+0000 7fa066e89f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:43.991 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:43.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:43.993 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030452.667146_918.1", 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:34:12.667146Z", 2023-09-29T23:34:43.994 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:43.995 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:43.996 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:43.997 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:43Z" 2023-09-29T23:34:43.998 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:43.999 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030452.667146_918.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:34:12.667146Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:43Z" } 2023-09-29T23:34:44.000 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030452.667146_918.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:34:12.667146Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:44.000 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:44.000 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:44.000 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:34:44.000 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:34:44.001 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:34:44.001 DEBUG:boto:Method: HEAD 2023-09-29T23:34:44.001 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:34:44.001 DEBUG:boto:Data: 2023-09-29T23:34:44.001 DEBUG:boto:Headers: {} 2023-09-29T23:34:44.001 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:34:44.001 DEBUG:boto:Port: 8000 2023-09-29T23:34:44.001 DEBUG:boto:Params: {} 2023-09-29T23:34:44.002 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:34:44.002 DEBUG:boto:Token: None 2023-09-29T23:34:44.002 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:44 GMT /hcnphz-74/ 2023-09-29T23:34:44.002 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k6L4QxB/+Hhat+zxZiauo3gGX7Y= 2023-09-29T23:34:44.003 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k6L4QxB/+Hhat+zxZiauo3gGX7Y=', 'Content-Length': '0'} 2023-09-29T23:34:44.012 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009b3848202b2fe5a7-0065175f14-2d72-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:44.012 DEBUG:boto:path=/hcnphz-74/obj3 2023-09-29T23:34:44.012 DEBUG:boto:auth_path=/hcnphz-74/obj3 2023-09-29T23:34:44.013 DEBUG:boto:Method: PUT 2023-09-29T23:34:44.013 DEBUG:boto:Path: /hcnphz-74/obj3 2023-09-29T23:34:44.013 DEBUG:boto:Data: 2023-09-29T23:34:44.013 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:34:44.013 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:34:44.013 DEBUG:boto:Port: 8000 2023-09-29T23:34:44.013 DEBUG:boto:Params: {} 2023-09-29T23:34:44.013 DEBUG:boto:Token: None 2023-09-29T23:34:44.014 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:34:44 GMT /hcnphz-74/obj3 2023-09-29T23:34:44.014 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:tm+Y+C87/yuk4vmVyd/VLsA7FCk= 2023-09-29T23:34:44.014 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:34:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tm+Y+C87/yuk4vmVyd/VLsA7FCk='} 2023-09-29T23:34:44.034 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004f4d38c0f05e82ba-0065175f14-2d72-a1'), ('Date', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:44.035 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:34:44.035 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:34:44.035 DEBUG:boto:Method: HEAD 2023-09-29T23:34:44.035 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:34:44.035 DEBUG:boto:Data: 2023-09-29T23:34:44.035 DEBUG:boto:Headers: {} 2023-09-29T23:34:44.035 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:34:44.035 DEBUG:boto:Port: 8000 2023-09-29T23:34:44.035 DEBUG:boto:Params: {} 2023-09-29T23:34:44.036 DEBUG:boto:Token: None 2023-09-29T23:34:44.036 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:34:44 GMT /hcnphz-74/ 2023-09-29T23:34:44.036 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:k6L4QxB/+Hhat+zxZiauo3gGX7Y= 2023-09-29T23:34:44.036 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:34:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:k6L4QxB/+Hhat+zxZiauo3gGX7Y=', 'Content-Length': '0'} 2023-09-29T23:34:44.039 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000273a61e536f926ba-0065175f14-2d72-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:44.040 DEBUG:boto:path=/hcnphz-74/obj4 2023-09-29T23:34:44.040 DEBUG:boto:auth_path=/hcnphz-74/obj4 2023-09-29T23:34:44.040 DEBUG:boto:Method: PUT 2023-09-29T23:34:44.040 DEBUG:boto:Path: /hcnphz-74/obj4 2023-09-29T23:34:44.040 DEBUG:boto:Data: 2023-09-29T23:34:44.040 DEBUG:boto:Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:34:44.040 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:34:44.040 DEBUG:boto:Port: 8000 2023-09-29T23:34:44.040 DEBUG:boto:Params: {} 2023-09-29T23:34:44.041 DEBUG:boto:Token: None 2023-09-29T23:34:44.041 DEBUG:boto:StringToSign: PUT qPXxZ/RPSWTmyZje6CcRDA== application/octet-stream Fri, 29 Sep 2023 23:34:44 GMT /hcnphz-74/obj4 2023-09-29T23:34:44.041 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:CrZhptzRUWlvOBSY5yJ9HcTKj60= 2023-09-29T23:34:44.041 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:34:44 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:CrZhptzRUWlvOBSY5yJ9HcTKj60='} 2023-09-29T23:34:44.086 DEBUG:boto:Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000987f90615bc408a6-0065175f14-2d72-a1'), ('Date', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:34:44.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.087 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:44.131 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:44.191 INFO:teuthology.orchestra.run.smithi112.stderr:No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:34:44.220 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:34:44.220 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.220 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:44.220 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:34:44.220 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030452.667146_918.1", 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:34:12.667146Z" 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.221 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.222 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:44.222 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "0.000000" 2023-09-29T23:34:44.222 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.222 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:34:44.223 DEBUG:rgw_multi.tests:master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:44.223 INFO:rgw_multi.tests:starting meta checkpoint for zone=a2 2023-09-29T23:34:44.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.223 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:44.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:44.267 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:44.321 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:44.323+0000 7fe840065f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:44.322 INFO:teuthology.orchestra.run.smithi112.stderr:2023-09-29T23:34:44.323+0000 7fe840065f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:34:44.345 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.346 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 1, 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:34:44.347 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "1_1696030452.667146_918.1", 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:44.348 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:34:12.667146Z", 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:44.349 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.350 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "state": 1, 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "", 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "0.000000", 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: "realm_epoch": 2 2023-09-29T23:34:44.351 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 1, 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 1 2023-09-29T23:34:44.352 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.353 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:44Z" 2023-09-29T23:34:44.353 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:44.354 DEBUG:rgw_multi.tests:current meta sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", "realm_epoch": 2 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "1_1696027855.273178_228.1", "next_step_marker": "", "total_entries": 1, "pos": 0, "timestamp": "2023-09-29T22:50:55.273178Z", "realm_epoch": 2 } }, { "key": 1, "val": { "state": 1, "marker": "1_1696030452.667146_918.1", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:34:12.667146Z", "realm_epoch": 2 } }, { "key": 2, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } }, { "key": 3, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 2 } } ] }, "full_sync": { "total": 1, "complete": 1 }, "current_time": "2023-09-29T23:34:44Z" } 2023-09-29T23:34:44.354 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696030452.667146_918.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:34:12.667146Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:34:44.354 DEBUG:rgw_multi.tests:log_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:44.354 DEBUG:rgw_multi.tests:sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696030452.667146_918.1', 2: '', 3: ''} 2023-09-29T23:34:44.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=a2 2023-09-29T23:34:44.354 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.355 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:44.440 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:44.440 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:44.542 INFO:teuthology.orchestra.run.smithi112.stdout:[ 2023-09-29T23:34:44.542 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052654", 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:34:44.036872Z" 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.543 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032059", 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout: "last_update": "2023-09-29T23:34:44.058938Z" 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.544 INFO:teuthology.orchestra.run.smithi112.stdout:] 2023-09-29T23:34:44.545 DEBUG:rgw_multi.tests:data markers for zone=a1 markers={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:34:44.545 INFO:rgw_multi.tests:starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:34:44.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:44.546 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:44.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:44.592 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:44.677 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:44.677 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.678 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.679 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.680 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.681 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:44.682 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:44Z" 2023-09-29T23:34:44.683 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:44.684 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:34:44Z" } 2023-09-29T23:34:44.685 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:34:44.685 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:34:44.685 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:44.685 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:34:49.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:49.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:49.686 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:49.734 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:49.735 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:49.812 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:49.812 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:49.812 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:49.812 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:49.813 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:49.814 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:49.815 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:49.816 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:49.817 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:49.818 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:49Z" 2023-09-29T23:34:49.819 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:49.820 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:34:49Z" } 2023-09-29T23:34:49.820 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:34:49.821 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:34:49.821 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:49.821 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:34:54.821 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:54.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:54.822 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:34:54.872 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:34:54.872 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:34:54.957 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:34:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:34:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:34:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:34:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:34:54.957 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:34:54.958 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:34:54.959 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:54.960 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:54.961 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:34:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:34:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:34:54.962 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:34:54.963 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:34:54.964 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:34:54Z" 2023-09-29T23:34:54.965 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:34:54.965 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:34:54Z" } 2023-09-29T23:34:54.969 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:34:54.969 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:34:54.969 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:34:54.969 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:34:59.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:59.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:34:59.971 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:00.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:00.018 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:00.106 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:00.106 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:00.107 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:00.108 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:00.109 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:00.110 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:00.111 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:00.112 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:00Z" 2023-09-29T23:35:00.113 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:00.114 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:00Z" } 2023-09-29T23:35:00.114 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:00.115 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:00.115 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:00.115 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:05.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:05.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:05.116 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:05.162 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:05.163 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:05.252 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:05.252 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:05.252 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:05.252 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:05.253 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:05.254 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:05.255 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:05.256 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:05.257 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:05.258 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:05.259 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:05.259 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:05Z" 2023-09-29T23:35:05.259 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:05.260 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:05Z" } 2023-09-29T23:35:05.260 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:05.260 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:05.260 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:05.260 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:10.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:10.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:10.262 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:10.315 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:10.399 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:10.399 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:10.399 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:10.399 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:10.399 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:10.399 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:10.400 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:10.401 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:10.402 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:10.403 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:10.404 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:10.405 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:10.405 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:10.405 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:10.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:10.405 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:10Z" 2023-09-29T23:35:10.405 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:10.406 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:10Z" } 2023-09-29T23:35:10.406 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:10.406 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:10.407 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:10.407 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:15.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:15.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:15.409 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:15.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:15.457 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:15.546 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:15.547 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:15.548 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:15.549 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:15.550 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:15.551 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:15.552 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:15Z" 2023-09-29T23:35:15.553 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:15.554 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:15Z" } 2023-09-29T23:35:15.554 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:15.554 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:15.554 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:15.554 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:20.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:20.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:20.556 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:20.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:20.691 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:20.691 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:20.691 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:20.691 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:20.691 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:20.691 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:20.692 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:20.693 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:20.694 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:20.695 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:20.696 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:20.697 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:20Z" 2023-09-29T23:35:20.697 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:20.698 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:20Z" } 2023-09-29T23:35:20.698 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:20.698 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:20.698 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:20.698 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:25.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:25.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:25.700 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:25.747 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:25.833 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:25.833 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:25.833 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:25.833 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:25.834 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:25.835 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:25.836 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:25.837 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:25.838 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:25Z" 2023-09-29T23:35:25.839 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:25.840 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:25Z" } 2023-09-29T23:35:25.840 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:25.840 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:25.841 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:25.841 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:30.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:30.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:30.842 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:30.892 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:30.893 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:30.985 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:30.985 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:30.985 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:30.985 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:30.985 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:30.986 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:30.987 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:30.988 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:30.989 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:30.990 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:30Z" 2023-09-29T23:35:30.991 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:30.992 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:30Z" } 2023-09-29T23:35:30.992 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:30.993 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:30.993 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:30.993 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:35.994 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:35.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:35.995 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:36.043 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:36.044 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:36.126 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:36.126 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:36.126 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:36.127 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:36.128 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:36.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:36.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:36.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:36Z" 2023-09-29T23:35:36.132 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:36.133 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:36Z" } 2023-09-29T23:35:36.134 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:36.134 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:36.134 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:36.134 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:41.135 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:41.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:41.135 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:41.182 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:41.183 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:41.269 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:41.269 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:41.269 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:41.270 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:41.271 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:41.272 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:41.273 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:41.274 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:41.275 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:41.276 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:41.276 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:41Z" 2023-09-29T23:35:41.276 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:41.277 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:41Z" } 2023-09-29T23:35:41.277 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:41.277 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:41.277 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:41.277 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:46.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:46.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:46.279 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:46.319 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:46.319 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:46.401 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:46.402 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:46.402 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:46.402 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:46.402 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:46.402 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:46.403 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:46.403 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:46.403 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:46.403 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:46.404 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:46.404 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:46.404 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:46.404 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:46.404 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:46.405 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:46.405 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:46.405 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:46.405 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:46.405 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:46.406 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:46.406 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:46.406 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:46.406 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:46.406 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:46.406 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:46.407 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:46.407 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:46.407 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:46.407 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:46.407 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:46.408 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:46.408 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:46.408 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:46.409 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:46.409 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:46.409 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:46.409 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:46.410 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:46.411 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:46.412 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:46.412 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:46.412 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:46.412 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:46Z" 2023-09-29T23:35:46.412 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:46.413 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:46Z" } 2023-09-29T23:35:46.414 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:46.414 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:46.414 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:46.414 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:51.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:51.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:51.416 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:51.462 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:51.463 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:51.548 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:51.548 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:51.549 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:51.550 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:51.551 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:51.552 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:51.553 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:51.554 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:51Z" 2023-09-29T23:35:51.555 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:51.555 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:51Z" } 2023-09-29T23:35:51.556 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:51.556 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:51.556 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:51.556 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:35:56.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:56.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:35:56.557 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:35:56.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:35:56.605 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:35:56.693 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:35:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:35:56.693 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:35:56.694 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:56.695 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:56.696 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:35:56.697 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:35:56.698 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:35:56.699 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:35:56Z" 2023-09-29T23:35:56.700 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:35:56.701 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:35:56Z" } 2023-09-29T23:35:56.701 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:35:56.701 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:35:56.701 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:35:56.701 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:36:01.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:01.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:01.703 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:01.752 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:01.752 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:01.842 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:36:01.842 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:36:01.843 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:01.844 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:01.845 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:01.846 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:01.847 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:36:01Z" 2023-09-29T23:36:01.848 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:36:01.849 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:36:01Z" } 2023-09-29T23:36:01.850 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:36:01.850 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:36:01.850 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:36:01.850 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:36:06.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:06.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:06.852 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:06.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:06.896 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:06.983 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:36:06.984 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:06.985 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:36:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:36:06.986 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:06.990 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:36:06.990 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:06.990 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:06.991 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:36:06.992 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:36:06.993 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:36:06.994 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:36:06.994 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:06.994 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:36:06Z" 2023-09-29T23:36:06.994 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:36:06.995 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:36:06Z" } 2023-09-29T23:36:06.995 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:36:06.995 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:36:06.995 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:36:06.995 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:36:11.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:11.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:11.997 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:12.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:12.045 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:12.128 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:36:12.128 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:12.129 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052513", 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.752312Z" 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:12.130 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:36:12.131 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:12.132 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000031918", 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:12.133 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:31:48.131473Z" 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:36:12.134 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:12.135 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:36:12Z" 2023-09-29T23:36:12.135 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:36:12.136 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052513", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:32:37.752312Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000031918", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:31:48.131473Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:36:12Z" } 2023-09-29T23:36:12.136 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052513', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.752312Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031918', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:31:48.131473Z'}}] 2023-09-29T23:36:12.136 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:36:12.136 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052513', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000031918'} 2023-09-29T23:36:12.136 WARNING:rgw_multi.tests:data of zone a2 behind zone a1: shard=0 master=00000000000000000000:00000000000000052654 target=00000000000000000000:00000000000000052513, shard=3 master=00000000000000000000:00000000000000032059 target=00000000000000000000:00000000000000031918 2023-09-29T23:36:17.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.138 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:17.187 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.187 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout:{ 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: "sync_status": { 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: "info": { 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "sync", 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: "num_shards": 4, 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: "instance_id": 6926381311200296959 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:17.271 INFO:teuthology.orchestra.run.smithi112.stdout: "markers": [ 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 0, 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000052654", 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 55, 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:17.272 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:34:44.036872Z" 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 1, 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000045306", 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 9, 2023-09-29T23:36:17.273 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:27:10.490774Z" 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 2, 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:17.274 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000046359", 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:32:37.779848Z" 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: { 2023-09-29T23:36:17.275 INFO:teuthology.orchestra.run.smithi112.stdout: "key": 3, 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "val": { 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "status": "incremental-sync", 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "marker": "00000000000000000000:00000000000000032059", 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "next_step_marker": "", 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "total_entries": 0, 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "pos": 0, 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: "timestamp": "2023-09-29T23:34:44.058938Z" 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:17.276 INFO:teuthology.orchestra.run.smithi112.stdout: } 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: ] 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: "full_sync": { 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: "total": 64, 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: "complete": 64 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: }, 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout: "current_time": "2023-09-29T23:36:17Z" 2023-09-29T23:36:17.277 INFO:teuthology.orchestra.run.smithi112.stdout:} 2023-09-29T23:36:17.278 DEBUG:rgw_multi.tests:current data sync status={ "sync_status": { "info": { "status": "sync", "num_shards": 4, "instance_id": 6926381311200296959 }, "markers": [ { "key": 0, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000052654", "next_step_marker": "", "total_entries": 55, "pos": 0, "timestamp": "2023-09-29T23:34:44.036872Z" } }, { "key": 1, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000045306", "next_step_marker": "", "total_entries": 9, "pos": 0, "timestamp": "2023-09-29T23:27:10.490774Z" } }, { "key": 2, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000046359", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:32:37.779848Z" } }, { "key": 3, "val": { "status": "incremental-sync", "marker": "00000000000000000000:00000000000000032059", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "2023-09-29T23:34:44.058938Z" } } ] }, "full_sync": { "total": 64, "complete": 64 }, "current_time": "2023-09-29T23:36:17Z" } 2023-09-29T23:36:17.279 DEBUG:rgw_multi.tests:sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000052654', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:34:44.036872Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045306', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:27:10.490774Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000046359', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:32:37.779848Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000032059', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:34:44.058938Z'}}] 2023-09-29T23:36:17.279 DEBUG:rgw_multi.tests:log_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:36:17.279 DEBUG:rgw_multi.tests:sync_status={0: '00000000000000000000:00000000000000052654', 1: '00000000000000000000:00000000000000045306', 2: '00000000000000000000:00000000000000046359', 3: '00000000000000000000:00000000000000032059'} 2023-09-29T23:36:17.279 INFO:rgw_multi.tests:finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:36:17.279 DEBUG:boto:path=/hcnphz-74/ 2023-09-29T23:36:17.279 DEBUG:boto:auth_path=/hcnphz-74/ 2023-09-29T23:36:17.280 DEBUG:boto:Method: HEAD 2023-09-29T23:36:17.280 DEBUG:boto:Path: /hcnphz-74/ 2023-09-29T23:36:17.280 DEBUG:boto:Data: 2023-09-29T23:36:17.280 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.280 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.280 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.280 DEBUG:boto:Params: {} 2023-09-29T23:36:17.280 DEBUG:boto:establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:36:17.281 DEBUG:boto:Token: None 2023-09-29T23:36:17.281 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-74/ 2023-09-29T23:36:17.281 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:mY88YcKhjiezpKf6SBZv8s3dmiE= 2023-09-29T23:36:17.281 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mY88YcKhjiezpKf6SBZv8s3dmiE=', 'Content-Length': '0'} 2023-09-29T23:36:17.290 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a02682f30026e4b4-0065175f71-2d76-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.290 DEBUG:boto:path=/hcnphz-74/obj3 2023-09-29T23:36:17.290 DEBUG:boto:auth_path=/hcnphz-74/obj3 2023-09-29T23:36:17.290 DEBUG:boto:Method: HEAD 2023-09-29T23:36:17.291 DEBUG:boto:Path: /hcnphz-74/obj3 2023-09-29T23:36:17.291 DEBUG:boto:Data: 2023-09-29T23:36:17.291 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.291 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.291 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.291 DEBUG:boto:Params: {} 2023-09-29T23:36:17.291 DEBUG:boto:Token: None 2023-09-29T23:36:17.291 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-74/obj3 2023-09-29T23:36:17.292 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:P1Wk1aEBOAEf5o5r8UlzFzIb5Gw= 2023-09-29T23:36:17.292 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:P1Wk1aEBOAEf5o5r8UlzFzIb5Gw=', 'Content-Length': '0'} 2023-09-29T23:36:17.295 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000e813c4c23369e643-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.295 DEBUG:boto:path=/hcnphz-74/obj3 2023-09-29T23:36:17.296 DEBUG:boto:auth_path=/hcnphz-74/obj3 2023-09-29T23:36:17.296 DEBUG:boto:Method: GET 2023-09-29T23:36:17.296 DEBUG:boto:Path: /hcnphz-74/obj3 2023-09-29T23:36:17.296 DEBUG:boto:Data: 2023-09-29T23:36:17.296 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.296 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.296 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.296 DEBUG:boto:Params: {} 2023-09-29T23:36:17.296 DEBUG:boto:Token: None 2023-09-29T23:36:17.297 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-74/obj3 2023-09-29T23:36:17.297 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:5SrfjoqwrMyv3zs1U37BHyV2m84= 2023-09-29T23:36:17.297 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5SrfjoqwrMyv3zs1U37BHyV2m84=', 'Content-Length': '0'} 2023-09-29T23:36:17.300 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx0000079432952eb912f9e-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.300 DEBUG:boto:path=/hcnphz-74/obj4 2023-09-29T23:36:17.300 DEBUG:boto:auth_path=/hcnphz-74/obj4 2023-09-29T23:36:17.300 DEBUG:boto:Method: HEAD 2023-09-29T23:36:17.300 DEBUG:boto:Path: /hcnphz-74/obj4 2023-09-29T23:36:17.300 DEBUG:boto:Data: 2023-09-29T23:36:17.301 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.301 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.301 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.301 DEBUG:boto:Params: {} 2023-09-29T23:36:17.301 DEBUG:boto:Token: None 2023-09-29T23:36:17.301 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-74/obj4 2023-09-29T23:36:17.301 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:DCw8x4lqMkrhknPFpiGMB20m3Gs= 2023-09-29T23:36:17.301 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DCw8x4lqMkrhknPFpiGMB20m3Gs=', 'Content-Length': '0'} 2023-09-29T23:36:17.305 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000eca4641706c07884-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.305 DEBUG:boto:path=/hcnphz-74/obj4 2023-09-29T23:36:17.306 DEBUG:boto:auth_path=/hcnphz-74/obj4 2023-09-29T23:36:17.306 DEBUG:boto:Method: GET 2023-09-29T23:36:17.306 DEBUG:boto:Path: /hcnphz-74/obj4 2023-09-29T23:36:17.306 DEBUG:boto:Data: 2023-09-29T23:36:17.306 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.306 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.306 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.306 DEBUG:boto:Params: {} 2023-09-29T23:36:17.307 DEBUG:boto:Token: None 2023-09-29T23:36:17.307 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-74/obj4 2023-09-29T23:36:17.307 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:ldxHfQfFohNMM9tNXFZ7tY9vG0o= 2023-09-29T23:36:17.307 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ldxHfQfFohNMM9tNXFZ7tY9vG0o=', 'Content-Length': '0'} 2023-09-29T23:36:17.310 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000cbe1c7f1f8980392-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.354 DEBUG:boto:path=/hcnphz-75/ 2023-09-29T23:36:17.355 DEBUG:boto:auth_path=/hcnphz-75/ 2023-09-29T23:36:17.355 DEBUG:boto:Method: HEAD 2023-09-29T23:36:17.355 DEBUG:boto:Path: /hcnphz-75/ 2023-09-29T23:36:17.355 DEBUG:boto:Data: 2023-09-29T23:36:17.355 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.355 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.355 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.356 DEBUG:boto:Params: {} 2023-09-29T23:36:17.356 DEBUG:boto:Token: None 2023-09-29T23:36:17.356 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-75/ 2023-09-29T23:36:17.356 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:t9tf0WqgEdDUKFprN/QxMNiZ7GI= 2023-09-29T23:36:17.356 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t9tf0WqgEdDUKFprN/QxMNiZ7GI=', 'Content-Length': '0'} 2023-09-29T23:36:17.359 DEBUG:boto:Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001f7e09a1195ccf23-0065175f71-2d76-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.360 DEBUG:boto:path=/hcnphz-75/obj3 2023-09-29T23:36:17.360 DEBUG:boto:auth_path=/hcnphz-75/obj3 2023-09-29T23:36:17.360 DEBUG:boto:Method: HEAD 2023-09-29T23:36:17.360 DEBUG:boto:Path: /hcnphz-75/obj3 2023-09-29T23:36:17.360 DEBUG:boto:Data: 2023-09-29T23:36:17.360 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.360 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.360 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.361 DEBUG:boto:Params: {} 2023-09-29T23:36:17.361 DEBUG:boto:Token: None 2023-09-29T23:36:17.361 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-75/obj3 2023-09-29T23:36:17.361 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:KqqrQm0hx7AxqwRD4Uqbek3YvVk= 2023-09-29T23:36:17.361 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KqqrQm0hx7AxqwRD4Uqbek3YvVk=', 'Content-Length': '0'} 2023-09-29T23:36:17.363 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000236353cf8e9a1013-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.364 DEBUG:boto:path=/hcnphz-75/obj3 2023-09-29T23:36:17.364 DEBUG:boto:auth_path=/hcnphz-75/obj3 2023-09-29T23:36:17.364 DEBUG:boto:Method: GET 2023-09-29T23:36:17.364 DEBUG:boto:Path: /hcnphz-75/obj3 2023-09-29T23:36:17.364 DEBUG:boto:Data: 2023-09-29T23:36:17.364 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.365 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.365 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.365 DEBUG:boto:Params: {} 2023-09-29T23:36:17.365 DEBUG:boto:Token: None 2023-09-29T23:36:17.365 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-75/obj3 2023-09-29T23:36:17.365 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:OqwfQ/YuORheUr0QK1H3sw50a/Y= 2023-09-29T23:36:17.365 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OqwfQ/YuORheUr0QK1H3sw50a/Y=', 'Content-Length': '0'} 2023-09-29T23:36:17.368 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000b440fd575c6fc7e0-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.410 DEBUG:boto:path=/hcnphz-75/obj4 2023-09-29T23:36:17.411 DEBUG:boto:auth_path=/hcnphz-75/obj4 2023-09-29T23:36:17.411 DEBUG:boto:Method: HEAD 2023-09-29T23:36:17.411 DEBUG:boto:Path: /hcnphz-75/obj4 2023-09-29T23:36:17.411 DEBUG:boto:Data: 2023-09-29T23:36:17.411 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.411 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.412 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.412 DEBUG:boto:Params: {} 2023-09-29T23:36:17.412 DEBUG:boto:Token: None 2023-09-29T23:36:17.412 DEBUG:boto:StringToSign: HEAD Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-75/obj4 2023-09-29T23:36:17.412 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:hNxRDslL57uyI4eE2AQ5d687qzE= 2023-09-29T23:36:17.413 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hNxRDslL57uyI4eE2AQ5d687qzE=', 'Content-Length': '0'} 2023-09-29T23:36:17.415 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx00000327893d94a83c070-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.416 DEBUG:boto:path=/hcnphz-75/obj4 2023-09-29T23:36:17.416 DEBUG:boto:auth_path=/hcnphz-75/obj4 2023-09-29T23:36:17.416 DEBUG:boto:Method: GET 2023-09-29T23:36:17.416 DEBUG:boto:Path: /hcnphz-75/obj4 2023-09-29T23:36:17.416 DEBUG:boto:Data: 2023-09-29T23:36:17.416 DEBUG:boto:Headers: {} 2023-09-29T23:36:17.417 DEBUG:boto:Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.417 DEBUG:boto:Port: 8001 2023-09-29T23:36:17.417 DEBUG:boto:Params: {} 2023-09-29T23:36:17.417 DEBUG:boto:Token: None 2023-09-29T23:36:17.417 DEBUG:boto:StringToSign: GET Fri, 29 Sep 2023 23:36:17 GMT /hcnphz-75/obj4 2023-09-29T23:36:17.417 DEBUG:boto:Signature: AWS P5GVQP2RXA6372D10DSL:vWxKHvIBnAAgKzuCAA2g/1SUIYM= 2023-09-29T23:36:17.418 DEBUG:boto:Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:36:17 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vWxKHvIBnAAgKzuCAA2g/1SUIYM=', 'Content-Length': '0'} 2023-09-29T23:36:17.420 DEBUG:boto:Response headers: [('Content-Length', '6'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:34:44 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-74:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.10950.1'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('x-amz-request-id', 'tx000000aae7b9d5fb1a0a2-0065175f71-2d76-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:36:17 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.421 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-bucket', '--bucket', 'hcnphz-74', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.421 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-bucket --bucket hcnphz-74 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:17.466 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.466 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.590 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:36:17.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'remove', '--group-id', 'sync-group', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.591 DEBUG:teuthology.orchestra.run.smithi112:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group remove --group-id sync-group --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:17.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.633 INFO:teuthology.orchestra.run.smithi112.stderr:ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.723 INFO:teuthology.orchestra.run.smithi112.stdout:{"sync_policy":{"groups":[]}} 2023-09-29T23:36:17.723 INFO:tasks.rgw_multisite_tests:test_sync_single_bucket_to_multiple: ... ok 2023-09-29T23:36:17.724 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.724 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-29T23:36:17.724 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_reshard_index_log_trim 2023-09-29T23:36:17.725 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1418, in test_bucket_reshard_index_log_trim 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests: assert(len(json_obj_5['layout']['logs']) == 1) 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hcnphz-55 2023-09-29T23:36:17.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.727 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:q5nS88jpzhz+cDuOEtsVB9LEt8c= 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q5nS88jpzhz+cDuOEtsVB9LEt8c=', 'Content-Length': '0'} 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000a5af1a94ff3aa535-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.729 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009d1833ca4b8204b7-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.731 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:nzYxennEmgYSJngboQg2GsyTQGs= 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nzYxennEmgYSJngboQg2GsyTQGs='} 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e565076d0204a9f1-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T23:36:17.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T23:36:17.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000fec53a1311be9b9a-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.737 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:s5fVA9Ks+vPJhj6l+2Cih1G8VTM= 2023-09-29T23:36:17.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:s5fVA9Ks+vPJhj6l+2Cih1G8VTM='} 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009320a2b0bab1fc9f-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T23:36:17.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ff7b148255168d6a-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:17.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.743 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:cT5AS5bcHaRRRceVlqM3jYpIbd4= 2023-09-29T23:36:17.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:cT5AS5bcHaRRRceVlqM3jYpIbd4='} 2023-09-29T23:36:17.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000094fb1f8a91bd61e4-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.745 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.745 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.745 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.745 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44= 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:713h7bzZRLzUXuEsr/pbrkIfj44=', 'Content-Length': '0'} 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000004bbc4b673f086c2-0065174efb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:17.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.747 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:03 GMT 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:lXa4qQ5E0MkupWuq/eFrB4U0W7Y= 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:03 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lXa4qQ5E0MkupWuq/eFrB4U0W7Y='} 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000064af09faacb5c16d-0065174efb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:17.748 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696024631.846323_86.1", 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T21:57:11.846323Z" 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.749 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696026363.077186_423.1", 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T22:26:03.077186Z" 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.750 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696024631.846323_86.1', 1: '1_1696026363.077186_423.1', 2: '', 3: ''} 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:03.449+0000 7fecc817df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:03.449+0000 7fecc817df40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.751 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:17.752 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696024631.846323_86.1", 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:17.753 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696026363.077186_423.1", 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:26:03.077186Z", 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:17.754 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:17.755 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:17.756 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:17.757 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T22:26:03Z" 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:17.758 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "marker": "1_1696024631.846323_86.1", 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T21:57:11.846323Z", 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:17.759 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "marker": "1_1696026363.077186_423.1", 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:26:03.077186Z", 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:17.760 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:17.761 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:17.762 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T22:26:03Z" 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.763 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696024631.846323_86.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T21:57:11.846323Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696026363.077186_423.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T22:26:03.077186Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696024631.846323_86.1', 1: '1_1696026363.077186_423.1', 2: '', 3: ''} 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696024631.846323_86.1', 1: '1_1696026363.077186_423.1', 2: '', 3: ''} 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hcnphz-55 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.764 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:03.601+0000 7ff9f42a5f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:08.605+0000 7ff9f42a5f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:13.605+0000 7ff9f42a5f40 1 bucket sync caught up with source: 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, 00000000001.39.6, , , , 00000000001.45.6, , , 00000000001.44.6, , 00000000001.62.6] 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, 00000000001.39.6, , , , 00000000001.45.6, , , 00000000001.44.6, , 00000000001.62.6] 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:13.605+0000 7ff9f42a5f40 0 bucket checkpoint complete 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hcnphz-55 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.765 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:13.757+0000 7fb89ff0ef40 1 waiting to reach incremental sync.. 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:18.757+0000 7fb89ff0ef40 1 waiting to reach incremental sync.. 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:23.761+0000 7fb89ff0ef40 1 bucket sync caught up with source: 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, 00000000001.162.6, , , , 00000000001.98.6, , , 00000000001.151.6, , 00000000001.282.6] 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, 00000000001.162.6, , , , 00000000001.98.6, , , 00000000001.151.6, , 00000000001.282.6] 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:23.761+0000 7fb89ff0ef40 0 bucket checkpoint complete 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hcnphz-55 zones={a2, a1} 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.767 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE= 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE=', 'Content-Length': '0'} 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b0c4b6b3425d6ce5-0065174f0f-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE= 2023-09-29T23:36:17.769 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2cxWJxCCZ09zVzz8e0fZzDjvJWE=', 'Content-Length': '0'} 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000021708d1e42bfca44-0065174f0f-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T23:36:17.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c3688c9cfee7cb-0065174f0f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.772 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-09-29T23:36:17.772 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.774 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bcfdcd8181897afa-0065174f0f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.778 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dbbb8db47dbdc58d-0065174f0f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ= 2023-09-29T23:36:17.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yUQZ1Px7lkfcv4/QcJp8tSN2/ZQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064747bd9250d8702-0065174f0f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000556e9e71cd0f812e-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.785 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000000186e148764912c-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.787 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY= 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY=', 'Content-Length': '0'} 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1a4f295e1d3f6d4-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.789 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY= 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uxeHtA9yPEs2i5WiyPOShoCQxUY=', 'Content-Length': '0'} 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004793b3d245b7c0b7-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.791 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009e3dd6b462c0db88-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.793 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM= 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:L5+AIyOHwbVODoNQRovWOkTT8CM=', 'Content-Length': '0'} 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f121ac886f09826e-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.795 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU= 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU=', 'Content-Length': '0'} 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c36d88d68ab2aa73-0065174f0f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.797 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU= 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NGQkrTaU1slBxqjDplyrxSaP/yU=', 'Content-Length': '0'} 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000de61e609e107117c-0065174f0f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI= 2023-09-29T23:36:17.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI=', 'Content-Length': '0'} 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007aff4f40335cb08d-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI= 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pXEzeCK+CDeO24J3NbOBo5Pt/jI=', 'Content-Length': '0'} 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee6572db531d4c9e-0065174f0f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.801 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:23 GMT 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2ZPiF+ixRoqydDlOtNHqKUYOm0c= 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:23 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2ZPiF+ixRoqydDlOtNHqKUYOm0c=', 'Content-Length': '0'} 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e8896b5089a6f76d-0065174f0f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:23 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.804 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:y/vGev4Yrisqgzvdy/Ebfzzrawo= 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:y/vGev4Yrisqgzvdy/Ebfzzrawo=', 'Content-Length': '0'} 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000068c92c5ebbdce28a-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.806 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009a11705bf58bf82b-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.808 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000323136e231503a1b-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.810 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e2f04a2c91a429-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.812 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fecef1af8d82c461-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo= 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo=', 'Content-Length': '0'} 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081df48b74cc2faa7-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.816 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo= 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HFLvcIvvuLsdAr4+UZEJI4Q1zCo=', 'Content-Length': '0'} 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026e103fe6216cc98-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.818 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008f59075dfe8399c0-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo= 2023-09-29T23:36:17.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hyq0ZKmPgMEpuhSKJd8tzW8m+Mo=', 'Content-Length': '0'} 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000426e6653b247995f-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000064b5558a6732076e-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bb2c6bd8d60a04e8-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU= 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU=', 'Content-Length': '0'} 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008d98010c367bce3f-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.827 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU= 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FRbQ9ZLaIMBvT/xpHMcd12X/IHU=', 'Content-Length': '0'} 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000055f14e85d2f0a37c-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.829 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg= 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg=', 'Content-Length': '0'} 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d696f6af47f5c57-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.831 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg= 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LxG9jj5W/IyO3scO8AJSCzAx/Dg=', 'Content-Length': '0'} 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000dbf938eb9e2eaa58-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002084c5edd7aa63cd-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e1f811b342c770a9-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.837 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f92545b697765bf2-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.839 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003a06443c7abb6dcd-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.841 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY= 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY=', 'Content-Length': '0'} 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000001d5116f0f831a9d-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY= 2023-09-29T23:36:17.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7SPwg1tFMz/UvWgaL4qQNdLF+QY=', 'Content-Length': '0'} 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000013b3507874034230-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T23:36:17.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cb4934f0b943c006-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ= 2023-09-29T23:36:17.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ve8lLRl1sEaLnW+Z9e9YRkdnLOQ=', 'Content-Length': '0'} 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000015647a6d19b18219-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b01c79d27a94b02e-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e7b6cad884a72be5-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.853 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk= 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk=', 'Content-Length': '0'} 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000006e70affb96a3515-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk= 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qceVsrBSoh70ltDofzwrrruarAk=', 'Content-Length': '0'} 2023-09-29T23:36:17.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bc156c9e54c94f6e-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4= 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4=', 'Content-Length': '0'} 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072bcb71cbc428351-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.858 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4= 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:30ZQjSfXRBUWLvPmXkAGwSTLFd4=', 'Content-Length': '0'} 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000faa14c897cce4f76-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.860 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e06fe1fc20feff31-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.862 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038b747dfaec32439-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f02da3c948053d3e-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.866 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e97ed4f365a5ea87-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.868 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc= 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc=', 'Content-Length': '0'} 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009ce46f0dbedeb085-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.870 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc= 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YB7EeXD564LgphSDUBMptJ0wvWc=', 'Content-Length': '0'} 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000099bc4a89ca949320-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.872 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b64df04e148228bb-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.874 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg= 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hPShJ3cav0tEgzRvOnHW0vMlHtg=', 'Content-Length': '0'} 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000058304e0e35e95cbf-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:17.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.876 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005287aae847adbbf5-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:17.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.878 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000087d40fbd475c9a7f-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI= 2023-09-29T23:36:17.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI=', 'Content-Length': '0'} 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000152b37d0bc0e6cc7-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI= 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wjS3nLbUsGZoHTW0ym/OmwUnziI=', 'Content-Length': '0'} 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000eafd0360e5e16036-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:17.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.883 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo= 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo=', 'Content-Length': '0'} 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094353bd415032ad8-0065174f10-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.884 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.885 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo= 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jOH/LXPsYnl1wNb1rYApTHALAWo=', 'Content-Length': '0'} 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000996fb76b05292edb-0065174f10-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:17.886 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.887 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ea1cd47ded15a8b2-0065174f10-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:17.888 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:24 GMT 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.889 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8= 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:24 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jhQASYu9i42QWjnhmrKJLdGJ7p8=', 'Content-Length': '0'} 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d96c07f1214e4fd1-0065174f10-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:26:24 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hcnphz-55 zones={a2, a1} 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding": "None", 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_index": { 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "Normal", 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "normal": { 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 11, 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.891 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "logs": [ 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 11, 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.892 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-55 --gen 0 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_id": "1#00000000001.39.6", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.16529626172099963377", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op": "write", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "object": "a", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance": "", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": "complete", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "index_ver": 1, 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:26:03.091518792Z", 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "ver": { 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pool": 9, 2023-09-29T23:36:17.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 3 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bilog_flags": 0, 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "versioned": false, 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner": "", 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner_display_name": "", 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones_trace": [ 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_id": "5#00000000001.45.6", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.7255028496961225069", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op": "write", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "object": "d", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance": "", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": "complete", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "index_ver": 1, 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:26:03.214573239Z", 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "ver": { 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pool": 9, 2023-09-29T23:36:17.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 8 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bilog_flags": 0, 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "versioned": false, 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner": "", 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner_display_name": "", 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones_trace": [ 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_id": "8#00000000001.44.6", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10735208816778764772", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op": "write", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "object": "c", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance": "", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": "complete", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "index_ver": 1, 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:26:03.164775568Z", 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "ver": { 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pool": 9, 2023-09-29T23:36:17.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 1 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bilog_flags": 0, 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "versioned": false, 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner": "", 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner_display_name": "", 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones_trace": [ 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_id": "10#00000000001.62.6", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op_tag": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5704.10601652402760318111", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "op": "write", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "object": "b", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance": "", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": "complete", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "index_ver": 1, 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:26:03.146528163Z", 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "ver": { 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pool": 9, 2023-09-29T23:36:17.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 3 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bilog_flags": 0, 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "versioned": false, 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner": "", 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "owner_display_name": "", 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones_trace": [ 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "entry": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20" 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-55 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 4 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:17.902 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding": "None", 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_index": { 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "Normal", 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "normal": { 2023-09-29T23:36:17.903 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "logs": [ 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:17.904 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 11, 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:17.905 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:17.906 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'hcnphz-55', '--gen', '1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket hcnphz-55 --gen 1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [] 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.907 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T23:36:17.908 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002b69294f5c8a4f4e-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.909 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Svm409yhFgka/uwmhV1UeO/nV+w= 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Svm409yhFgka/uwmhV1UeO/nV+w='} 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000eba0800a3dede4b7-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.910 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.911 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e6a473b7d2c41841-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:17.912 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.913 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:bGcFvmTC8RQ0eyZ88hb/DOJyNos= 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bGcFvmTC8RQ0eyZ88hb/DOJyNos='} 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e4844afac09f9871-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.914 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T23:36:17.915 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000079d2d0f797e0d33e-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.916 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:vxRRysFk5R7Sh7pWE1fs65OhyBE= 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:vxRRysFk5R7Sh7pWE1fs65OhyBE='} 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000076a5fb8e3bb664a3-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.917 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.918 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I= 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:25L5KYPKVZMyjyya763WBDZiQ6I=', 'Content-Length': '0'} 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000088dde7299d71666d-0065174f11-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:17.919 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:17.920 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:26:25 GMT 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Tg3VGe3/MykbD4TlX5WE7F4WKdk= 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 22:26:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Tg3VGe3/MykbD4TlX5WE7F4WKdk='} 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001765f7ee71591017-0065174f11-1648-a1'), ('Date', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hcnphz-55 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:25.680+0000 7fa658d2ff40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:30.680+0000 7fa658d2ff40 1 waiting to reach latest gen 1, on 0.. 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:35.684+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , ] 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:40.684+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , ] 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:45.688+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , ] 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.922 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi131.stdout: INFO: 2023-09-29T22:26:50.125+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:50.688+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , ] 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:26:55.688+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:00.692+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:05.691+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:10.691+0000 7fa658d2ff40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, , 00000000002.27.6] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:15.695+0000 7fa658d2ff40 1 bucket sync caught up with source: 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [00000000003.25.6, 00000000002.45.6, 00000000002.27.6] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:15.695+0000 7fa658d2ff40 0 bucket checkpoint complete 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hcnphz-55 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.924 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:15.843+0000 7f66bfe8bf40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, 00000000001.162.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:20.847+0000 7f66bfe8bf40 1 waiting for incremental sync to catch up: 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, 00000000001.162.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:25.851+0000 7f66bfe8bf40 1 bucket sync caught up with source: 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:17.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:25.851+0000 7f66bfe8bf40 0 bucket checkpoint complete 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hcnphz-55 zones={a2, a1} 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.926 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.927 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA= 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA=', 'Content-Length': '0'} 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009fecfb967f412c96-0065174f4d-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.928 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA= 2023-09-29T23:36:17.929 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ya7CeLIvahcrMfINh9UPfIiI6sA=', 'Content-Length': '0'} 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e128b751eb976b25-0065174f4d-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.930 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.931 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e548903fe8cdee2c-0065174f4d-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-09-29T23:36:17.932 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T23:36:17.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000037a03ba692d82dfb-0065174f4d-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-09-29T23:36:17.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.937 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009237124e724bd7b6-0065174f4d-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.939 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg= 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DpBrECK0qOCVWw/mXc7CQPbK9bg=', 'Content-Length': '0'} 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ee75dd4fbe556e36-0065174f4d-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006de4fe1d723c45fe-0065174f4d-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.943 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081c168677a57750e-0065174f4d-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.944 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.945 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g= 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g=', 'Content-Length': '0'} 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f891bfac70d0b1-0065174f4d-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.946 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.947 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g= 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kxr+pI8H1HbYihEPAxBkL+4sc9g=', 'Content-Length': '0'} 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005901b8b89c07725f-0065174f4d-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:17.948 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.949 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T23:36:17.950 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ed84c3ad98861e6b-0065174f4d-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.951 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI= 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FJVr/NWUITA/XNMh5OTCJxYxXMI=', 'Content-Length': '0'} 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000051472a26e9257e3d-0065174f4d-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:25 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.952 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.953 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:25 GMT 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:fzsOvyEi/mRElMsoGSaihtzGKW8= 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:25 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fzsOvyEi/mRElMsoGSaihtzGKW8=', 'Content-Length': '0'} 2023-09-29T23:36:17.954 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000aa954738329afa59-0065174f4d-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.955 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.956 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb3c1c0281b8c12d-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.957 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI= 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI=', 'Content-Length': '0'} 2023-09-29T23:36:17.958 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b0f6383da5f67cfb-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.959 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI= 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:9fM9wa53+B7zWIT5X5zs2uIsfZI=', 'Content-Length': '0'} 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f8f31b1ddd2a022a-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.960 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.961 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA= 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA=', 'Content-Length': '0'} 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001514818696359ba6-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:17.962 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.963 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA= 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:nHw/xZiNt+Cx8wLPg7Kh8kE82ZA=', 'Content-Length': '0'} 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006a0e5c6d01743389-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.964 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.965 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006bf77a5dd9293029-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.966 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.967 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005c4b7587caf5f694-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.968 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.969 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007e02c34fd6c300c-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.970 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.971 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000956cfeea77f61419-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.972 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.973 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc= 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc=', 'Content-Length': '0'} 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005b87527734526186-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.974 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.975 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc= 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nr9Pu8K35biXDHWAPWoytN+3zlc=', 'Content-Length': '0'} 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000035faf4ce85d129db-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.976 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.977 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005f62fd143a200ddd-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.978 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.979 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M= 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SW5sZ6VKO8OgM99jhWWDnBw9y8M=', 'Content-Length': '0'} 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000825689ff75a3b1f1-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.980 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.981 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e359ddd0b490fb3-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.982 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.983 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004d83ef19eb7ec19f-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.984 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.985 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q= 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q=', 'Content-Length': '0'} 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000061cb748ee176493e-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.986 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q= 2023-09-29T23:36:17.987 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kD85agXsp/cuHDva192urJyGI3Q=', 'Content-Length': '0'} 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000acc2f81c3554ca7d-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.988 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c= 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c=', 'Content-Length': '0'} 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006dbe54629e3484c8-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.989 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.990 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c= 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:snDCDfRaF4wQAHSSi0W/OArZX0c=', 'Content-Length': '0'} 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006d067dbd563a095a-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.991 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.992 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb46ea7c0c809cd7-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:17.993 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.994 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000505702c7bf947ed1-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.995 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:17.996 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T23:36:17.997 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007ef5d6abffebdc30-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:17.998 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T23:36:17.999 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c1f8d666e7a7c494-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.000 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI= 2023-09-29T23:36:18.001 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI=', 'Content-Length': '0'} 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086bbbe4784a5ece9-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.002 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.003 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI= 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mpOS0O3z9IaYwYPClqbGfw77FeI=', 'Content-Length': '0'} 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001a2b98505f8b9543-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.004 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.005 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000018bac9aabb312772-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.006 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.007 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s= 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7vQCe+P4OhaSVrqagMrkqTAVT/s=', 'Content-Length': '0'} 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004d61bb48e117d412-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.008 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.009 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005790c1cce3054cac-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.010 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.011 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5560d4c73b0c4ce-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.012 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE= 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE=', 'Content-Length': '0'} 2023-09-29T23:36:18.013 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005d2cd8ad61d8d0d2-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.014 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE= 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:X8nIen/GG2tWgxs6QEC3udpYJRE=', 'Content-Length': '0'} 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008b44a53162626f34-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.015 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.016 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo= 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo=', 'Content-Length': '0'} 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e125f6468202fcf5-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.017 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.018 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo= 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:H0V78EwLAg3iOcy8fiOt9W4H8fo=', 'Content-Length': '0'} 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000130f40b75086f0be-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.019 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.020 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000034656365072c233c-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.021 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.022 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002aff89b531561cb8-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.023 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.024 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000531c778e176994e4-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.025 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.026 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d4a6141d745053a3-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.027 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.028 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ= 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007cd125531a91538-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.029 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.030 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ= 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:67N7a7wjEtDGAshoyJX9YU9pHPQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000065f89fbab39c4c3f-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.031 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T23:36:18.032 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1a7840c93025b1f-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.033 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k= 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tpE8qeZzWBK/J7GhvaTwgcFfs1k=', 'Content-Length': '0'} 2023-09-29T23:36:18.034 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040d161916174affe-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.035 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000006412da55932e8c1b-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.036 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.037 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002fd453a2fa2d4a79-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.038 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.039 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo= 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo=', 'Content-Length': '0'} 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005268d57ef78f0ed6-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.040 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.041 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo= 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lqmrN+HxTfHtGpuFA8lL+fGIDqo=', 'Content-Length': '0'} 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009c44482f9cafd9b5-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.042 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.043 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI= 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI=', 'Content-Length': '0'} 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b178b5269199d3db-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.044 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.045 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI= 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LRyNUwvSs87N8tzpRPIidAnX7HI=', 'Content-Length': '0'} 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bb0dddf2aa984e4d-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.046 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000b9592e0e365e592-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.047 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.048 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.049 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.050 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000005100f854695c5f9c-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.051 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.052 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d85062c1a9cee4b-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.053 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.054 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000030ee317f882f42d7-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.055 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.056 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U= 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U=', 'Content-Length': '0'} 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000037050c2ffcb2d19b-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.057 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.058 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U= 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bP4iSqQAerkFU8JdHN+a5Wl5u1U=', 'Content-Length': '0'} 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e20b2d438f685a02-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.059 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.060 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a9e88abb1c388686-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.061 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.062 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I= 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/Ij2Mu6CWoGQtZIku5TRqq64D2I=', 'Content-Length': '0'} 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fc33487ca4eebff6-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.063 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.064 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000038bea6fcf69079a2-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.065 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.066 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001666328bc5689918-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.067 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.068 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ= 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f7b5671c6e3733f5-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.069 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ= 2023-09-29T23:36:18.070 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K0D5qoLSPPlp7Zercg7RuXriEYQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000003825abf381e3d75-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.071 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E= 2023-09-29T23:36:18.072 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E=', 'Content-Length': '0'} 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047880adfcae4d312-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.073 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E= 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RPRzZGQMQdzyJVybf7+1mHYgp1E=', 'Content-Length': '0'} 2023-09-29T23:36:18.074 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e5f177f588330b76-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.075 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.076 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004e01403d303d1f3d-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.077 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.078 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000420c74cbf75f2faa-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.079 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.080 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e2360ce10c46a3a-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.081 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.082 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fad1bb6c538c8646-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.083 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.084 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ= 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.085 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003052224fff4d8ffc-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.086 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ= 2023-09-29T23:36:18.087 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jeOQJTHk2loOajGizeCIwFiXuMQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000089cd842e800e59d5-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.088 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.089 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000594a49ca533d5638-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.090 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.091 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk= 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:NF1QgANNEfrzKkxhpdNkgwHTEqk=', 'Content-Length': '0'} 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ab65709473f1456c-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.092 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.093 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000a0ef0328d3b9210-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.094 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.095 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001eadcce9ce195ac2-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.096 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.097 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY= 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY=', 'Content-Length': '0'} 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000605a1d19f6f220f1-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.098 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.099 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY= 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JdDCJ5d/FT6aTSPP4Qo88NovLoY=', 'Content-Length': '0'} 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008cbbc72845336575-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.100 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.101 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU= 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU=', 'Content-Length': '0'} 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c67d6c250e9b8dc8-0065174f4e-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.102 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU= 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pz+JH842GSQJX9rr4x37+ooXOSU=', 'Content-Length': '0'} 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000df76d1ed65c8acdb-0065174f4e-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.103 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.104 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cccb1e00a3145032-0065174f4e-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.105 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.106 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:26 GMT 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo= 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:26 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KPTKyU5rdpBLNi9NDwkeeAup/uo=', 'Content-Length': '0'} 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a364616318dc8467-0065174f4e-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:26 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-09-29T23:36:18.107 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.108 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T23:36:18.109 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000075ebc229ba835f57-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.110 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.111 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fa8b45b530f1da18-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.112 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.113 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y= 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y=', 'Content-Length': '0'} 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000389f9457070a2386-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.114 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.115 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y= 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:DQqCbQjA2pzYXEaiG0FML3bug0Y=', 'Content-Length': '0'} 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000047b3df3bc53b49d7-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.116 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.117 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072eeb96a61e4d66b-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.118 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.119 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I= 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AyU5jekGPvQj3lH8LUG0DYo/+9I=', 'Content-Length': '0'} 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000120494d18d7a8d30-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.120 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.121 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003fd40d9ff6459504-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.122 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.123 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd885df891846494-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.124 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.125 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU= 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU=', 'Content-Length': '0'} 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000829e015d9c618f0c-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.126 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU= 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:tLsCs6KBWCtQ80haNrrocF/pRiU=', 'Content-Length': '0'} 2023-09-29T23:36:18.127 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000801eb992c1d9eace-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.128 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08= 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08=', 'Content-Length': '0'} 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daba551c938daea-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.129 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.130 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08= 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:azUaInQXkcSMYviw1zOBr/P4K08=', 'Content-Length': '0'} 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c047c6a3cf686b7a-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.131 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.132 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000002bc8d416ec95c3db-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.133 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.134 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b5795ae68e9b0176-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.135 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.136 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0a4e43ab0774ee9-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.137 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.138 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1b62a86d66ec79f-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.139 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.140 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI= 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI=', 'Content-Length': '0'} 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086df69ba4fb113a6-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.141 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.142 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI= 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JBmRANrr46K9kVOdGGgmfZgy5kI=', 'Content-Length': '0'} 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d2919d9261bcec53-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.143 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.144 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a6b27edd8686bcb4-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.145 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.146 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ= 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6gZB65lJDruC80AAOFjzNNtxaAQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005163f5402d65fe94-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.147 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.148 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f019894178a1f9d9-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.149 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.150 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003d9ee89020f1edc6-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.151 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E= 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E=', 'Content-Length': '0'} 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020d54e28a4a7f62e-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.152 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.153 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E= 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:kNVeR+Ky39DlvPLJRdpyzHx/y4E=', 'Content-Length': '0'} 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005e200e5f1ca06d40-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.154 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.155 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc= 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc=', 'Content-Length': '0'} 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000081e32019c516a4a4-0065174f4f-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.156 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.157 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc= 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ppb2yCGrnFdoN1JRju87DGlkJuc=', 'Content-Length': '0'} 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ebe439fa79a7b602-0065174f4f-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.158 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.159 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000e155b123fc6e59ad-0065174f4f-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.160 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:27 GMT 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.161 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk= 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:A2xddVAu1dV5k/Dswhf5VCIL3tk=', 'Content-Length': '0'} 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c527c45bd6fcefaa-0065174f4f-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hcnphz-55 zones={a2, a1} 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--bucket', 'hcnphz-55', '--num-shards', '3', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --bucket hcnphz-55 --num-shards 3 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-55 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 8 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding": "None", 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_index": { 2023-09-29T23:36:18.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "Normal", 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "normal": { 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "logs": [ 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:18.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 0, 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 11, 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.166 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.167 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hcnphz-55 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.168 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:27.835+0000 7ff139370f40 1 waiting to reach latest gen 2, on 1.. 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:32.839+0000 7ff139370f40 1 bucket sync caught up with empty source 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:32.839+0000 7ff139370f40 0 bucket checkpoint complete 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hcnphz-55 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-55', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-55 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:32.987+0000 7efe74591f40 1 bucket sync caught up with source: 2023-09-29T23:36:18.169 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, 00000000002.166.6, , , 00000000001.117.6, 00000000001.98.6, , , 00000000002.155.6, , 00000000002.286.6] 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T22:27:32.987+0000 7efe74591f40 0 bucket checkpoint complete 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hcnphz-55 zones={a2, a1} 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.170 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI= 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI=', 'Content-Length': '0'} 2023-09-29T23:36:18.171 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008f3a0f631caa7a12-0065174f55-1642-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/ 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.172 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI= 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YxNoohrHZi9K2VW3MK1EDj/E4RI=', 'Content-Length': '0'} 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000001a359af101cbcea-0065174f55-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.173 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.174 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fd4d02f120c84444-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-09-29T23:36:18.175 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.176 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.177 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000491dd0bce4705f3a-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=a 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=b 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=c 2023-09-29T23:36:18.178 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=d 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=e 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=f 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=g 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=h 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.179 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:18.180 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000795d564a5824d5c4-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?versions&max-keys=999 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.181 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:/hcnphz-55/?versions 2023-09-29T23:36:18.182 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I= 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pLqQdng9AElk9a0sDpmm5+uHQ+I=', 'Content-Length': '0'} 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001cd49bf121420788-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55999falseanulltrue2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalbnulltrue2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalcnulltrue2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormaldnulltrue2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalenulltrue2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalfnulltrue2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalgnulltrue2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalhnulltrue2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:18.183 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.184 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001c58cf9523e5e071-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.185 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.186 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007a444942493bfa61-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.187 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.188 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c= 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c=', 'Content-Length': '0'} 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b1402cca137b1364-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.189 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c= 2023-09-29T23:36:18.190 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ERx8iUTFJ29qT8nlAIK5zPKO21c=', 'Content-Length': '0'} 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009dee43a1937de05e-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.191 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T23:36:18.192 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009fa07df144d96b96-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a?versionId=null 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.193 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a?versionId=null 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII= 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:U0omMVbbvbwleTj2UVpC0VT/OII=', 'Content-Length': '0'} 2023-09-29T23:36:18.194 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec5c240879168889-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.195 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b54a625277d36a7e-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.196 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.197 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cd6cd91a9492b45c-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.198 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.199 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0= 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0=', 'Content-Length': '0'} 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005683bfaeed2d54ec-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.200 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.201 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0= 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9Eh4ZEZPeWjy98KY8ff3YnFY/0=', 'Content-Length': '0'} 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ee593d161011b0fa-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=a 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:18.202 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:18.203 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw= 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw=', 'Content-Length': '0'} 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ca32b5a3e59ecd4c-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/a 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/a 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/a 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.204 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:/hcnphz-55/a 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw= 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:TSSxRRbGIAiulVWX9I4yFptEbJw=', 'Content-Length': '0'} 2023-09-29T23:36:18.205 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000072538d194d5c9362-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.206 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8d9106c825a0a8-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.207 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=a 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=a 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.208 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000292459b27e68f298-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55a1000falsea2023-09-29T22:26:03.091Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.209 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.210 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004005a556e53e56c4-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.211 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.212 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000020790208cf05948f-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.213 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.214 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk= 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk=', 'Content-Length': '0'} 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001d26e03a51fb1ff6-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.215 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.216 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk= 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:M6sFJgc3YIvbAWhdcrGoGd1ceQk=', 'Content-Length': '0'} 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ec8729410931f48a-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.217 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.218 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000088d73058abf39afc-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.219 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b?versionId=null 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.220 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b?versionId=null 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY= 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/g/DnlGq42YkyMLYuxrmbMDSNmY=', 'Content-Length': '0'} 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086ea020abeb24c76-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.221 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.222 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000089c8d45918b94599-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.223 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.224 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d12f3e87a4aad0b9-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:18.225 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:18.226 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo= 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo=', 'Content-Length': '0'} 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000bfdb4630aa502f26-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.227 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo= 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pcxy6H3lLycH3ugTUaWBbLqGWWo=', 'Content-Length': '0'} 2023-09-29T23:36:18.228 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000087e22e0d84f48f53-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=b 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.229 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw= 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw=', 'Content-Length': '0'} 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69ed9bf335b0a6e-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.230 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/b 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/b 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/b 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.231 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:/hcnphz-55/b 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw= 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MHvcbHp+Nb6+D7hgbbaCdliRBlw=', 'Content-Length': '0'} 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000004c5c629759295b37-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.232 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.233 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042f84529e72985c8-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.234 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=b 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=b 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.235 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003174b99586b5dbbd-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55b1000falseb2023-09-29T22:26:03.146Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:18.236 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.237 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000267be75d8736644c-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.238 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.239 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001e96c5c7fecae4d6-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.240 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.241 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I= 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I=', 'Content-Length': '0'} 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ea9d7ae0676a292f-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.242 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.243 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I= 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:39QK80a9URwi6jSe9rT8bRce44I=', 'Content-Length': '0'} 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cb4c65fedc62f7f-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.244 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.245 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000062246877fffd4799-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.246 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c?versionId=null 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.247 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c?versionId=null 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28= 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:r/0b+L32Ld6gR1xFS34LmhLNL28=', 'Content-Length': '0'} 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d0b01a62f06bb02a-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.248 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.249 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000055b50f0bed0adbfa-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.250 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.251 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b7c894924404b38d-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.252 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.253 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q= 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q=', 'Content-Length': '0'} 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d241dadd7a0c04af-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.254 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.257 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q= 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:03E0i8vN6WVuqBSXdsjIxdT7k9Q=', 'Content-Length': '0'} 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008ee43edd9a27e2a3-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=c 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.258 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU= 2023-09-29T23:36:18.259 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU=', 'Content-Length': '0'} 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e27dd016d9dd6475-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/c 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/c 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/c 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.260 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:/hcnphz-55/c 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU= 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Bh7RpvfbqFl6/gXuximliaHsRpU=', 'Content-Length': '0'} 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c48ef98a7b095ba4-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.261 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.262 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000a09c060b4434121c-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.263 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=c 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=c 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.264 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000042397a830e4fbd64-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55c1000falsec2023-09-29T22:26:03.164Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.265 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.266 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c69ee7c73a8d30be-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.267 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.268 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000097ceea3c43af123f-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.269 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.270 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g= 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g=', 'Content-Length': '0'} 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b860e2d160158f5f-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.271 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.272 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g= 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:un6o8K9iX0GsDbcfuTvlBudNT5g=', 'Content-Length': '0'} 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039859f20810e5f31-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.273 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.274 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c1a824e50598d52-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.275 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d?versionId=null 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.276 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d?versionId=null 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g= 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4k0el1XFSXFmJ96DefKwwzXhN9g=', 'Content-Length': '0'} 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000927b41e8ea5daa25-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.277 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.278 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000019336b2898b30eb4-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.279 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.280 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000ba4fc564ad14073-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.281 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.282 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs= 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs=', 'Content-Length': '0'} 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000082bea5cdd9f6678d-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.283 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs= 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3n60Zbvz7xfUd6zWQLiVsLjgfSs=', 'Content-Length': '0'} 2023-09-29T23:36:18.284 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003dd6542d0d086061-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=d 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.285 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8= 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8=', 'Content-Length': '0'} 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e4c722580826b23d-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/d 2023-09-29T23:36:18.286 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/d 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/d 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.287 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:/hcnphz-55/d 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8= 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PTOqtkuf13dwOCIetwyOsYEH2E8=', 'Content-Length': '0'} 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:03 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000021b8f28ae393a16b-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.288 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.289 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000720fb72a03472d61-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=d 2023-09-29T23:36:18.290 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=d 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.291 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000003eab18f1ab39cf88-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55d1000falsed2023-09-29T22:26:03.214Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.292 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.293 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001ac882e158ef96ce-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.294 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.295 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000cd9210fd2af0d74-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.296 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.297 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg= 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg=', 'Content-Length': '0'} 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000028be9a0d4feee27-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.298 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.299 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg= 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:w0joHvVPN2npfptYIwL60t79tsg=', 'Content-Length': '0'} 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c564e959b2979b55-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.300 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.301 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000705ca5d3e3a526c7-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e?versionId=null 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.302 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e?versionId=null 2023-09-29T23:36:18.303 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo= 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:qTJK0BAiAhwiSXWIXkJS3h0lKjo=', 'Content-Length': '0'} 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ae592a3b866567b9-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.304 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.305 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000041738e0d0d19007d-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.306 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.307 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000013bf3d5c91ae6824-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.308 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg= 2023-09-29T23:36:18.309 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg=', 'Content-Length': '0'} 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000507b0893b8f54e48-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.310 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg= 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:I6u6Spp6H6hu8SVCK0bK30tUjQg=', 'Content-Length': '0'} 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000abb3156e45c14ce-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=e 2023-09-29T23:36:18.311 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.312 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8= 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8=', 'Content-Length': '0'} 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000009f95ba8fbfc45bb-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/e 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/e 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/e 2023-09-29T23:36:18.313 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:/hcnphz-55/e 2023-09-29T23:36:18.314 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8= 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/fZvGoAj6vrMxWFq8j5UxMWHHK8=', 'Content-Length': '0'} 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000884de15f40dbebf-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.315 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.316 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000007b0acaaf16d82baf-0065174f55-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=e 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=e 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.317 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE= 2023-09-29T23:36:18.318 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:p3icRd7K9qyYb5lN2oQk+gnXUoE=', 'Content-Length': '0'} 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000080e8150405d9b09-0065174f55-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55e1000falsee2023-09-29T22:26:25.435Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.319 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.320 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000080844cde59962189-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.321 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T23:36:18.322 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002e797f529b46b850-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.323 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA= 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA=', 'Content-Length': '0'} 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000478f9b2fabed7a3a-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.324 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.325 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA= 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:V6a/NfaRvnN3dcRZZM4AXVG5btA=', 'Content-Length': '0'} 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000086a28a4697c06185-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-09-29T23:36:18.326 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.327 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ad80a629fc3a454c-0065174f55-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.328 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f?versionId=null 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.329 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:33 GMT 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f?versionId=null 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8= 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:33 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Mo+9/HMdB9v+Kp51pt4PoqMihb8=', 'Content-Length': '0'} 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003144f66e080a3ba1-0065174f55-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:33 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.330 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.331 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000662395ca3bf15b66-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.332 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.333 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ba7f010ee643c7d2-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.334 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.335 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc= 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc=', 'Content-Length': '0'} 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c0eaf47e7f0abe70-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.336 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc= 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VKSj4HiVtl7NVcMY1Os3wjusYfc=', 'Content-Length': '0'} 2023-09-29T23:36:18.337 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d76e567a3c291c60-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=f 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.338 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU= 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU=', 'Content-Length': '0'} 2023-09-29T23:36:18.339 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000032362e3dbd53ceb9-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/f 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/f 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/f 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.340 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:/hcnphz-55/f 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU= 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IffvYq6rWUo9uANMbq1lYPxUgOU=', 'Content-Length': '0'} 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000045100783aa7bc3f2-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.341 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.342 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.343 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000caf26b803718f3a8-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=f 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=f 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.344 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.345 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001393a5ce3686eeec-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55f1000falsef2023-09-29T22:26:25.451Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.346 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.347 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001db05efbd596c8f2-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.348 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.349 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b4c9a4e0f1ab92ab-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.350 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.351 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk= 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk=', 'Content-Length': '0'} 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094c736e4f2af7158-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.352 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.353 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk= 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:mqmQpsl8LN5ag1dCFDU/sL0fBTk=', 'Content-Length': '0'} 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000539186061fa72c0c-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.354 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.355 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000046bb0656264f260-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.356 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g?versionId=null 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.357 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g?versionId=null 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI= 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JwGWqxX5CKRz+HnR4c5F/1iVLMI=', 'Content-Length': '0'} 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003f6dec1395953e24-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.358 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.359 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000da30a0dbf46283e3-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.360 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.361 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.362 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000510c8ef9dacc9bff-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.363 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ= 2023-09-29T23:36:18.364 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006cd504c77707300b-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.365 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ= 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:t0c/EJS4frMQqyA+NJgx2UlaEUQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.366 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000026b5e79fc61c3d9b-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=g 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.367 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ= 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.368 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000adb1084e4c6f518b-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/g 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/g 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/g 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.369 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:/hcnphz-55/g 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ= 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ow3dSu4D1w9+Beo+VLCdcd5yRDQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fca2ab5744b72311-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.370 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.371 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000f2331719a79a8470-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.372 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=g 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=g 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.373 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000bd637e13a8497d3c-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55g1000falseg2023-09-29T22:26:25.504Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.374 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.375 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b8783bde5966f439-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.376 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.377 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000be4b155f6da92b5-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.378 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.379 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4= 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4=', 'Content-Length': '0'} 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b74675bf08a18f64-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.380 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.381 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4= 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RSdcY9Aq3CZLBiJkx6Cjn7gBzL4=', 'Content-Length': '0'} 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cbe49d9cecc04803-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.382 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.383 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000009b1ee306251c5e09-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.384 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h?versionId=null 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.385 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h?versionId=null 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA= 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:sbN5YfjHQ93zIu0qwcXwN46QZYA=', 'Content-Length': '0'} 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000846d75b3cae133c3-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.386 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.387 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000017a70cd951e0711-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.388 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.389 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000002d993db298e12e0-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.390 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk= 2023-09-29T23:36:18.391 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk=', 'Content-Length': '0'} 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006284388aa328e4d4-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.392 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk= 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IBCHbpBdoLycYyJeUI5afVY06uk=', 'Content-Length': '0'} 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000010047e836e0a2e97-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=h 2023-09-29T23:36:18.393 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.394 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4= 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4=', 'Content-Length': '0'} 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-55:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.20'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000001244e8d30b0f9a61-0065174f56-1642-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/h 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/h 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.395 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/h 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.396 INFO:tasks.rgw_multisite_tests:/hcnphz-55/h 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4= 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:4bFP0n6QGL6NEpSsicegdMLCTa4=', 'Content-Length': '0'} 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 22:26:25 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c78b3cedee70c3f7-0065174f56-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.397 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.398 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c7cb879d673e2974-0065174f56-1642-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/ 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/ 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-55/?prefix=h 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-55/?prefix=h 2023-09-29T23:36:18.399 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 22:27:34 GMT 2023-09-29T23:36:18.400 INFO:tasks.rgw_multisite_tests:/hcnphz-55/ 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4= 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 22:27:34 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:i80YNJmSXe7qap3vENjzzihDoX4=', 'Content-Length': '0'} 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000b26cb66cb4c90511-0065174f56-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 22:27:34 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-55h1000falseh2023-09-29T22:26:25.521Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hcnphz-55 zones={a2, a1} 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.401 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:34.655+0000 7f39009db700 -1 req 17317900272779064789 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-3:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.3,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-3:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.3,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:34.655+0000 7f38e49a3700 -1 req 2720698406258693920 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-24:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.24,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-24:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.24,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:34.663+0000 7f3935244700 -1 req 7328178435526173355 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-26:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.26,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-26:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.26,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:34.663+0000 7f38ec9b3700 -1 req 8345277035345697252 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-29:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.29,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-29:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.29,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:34.679+0000 7f38ce977700 -1 req 7922580692162518313 0.003999894s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-20:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.20,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-20:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.20,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:34.679+0000 7f38bd154700 -1 req 17929215227216565450 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-51:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.16,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-51:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.16,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.402 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding": "None", 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_index": { 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "Normal", 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "normal": { 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.403 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "logs": [ 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.404 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.405 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.406 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.027+0000 7f389810a700 -1 req 16906345383139285555 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-5:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.5,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-5:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.5,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.027+0000 7f38a291f700 -1 req 3358093351381680342 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-39:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.4,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-39:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.4,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.027+0000 7f38a111c700 -1 req 6006448330050620975 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-28:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.28,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-28:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.28,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.035+0000 7f38a011a700 -1 req 11845390325519182613 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-30:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.30,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-30:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.30,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.039+0000 7f389890b700 -1 req 10070599165353365148 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-4:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.4,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-4:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.4388.4,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.043+0000 7f38908fb700 -1 req 16402390693735243024 0.000000000s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-37:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.2,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-37:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.2,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T22:27:35.047+0000 7f38768c7700 -1 req 14765125288640428672 0.003999894s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=hcnphz-38:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.3,z=a75fe7d4-5635-4f57-8059-26c1aacd3a0f,az=0},d={b=hcnphz-38:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.3,z=272b60f1-3a4d-4009-a171-ab410395ba9a,az=0}} returned ret=-2 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'layout', '--bucket', 'hcnphz-55', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket layout --bucket hcnphz-55 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding": "None", 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_index": { 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "Normal", 2023-09-29T23:36:18.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "normal": { 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "logs": [ 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 1, 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "type": "InIndex", 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "in_index": { 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "gen": 2, 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "layout": { 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 3, 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hash_type": "Mod" 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.412 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-29T23:36:18.412 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.412 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-29T23:36:18.412 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-09-29T23:36:18.412 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1842, in test_resharded_bucket_incremental_sync_latest_after_data_sync_init 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-29T23:36:18.413 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.414 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:40 GMT 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.415 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:0njbPnIqthjxda0rs1Pyi/mKpCg= 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:40 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0njbPnIqthjxda0rs1Pyi/mKpCg=', 'Content-Length': '0'} 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000cdc3c071a61318db-0065175790-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:40 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hcnphz-63 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:36:18.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696028560.644899_607.1", 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:02:40.644899Z" 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.417 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.418 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:40.876+0000 7ff913fb5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:40.876+0000 7ff913fb5f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.419 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.420 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696028560.633902_604.1", 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:02:40.633902Z", 2023-09-29T23:36:18.421 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.422 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.423 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:02:40Z" 2023-09-29T23:36:18.424 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.425 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.426 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028560.633902_604.1", 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:02:40.633902Z", 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.427 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.428 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.429 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:40Z" 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028560.633902_604.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:02:40.633902Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.633902_604.1', 2: '', 3: ''} 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696028560.644899_607.1 target=1_1696028560.633902_604.1 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.430 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:46.016+0000 7fcdb62ccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:46.020+0000 7fcdb62ccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.431 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.432 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696028560.644899_607.1", 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.433 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:02:40.644899Z", 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.434 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.435 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.436 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:02:46Z" 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.437 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.438 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028560.644899_607.1", 2023-09-29T23:36:18.439 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:02:40.644899Z", 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.440 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.441 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.442 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:02:46Z" 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028560.644899_607.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:02:40.644899Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028560.644899_607.1', 2: '', 3: ''} 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-29T23:36:18.443 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.444 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008db91138ab5140be-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj11 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj11 2023-09-29T23:36:18.445 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj11 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.446 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj11 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3VsyJq7RoXQdBcFwx1Bp+lIsl9w= 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3VsyJq7RoXQdBcFwx1Bp+lIsl9w='} 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000049fe953529f292d0-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.447 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.448 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000009f3be63aae232cee-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj22 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj22 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj22 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.449 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj22 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.450 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Hrt2jr85Go8VfWHaw45mbebQ23I= 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Hrt2jr85Go8VfWHaw45mbebQ23I='} 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000073e66f286b822ad-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.451 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003d6d495b2e21c941-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj33 2023-09-29T23:36:18.452 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj33 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj33 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.453 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj33 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:GrMFlL0/J+dh7SDXMURGoqHMymU= 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GrMFlL0/J+dh7SDXMURGoqHMymU='} 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008e4559ee9d1367ea-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.454 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.455 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005557cf311a69554c-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj44 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj44 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj44 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.456 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj44 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:YYorWEm0PsTcrvNRmM1CHBmIeqs= 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YYorWEm0PsTcrvNRmM1CHBmIeqs='} 2023-09-29T23:36:18.457 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000657f1479de37c7f8-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.458 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a7ea29185cb9e25c-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj55 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj55 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.459 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj55 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.461 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj55 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ZXBg8+Cg1AauOratjNMqhqQYIPc= 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ZXBg8+Cg1AauOratjNMqhqQYIPc='} 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c8022be96d1c8ba4-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.462 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-63 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 5 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.463 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.464 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000095f241fe4fceb46a-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj17 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj17 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj17 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.465 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj17 2023-09-29T23:36:18.466 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:0zKe4BeCSVEnP1LSlP/HhUZxjjA= 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0zKe4BeCSVEnP1LSlP/HhUZxjjA='} 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b66d44bb25c4575d-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.467 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.468 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cfe5bf73a964d79f-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj34 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj34 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj34 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.469 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj34 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ODDMH7IInvwpT5cyz2YtAf6SDkI= 2023-09-29T23:36:18.470 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ODDMH7IInvwpT5cyz2YtAf6SDkI='} 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026c1b553d3c8cfaf-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.471 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.472 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000611594259b0daf2a-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj51 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj51 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj51 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.473 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj51 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:iPfiH3ImjZDxHOUhEys8rS7aaXc= 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iPfiH3ImjZDxHOUhEys8rS7aaXc='} 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000052731fbdbe86c217-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.474 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.475 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000065c3e862aa8d4499-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj68 2023-09-29T23:36:18.476 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj68 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj68 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.477 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj68 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:wPunL7oDPNxFjYU1g50Uwl752O0= 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wPunL7oDPNxFjYU1g50Uwl752O0='} 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001e15e93de8b0866e-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.478 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.479 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003c02f48fdc01227a-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj85 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj85 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.480 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj85 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.481 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj85 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:6s0MFTX73hHVwLWXkhP7Nz9q8RU= 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:6s0MFTX73hHVwLWXkhP7Nz9q8RU='} 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ba926529d81888b4-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.482 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-63 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 10 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.483 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d9e09dc26bab91b2-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj19 2023-09-29T23:36:18.484 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj19 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj19 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.485 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj19 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Ps5YyJST8EgnFDUDmU04vH55QPw= 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Ps5YyJST8EgnFDUDmU04vH55QPw='} 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d2b1b54557245f32-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.486 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.487 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000046f689522b89f716-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj38 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj38 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj38 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.488 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj38 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:wwznUoV+YviHjspkK27h4TMeTNU= 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wwznUoV+YviHjspkK27h4TMeTNU='} 2023-09-29T23:36:18.489 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009c4ed7870716cde5-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.490 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002719abea70655cb0-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj57 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj57 2023-09-29T23:36:18.491 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj57 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.492 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj57 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JyFZNiQ68ra64ZZYmqTYUULpRe0= 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JyFZNiQ68ra64ZZYmqTYUULpRe0='} 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000a072f0513237261-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.493 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.494 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000037d3784059aa80b4-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj76 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj76 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj76 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.495 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj76 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RBHcvAog+V1s/K2OXdpH+KBSgck= 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBHcvAog+V1s/K2OXdpH+KBSgck='} 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b10a05d1cde2e77d-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.496 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.497 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0= 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:zX23WGxIqUpgz71mAds7dkunnM0=', 'Content-Length': '0'} 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a3fbb75fb59b957f-0065175796-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj95 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj95 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.498 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj95 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.499 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:46 GMT 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj95 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:v+1bcK2g0QQCbFB3D8ThV97J5Gs= 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:46 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:v+1bcK2g0QQCbFB3D8ThV97J5Gs='} 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000817563f5fe181771-0065175796-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:46 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.500 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-63 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 15 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.501 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ca98c4153c82cd5e-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj23 2023-09-29T23:36:18.502 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj23 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj23 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.503 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj23 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:MVLCoVzTF3UsFvyn52H9Jjm1VBU= 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MVLCoVzTF3UsFvyn52H9Jjm1VBU='} 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009dd149ceb746b042-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.504 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.505 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000eaecc0358eceb312-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj46 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj46 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj46 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.506 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj46 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ctMCAtc22knIfa5uwCLqvZbqCuI= 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ctMCAtc22knIfa5uwCLqvZbqCuI='} 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000005c7cc7f496659149-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.507 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.508 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000640776d3bbf8427b-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj69 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj69 2023-09-29T23:36:18.509 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj69 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.510 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj69 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:8M20CrRUTC/suQZInGbrNpaMRpc= 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:8M20CrRUTC/suQZInGbrNpaMRpc='} 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003b8fb71a423e3d8d-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.511 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.512 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002428654c9f8415ff-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj92 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj92 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj92 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.513 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj92 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:0etSwld5W7yrU4u9m2iA41CPv6o= 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0etSwld5W7yrU4u9m2iA41CPv6o='} 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c797b20651f5e4f3-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.514 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.515 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dfe54fa6100eafc7-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj115 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj115 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.516 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj115 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.517 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj115 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:fEuYBInLTwKQgQFQTEkBh3bjY40= 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fEuYBInLTwKQgQFQTEkBh3bjY40='} 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000efeb4afdc1676fa7-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.518 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-63 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 20 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.519 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006808f738291ad124-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj29 2023-09-29T23:36:18.520 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj29 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj29 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.521 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj29 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7YdFd52UB0HpOMadJN+s7RreNs0= 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7YdFd52UB0HpOMadJN+s7RreNs0='} 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000d13fa03a8b65a788-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.522 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.523 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000097e27d3e4d9196e7-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj58 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj58 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj58 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.524 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj58 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:xbawLMafL0BXIIvQBT3xqLnxR0o= 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xbawLMafL0BXIIvQBT3xqLnxR0o='} 2023-09-29T23:36:18.525 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035e60c164cafbc96-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.526 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d28d65eb380a6109-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj87 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj87 2023-09-29T23:36:18.527 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj87 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.528 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj87 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gySpIxGqMwVe3aseqv38+WWXyAY= 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gySpIxGqMwVe3aseqv38+WWXyAY='} 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d8354ef3c6f88d7-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.529 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.530 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032ed4e00a783852d-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj116 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj116 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj116 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.531 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj116 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:KNRLhzYaTC8DDpDm4jkOXTwT2dg= 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:KNRLhzYaTC8DDpDm4jkOXTwT2dg='} 2023-09-29T23:36:18.532 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000033c941bb74d0ab1d-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.533 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU= 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XoUJgOPvIbx4e5zwtvIhcQVDAlU=', 'Content-Length': '0'} 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000336333591df41abb-0065175797-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj145 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj145 2023-09-29T23:36:18.534 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj145 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.535 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:47 GMT 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj145 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:PG6RNzC3aDTjvsPohNwmvlVQB70= 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:47 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:PG6RNzC3aDTjvsPohNwmvlVQB70='} 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000021857fd48101569b-0065175797-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:47 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.536 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-63 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 25 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.537 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005766a85acbd49b8d-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj31 2023-09-29T23:36:18.538 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj31 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj31 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.539 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj31 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:fKrKd+s7mIjwM/T3q57SC4R3h7M= 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:fKrKd+s7mIjwM/T3q57SC4R3h7M='} 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000914d35a982c9a014-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.540 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.541 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000045f974f3e0f5b741-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj62 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj62 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj62 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.542 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj62 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gL7iKGuacMWNP8bAnaANSrPUl1Y= 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gL7iKGuacMWNP8bAnaANSrPUl1Y='} 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000784d55ef5956295f-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.543 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.544 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000047a481ace3ea5317-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj93 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj93 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj93 2023-09-29T23:36:18.545 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj93 2023-09-29T23:36:18.546 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:BN2tl3FwzzXMU+XE1JJDr0C2ozc= 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:BN2tl3FwzzXMU+XE1JJDr0C2ozc='} 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b4c96c74d4bbff73-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.547 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000087d64ba36ca8dea3-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.548 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj124 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj124 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj124 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.549 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj124 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:MCPIIkV2d1x6wgMjtqudDN0nYf8= 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MCPIIkV2d1x6wgMjtqudDN0nYf8='} 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006aad2951f2349d5e-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.550 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.551 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d4963863d24ae374-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj155 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj155 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.552 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj155 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.553 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj155 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Nc2UZt1lezJ0Yq8Z1RV59NsuJ9g= 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Nc2UZt1lezJ0Yq8Z1RV59NsuJ9g='} 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001392567c8a8f0b36-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-63', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-63 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.554 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-63 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 30 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.555 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000025b6e1f7fc7fe3ab-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.556 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj37 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj37 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj37 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.557 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj37 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:f4AsM2zQVwu+sN1YkgYrfYIMOqs= 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:f4AsM2zQVwu+sN1YkgYrfYIMOqs='} 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b48b884e97185892-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.558 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.559 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000d8cefdbb17af77ee-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj74 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj74 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj74 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.560 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj74 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.561 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:YwLjbXbtDdz+2yH1D6DbRjtU5ec= 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:YwLjbXbtDdz+2yH1D6DbRjtU5ec='} 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000001ecbe35e3c49760e-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.562 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000006f7e2d324b9ba2b4-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.563 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj111 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj111 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj111 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.564 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj111 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:bgKW9zhmkCIky+Yh7aTIerigVCU= 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bgKW9zhmkCIky+Yh7aTIerigVCU='} 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000093960dc7f26fb6bd-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.565 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.566 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000075ddacd481867411-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj148 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj148 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj148 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.567 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj148 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:VXI1Qq0yn4YICiMdB0QFLF8bwGs= 2023-09-29T23:36:18.568 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VXI1Qq0yn4YICiMdB0QFLF8bwGs='} 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000008742674c941057e5-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/ 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/ 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/ 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.569 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:/hcnphz-63/ 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA= 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7Dr9Jwahm1Co0l5QY5yfh1puHzA=', 'Content-Length': '0'} 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000032b62e1be2425f68-0065175798-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-63/obj185 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-63/obj185 2023-09-29T23:36:18.570 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-63/obj185 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.571 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:02:48 GMT 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:/hcnphz-63/obj185 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:EWroXpbMFg9Ex/gnfuKnAw3jGDk= 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:02:48 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:EWroXpbMFg9Ex/gnfuKnAw3jGDk='} 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000006f1cfffbf56272-0065175798-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:02:48 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hcnphz-63 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-63 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.572 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:48.928+0000 7fbb8821cf40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:53.928+0000 7fbb8821cf40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:58.936+0000 7fbb8821cf40 1 bucket sync caught up with source: 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, , , , , , 00000000002.118.6, , , , , , 00000000002.115.6, , , , , 00000000002.105.6, , , , , , , , , , , , , , , 00000000001.131.6, , , , 00000000001.107.6] 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, , , , , , 00000000002.118.6, , , , , , 00000000002.115.6, , , , , 00000000002.105.6, , , , , , , , , , , , , , , 00000000001.131.6, , , , 00000000001.107.6] 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:58.936+0000 7fbb8821cf40 0 bucket checkpoint complete 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hcnphz-63 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-63', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-63 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.573 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:02:59.080+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:04.080+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:09.080+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:14.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:19.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:24.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:29.083+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:34.087+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:39.087+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:44.091+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:49.091+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.574 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:54.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:03:59.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:04.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:09.090+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:14.094+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:19.094+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:24.094+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:29.093+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:34.097+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:39.101+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:44.101+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.575 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:49.101+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:54.105+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:04:59.105+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:04.105+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:09.108+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:14.108+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:19.108+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:24.112+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:29.112+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:34.112+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:39.116+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:44.115+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.576 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:49.115+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:54.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:05:59.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:04.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:09.119+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:14.123+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:19.123+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:24.122+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:29.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:34.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:39.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:44.126+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.577 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:49.130+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:54.130+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:06:59.130+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:04.133+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:09.133+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:14.137+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:19.137+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:24.141+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:29.141+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi131.stdout: INFO: 2023-09-29T23:07:30.251+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:34.141+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:39.144+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.578 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:44.144+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:49.144+0000 7fd4d3a25f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:54.144+0000 7fd4d3a25f40 -1 bucket checkpoint timed out waiting to reach incremental sync 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:54.144+0000 7fd4d3a25f40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:54.144+0000 7fd4d3a25f40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-29T23:36:18.579 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 1904, in test_resharded_bucket_incremental_sync_oldest_after_data_sync_init 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests: zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) 2023-09-29T23:36:18.580 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 395, in zonegroup_bucket_checkpoint 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket_name) 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1) 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite.py", line 197, in admin 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests: assert r == 0 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests:AssertionError: 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.581 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.582 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gHVhHSDGwpvcGz+geqan135uQKs= 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gHVhHSDGwpvcGz+geqan135uQKs=', 'Content-Length': '0'} 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000363e896aae6634de-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: created bucket=hcnphz-64 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.583 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696028874.218108_635.1", 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:07:54.218108Z" 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.584 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696028874.218108_635.1', 2: '', 3: ''} 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.585 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:54.440+0000 7f018c4e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:54.440+0000 7f018c4e7f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.586 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.587 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696028874.218108_635.1", 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.588 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:07:54.218108Z", 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.589 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.590 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.591 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:07:54Z" 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.592 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.593 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "marker": "1_1696028874.218108_635.1", 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:07:54.218108Z", 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.594 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.595 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.596 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:07:54Z" 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.597 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696028874.218108_635.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:07:54.218108Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696028874.218108_635.1', 2: '', 3: ''} 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696028874.218108_635.1', 2: '', 3: ''} 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.598 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:36:18.599 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000b3eb7c7582c835a-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.600 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:oeLb5xZ/52xdRBkcafX5wDJG0EA= 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:oeLb5xZ/52xdRBkcafX5wDJG0EA='} 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000c5448adb18e18f2-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.601 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.602 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '1'), ('X-RGW-Bytes-Used', '3'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f163f178eb7881b7-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22 2023-09-29T23:36:18.603 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22 2023-09-29T23:36:18.604 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:uyye1TxlBZ9FKkRsHP4j4swNt5M= 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uyye1TxlBZ9FKkRsHP4j4swNt5M='} 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004ead9b5914aeca06-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.605 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:36:18.606 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '2'), ('X-RGW-Bytes-Used', '6'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000a589ad1c4d641514-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.607 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:hvAKVPFQaAom5Qr+MUQFgAeo5n4= 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:hvAKVPFQaAom5Qr+MUQFgAeo5n4='} 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c892055b32f67b86-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.608 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.609 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '3'), ('X-RGW-Bytes-Used', '9'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000bc552f8587f3db75-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.610 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.611 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HmfdsPwFbA17aMgeLKssyU/8BtE= 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HmfdsPwFbA17aMgeLKssyU/8BtE='} 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000000966a5a23e6e121e-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.612 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs= 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:RBMVjo4D5sOJrF1PqaRY+BOBrYs=', 'Content-Length': '0'} 2023-09-29T23:36:18.613 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '4'), ('X-RGW-Bytes-Used', '12'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e507b653268aea0d-00651758ca-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.614 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:07:54 GMT 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:eb161C01IjoPMiIYcqjHiAhdvyU= 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:07:54 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:eb161C01IjoPMiIYcqjHiAhdvyU='} 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000082958461c1f87a2b-00651758ca-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hcnphz-64 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.615 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:54.756+0000 7f211af40f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:59.760+0000 7f211af40f40 1 bucket sync caught up with source: 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, , , , , , , , 00000000001.70.6, 00000000002.112.6, 00000000002.137.6] 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, , , , , , , , 00000000001.70.6, 00000000002.112.6, 00000000002.137.6] 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:59.760+0000 7f211af40f40 0 bucket checkpoint complete 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a2 target=a1 bucket=hcnphz-64 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.616 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-64 --source-zone a2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:07:59.952+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:04.952+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:09.956+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:14.956+0000 7ff052e21f40 1 waiting to reach incremental sync.. 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:19.959+0000 7ff052e21f40 1 bucket sync caught up with source: 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: local status: [, , , , , , , , 00000000001.159.6, 00000000002.316.6, 00000000002.196.6] 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: remote markers: [, , , , , , , , 00000000001.159.6, 00000000002.316.6, 00000000002.196.6] 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:19.959+0000 7ff052e21f40 0 bucket checkpoint complete 2023-09-29T23:36:18.617 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: comparing bucket=hcnphz-64 zones={a2, a1} 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:36:18.618 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:19 GMT 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo= 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo=', 'Content-Length': '0'} 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f5c767151567594f-00651758e3-1ef7-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:19 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.619 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.620 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:19 GMT 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo= 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:19 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ITNj77ya6Ouxvfy0g+drnJIMYDo=', 'Content-Length': '0'} 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000287908df22875a45-00651758e3-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket1 objects: 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.621 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.622 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:/hcnphz-64/?versions 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c50596144f6cf85c-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-09-29T23:36:18.623 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: bucket2 objects: 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.624 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:/hcnphz-64/?versions 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.625 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000c6d2dd516037c8d7-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj11 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj22 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj33 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj44 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: o=obj55 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.626 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.627 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:/hcnphz-64/?versions 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000028d11ed34f398f27-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.628 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?versions&max-keys=999 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.629 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:/hcnphz-64/?versions 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc= 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:HPb3yYJMocq+71Zkwtv/LewYTJc=', 'Content-Length': '0'} 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d4e5e35a2dce7287-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64999falseobj11nulltrue2023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj22nulltrue2023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj33nulltrue2023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj44nulltrue2023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormalobj55nulltrue2023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.630 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.631 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000d1843eec3c709d43-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.632 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.633 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003e8e65707312a77b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.634 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.635 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY= 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY=', 'Content-Length': '0'} 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000246718f0c790efa7-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.636 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.637 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY= 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XGFI2/3m8opJ8Uo8X/ZYpjSHzYY=', 'Content-Length': '0'} 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000074cf7cd370dc0fb2-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.638 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.639 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000091d8b7900e9620c4-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.640 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11?versionId=null 2023-09-29T23:36:18.641 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ= 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JLNn97d9YZmIeYHFs5Z4G18fJpQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f98c48ba6bc84784-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.642 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.643 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000018804391b7a798fc-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.644 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.645 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000008f57d318606aed28-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.646 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8= 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8=', 'Content-Length': '0'} 2023-09-29T23:36:18.647 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000059267b1b76200a7d-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.648 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8= 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rhx/UBR0uhcU0/kmjDR4vb4Z0g8=', 'Content-Length': '0'} 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000092c0ccc79487aa3d-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj11 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.649 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.650 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA= 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA=', 'Content-Length': '0'} 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000008daa19247e0b4ee7-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj11 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj11 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj11 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.651 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj11 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.652 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA= 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:N3YVYUawmRj4BlVKvjCpEmbuHnA=', 'Content-Length': '0'} 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe8aa16084aade9f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.653 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.654 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000007cec1790ec521a2-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj11 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.655 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.656 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000000f8340eb2629567d-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj111000falseobj112023-09-29T23:07:54.487Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.657 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.658 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c295662032753d54-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.659 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.660 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a14a40ad1e91eba3-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.661 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.662 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY= 2023-09-29T23:36:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY=', 'Content-Length': '0'} 2023-09-29T23:36:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000011773f0a91baec19-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.663 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.665 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY= 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:AcUv1sbnxt+FrQGJtd1y5tyH0DY=', 'Content-Length': '0'} 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000650467a1744f490b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.666 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.667 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000448f21448f66f838-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.668 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.669 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22?versionId=null 2023-09-29T23:36:18.670 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE= 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3wEZAeEP3DadMuB2awUdTwEYChE=', 'Content-Length': '0'} 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000af5f1de7ddcf1e23-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.671 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.672 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000fef5b6ad6f856e4f-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.673 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.674 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000d62eb2f1cf14fc23-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.675 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.676 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew= 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew=', 'Content-Length': '0'} 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000005ca8e46435b2d831-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.677 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.678 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew= 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:GsZVtPecwtae+lwNR9iSG8ndCew=', 'Content-Length': '0'} 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000b6f8c86d327cc106-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj22 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.679 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22 2023-09-29T23:36:18.680 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ= 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c50885bb50e8b3a5-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj22 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj22 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj22 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.681 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj22 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ= 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:wYSCytS0gqqjL6eh45GZZnXqnyQ=', 'Content-Length': '0'} 2023-09-29T23:36:18.682 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000baf80f887baa23ed-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.683 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000464b265d465bfc94-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.684 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj22 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.685 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000ca1ff4eb47946cbe-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.686 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj221000falseobj222023-09-29T23:07:54.507Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.687 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:36:18.688 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c33d996ef5557fd7-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.689 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000e1444c1c10b40d4b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.690 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.691 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI= 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI=', 'Content-Length': '0'} 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000ea4984875816f20-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.692 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.693 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI= 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iaoUM+UJlgHvOZwqySfyegBx2VI=', 'Content-Length': '0'} 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000a8ff99f125ab8d33-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.694 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.695 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f29d011778f52249-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.696 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.697 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33?versionId=null 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA= 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uy5l8Tl5jEM6ZyyyF+2g4DdqWqA=', 'Content-Length': '0'} 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c03ac08b21861ed8-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.698 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.699 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000dccdc70020543a9f-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.700 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.701 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000196cbcd37e0f2b5d-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.702 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.703 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8= 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8=', 'Content-Length': '0'} 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000079fbd2df1fb51879-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.704 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33 2023-09-29T23:36:18.705 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8= 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lT1ps/InXroQLQgsHtH1oeB+hf8=', 'Content-Length': '0'} 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000040c0b3a8d3a47138-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj33 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.706 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c= 2023-09-29T23:36:18.707 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c=', 'Content-Length': '0'} 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000002489d0f695c53387-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj33 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj33 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj33 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.708 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj33 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c= 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:JSHUW4jIGmxTE4TzO8HotF9Dc7c=', 'Content-Length': '0'} 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006f80dd6cac90417a-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.709 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.710 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000060cf2bc87f4bed00-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.711 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj33 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.712 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000001b347724b622bd40-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.713 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj331000falseobj332023-09-29T23:07:54.559Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.714 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.715 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000e9e5c49430058be-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.716 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.717 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000f1130b7aee650bf8-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.718 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.719 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo= 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo=', 'Content-Length': '0'} 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000c5e40215e0fcf62f-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.720 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.721 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo= 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FKeTcmzyrw8m3H/Q/kElQ1V+IIo=', 'Content-Length': '0'} 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000039615b7e67f6ec38-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.722 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.723 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000094861147099ab769-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.724 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.725 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44?versionId=null 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0= 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OvDcKMA4Y/BnCq9HZVADU2t9jI0=', 'Content-Length': '0'} 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000049e66d8746a9106b-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.726 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.727 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000011cc2f53213d7ae5-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.728 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.729 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000004bad3ca7861e9323-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.730 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.731 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc= 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc=', 'Content-Length': '0'} 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000952badfbf8aaa365-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.732 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44 2023-09-29T23:36:18.733 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc= 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:rAAoEWm4Oi90zKrbCEp55qzdBBc=', 'Content-Length': '0'} 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000339fd83137d68b0f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj44 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.734 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs= 2023-09-29T23:36:18.735 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs=', 'Content-Length': '0'} 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000006c4a6f54647e233f-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj44 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj44 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj44 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.736 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj44 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs= 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:VFbavUr4BkWGc/Ai5NvnzZbwkZs=', 'Content-Length': '0'} 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000007c81ef947b2a0661-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.737 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.738 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000092a5aaca2a4b1e5a-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.739 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj44 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.740 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000cb8551c08d628464-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.741 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj441000falseobj442023-09-29T23:07:54.578Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.742 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:36:18.743 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000edf29e7cef3f8d57-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.744 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000003d79c93d172f66a2-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.745 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.746 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM= 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM=', 'Content-Length': '0'} 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx000000b9cf5582ceb40ff-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.747 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.748 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM= 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:3Myvm7cwO0W2SQypd/ykmzUPQqM=', 'Content-Length': '0'} 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000076f03ade4f92a20f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-29T23:36:18.749 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.750 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000fe6cefb24a297835-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.751 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.752 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55?versionId=null 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg= 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aOuuNCe/oFhi3WLL41FkTGRPcGg=', 'Content-Length': '0'} 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-amz-version-id', 'null'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000ded2b2933a4c8e2f-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.753 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.754 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009eca33e05f014ed0-00651758e4-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.755 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.756 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY= 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:2pzq2RaePC+OlT9Yj4Wzp5mGJMY=', 'Content-Length': '0'} 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx00000df00100b7e3dfb0d-00651758e4-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.757 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.758 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI= 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI=', 'Content-Length': '0'} 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000142c6d5a10cd005b-00651758e4-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55 2023-09-29T23:36:18.759 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:20 GMT 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55 2023-09-29T23:36:18.760 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI= 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:20 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:W3MbVOU5yvuH5HBK0f6LB7BkzvI=', 'Content-Length': '0'} 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000007f747b6572d266d-00651758e4-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:20 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: DEBUG: comparing key name=obj55 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.761 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:21 GMT 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE= 2023-09-29T23:36:18.762 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE=', 'Content-Length': '0'} 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('x-amz-replication-status', 'REPLICA'), ('x-rgw-replicated-from', 'a75fe7d4-5635-4f57-8059-26c1aacd3a0f:hcnphz-64:a75fe7d4-5635-4f57-8059-26c1aacd3a0f.5706.29'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx00000cde0e29296f5d983-00651758e5-1ef7-a2'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj55 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj55 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj55 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.763 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:21 GMT 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj55 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE= 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:aWXGSrOAy/5C3LRjSPDRP2bglQE=', 'Content-Length': '0'} 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '3'), ('Accept-Ranges', 'bytes'), ('Last-Modified', 'Fri, 29 Sep 2023 23:07:54 GMT'), ('x-rgw-object-type', 'Normal'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('x-amz-request-id', 'tx0000036e7d9a98367631c-00651758e5-1648-a1'), ('Content-Type', 'application/octet-stream'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.764 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.765 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:21 GMT 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY= 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY=', 'Content-Length': '0'} 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx0000008fcdcecfe256e60-00651758e5-1ef7-a2'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.766 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: GET 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/?prefix=obj55 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.767 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:GET 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:21 GMT 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY= 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:21 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IaW+tiSiXwadhHb47WpCbiHF5eY=', 'Content-Length': '0'} 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Transfer-Encoding', 'chunked'), ('x-amz-request-id', 'tx000009431464e0d3aa018-00651758e5-1648-a1'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:08:21 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.768 INFO:tasks.rgw_multisite_tests:boto: DEBUG: b'hcnphz-64obj551000falseobj552023-09-29T23:07:54.631Z"acbd18db4cc2f85cedef654fccc4a4d8"3STANDARDrgw-multisite-test-userMultisite Test UserNormal' 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.zone_rados: INFO: success, bucket identical: bucket=hcnphz-64 zones={a2, a1} 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: DEBUG: waiting for process to exit 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: INFO: waiting for 300 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T23:08:21.071+0000 7f4a52ee1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 79272) UID: 0 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T23:08:21.071+0000 7f4a56740f80 -1 shutting down 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Stopped 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '17', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.769 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 17 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-64 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 5 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.770 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.771 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '5'), ('X-RGW-Bytes-Used', '15'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000cff3d762e3b75169-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj17 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj17 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj17 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.772 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj17 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:MrET2tVQQMpyzFXqkPaqZOMu76k= 2023-09-29T23:36:18.773 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:MrET2tVQQMpyzFXqkPaqZOMu76k='} 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000006ce1e67f1244ab1-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.774 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '6'), ('X-RGW-Bytes-Used', '18'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004930a8520e33ef5f-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.775 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj34 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj34 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj34 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.776 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj34 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:uNsLZa7sWvzgjO5PTjwO1kRzKg0= 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uNsLZa7sWvzgjO5PTjwO1kRzKg0='} 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000035e847b471d6d244-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.777 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.778 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '7'), ('X-RGW-Bytes-Used', '21'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000002000272854156dae-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj51 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj51 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj51 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.779 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj51 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FHN3efPYJCcZ6n4Jxp0Ng/BrVIQ= 2023-09-29T23:36:18.780 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FHN3efPYJCcZ6n4Jxp0Ng/BrVIQ='} 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000006cc0cec6198ccf07-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.781 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '8'), ('X-RGW-Bytes-Used', '24'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000012d689b03a79737e-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj68 2023-09-29T23:36:18.782 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj68 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj68 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.783 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj68 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:FiqbqHGdJJWkQVwAopGsVKgzzU4= 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:FiqbqHGdJJWkQVwAopGsVKgzzU4='} 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000b979194614f2ebab-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.784 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.785 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '9'), ('X-RGW-Bytes-Used', '27'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008347f07c42dd4a0b-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj85 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj85 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj85 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.786 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj85 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:SShh1zayCDhUb1AA5Ia+eyJu9bc= 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SShh1zayCDhUb1AA5Ia+eyJu9bc='} 2023-09-29T23:36:18.787 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059e9f906b811f3d9-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '19', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 19 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-64 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 10 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.788 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.789 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '10'), ('X-RGW-Bytes-Used', '30'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000005580344351cf044f-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj19 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj19 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj19 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.790 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj19 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:7LWU167bq48cKm6Ly/X7/9xRZ5w= 2023-09-29T23:36:18.791 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:7LWU167bq48cKm6Ly/X7/9xRZ5w='} 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000fcdf75ec71f3f161-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.792 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '11'), ('X-RGW-Bytes-Used', '33'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000520dbac2be8b2021-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj38 2023-09-29T23:36:18.793 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj38 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj38 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.794 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj38 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:q9sx11XIQu4AdH/if0/h1GAPobQ= 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:q9sx11XIQu4AdH/if0/h1GAPobQ='} 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000096d8c0e4855f15d-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.795 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.796 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '12'), ('X-RGW-Bytes-Used', '36'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000ba616302b5d3f43c-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj57 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj57 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj57 2023-09-29T23:36:18.797 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.798 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj57 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:lDP7LbJYU4c2KCwU3PhvwhtEdHM= 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:lDP7LbJYU4c2KCwU3PhvwhtEdHM='} 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004efec346b392dd80-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.799 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.800 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '13'), ('X-RGW-Bytes-Used', '39'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003f128894b7407e10-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj76 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj76 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj76 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.801 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj76 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.802 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:IzH5d+0rvAV+Ts95LMabeDOneJM= 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:IzH5d+0rvAV+Ts95LMabeDOneJM='} 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f99c6838308edc01-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.803 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM= 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Xux6ROz2yhAJgsB1UP+1POYuwOM=', 'Content-Length': '0'} 2023-09-29T23:36:18.804 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '14'), ('X-RGW-Bytes-Used', '42'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000628e1208b04d3d4a-00651758eb-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj95 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj95 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj95 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.805 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:27 GMT 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj95 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:5+Ve9lauabMQalYkYEmsWVl7XH0= 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:27 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:5+Ve9lauabMQalYkYEmsWVl7XH0='} 2023-09-29T23:36:18.806 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003650fe82a5a7143b-00651758eb-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:27 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '23', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 23 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-64 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 15 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.807 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.808 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '15'), ('X-RGW-Bytes-Used', '45'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000008293c57f5cc96530-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj23 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj23 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj23 2023-09-29T23:36:18.809 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.810 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj23 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:dgDHy46sV2hP+wCnOm0XgOFFKio= 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:dgDHy46sV2hP+wCnOm0XgOFFKio='} 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000010d626fcf3db0f0d-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.811 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.812 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '16'), ('X-RGW-Bytes-Used', '48'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000dde7ae51b32fcdcf-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj46 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj46 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj46 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.813 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj46 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.814 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:XdRDq9QCgPjpA4s+Qklp+LRzcyg= 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:XdRDq9QCgPjpA4s+Qklp+LRzcyg='} 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000059a13ef26bd3ac21-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.815 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.816 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '17'), ('X-RGW-Bytes-Used', '51'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000001bd16b4eb472d005-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj69 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj69 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj69 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.817 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj69 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:uvFUKcJ1PWYvrpsFave8AbrGTCo= 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:uvFUKcJ1PWYvrpsFave8AbrGTCo='} 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000be7658a89d96ec68-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.818 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.819 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '18'), ('X-RGW-Bytes-Used', '54'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041700fd4c27f9aa4-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj92 2023-09-29T23:36:18.820 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj92 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj92 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.821 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj92 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ox6yM9WyH9QenmachEelKuAbWZ0= 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ox6yM9WyH9QenmachEelKuAbWZ0='} 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000009d84b5ca71659926-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.822 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.823 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '19'), ('X-RGW-Bytes-Used', '57'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000043351201958f4477-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj115 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj115 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.824 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj115 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.825 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj115 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yXJoh+VBQQmStytwBpqC7xB0xCU= 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yXJoh+VBQQmStytwBpqC7xB0xCU='} 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000004641adc5eae9dbe5-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '29', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 29 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.826 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-64 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 20 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.827 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.828 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '20'), ('X-RGW-Bytes-Used', '60'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b52fa273bfe91892-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj29 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj29 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj29 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.829 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj29 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Uwnj4mjedUputwQ+K+3F8M3kOII= 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Uwnj4mjedUputwQ+K+3F8M3kOII='} 2023-09-29T23:36:18.830 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000037fca4e37768d0fc-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.831 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '21'), ('X-RGW-Bytes-Used', '63'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000adf31fe4319ba385-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj58 2023-09-29T23:36:18.832 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj58 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj58 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.833 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj58 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:OmDsrddYYCrJRsUnU+oL+Mbfgx4= 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OmDsrddYYCrJRsUnU+oL+Mbfgx4='} 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000c7f0563ac79c39fb-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.834 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.835 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '22'), ('X-RGW-Bytes-Used', '66'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000950368f051b1cb69-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj87 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj87 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.836 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj87 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.837 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj87 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:ruMT9q9TUznBT+Vh/q+TiGMP9TA= 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:ruMT9q9TUznBT+Vh/q+TiGMP9TA='} 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000095a27bb2e646bfc5-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.838 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.839 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '23'), ('X-RGW-Bytes-Used', '69'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003dd4dc534c472d5b-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj116 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj116 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj116 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.840 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj116 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:0cVGTpDjY3A33WOHnY65E6sv0bU= 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:0cVGTpDjY3A33WOHnY65E6sv0bU='} 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000bd3f785daef3b47b-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.841 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.842 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE= 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Y+7o4fFpkAS6zmnLvyhfZN0ELTE=', 'Content-Length': '0'} 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '24'), ('X-RGW-Bytes-Used', '72'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000008e6aad89c1a76dc-00651758ec-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj145 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj145 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj145 2023-09-29T23:36:18.843 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:28 GMT 2023-09-29T23:36:18.844 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj145 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:J+Tu5687J7vKbwYyZe5uAca0mNU= 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:28 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:J+Tu5687J7vKbwYyZe5uAca0mNU='} 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e64f88adc5f20906-00651758ec-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:28 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '31', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 31 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.845 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-64 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 25 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.846 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '25'), ('X-RGW-Bytes-Used', '75'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000000f1f0d68c2d6aadd-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.847 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj31 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj31 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj31 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.848 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj31 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:xFKAiNuF2JhqXFCMnnB+oJmcCoY= 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:xFKAiNuF2JhqXFCMnnB+oJmcCoY='} 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000e6bf34768b218c93-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.849 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.850 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '26'), ('X-RGW-Bytes-Used', '78'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e1117808ff7d0699-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj62 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj62 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj62 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.851 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj62 2023-09-29T23:36:18.852 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:E5kPQwCvfIqMpaSSvQV6Jawzf9M= 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:E5kPQwCvfIqMpaSSvQV6Jawzf9M='} 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000601211c5046f97-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.853 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.854 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '27'), ('X-RGW-Bytes-Used', '81'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000003608d865a54b388f-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj93 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj93 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj93 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.855 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj93 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:z7W92xRkFg+mE4e0AHfYI49MJOE= 2023-09-29T23:36:18.856 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:z7W92xRkFg+mE4e0AHfYI49MJOE='} 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000000fdd5682d942aba-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.857 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.858 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '28'), ('X-RGW-Bytes-Used', '84'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b92b82a4ca70fdd2-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj124 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj124 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj124 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.859 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj124 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:Q1F66Vm/gXtFwmgPXreIslGXyew= 2023-09-29T23:36:18.860 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:Q1F66Vm/gXtFwmgPXreIslGXyew='} 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000039afa990908cb3bc-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.861 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.862 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '29'), ('X-RGW-Bytes-Used', '87'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000f6431d1a874ec7e5-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj155 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj155 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj155 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.863 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj155 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:iumiTN4f7/qkLdyN0VuF2ybHYv4= 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:iumiTN4f7/qkLdyN0VuF2ybHYv4='} 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000026cd8deb670b33ac-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'reshard', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--bucket', 'hcnphz-64', '--num-shards', '37', '--yes-i-really-mean-it', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.864 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket reshard --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --bucket hcnphz-64 --num-shards 37 --yes-i-really-mean-it --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:tasks.daemonwatchdog.daemon_watchdog: INFO: daemon c1.rgw.client.1 has been restored 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: tenant: 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: bucket name: hcnphz-64 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: total entries: 30 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.865 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.866 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '30'), ('X-RGW-Bytes-Used', '90'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000004d4bfad0b95087a6-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj37 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj37 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj37 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.867 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj37 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:pq9fua7yfSlwKc354hDYdvZaEgA= 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:pq9fua7yfSlwKc354hDYdvZaEgA='} 2023-09-29T23:36:18.868 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000070f35307a9d492c4-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.869 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '31'), ('X-RGW-Bytes-Used', '93'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000b3164dfc2e5d43ba-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj74 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj74 2023-09-29T23:36:18.870 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj74 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.871 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj74 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yCafieqdpd0V3sM8DLoC0rjlpD0= 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yCafieqdpd0V3sM8DLoC0rjlpD0='} 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000ca92505fed6fbc06-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.872 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.873 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '32'), ('X-RGW-Bytes-Used', '96'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000057d404f0e1d22159-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj111 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj111 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj111 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.874 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj111 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:LntyddlroZBjqNST93aVFituHFU= 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:LntyddlroZBjqNST93aVFituHFU='} 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000f5cf61a9acf0076a-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.875 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.876 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '33'), ('X-RGW-Bytes-Used', '99'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000c98bbc2ec3b39029-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj148 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj148 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.877 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj148 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.878 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj148 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yI75IkBDSiFKr6/nP0G8rHEu0xw= 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yI75IkBDSiFKr6/nP0G8rHEu0xw='} 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000a0c34c4de88268fd-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/ 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/ 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/ 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.879 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:/hcnphz-64/ 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.880 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE= 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:gw9YqS28P6mDObuxhV9ARMz0IKE=', 'Content-Length': '0'} 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '34'), ('X-RGW-Bytes-Used', '102'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx00000e10db53e11b8d6e7-00651758ed-1648-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-64/obj185 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-64/obj185 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-64/obj185 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue'} 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.881 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:rL0Y20zC+Fzt72VPzMSk2A== 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:08:29 GMT 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:/hcnphz-64/obj185 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:jbyEuWCKZk2eHZ2HWc1P4DItY3o= 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'rL0Y20zC+Fzt72VPzMSk2A==', 'Content-Length': '3', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:08:29 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:jbyEuWCKZk2eHZ2HWc1P4DItY3o='} 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"acbd18db4cc2f85cedef654fccc4a4d8"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx00000cb3c73ebaece6587-00651758ed-1648-a1'), ('Date', 'Fri, 29 Sep 2023 23:08:29 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.882 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'init', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync init --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:29.891+0000 7ff864fd5f40 5 RGW-SYNC:data: waiting on data sync status lease 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:29.891+0000 7ff864fd5f40 5 RGW-SYNC:data: acquired data sync status lease 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Restarting daemon 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1: INFO: Started 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.883 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 7 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: curl: (7) Failed to connect to smithi112.front.sepia.ceph.com port 8001: Connection refused 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > curl http://smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: % Total % Received % Xferd Average Speed Time Time Time Current 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: Dload Upload Total Spent Left Speed 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 71333 0 --:--:-- --:--:-- --:--:-- 71333 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: anonymous 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: bucket checkpoint: source=a1 target=a2 bucket=hcnphz-64 2023-09-29T23:36:18.884 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hcnphz-64', '--source-zone', 'a1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hcnphz-64 --source-zone a1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:31.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:36.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:41.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:46.187+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:51.191+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:08:56.190+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:01.190+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.885 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:06.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:11.190+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:16.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:21.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:26.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:31.194+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:36.197+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:41.197+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:46.201+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:51.205+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:09:56.205+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 0.. 2023-09-29T23:36:18.886 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:01.205+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:06.209+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:11.208+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:16.208+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:21.208+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:26.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:31.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:36.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:41.212+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:46.216+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:51.215+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:10:56.215+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.887 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:01.219+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:06.215+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:11.219+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:16.219+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:21.223+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:26.223+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:31.222+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:36.222+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:41.222+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:46.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:51.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.888 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:11:56.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:01.226+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:06.225+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:11.229+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:16.229+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:21.233+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:26.233+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi131.stdout: INFO: 2023-09-29T23:12:30.389+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:31.237+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:36.237+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:41.237+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:46.236+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.889 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:51.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:12:56.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:01.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:06.240+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:11.244+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:16.244+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:21.244+0000 7ff58b31cf40 1 waiting to reach latest gen 6, on 1.. 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:26.243+0000 7ff58b31cf40 -1 bucket checkpoint timed out waiting to reach latest generation 6 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:26.243+0000 7ff58b31cf40 0 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:13:26.243+0000 7ff58b31cf40 -1 bucket sync checkpoint failed: (110) Connection timed out 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run: DEBUG: got remote process result: 110 2023-09-29T23:36:18.890 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-29T23:36:18.891 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.891 INFO:tasks.rgw_multisite_tests:====================================================================== 2023-09-29T23:36:18.891 INFO:tasks.rgw_multisite_tests:FAIL: test_sync_flow_symmetrical_zonegroup_all: 2023-09-29T23:36:18.891 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2177, in test_sync_flow_symmetrical_zonegroup_all 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests: check_object_exists(bucket, objnames[0], content) 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/../src/test/rgw/rgw_multi/tests.py", line 2041, in check_object_exists 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests: assert_not_equal(k, None) 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests:AssertionError: None == None 2023-09-29T23:36:18.892 INFO:tasks.rgw_multisite_tests:-------------------- >> begin captured logging << -------------------- 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:36:18.893 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:14:52.567201Z" 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.894 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:14.853+0000 7f365aac0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:14.869+0000 7f365aac0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.895 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.896 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.897 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.898 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.899 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.900 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:14Z" 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.901 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.902 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.903 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.904 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.905 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:14Z" 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.906 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[]} 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.907 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'create', '--group-id', 'sync-group', '--status', 'allowed', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group create --group-id sync-group --status allowed --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{},"pipes":[],"status":"allowed"}]} 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'flow', 'create', '--group-id', 'sync-group', '--flow-id', 'sync-flow1', '--flow-type', 'symmetrical', '--zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group flow create --group-id sync-group --flow-id sync-flow1 --flow-type symmetrical --zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[],"status":"allowed"}]} 2023-09-29T23:36:18.908 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'pipe', 'create', '--group-id', 'sync-group', '--pipe-id', 'sync-pipe', '--source-zones=a1,a2', '--dest-zones=a1,a2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group pipe create --group-id sync-group --pipe-id sync-pipe --source-zones=a1,a2 --dest-zones=a1,a2 --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"allowed"}]} 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'group', 'modify', '--group-id', 'sync-group', '--status', 'enabled', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync group modify --group-id sync-group --status enabled --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.909 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "epoch": 15, 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "predecessor_uuid": "cbc12b34-1e27-40f2-8eb8-fd649eede431", 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": [], 2023-09-29T23:36:18.910 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period_map": { 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zonegroups": [ 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "a", 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a", 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "api_name": "", 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "is_master": true, 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames": [], 2023-09-29T23:36:18.911 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames_s3website": [], 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a2", 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8001" 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-09-29T23:36:18.912 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.913 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "a1", 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi112.front.sepia.ceph.com:8000" 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-09-29T23:36:18.914 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "placement_targets": [ 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "default-placement", 2023-09-29T23:36:18.915 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [], 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "storage_classes": [ 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "STANDARD" 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "default_placement": "default-placement", 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_policy": { 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "groups": [ 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-group", 2023-09-29T23:36:18.916 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "data_flow": { 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "symmetrical": [ 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-flow1", 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pipes": [ 2023-09-29T23:36:18.917 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "sync-pipe", 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "*", 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": { 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket": "*", 2023-09-29T23:36:18.918 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "a75fe7d4-5635-4f57-8059-26c1aacd3a0f" 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "params": { 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "source": { 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "filter": { 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [] 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "dest": {}, 2023-09-29T23:36:18.919 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "priority": 0, 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "mode": "system", 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user": "" 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "enabled" 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled_features": [ 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-09-29T23:36:18.920 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "b", 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b", 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "api_name": "", 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "is_master": false, 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames": [], 2023-09-29T23:36:18.921 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "hostnames_s3website": [], 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "zones": [ 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b2", 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi131.front.sepia.ceph.com:8001" 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-09-29T23:36:18.922 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.923 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "id": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "b1", 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "endpoints": [ 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "http://smithi131.front.sepia.ceph.com:8000" 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_meta": false, 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "log_data": true, 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_index_max_shards": 11, 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "read_only": false, 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tier_type": "", 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from_all": true, 2023-09-29T23:36:18.924 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_from": [], 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "redirect_zone": "", 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "supported_features": [ 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "compress-encrypted", 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "placement_targets": [ 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "name": "default-placement", 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "tags": [], 2023-09-29T23:36:18.925 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "storage_classes": [ 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "STANDARD" 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "default_placement": "default-placement", 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_policy": { 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "groups": [] 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled_features": [ 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "resharding" 2023-09-29T23:36:18.926 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ], 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "short_zone_ids": [ 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "272b60f1-3a4d-4009-a171-ab410395ba9a", 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 1230604351 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "558b41d2-5c02-4685-b471-4770c7b56dfc", 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 192990612 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.927 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "794315e0-2f8f-40cc-bb4b-5c0cda8a89b2", 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 3868157510 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": 3032081421 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zonegroup": "a", 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "master_zone": "a75fe7d4-5635-4f57-8059-26c1aacd3a0f", 2023-09-29T23:36:18.928 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period_config": { 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_quota": { 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "check_on_raw": false, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size": -1, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size_kb": 0, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_objects": -1 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user_quota": { 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "check_on_raw": false, 2023-09-29T23:36:18.929 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size": -1, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_size_kb": 0, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_objects": -1 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "user_ratelimit": { 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "bucket_ratelimit": { 2023-09-29T23:36:18.930 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "anonymous_ratelimit": { 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_ops": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_ops": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_read_bytes": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "max_write_bytes": 0, 2023-09-29T23:36:18.931 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "enabled": false 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_id": "afcff763-1d9e-43af-a674-cef7f31c7813", 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_name": "test-realm", 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'sync', 'policy', 'get', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 sync policy get --cluster c1 --debug-rgw 0 --debug-ms 0 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.932 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: {"groups":[{"id":"sync-group","data_flow":{"symmetrical":[{"id":"sync-flow1","zones":["a2","a1"]}]},"pipes":[{"id":"sync-pipe","source":{"bucket":"*","zones":["a2","a1"]},"dest":{"bucket":"*","zones":["a2","a1"]},"params":{"source":{"filter":{"tags":[]}},"dest":{},"priority":0,"mode":"system","user":""}}],"status":"enabled"}]} 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a1 name=hcnphz-66 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-66/ 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-66/ 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-66/ 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.933 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8000} 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:15:15 GMT 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:/hcnphz-66/ 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:yqT2WNtcCDqP+xtooI/q36ZoUIg= 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:15 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:yqT2WNtcCDqP+xtooI/q36ZoUIg=', 'Content-Length': '0'} 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000cde69bf3867a1293-0065175a83-2217-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.934 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-66/ 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-66/ 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-66/ 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.935 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:15:16 GMT 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:/hcnphz-66/ 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:bV/MxvkPOKI2058Y3erXXVQaMng= 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:bV/MxvkPOKI2058Y3erXXVQaMng=', 'Content-Length': '0'} 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000041c475760ae6a02a-0065175a84-2217-a1'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-66/obj1 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-66/obj1 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.936 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-66/obj1 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8000 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8000 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.937 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:15:16 GMT 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:/hcnphz-66/obj1 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:toS9Pm3kwkPNtC7dVI3QyO+m6H8= 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:toS9Pm3kwkPNtC7dVI3QyO+m6H8='} 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx000003210490ace1674d4-0065175a84-2217-a1'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: create bucket zone=a2 name=hcnphz-67 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-67/ 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-67/ 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.938 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-67/ 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.939 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:15:16 GMT 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:/hcnphz-67/ 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:QRvCHnh35mrMHpb5lywU9ccrFag= 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:QRvCHnh35mrMHpb5lywU9ccrFag=', 'Content-Length': '0'} 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('x-amz-request-id', 'tx00000b5963b7d5de1323a-0065175a84-221d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-67/ 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-67/ 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-67/ 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:18.940 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:15:16 GMT 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:/hcnphz-67/ 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.941 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:/XoqtsCzRqKYmAkEfXtpKWJ4kBg= 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:/XoqtsCzRqKYmAkEfXtpKWJ4kBg=', 'Content-Length': '0'} 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx000007ea2e93878711df8-0065175a84-221d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-67/obj2 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-67/obj2 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: PUT 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-67/obj2 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue'} 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:18.942 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:PUT 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:qPXxZ/RPSWTmyZje6CcRDA== 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:application/octet-stream 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:15:16 GMT 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:/hcnphz-67/obj2 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:SXSD8vSZVrMHHyyBO4o+V1CW1dw= 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'qPXxZ/RPSWTmyZje6CcRDA==', 'Content-Length': '6', 'Expect': '100-Continue', 'Date': 'Fri, 29 Sep 2023 23:15:16 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:SXSD8vSZVrMHHyyBO4o+V1CW1dw='} 2023-09-29T23:36:18.943 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '0'), ('ETag', '"a8f5f167f44f4964e6c998dee827110c"'), ('Accept-Ranges', 'bytes'), ('x-amz-request-id', 'tx0000015de03fa6e36adfd-0065175a84-221d-a2'), ('Date', 'Fri, 29 Sep 2023 23:15:16 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: No --period given, using current period=b4e33480-bfb8-4364-b1d7-649c06bc8c56 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T22:50:55.273178Z" 2023-09-29T23:36:18.944 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:15:16.160461Z" 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.945 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "0.000000" 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: master meta markers={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting meta checkpoint for zone=a2 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:16.437+0000 7f9ea52fff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.946 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:16.441+0000 7f9ea52fff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.947 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.948 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.949 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.950 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.951 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:16Z" 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.952 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.953 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.954 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.955 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.956 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.957 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:16Z" 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi112.stdout: INFO: 2023-09-29T23:15:17.141+0000 7f4525c2c700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:36:18.958 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:21.576+0000 7fa085659f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:21.580+0000 7fa085659f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.959 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.960 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.961 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.962 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.963 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.964 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:21Z" 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.965 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.966 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.967 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.968 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.969 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:21Z" 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.970 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:26.716+0000 7f378444ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:26.720+0000 7f378444ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.971 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.972 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.973 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.974 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.975 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.976 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.977 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:26Z" 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.978 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.979 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.980 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.981 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.982 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.983 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:26Z" 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:18.984 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:31.856+0000 7f06e6c9bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:31.860+0000 7f06e6c9bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.985 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.986 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.987 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.988 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.989 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:18.990 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:31Z" 2023-09-29T23:36:18.991 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:18.992 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.993 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:18.994 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.995 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:18.996 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:18.997 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:31Z" 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.998 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:37.004+0000 7fb3e6e11f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:37.004+0000 7fb3e6e11f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:18.999 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.000 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.001 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.002 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.003 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.004 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:37Z" 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.005 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.006 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.007 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.008 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.009 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.010 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.011 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:37Z" 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.012 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:42.140+0000 7f5cd6dccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:42.144+0000 7f5cd6dccf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.013 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.014 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.015 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.016 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.017 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.018 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.019 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:42Z" 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.020 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.021 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.022 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.023 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.024 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.025 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.026 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:42Z" 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.027 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:47.280+0000 7f9acaaa1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:47.284+0000 7f9acaaa1f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.028 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.029 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.030 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.031 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.032 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.033 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:47Z" 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.034 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.035 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.036 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.037 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.038 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.039 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:47Z" 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.040 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:52.424+0000 7f3cf48f0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.041 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:52.424+0000 7f3cf48f0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.042 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.043 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.044 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.045 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.046 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.047 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:52Z" 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.048 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.049 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.050 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.051 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.052 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.053 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:52Z" 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.054 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:57.564+0000 7faec6197f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:15:57.568+0000 7faec6197f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.055 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.056 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.057 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.058 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.059 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.060 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.061 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:15:57Z" 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.062 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.063 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.064 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.065 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.066 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.067 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:15:57Z" 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.068 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:02.703+0000 7f4912670f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.069 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:02.707+0000 7f4912670f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.070 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.071 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.072 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.073 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.074 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:02Z" 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.075 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.076 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.077 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.078 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.079 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.081 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.082 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:02Z" 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.083 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.084 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.084 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:07.843+0000 7f46dbbaef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:07.843+0000 7f46dbbaef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.085 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.086 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.087 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.088 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.089 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.090 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.091 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.092 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.093 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.094 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.095 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:07Z" 2023-09-29T23:36:19.096 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.096 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.133 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.134 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.135 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.136 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.137 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.138 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.139 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.140 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:07Z" 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.141 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:12.987+0000 7f27f0cbef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:12.987+0000 7f27f0cbef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.142 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.143 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.144 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.145 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.146 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.147 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.148 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:12Z" 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.149 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.150 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.151 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.152 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.153 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.154 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.155 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:12Z" 2023-09-29T23:36:19.156 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.157 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:18.135+0000 7f7ef2665f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:18.135+0000 7f7ef2665f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.158 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.159 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.160 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.161 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.162 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.163 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.164 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:18Z" 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.165 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.166 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.167 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.168 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.169 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.170 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.171 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:18Z" 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.172 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:23.275+0000 7ff3a0e37f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:23.275+0000 7ff3a0e37f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.173 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.174 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.175 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.176 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.177 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.178 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.179 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.180 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:23Z" 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.181 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.182 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.183 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.184 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.185 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.186 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.187 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.188 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:23Z" 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.189 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:28.419+0000 7fc9f73b3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:28.419+0000 7fc9f73b3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.190 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.191 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.192 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.193 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.194 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.195 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.196 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.197 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:28Z" 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.198 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.199 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.200 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.201 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.202 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.203 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.204 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:28Z" 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.205 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:33.563+0000 7fdf1186ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.206 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:33.563+0000 7fdf1186ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.207 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.208 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.209 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.210 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.211 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.212 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:33Z" 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.213 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.214 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.215 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.216 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.217 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.218 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.219 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:33Z" 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.220 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:38.706+0000 7f540134ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:38.706+0000 7f540134ef40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.221 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.222 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.223 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.224 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.225 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.226 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.227 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:38Z" 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.228 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.229 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.230 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.231 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.232 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.233 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.234 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:38Z" 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.235 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:43.846+0000 7f8576554f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:43.850+0000 7f8576554f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.236 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.237 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.238 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.239 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.240 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.241 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.242 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:43Z" 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.243 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.244 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.245 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.246 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.247 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.248 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.249 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:43Z" 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.250 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.251 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:48.994+0000 7f29b8d2ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:48.998+0000 7f29b8d2ff40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.252 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.253 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.254 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.255 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.256 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.257 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.258 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:49Z" 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.259 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.260 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.261 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.262 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.263 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.264 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.265 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.266 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:49Z" 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.267 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:54.138+0000 7f3080855f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:54.138+0000 7f3080855f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.268 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.269 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.270 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.271 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.272 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.273 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.274 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.275 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.276 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:54Z" 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.277 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.278 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.279 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.280 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.281 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.282 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.283 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:54Z" 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.284 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:59.286+0000 7f6004011f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:16:59.286+0000 7f6004011f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.285 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.286 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.289 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.290 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.291 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.292 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.293 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.294 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:16:59Z" 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.295 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.296 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.297 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.298 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.299 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.300 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.301 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.302 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.302 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.302 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.302 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.302 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.302 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:16:59Z" 2023-09-29T23:36:19.303 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.304 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:04.422+0000 7f38dabf3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:04.426+0000 7f38dabf3f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.305 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.306 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.307 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.308 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.309 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.310 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.311 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.312 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:04Z" 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.313 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.314 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.315 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.316 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.317 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.318 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.319 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.320 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:04Z" 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.321 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.322 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:09.566+0000 7f7490621f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:09.570+0000 7f7490621f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.323 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.324 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.325 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.326 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.327 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.328 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.329 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.330 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:09Z" 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.331 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.332 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.333 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.334 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.335 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.336 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.337 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.338 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:09Z" 2023-09-29T23:36:19.339 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.340 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:14.710+0000 7f6a53d5bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:14.710+0000 7f6a53d5bf40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.341 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.342 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.343 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.344 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.345 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.346 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.347 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.348 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:14Z" 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.349 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.350 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.351 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.352 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.353 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.354 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.355 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.356 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:14Z" 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.357 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.1.smithi112.stdout: INFO: 2023-09-29T23:17:17.169+0000 7f87f9cdf700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi131.stdout: INFO: 2023-09-29T23:17:17.379+0000 7f270e947700 -1 WARNING: The OSD cluster is overloaded and struggling to complete ops. You need more capacity to serve this level of demand. 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.358 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:19.853+0000 7f4bb9237f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:19.853+0000 7f4bb9237f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.359 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.360 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.361 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.362 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.363 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.364 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.365 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.366 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:19Z" 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.367 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.368 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.369 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.370 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.371 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.372 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.373 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:19Z" 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.374 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:24.993+0000 7f4cbf1c0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.375 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:24.993+0000 7f4cbf1c0f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.376 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.377 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.378 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.379 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.380 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.381 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.382 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.383 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:25Z" 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.384 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.385 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.386 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029292.567201_712.1", 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:14:52.567201Z", 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.387 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.388 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.389 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.390 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:25Z" 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029292.567201_712.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:14:52.567201Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029292.567201_712.1', 2: '', 3: ''} 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: WARNING: zone a2 behind master: shard=1 master=1_1696029316.160461_736.1 target=1_1696029292.567201_712.1 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.391 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:30.137+0000 7fc8a34c9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: 2023-09-29T23:17:30.137+0000 7fc8a34c9f40 20 RGW-SYNC:meta: read sync status 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.392 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.393 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 1, 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.394 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:15:16.160461Z", 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.395 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.396 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "state": 1, 2023-09-29T23:36:19.397 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "", 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "0.000000", 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "realm_epoch": 2 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.398 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 1, 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 1 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current meta sync status={ 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.399 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "period": "b4e33480-bfb8-4364-b1d7-649c06bc8c56", 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.400 INFO:tasks.rgw_multisite_tests: "marker": "1_1696027855.273178_228.1", 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: "total_entries": 1, 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T22:50:55.273178Z", 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.401 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "marker": "1_1696029316.160461_736.1", 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:15:16.160461Z", 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.402 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.403 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: "state": 1, 2023-09-29T23:36:19.404 INFO:tasks.rgw_multisite_tests: "marker": "", 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: "timestamp": "0.000000", 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: "realm_epoch": 2 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.405 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests: "total": 1, 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests: "complete": 1 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'state': 1, 'marker': '1_1696027855.273178_228.1', 'next_step_marker': '', 'total_entries': 1, 'pos': 0, 'timestamp': '2023-09-29T22:50:55.273178Z', 'realm_epoch': 2}}, {'key': 1, 'val': {'state': 1, 'marker': '1_1696029316.160461_736.1', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.160461Z', 'realm_epoch': 2}}, {'key': 2, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}, {'key': 3, 'val': {'state': 1, 'marker': '', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '0.000000', 'realm_epoch': 2}}] 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.406 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '1_1696027855.273178_228.1', 1: '1_1696029316.160461_736.1', 2: '', 3: ''} 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finish meta checkpoint for zone=a2 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'a1', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: [ 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.407 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:36:19.408 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "last_update": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: data markers for zone=a1 markers={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: starting data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: INFO: rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:tasks.util.rgw: DEBUG: rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'a2', '--rgw-zonegroup', 'a', '--rgw-realm', 'test-realm', '--source-zone', 'a1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2023-09-29T23:36:19.409 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112: DEBUG: > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm --source-zone a1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setuser ceph since I am not root 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stderr: INFO: ignoring --setgroup ceph since I am not root 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "sync_status": { 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "info": { 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "sync", 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "num_shards": 4, 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "instance_id": 6926381311200296959 2023-09-29T23:36:19.410 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "markers": [ 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 0, 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 55, 2023-09-29T23:36:19.411 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 1, 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:36:19.412 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 9, 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 2, 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-09-29T23:36:19.413 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: { 2023-09-29T23:36:19.414 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "key": 3, 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "val": { 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "status": "incremental-sync", 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "next_step_marker": "", 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total_entries": 0, 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "pos": 0, 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.415 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: ] 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "full_sync": { 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "total": 64, 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "complete": 64 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: }, 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:teuthology.orchestra.run.smithi112.stdout: INFO: } 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: current data sync status={ 2023-09-29T23:36:19.416 INFO:tasks.rgw_multisite_tests: "sync_status": { 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "info": { 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "status": "sync", 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "num_shards": 4, 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "instance_id": 6926381311200296959 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "markers": [ 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "key": 0, 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.417 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000050969", 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "total_entries": 55, 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:15:16.048128Z" 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "key": 1, 2023-09-29T23:36:19.418 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045024", 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "total_entries": 9, 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:08:29.597662Z" 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.419 INFO:tasks.rgw_multisite_tests: "key": 2, 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000045517", 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:15:19.183068Z" 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.420 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: { 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "key": 3, 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "val": { 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "status": "incremental-sync", 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "marker": "00000000000000000000:00000000000000031496", 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "next_step_marker": "", 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "total_entries": 0, 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "pos": 0, 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: "timestamp": "2023-09-29T23:08:33.687356Z" 2023-09-29T23:36:19.421 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: } 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: ] 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: "full_sync": { 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: "total": 64, 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: "complete": 64 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: }, 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests: "current_time": "2023-09-29T23:17:30Z" 2023-09-29T23:36:19.422 INFO:tasks.rgw_multisite_tests:} 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_markers=[{'key': 0, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000050969', 'next_step_marker': '', 'total_entries': 55, 'pos': 0, 'timestamp': '2023-09-29T23:15:16.048128Z'}}, {'key': 1, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045024', 'next_step_marker': '', 'total_entries': 9, 'pos': 0, 'timestamp': '2023-09-29T23:08:29.597662Z'}}, {'key': 2, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000045517', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:15:19.183068Z'}}, {'key': 3, 'val': {'status': 'incremental-sync', 'marker': '00000000000000000000:00000000000000031496', 'next_step_marker': '', 'total_entries': 0, 'pos': 0, 'timestamp': '2023-09-29T23:08:33.687356Z'}}] 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: log_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: DEBUG: sync_status={0: '00000000000000000000:00000000000000050969', 1: '00000000000000000000:00000000000000045024', 2: '00000000000000000000:00000000000000045517', 3: '00000000000000000000:00000000000000031496'} 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:rgw_multi.tests: INFO: finished data checkpoint for target_zone=a2 source_zone=a1 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-66/ 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-66/ 2023-09-29T23:36:19.423 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-66/ 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: establishing HTTP connection: kwargs={'timeout': 70, 'port': 8001} 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:19.424 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:17:30 GMT 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests:/hcnphz-66/ 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:K3LO7PDMrdsjmd1E0blhb1sx6hI= 2023-09-29T23:36:19.425 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:K3LO7PDMrdsjmd1E0blhb1sx6hI=', 'Content-Length': '0'} 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('X-RGW-Object-Count', '0'), ('X-RGW-Bytes-Used', '0'), ('X-RGW-Quota-User-Size', '-1'), ('X-RGW-Quota-User-Objects', '-1'), ('X-RGW-Quota-Max-Buckets', '1000'), ('X-RGW-Quota-Bucket-Size', '-1'), ('X-RGW-Quota-Bucket-Objects', '-1'), ('x-amz-request-id', 'tx0000061589254f67d3604-0065175b0a-221d-a2'), ('Content-Length', '0'), ('Date', 'Fri, 29 Sep 2023 23:17:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: path=/hcnphz-66/obj1 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: auth_path=/hcnphz-66/obj1 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Method: HEAD 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Path: /hcnphz-66/obj1 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Data: 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Headers: {} 2023-09-29T23:36:19.426 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Host: smithi112.front.sepia.ceph.com:8001 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Port: 8001 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Params: {} 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Token: None 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests:boto: DEBUG: StringToSign: 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests:HEAD 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.427 INFO:tasks.rgw_multisite_tests:Fri, 29 Sep 2023 23:17:30 GMT 2023-09-29T23:36:19.428 INFO:tasks.rgw_multisite_tests:/hcnphz-66/obj1 2023-09-29T23:36:19.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Signature: 2023-09-29T23:36:19.428 INFO:tasks.rgw_multisite_tests:AWS P5GVQP2RXA6372D10DSL:OrOb49b1R+pw/3q4hMndFfGdP9Y= 2023-09-29T23:36:19.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Final headers: {'User-Agent': 'Boto/2.49.0 Python/3.8.10 Linux/5.4.0-163-generic', 'Date': 'Fri, 29 Sep 2023 23:17:30 GMT', 'Authorization': 'AWS P5GVQP2RXA6372D10DSL:OrOb49b1R+pw/3q4hMndFfGdP9Y=', 'Content-Length': '0'} 2023-09-29T23:36:19.428 INFO:tasks.rgw_multisite_tests:boto: DEBUG: Response headers: [('Content-Length', '219'), ('x-amz-request-id', 'tx0000074e5ab68511bc5db-0065175b0a-221d-a2'), ('Accept-Ranges', 'bytes'), ('Content-Type', 'application/xml'), ('Date', 'Fri, 29 Sep 2023 23:17:30 GMT'), ('Connection', 'Keep-Alive')] 2023-09-29T23:36:19.428 INFO:tasks.rgw_multisite_tests:--------------------- >> end captured logging << --------------------- 2023-09-29T23:36:19.429 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.429 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2023-09-29T23:36:19.429 INFO:tasks.rgw_multisite_tests:Ran 47 tests in 6101.225s 2023-09-29T23:36:19.429 INFO:tasks.rgw_multisite_tests: 2023-09-29T23:36:19.429 INFO:tasks.rgw_multisite_tests:FAILED (SKIP=4, failures=4) 2023-09-29T23:36:19.430 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-09-29T23:36:19.682 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6359c3d409734325a0f99e626a8026ed Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/github.com_ceph_ceph-c_38c3646d535f6f917a872ef9baef468589ba9f32/qa/tasks/rgw_multisite_tests.py", line 95, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2023-09-29T23:36:19.683 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2023-09-29T23:36:19.701 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2023-09-29T23:36:19.720 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-09-29T23:36:19.729 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2023-09-29T23:36:19.729 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:19.732 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-09-29T23:36:19.732+0000 7f4544469700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone a1 --rgw-zonegroup a --rgw-realm test-realm (PID: 35309) UID: 0 2023-09-29T23:36:19.732 INFO:tasks.rgw.c1.client.0.smithi112.stdout:2023-09-29T23:36:19.732+0000 7f4547cc8f80 -1 shutting down 2023-09-29T23:36:21.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2023-09-29T23:36:25.831 INFO:tasks.rgw.c1.client.0:Stopped 2023-09-29T23:36:25.832 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2023-09-29T23:36:25.839 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/vault-root-token 2023-09-29T23:36:25.896 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2023-09-29T23:36:25.896 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:25.897 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:36:25.900+0000 7f880bd03700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone a2 --rgw-zonegroup a --rgw-realm test-realm (PID: 82699) UID: 0 2023-09-29T23:36:25.897 INFO:tasks.rgw.c1.client.1.smithi112.stdout:2023-09-29T23:36:25.900+0000 7f880f562f80 -1 shutting down 2023-09-29T23:36:26.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2023-09-29T23:36:26.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2023-09-29T23:36:31.998 INFO:tasks.rgw.c1.client.1:Stopped 2023-09-29T23:36:31.999 DEBUG:teuthology.orchestra.run.smithi112:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock 2023-09-29T23:36:32.006 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f /etc/ceph/vault-root-token 2023-09-29T23:36:32.064 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2023-09-29T23:36:32.064 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:32.065 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:36:32.059+0000 7f272c98b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone b1 --rgw-zonegroup b --rgw-realm test-realm (PID: 34829) UID: 0 2023-09-29T23:36:32.065 INFO:tasks.rgw.c2.client.0.smithi131.stdout:2023-09-29T23:36:32.059+0000 7f27301eaf80 -1 shutting down 2023-09-29T23:36:32.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2023-09-29T23:36:38.166 INFO:tasks.rgw.c2.client.0:Stopped 2023-09-29T23:36:38.166 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2023-09-29T23:36:38.337 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/vault-root-token 2023-09-29T23:36:38.354 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2023-09-29T23:36:38.486 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2023-09-29T23:36:38.486 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:38.487 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T23:36:38.483+0000 7efd57c4f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone b2 --rgw-zonegroup b --rgw-realm test-realm (PID: 35931) UID: 0 2023-09-29T23:36:38.487 INFO:tasks.rgw.c2.client.1.smithi131.stdout:2023-09-29T23:36:38.483+0000 7efd5b4aef80 -1 shutting down 2023-09-29T23:36:43.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2023-09-29T23:36:43.858 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2023-09-29T23:36:44.588 INFO:tasks.rgw.c2.client.1:Stopped 2023-09-29T23:36:44.589 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2023-09-29T23:36:44.596 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/vault-root-token 2023-09-29T23:36:44.613 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-29T23:36:44.624 DEBUG:teuthology.orchestra.run.smithi131:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2023-09-29T23:36:45.522 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-29T23:36:45.523 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-29T23:36:45.523 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2023-09-29T23:36:45.523 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:45.593 INFO:tasks.ceph.c2.osd.0:Stopped 2023-09-29T23:36:45.594 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2023-09-29T23:36:45.594 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:49.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 is failed for ~0s 2023-09-29T23:36:49.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2023-09-29T23:36:51.696 INFO:tasks.ceph.c2.osd.1:Stopped 2023-09-29T23:36:51.696 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2023-09-29T23:36:51.696 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:54.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 is failed for ~0s 2023-09-29T23:36:54.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.1 has been restored 2023-09-29T23:36:57.798 INFO:tasks.ceph.c2.osd.2:Stopped 2023-09-29T23:36:57.799 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-29T23:36:57.799 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2023-09-29T23:36:57.799 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:57.833 INFO:tasks.ceph.c2.mgr.x:Stopped 2023-09-29T23:36:57.833 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-29T23:36:57.833 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2023-09-29T23:36:57.834 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:36:57.868 INFO:tasks.ceph.c2.mon.a:Stopped 2023-09-29T23:36:57.868 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-29T23:36:57.868 DEBUG:teuthology.orchestra.run.smithi131:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-29T23:36:58.003 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T23:36:58.003 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2023-09-29T23:36:58.745 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T23:36:58.745 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2023-09-29T23:36:59.041 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T23:36:59.041 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2023-09-29T23:36:59.281 INFO:tasks.ceph:Archiving mon data... 2023-09-29T23:36:59.282 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/data/c2.mon.a.tgz 2023-09-29T23:36:59.283 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2023-09-29T23:36:59.303 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.ibFf1CuX3i 2023-09-29T23:36:59.304 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.ibFf1CuX3i 2023-09-29T23:36:59.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.osd.2 has been restored 2023-09-29T23:36:59.441 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.ibFf1CuX3i 2023-09-29T23:36:59.525 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.ibFf1CuX3i is 427KB 2023-09-29T23:36:59.547 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.ibFf1CuX3i 2023-09-29T23:36:59.552 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-29T23:36:59.553 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:36:59.555 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:36:59.625 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-29T23:36:59.636 DEBUG:teuthology.orchestra.run.smithi112:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2023-09-29T23:36:59.971 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-29T23:36:59.971 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-29T23:36:59.971 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2023-09-29T23:36:59.971 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:37:00.046 INFO:tasks.ceph.c1.osd.0:Stopped 2023-09-29T23:37:00.047 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2023-09-29T23:37:00.047 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:37:00.129 INFO:tasks.ceph.c1.osd.1:Stopped 2023-09-29T23:37:00.129 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2023-09-29T23:37:00.129 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:37:00.204 INFO:tasks.ceph.c1.osd.2:Stopped 2023-09-29T23:37:00.204 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-29T23:37:00.205 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2023-09-29T23:37:00.205 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:37:00.233 INFO:tasks.ceph.c1.mgr.x:Stopped 2023-09-29T23:37:00.234 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-29T23:37:00.234 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2023-09-29T23:37:00.234 INFO:teuthology.orchestra.run:waiting for 300 2023-09-29T23:37:00.266 INFO:tasks.ceph.c1.mon.a:Stopped 2023-09-29T23:37:00.266 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-29T23:37:00.267 DEBUG:teuthology.orchestra.run.smithi112:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-29T23:37:00.282 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T23:37:00.283 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2023-09-29T23:37:00.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T23:37:00.657 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2023-09-29T23:37:00.893 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T23:37:00.893 DEBUG:teuthology.orchestra.run.smithi112:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2023-09-29T23:37:01.084 INFO:tasks.ceph:Archiving mon data... 2023-09-29T23:37:01.084 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/data/c1.mon.a.tgz 2023-09-29T23:37:01.085 DEBUG:teuthology.orchestra.run.smithi112:> mktemp 2023-09-29T23:37:01.090 INFO:teuthology.orchestra.run.smithi112.stdout:/tmp/tmp.ElTiICgVDl 2023-09-29T23:37:01.091 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.ElTiICgVDl 2023-09-29T23:37:01.227 DEBUG:teuthology.orchestra.run.smithi112:> sudo chmod 0666 /tmp/tmp.ElTiICgVDl 2023-09-29T23:37:01.295 DEBUG:teuthology.orchestra.remote:smithi112:/tmp/tmp.ElTiICgVDl is 460KB 2023-09-29T23:37:01.359 DEBUG:teuthology.orchestra.run.smithi112:> rm -fr /tmp/tmp.ElTiICgVDl 2023-09-29T23:37:01.366 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-29T23:37:01.366 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:37:01.368 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2023-09-29T23:37:01.385 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-29T23:37:01.385 DEBUG:teuthology.orchestra.run.smithi112:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:37:01.426 DEBUG:teuthology.orchestra.run.smithi131:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-29T23:37:01.449 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:37:01.459 INFO:teuthology.orchestra.run.smithi131.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-29T23:37:01.465 INFO:tasks.ceph:Archiving crash dumps... 2023-09-29T23:37:01.467 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/remote/smithi112/crash 2023-09-29T23:37:01.468 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:37:01.486 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/remote/smithi131/crash 2023-09-29T23:37:01.486 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-29T23:37:01.529 INFO:tasks.ceph:Compressing logs... 2023-09-29T23:37:01.530 DEBUG:teuthology.orchestra.run.smithi112:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:37:01.532 DEBUG:teuthology.orchestra.run.smithi131:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:50:34.218 INFO:tasks.ceph:Archiving logs... 2023-09-29T23:50:34.219 DEBUG:teuthology.misc:Transferring archived files from smithi112:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/remote/smithi112/log 2023-09-29T23:50:34.220 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:50:50.764 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/remote/smithi131/log 2023-09-29T23:50:50.765 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-29T23:52:22.611 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-29T23:52:22.621 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-29T23:52:22.621 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:52:22.624 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-29T23:52:22.711 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-29T23:52:22.722 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-29T23:52:22.722 DEBUG:teuthology.orchestra.run.smithi112:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:52:22.724 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-29T23:52:22.747 INFO:teuthology.orchestra.run.smithi112.stdout: remote refid st t when poll reach delay offset jitter 2023-09-29T23:52:22.748 INFO:teuthology.orchestra.run.smithi112.stdout:============================================================================== 2023-09-29T23:52:22.748 INFO:teuthology.orchestra.run.smithi112.stdout:*hv01.front.sepi 44.4.53.2 2 u 202 256 377 0.189 -2.873 0.334 2023-09-29T23:52:22.748 INFO:teuthology.orchestra.run.smithi112.stdout:+hv02.front.sepi 63.231.80.2 3 u 172 256 377 0.189 -2.577 0.334 2023-09-29T23:52:22.748 INFO:teuthology.orchestra.run.smithi112.stdout:+hv03.front.sepi 74.6.168.72 3 u 146 256 377 0.199 -4.478 0.515 2023-09-29T23:52:22.748 INFO:teuthology.orchestra.run.smithi112.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-09-29T23:52:22.846 INFO:teuthology.orchestra.run.smithi131.stdout: remote refid st t when poll reach delay offset jitter 2023-09-29T23:52:22.847 INFO:teuthology.orchestra.run.smithi131.stdout:============================================================================== 2023-09-29T23:52:22.847 INFO:teuthology.orchestra.run.smithi131.stdout:*hv01.front.sepi 44.4.53.2 2 u 62 256 377 0.193 0.939 0.561 2023-09-29T23:52:22.847 INFO:teuthology.orchestra.run.smithi131.stdout:+hv02.front.sepi 63.231.80.2 3 u 218 256 377 0.171 1.428 0.258 2023-09-29T23:52:22.847 INFO:teuthology.orchestra.run.smithi131.stdout:+hv03.front.sepi 74.6.168.72 3 u 28 256 377 0.189 -0.019 0.132 2023-09-29T23:52:22.847 INFO:teuthology.orchestra.run.smithi131.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-09-29T23:52:22.848 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-29T23:52:22.859 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-29T23:52:22.860 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-29T23:52:22.870 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-29T23:52:22.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-29T23:52:22.891 INFO:teuthology.task.internal:Duration was 7814.841238 seconds 2023-09-29T23:52:22.891 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-29T23:52:22.902 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-29T23:52:22.902 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:52:22.905 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-29T23:52:23.159 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-29T23:52:23.159 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi112.front.sepia.ceph.com 2023-09-29T23:52:23.160 DEBUG:teuthology.orchestra.run.smithi112:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:52:23.176 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2023-09-29T23:52:23.176 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-29T23:52:23.203 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-29T23:52:23.204 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:52:23.206 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-29T23:52:23.257 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-29T23:52:23.258 DEBUG:teuthology.orchestra.run.smithi112:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:52:23.260 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-29T23:52:23.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-29T23:52:23.517 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-29T23:52:23.517 DEBUG:teuthology.orchestra.run.smithi112:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:52:23.534 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-29T23:52:23.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-29T23:52:23.573 DEBUG:teuthology.orchestra.run.smithi112:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:52:23.576 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:52:23.587 INFO:teuthology.orchestra.run.smithi112.stdout:kernel.core_pattern = core 2023-09-29T23:52:23.605 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2023-09-29T23:52:23.674 DEBUG:teuthology.orchestra.run.smithi112:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:52:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:52:23.680 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-29T23:52:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-29T23:52:23.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-29T23:52:23.696 INFO:teuthology.task.internal:Transferring archived files... 2023-09-29T23:52:23.697 DEBUG:teuthology.misc:Transferring archived files from smithi112:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/remote/smithi112 2023-09-29T23:52:23.697 DEBUG:teuthology.orchestra.run.smithi112:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:52:23.744 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284/remote/smithi131 2023-09-29T23:52:23.744 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-29T23:52:23.767 INFO:teuthology.task.internal:Removing archive directory... 2023-09-29T23:52:23.767 DEBUG:teuthology.orchestra.run.smithi112:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:52:23.770 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-29T23:52:23.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-29T23:52:23.787 INFO:teuthology.task.internal:Not uploading archives. 2023-09-29T23:52:23.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-29T23:52:23.798 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-29T23:52:23.798 DEBUG:teuthology.orchestra.run.smithi112:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:52:23.819 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-29T23:52:23.824 INFO:teuthology.orchestra.run.smithi112.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:52 /home/ubuntu/cephtest 2023-09-29T23:52:23.829 INFO:teuthology.orchestra.run.smithi131.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 29 23:52 /home/ubuntu/cephtest 2023-09-29T23:52:23.831 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-29T23:52:23.842 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-29T23:52:23.864 INFO:teuthology.nuke:Checking targets against current locks 2023-09-29T23:52:23.890 DEBUG:teuthology.nuke:shortname: smithi112 2023-09-29T23:52:23.890 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:52:23.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi112.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:33:23.921292', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-29T23:52:23.918 DEBUG:teuthology.nuke:shortname: smithi131 2023-09-29T23:52:23.918 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-29T23:52:23.937 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_12:36:11-rgw-wip-yuri11-testing-2023-09-28-1611-reef-distro-default-smithi/7406284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-29 21:33:23.923073', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-09-29T23:52:23.967 INFO:teuthology.orchestra.console.smithi112:Power off 2023-09-29T23:52:23.968 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:52:23.981 INFO:teuthology.orchestra.console.smithi131:Power off 2023-09-29T23:52:23.982 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-29T23:52:23.992 DEBUG:teuthology.orchestra.console.smithi112:power off output: Chassis Power Control: Down/Off 2023-09-29T23:52:23.993 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:24.005 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2023-09-29T23:52:24.005 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:24.016 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-09-29T23:52:24.031 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-09-29T23:52:28.018 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:28.033 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:28.135 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-09-29T23:52:28.143 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-09-29T23:52:32.139 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:32.150 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:32.253 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is on 2023-09-29T23:52:32.262 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2023-09-29T23:52:36.254 DEBUG:teuthology.orchestra.console.smithi112:pexpect command: ipmitool -H smithi112.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:36.263 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-29T23:52:36.366 DEBUG:teuthology.orchestra.console.smithi112:check power output: Chassis Power is off 2023-09-29T23:52:36.374 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2023-09-29T23:52:36.467 INFO:teuthology.orchestra.console.smithi112:Power off completed 2023-09-29T23:52:36.475 INFO:teuthology.orchestra.console.smithi131:Power off completed 2023-09-29T23:52:36.580 INFO:teuthology.run:Summary data: description: rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_20.04} tasks/test_multi} duration: 7814.841238021851 failure_reason: rgw multisite test failures flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6359c3d409734325a0f99e626a8026ed status: fail success: false 2023-09-29T23:52:36.580 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-29T23:52:36.656 INFO:teuthology.run:FAIL